C H A P T E R

Troubleshooting Router Switch Fabric

This chapter describes techniques that you can use to troubleshoot router switch fabric. It includes the
following sections:

CRS-1 Switch Fabric Overview, page 4-91

Flags, page 4-93
Verifying and Troubleshooting the Fabric Plane State, page 4-94
Verifying and Troubleshooting Up Fabric Planes, page 4-99
Troubleshooting Down Fabric Planes, page 4-101

CRS-1 Switch Fabric Overview

The switch fabric is constructed of eight independent fabric planes. Each plane consists of at least three
stage switch element ASICs (SEAs), which are collectively known as known as switch fabric elements
(SFE). While all SEAs are identical, the operational behavior of a SEA is defined according to the
position of the SEA in the fabric. A SEA set for Stage 1 operation is known as an S1 ASIC. A SEA set
for Stage 2 operation is known as an S2 ASIC. A SEA set for Stage 3 operation is known as an S3 ASIC.
The complete fabric is composed of the ingress queue ASIC on the ingress modular services card (MSC)
or designated route processor (D)RP, the S1, S2, and S3 ASICS on the switch fabric planes and one of
more fabric queue ASICs on the destination MSC or (D)RP.

Note

For Cisco CRS-1 Multishelf Systems, connect to the route processor (RP) for the owner logical router
(LR) when troubleshooting the fabric plane. When you are connected to the owner LR, you have control
over the entire system and all cards assigned to the owner LR.

Note

Line cards in Cisco CRS-1s are called modular services cards (MSCs).
Between ingress queue, S1, S2, S3, and fabric queue ASICs are communication paths known as fabric
links. An individual fabric link has a bandwidth capacity of 2.5 Gbps. A link is unidirectional.
An ingress MSC has multiple 2.5 Gbps links from the ingress queue ASIC on the MSC to an individual
fabric plane and to an individual S1 ASIC on that fabric plane. Connections are provided in the same
fashion to each of the eight fabric planes that are present in a system in normal operation.
All S1 ASICs have at least one 2.5 Gbps link to every S2 ASIC present on that fabric plane. An S1-to-S2
link does not and cannot span across fabric planes.

Cisco IOS XR Troubleshooting

OL-11138-02

4-91

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

All S2 ASICs have at least one 2.5 Gbps link to every S3 ASIC present on that fabric plane. An S2-to-S3
link does not and cannot span across fabric planes.
An egress MSC has multiple 2.5 Gbps links from the S3 ASIC to one or two fabric queue ASICs present
on that MSC.
In a 16-slot single-chassis system, the eight switch fabric planes are contained on eight individual fabric
cards known as FC/S or S123. Each of the fabric cards contains two S1, two S2, and four S3 ASICs.
In an 8-slot system, the eight switch fabric planes are contained on four individual fabric cards known
as FC/S or HS123. Each fabric card contains two planes. Each plane is composed of one S1, one S2, and
one S3 ASIC.
In a 4-slot system, there are four fabric planes. The fabric planes are contained on four individual fabric
cards known as FC/s or QS123. Each plane is composed of one S1, one S2, and one S3 ASIC.
Each S13 card contains two S1 and four S3 ASICs and corresponds to one switch fabric plane. Each
fabric card chassis has one or more switch fabric planes and each fabric plane consists of one or more
S2 fabric cards. Each S2 fabric card contains six S2 ASICs.
Data is not transmitted across the fabric in packets. Packets are segmented into cells by the ingress queue
ASIC and reassembled back to packets by the fabric queue ASIC. After a cell is placed on a plane, it will
remain on that plane until it reaches the egress MSC. While the system is operational, cells are always
being sent and received even though they may not contain any data. Cells with no data are termed as idle
cells.
Cell forwarding on the S3 ASIC on both the Cisco CRS-1 4-slot, 8-slot, and 16-slot systems involves
examination of the cell header. Using the cell header, the S3 is able to determine which fabric queue
ASIC the cells should be sent to. The S3 ASIC injects cells in a round-robin fashion across the multiple
links connecting the S3 ASIC to the fabric queue ASIC.
As the cell is created, forward error correction (FEC) data is calculated and appended. The FEC code is
used on the transmission links to ensure the integrity of the data as it is transmitted and received. The
FEC data is removed, checked, recalculated, and reapplied by each SFE in the path until it is delivered
to the egress MSC. The FEC value is checked before a cell is processed by the SFE. If an error is
detected, the FEC code can be used to recover the cell. Depending on the nature of the error, recovery
may not be possible, in which case, the cell is discarded.
A system can operate with seven planes with out performance degradation. As further planes are
removed, overall capacity degrades but the system remains operational. A system requires a minimum
of two planes to maintain service. One plane must be odd numbered and the other plane must be even
numbered.

Note

A 4-slot system operates on four planes. The loss of a plane reduces the usable forwarding capacity by
approximately 6Gbps.
A plane marked MCAST_DOWN means that as far as the ingress line cards are concerned, not all
destination FabricQ ASICs can be reached via the plane. Since the destinations of multicast traffic
cannot be predetermined due to their dynamic nature, the plane is termed as MCAST_DOWN. Multicast
traffic will use the other fabric planes to deliver date to the appropriate egress line cards. Unicast traffic
will continue to use the plane if traffic is destined to a FabricQ ASIC that is still reachable. If traffic is
destined for a FabricQ ASIC that is not unreachable, the other available plane is used instead of the plane
in the MCAST_DOWN state.

Cisco IOS XR Troubleshooting

4-92

OL-11138-02

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

Flags
A Flags field is provided in some fabric-related show commands. The flag is an abbreviated reason for
a link being placed in the down state. Table 4-1 provides descriptions of the flags.
Table 4-1

Fabric show Command Flags

P - plane admin downplane has been taken out p - plane oper downplane is not able to operate
of service by administrative action
or has been taken out of service by administrative
action
C - card admin downcard has been taken out of c - card oper downflag is set in response to
service by administrative action (reserved for
system notification that a card will soon be
future use)
powered off. This is a transient condition lasting a
few seconds before power is withdrawn. Unusual
to see this flag in practice
L - link port admin downlink has been taken out l - linkport oper downflag is set or cleared in
of service by administrative action
response to messages from the ASIC driver code.
Flags are enabled when the asic is first
discovered. Flag is set if the ASIC driver has
signaled that an individual link port is not usable
for fabric traffic, or that the driver has confirmed
that the link is ready. Two common reasons for
seeing an l flag that are initialization is not
completed yet or that sufficient errors have been
seen at a receive port so that the ASIC driver shuts
the port down
A - asic admin down ASIC has been taken out a - asic oper down flag is set or cleared in
of service by administrative action. If the A flag is response to messages from the ASIC driver code.
set, it is seen on all links connected to that ASIC Flags are enabled when the ASIC is first
discovered. Flag is set if the ASIC driver has
signaled that an SEA ASIC is not usable for fabric
traffic. If the a flag is set, it is seen on all links
connected to that ASIC
B - bundle port admin downbundle port has
been taken out of service by administrative action.
If the B flag is set, it is seen on all links associated
to that bundle

b - bundle port oper downbundle port cannot

operate or has been taken out of service by
administrative action. If the b flag is set, it is seen
on all links associated to that bundle

I - bundle admin downbundle has been taken

out of service by administrative action

i - bundle oper downbundle cannot operate or

has been taken out of service by administrative
action

N - node admin downNode has been taken out

of service by administrative action

n - node downnode cannot operate or has been

taken out of service by administrative action

o - other end of link downonly applies to TX

links. Indicates that receiver link port is down

d - data down idle cells are being transmitted or

received, but link is not able to transport data
cells. Flag is set in response to a message from the
ASIC driver. Link ports go into this state during
link startup. It is typically transient and should
only persist if something goes wrong with the
startup process

Cisco IOS XR Troubleshooting

OL-11138-02

4-93

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

Table 4-1

Fabric show Command Flags (continued)

f - failed component downstreamflag is set by

fabric status database if it decides to stop using a
link because of some downstream failure
m - plane multicast down not all fabric queue
ASICs present on the plane are reachable;
multicast forwarding is not operational on this
plane. Unicast traffic is unaffected.

Verifying and Troubleshooting the Fabric Plane State

To verify and troubleshoot the fabric plane state, perform the following procedure.

Note

All fabric troubleshooting should be performed in administration executive (admin EXEC) mode while
you are logged into the default logical router. This allows you to view system-wide parameters.

SUMMARY STEPS
1.

admin

2.

show controllers fabric plane all detail

3.

show controllers fabric plane all statistics

4.

show controllers fabric plane plane_id statistics detail

5.

show control fabric sfe s1 all | include UP.*DOWN

show control fabric sfe s2 all | include UP.*DOWN
show control fabric sfe s3 all | include UP.*DOWN
show control fabric sfe ingress all | include UP.*DOWN (for MSC fabric interface ASICs)
show control fabric sfe fabricq all | include UP.*DOWN (for MSC fabric interface ASICs)

6.

Contact Cisco Technical Support if the problem is not resolved

Cisco IOS XR Troubleshooting

4-94

OL-11138-02

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

DETAILED STEPS

Step 1

Command or Action

Purpose

admin

Enters administration executive (admin EXEC) mode.

Example:
RP/0/RP0/CPU0:router#

Step 2

admin

show controllers fabric plane all detail

Displays system fabric plane information from all fabric

planes.

Example:

Check the number of planes in operation. If there are any

planes that are down (either the Admin State or Oper State
is DOWN), proceed to the Troubleshooting Down Fabric
Planes section on page 4-101.

RP/0/RP0/CPU0:router(admin)# show controllers

fabric plane all

A system can operate with seven planes without

performance degradation. As further planes are removed,
overall capacity degrades, but the system remains
operational. A system must have a minimum of two
operational planes to maintain service. One must be an
odd-numbered plane, the other must be an even-numbered
plane.
Step 3

show controllers fabric plane all statistics

Displays controller fabric statistics for all planes.

Errors are indicated in the last three columns in the output:

Example:
RP/0/RP0/CPU0:router(admin)# show controllers
fabric plane all statistics

CE=correctable errors

UCE=uncorrectable errors

PE=parity errors (in an ASIC memory)

The Out Cells value should be equal or greater than the In

Cells value because of multicast copies created in the fabric.
If the Out Cells value is equal to or greater than the In Cells
value, proceed to Step 4.
If the values in the Out Cells column are much less than the
values in the In Cells column, cells are being dropped. If the
Out Cells value is less than the In Cells value, proceed to the
Verifying and Troubleshooting Up Fabric Planes section
on page 4-99.
If the CE, UCE or PE values are incrementing, contact
Cisco Technical Support. For contact information for
Cisco Technical Support, see the Obtaining Technical
Assistance section on page xi in the Preface.

Cisco IOS XR Troubleshooting

OL-11138-02

4-95

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

Step 4

Command or Action

Purpose

show controllers fabric plane plane_id

statistics detail

Displays controller fabric statistics for a specific plane.

Example:
RP/0/RP0/CPU0:router(admin)# show controllers
fabric plane 0 statistics detail

The output is displayed for unicast and multicast. This

allows you to determine if a unicast problem was hidden by
multicast cells. If the Total received unicast data cells value
is much lower than the Total transmitted unicast data cells,
there are transmission problems. If there is a large drop in
the number of Total transmitted unicast data cells compared
to the Total received unicast data cells, proceed to the
Verifying and Troubleshooting Up Fabric Planes section
on page 4-99.
The Total unicast lost cells and Total multicast lost cells
displays the number of cells that were received by an ASIC
but could not be handled. Dropped multicast cells can be
due to congestion in the fabric. Dropped unicast cells
indicate a more serious problem. If dropped unicast cells are
detected, contact Cisco Technical Support. For Cisco
Technical Support contact information, see the Obtaining
Technical Assistance section on page xi in the Preface.

Step 5

show controllers
UP.*DOWN
show controllers
UP.*DOWN
show controllers
UP.*DOWN
show controllers
include UP.*DOWN
show controllers
include UP.*DOWN

fabric sfe s1 all | include

fabric sfe s2 all | include
fabric sfe s3 all | include
fabric sfe ingress all |

Use these commands on the ASICs in the affected planes to

help isolate the problem to a specific board.
Once you have isolated the problem to a specific board,
contact Cisco Technical Support. For Cisco Technical
Support contact information, see the Obtaining Technical
Assistance section on page xi in the Preface.

fabric sfe fabricq all |

Example:
RP/0/RP0/CPU0:router(admin)# show controllers
fabric sfe s1 all | include UP.*DOWN
RP/0/RP0/CPU0:router(admin)# show controllers
fabric sfe s2 all | include UP.*DOWN
RP/0/RP0/CPU0:router(admin)# show controllers
fabric sfe s3 all | include UP.*DOWN
RP/0/RP0/CPU0:router(admin)# show controllers
fabric sfe ingress all | include UP.*DOWN
RP/0/RP0/CPU0:router(admin)# show controllers
fabric sfe fabricq all | include UP.*DOWN

Step 6

Contact Cisco Technical Support.

If the problem is not resolved, contact Cisco Technical

Support. For Cisco Technical Support contact information,
see the Obtaining Technical Assistance section on
page xi in the Preface.

Cisco IOS XR Troubleshooting

4-96

OL-11138-02

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

Examples
The following example shows how to confirm the number of fabric planes (number of fabric planes in
operation):
RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all detail
Flags: P C
L
A
B
I
N
o
f
m

plane admin down,

p - plane oper down
- card admin down,
c - card oper down
- link port admin down,
l - linkport oper down
- asic admin down,
a - asic oper down
- bundle port admin Down, b - bundle port oper down
- bundle admin down,
i - bundle oper down
- node admin down,
n - node down
- other end of link down d - data down
- failed component downstream
- plane multicast down

Plane Admin
Oper
Down
Total
Down
Id
State
State
Flags
Bundles
Bundles
-----------------------------------------------------0
UP
UP
9
3
1
UP
UP
9
3
2
UP
UP
9
3
3
UP
UP
9
3
4
UP
UP
9
3
5
UP
UP
9
3
6
UP
UP
9
3
7
UP
UP
9
3

The output above shows all eight fabric planes in the up state.
The following output shows Fabric Plane 1 marked as MCAST_DOWN:
RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all detail
Flags: P C
L
A
B
I
N
o
f
m

plane admin down,

p - plane oper down
- card admin down,
c - card oper down
- link port admin down,
l - linkport oper down
- asic admin down,
a - asic oper down
- bundle port admin Down, b - bundle port oper down
- bundle admin down,
i - bundle oper down
- node admin down,
n - node down
- other end of link down d - data down
- failed component downstream
- plane multicast down

Plane Admin
Oper
Down
Total
Down
Id
State
State
Flags
Bundles
Bundles
-----------------------------------------------------0
UP
UP
0
0
1
UP
MCAST_DOWN m
0
0
3
UP
UP
0
0
4
UP
UP
0
0
5
UP
UP
0
0
6
UP
UP
0
0
7
UP
UP
0
0

The fabric plane is operational, but not all destination fabric queue application-specific integrated circuit
(ASICs) can be reached through this fabric plane, as far as the ingress modular services cards (MSCs)
are concerned. Because destinations of multicast traffic cannot be predetermined due to their dynamic
nature, the fabric plane is shown as MCAST_DOWN. Multicast traffic uses the other fabric planes to
deliver data to the appropriate egress MSCs. Unicast traffic continues to use the fabric plane if traffic is

Cisco IOS XR Troubleshooting

OL-11138-02

4-97

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

destined to a fabric queue ASIC that is still reachable. If traffic is destined for a fabric queue ASIC that
is deemed unreachable, the other available fabric plane is used instead of the plane in the
MCAST_DOWN state. To change the operational state of a fabric plane from MCAST_DOWN to up,
contact Cisco Technical Support. See the Obtaining Technical Assistance section on page xi in the
Preface.
The following output shows Fabric Plane 0 as administratively shut down:
RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all detail
Flags: P C
L
A
B
I
N
o
f
m

plane admin down,

p - plane oper down
- card admin down,
c - card oper down
- link port admin down,
l - linkport oper down
- asic admin down,
a - asic oper down
- bundle port admin Down, b - bundle port oper down
- bundle admin down,
i - bundle oper down
- node admin down,
n - node down
- other end of link down d - data down
- failed component downstream
- plane multicast down

Plane Admin
Oper
Down
Total
Down
Id
State
State
Flags
Bundles
Bundles
-----------------------------------------------------0
DOWN
DOWN
P
0
0
1
UP
UP
0
0
2
UP
UP
0
0
3
UP
UP
0
0
4
UP
UP
0
0
5
UP
UP
0
0
6
UP
UP
0
0
7
UP
UP
0
0

The following output shows Fabric Plane 4 as administratively shutdown and powered off:
RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all detail
Flags: P C
L
A
B
I
N
o
f
m

plane admin down,

p - plane oper down
- card admin down,
c - card oper down
- link port admin down,
l - linkport oper down
- asic admin down,
a - asic oper down
- bundle port admin Down, b - bundle port oper down
- bundle admin down,
i - bundle oper down
- node admin down,
n - node down
- other end of link down d - data down
- failed component downstream
- plane multicast down

Plane Admin
Oper
Down
Total
Down
Id
State
State
Flags
Bundles
Bundles
-----------------------------------------------------0
UP
UP
0
0
1
UP
UP
0
0
2
UP
UP
0
0
3
UP
UP
0
0
4
DOWN
DOWN
pPm
0
0
5
UP
UP
0
0
6
UP
UP
0
0
7
UP
UP
0
0

Cisco IOS XR Troubleshooting

4-98

OL-11138-02

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

The following example shows how to display statistics for all fabric planes:
RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all statistics
In
Out
CE
UCE
PE
Plane
Cells
Cells
Cells
Cells
Cells
-------------------------------------------------------------------------------0
736182590951
736183103374
0
0
0
1
736324880091
736325098791
0
0
0
2
736315213586
736315442992
0
0
0
3
736299535716
736299764252
0
0
0
4
736300399513
736300627048
0
0
0
5
736295116556
736295346246
0
0
0
6
736311177372
736311406904
0
0
0
7
736296917574
736297149734
0
0
0

CE Cells= Errored cell with a correctable error detected using FEC code
UCE Cells= Errored cell with an uncorrectable error detected using FEC code
PE Cells= Parity error present within a cell when processed by SFE
The following example shows to display detailed fabric plane statistics for a specified fabric plane:
RP/0/RP0/CPU0:router(admin)# show controllers fabric plane 0 statistics detail
The fabric plane number is 0
Total number of providers for the statistics: 1
Total received data cells: 749351837428
Total received unicast data cells: 749351660427
Total received multicast data cells: 177001
Total transmitted data cells: 749352358340
Total transmitted unicast data cells: 749351671469
Total transmitted multicast data cells: 686871
Total received correctable errored cells: 0
Total received uncorrectable errored cells: 0
Total received parity error cells: 0
Total unicast lost cells: 0
Total multicast lost cells: 0
Last clearing of "show controller fabric plane" counters never

Verifying and Troubleshooting Up Fabric Planes

To verify and troubleshoot the up fabric planes, perform the following procedure.
For Cisco CRS-1 Multishelf Systems the S1 to S2 links and the S2 to S3 links are inter-chassis links. For
Cisco CRS-1 single-shelf systems the links are internal chassis links. The commands used are the same.

SUMMARY STEPS
1.

admin

2.

show controllers fabric link port

show controllers fabric link port
show controllers fabric link port
show controllers fabric link port

3.

show controllers fabric link port s2rx all stat | exclude \ 0 +0 +0$
show controllers fabric link port s3rx all stat | exclude \ 0 +0 +0$

4.

config

s2rx all | include UP.*DOWN.*SM

s3rx all | include UP.*DOWN.*SM
s2tx all | include UP.*DOWN.*SM
s3tx all | include UP.*DOWN.*SM

Cisco IOS XR Troubleshooting

OL-11138-02

4-99

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

5.

controllers fabric link port {s1rx | s1tx | s2rx | s2tx | s3rx | s3tx} location link-id shutdown

6.

Reload the fabric card.

DETAILED STEPS

Step 1

Command or Action

Purpose

admin

Enters administration executive (admin EXEC) mode.

Example:
RP/0/RP0/CPU0:router#

Step 2

admin

show controllers fabric

include UP.*DOWN.*SM
show controllers fabric
include UP.*DOWN.*SM
show controllers fabric
include UP.*DOWN.*SM
show controllers fabric
include UP.*DOWN.*SM

link port s2rx all |

link port s3rx all |
link port s2tx all |
link port s3tx all |

RP/0/RP0/CPU0:router(admin)# show controllers

fabric link port s2rx all | include
UP.*DOWN.*SM
RP/0/RP0/CPU0:router(admin)# show controllers
fabric link port s3rx all | include
UP.*DOWN.*SM
RP/0/RP0/CPU0:router(admin)# show controllers
fabric link port s2tx all | include
UP.*DOWN.*SM
RP/0/RP0/CPU0:router(admin)# show controllers
fabric link port s3tx all | include
UP.*DOWN.*SM
show controllers
| exclude \ 0 +0
show controllers
| exclude \ 0 +0

Repeat this command for each stage and direction and

check if a fabric port link is down (ADMIN UP and OPER
DOWN).
The following down flags may be displayed:

Example:

Step 3

Displays the link state of a fiber port. The command output

shows interchassis link ports that are operationally down,
excluding those that are administratively down and those
that are completely disconnected.

fabric link port s2rx all stat

+0$
fabric link port s3rx all stat
+0$

lLink is down

oDowned link port due to the other end of the link

being down

aASIC is down

A or LASIC (A) or link (L) is administratively shut

down

If the link state of a fabric port is down, proceed to Step 5

to shut down the link.
Displays any link that has an error count, indicating a
problem link even if the link has not been shut down.
If there are UCE errors on a link, shut down the link to avoid
traffic loss. Proceed to Step 5 to shut down the link.

Example:
RP/0/RP0/CPU0:router(admin)# show control
fabric link port s2rx all stat | exclude \ 0 +0
+0$
RP/0/RP0/CPU0:router(admin)# show control
fabric link port s3rx all stat | exclude \ 0 +0
+0$

Cisco IOS XR Troubleshooting

4-100

OL-11138-02

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

Step 4

Command or Action

Purpose

config

Enters administration configuration mode.

Example:
RP/0/RP0/CPU0:router(admin)# config

Step 5

controllers fabric link port {s1rx | s1tx |

s2rx | s2tx | s3rx | s3tx} location link-id
shutdown

Places a link out of service. This command closes both ends

of the link.

Note that performing such an operation should be

conducted in conjunction with Cisco Technical Support
Example:
engineers. For Cisco Technical Support contact
RP/0/RP0/CPU0:router(admin-config)# controllers
information, see the Obtaining Technical Assistance
fabric link port s1rx 0/0/cpu0 1 shutdown
section on page xi in the Preface.
Step 6

Reload the fabric card containing the out of service

link.

See the Managing the Router Hardware module of

Cisco IOS XR System Management Configuration Guide for
information on reloading the fabric card.
If the problem is not resolved, Cisco Technical Support. For
Cisco Technical Support contact information, see the
Obtaining Technical Assistance section on page xi in the
Preface.

Troubleshooting Down Fabric Planes

An individual link placed in a down state can be sustained without impact to the overall operation or
forwarding capacity of the Cisco CRS-1 Multishelf System. At each stage of the fabric, there is a greater
bandwidth capacity than can actually be generated by the modular services cards (MSCs). The egress
side of the switch fabric has at least double the capacity of the ingress.
The loss of a single link reduces the switch fabric capacity on a single plane by 2.5 Gbps or
approximately 1.5625 Gbps when considering 8b/10b encoding and cell tax overheads. The actual
effective loss of bandwidth varies depending on where the link loss takes place. If a link were to be lost
between the ingress queue and S1 stage or S1 and S2 stage, the reduction is against a capacity of 80 Gbps
or 50 Gbps effective bandwidth. A link loss between S2 and S3 stages or S3 and fabric queue stages
means that the reduction is against a capacity of 160 Gbps or 100 Gbps effective bandwidth.
Consider a situation in which a set of S1 links was lost on a single plane. Six or seven links would need
to be taken out of service in order to reduce the effective bandwidth capacity by approximately 10 Gbps.
It is, therefore, possible to contend that because the MSC is capable of transmitting at 40 Gbps, there is
still sufficient capacity to carry the traffic without loss.
The system is designed to operate with seven out of eight fabric planes in operation and provide full
line-rate forwarding capability, because it provides approximately 42 Gbps to each MSC. If another
fabric plane is removed, forwarding capacity would be reduced to approximately 34 Gbps.

SUMMARY STEPS
1.

admin

2.

show controllers fabric plane all detail

3.

Bring up the plane:

a. configure

Cisco IOS XR Troubleshooting

OL-11138-02

4-101

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

b. no controllers fabric plane plane-id shutdown

c. end
4.

show running-config

5.

show controllers fabric connectivity all

6.

show controllers fabric sfe {s1 | s2 | s3 | ingressq | fabricq} all

7.

show controllers fabric bundle port all

DETAILED STEPS

Step 1

Command or Action

Purpose

admin

Enters administration executive (admin EXEC) mode.

Example:
RP/0/RP0/CPU0:router#

Step 2

admin

show controllers fabric plane all detail

Displays information about the fabric plane.

Example:

Verify that the down plane is not administratively down. If the

Admin State is down, the operational state should also be down.

RP/0/RP0/CPU0:router(admin)# show
controllers fabric plane all detail

Step 3

configure
no controllers fabric plane plane-id
shutdown
end

If the plane is administratively down and the only flag is P,

bring the plane up. Proceed to Step 3 to bring up the plane. See
the Flags section on page 4-93 for detailed information on
flags.

If the plane is administratively up but the flag is m, the plane is

multicast down and some destinations (for example, MSCs and
route processors) are having connectivity issues to the fabric on
that plane. Proceed to Step 5. See the Flags section on
page 4-93 for detailed information on flags.

If the plane is administratively down and the flag is p, the plane

is operationally down. Proceed to Step 6. See the Flags
section on page 4-93 for detailed information on flags.

The following tasks must be completed to bring up the plane:

configureEnters administration configuration mode

no controllers fabric plane plane-id shutdownPerforms a

graceful shutdown of the fabric plane to ensure that data is no
longer flowing through the plane before a fabric
reconfiguration or fabric plane migration

endSaves the configuration change

Example:
RP/0/RP0/CPU0:router(admin)#configure
RP/0/RP0/CPU0:router(admin_config)# no
controllers fabric plane 3 shutdown
RP/0/RP0/CPU0:router(admin_config)# end

Cisco IOS XR Troubleshooting

4-102

OL-11138-02

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

Step 4

Command or Action

Purpose

show running-config

For Cisco CRS-1 Multishelf Systems only.

Displays the contents of the administrative running configuration.

Example:
RP/0/RP0/CPU0:router(admin)# show
running-config

Verify that the plane is set. In the Cisco CRS-1 Multishelf System,
the command output indicates which fiber module in the fabric card
chassis (FCC) is connected from the plane.
The following is an example of the output:
controllers fabric plane 0 topology single-module location
F0/SM4/FM

Step 5

show controllers fabric connectivity all

Displays controller fabric connectivity information for all fabric

ports.

Example:

For the down plane, check if the MSCs and route processors have
receive and transmit connectivity to the fabric.

RP/0/RP0/CPU0:router(admin)# show
controllers fabric connectivity all

If there is no transmit and receive connectivity to the fabric,

contact Cisco Technical Support. For Cisco Technical Support
contact information, see the Obtaining Technical Assistance
section on page xi in the Preface.

If there is no transmit and receive connectivity to a MSC that is

not required, remove it from the system or power it down. This
will bring the plane back into the up state. See Installing the
Cisco CRS-1 Carrier Routing System 16-Slot Line Card
Chassis at the following URL for information on removing
MSCs:
[Link]
upport_series_home.html

Step 6

show controllers fabric sfe {s1 | s2 |

s3 | ingressq | fabricq} all

Displays ASIC information.

Verify that all the ASICs are in the administrative and operational
up state.

Example:
RP/0/RP0/CPU0:router(admin)# show
controllers fabric sfe s1 all

Step 7

show controllers fabric bundle port all

If all ASICs are in the up state, proceed to Step 7.

If all the ASICs are not in the up state, contact Cisco Technical
Support. For Cisco Technical Support contact information, see
the Obtaining Technical Assistance section on page xi in the
Preface.

For Cisco CRS-1 Multishelf Systems only.

Displays bundle port information.

Example:
RP/0/RP0/CPU0:router(admin)# show
controllers fabric bundle port all

Verify that all bundle ports are in the operational up state.

Note

If a bundle port is in the operational down state, contact Cisco

Technical Support, see the Obtaining Technical Assistance
section on page xi in the Preface.
For this release, three of the nine ports will be down
because there is no third rack connected to the fabric card
chassis. The down ports are 6, 7, and 8.
If a bundle port is in the administrative down state, proceed to
Step 3 to bring up the plane.

Cisco IOS XR Troubleshooting

OL-11138-02

4-103

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

Examples
The output from the show controllers fabric plane all detail command displays information on the
fabric plane:
RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all detail
Flags: P - plane admin down,
p - plane oper down
C - card admin down,
c - card oper down
L - link port admin down,
l - linkport oper down
A - asic admin down,
a - asic oper down
B - bundle port admin Down, b - bundle port oper down
I - bundle admin down,
i - bundle oper down
N - node admin down,
n - node down
o - other end of link down d - data down
f - failed component downstream
m - plane multicast down
Plane Admin
Oper
Down
Total
Down
Id
State
State
Flags
Bundles
Bundles
-----------------------------------------------------0
UP
UP
9
3
1
UP
UP
9
3
2
UP
DOWN
p
0
0
3
UP
DOWN
p
0
0
4
UP
MCAST_DOWN m
9
3
5
UP
UP
9
3
6
UP
DOWN
p
0
0
7
UP
DOWN
p
0
0

The output from the show controllers fabric bundle port all command displays the status of the
bundle ports:
RP/0/RP0/CPU0:router(admin)# show controllers fabric bundle port all
Flags: P C
L
A
B
I
N
o
f
m

plane admin down,

p - plane oper down
- card admin down,
c - card oper down
- link port admin down,
l - linkport oper down
- asic admin down,
a - asic oper down
- bundle port admin Down, b - bundle port oper down
- bundle admin down,
i - bundle oper down
- node admin down,
n - node down
- other end of link down d - data down
- failed component downstream
- plane multicast down

Bundle Port
Admin
Oper
R/S/M/P
State
State
-----------------------------0/SM0/SP/0
UP
UP
0/SM0/SP/1
UP
UP
0/SM0/SP/2
UP
UP
0/SM1/SP/0
UP
UP
0/SM1/SP/1
UP
UP
0/SM1/SP/2
UP
UP
0/SM4/SP/0
UP
UP
0/SM4/SP/1
UP
UP
0/SM4/SP/2
UP
UP
0/SM5/SP/0
UP
UP
0/SM5/SP/1
UP
UP
0/SM5/SP/2
UP
UP
1/SM0/SP/0
UP
UP
1/SM0/SP/1
UP
UP

Cisco IOS XR Troubleshooting

4-104

OL-11138-02

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

1/SM0/SP/2
1/SM1/SP/0
1/SM1/SP/1
1/SM1/SP/2
1/SM4/SP/0
1/SM4/SP/1
1/SM4/SP/2
1/SM5/SP/0
1/SM5/SP/1
1/SM5/SP/2
F0/SM0/FM/0

UP
UP
UP
UP
UP
UP
UP
UP
UP
UP
UP

UP
UP
UP
UP
UP
UP
UP
UP
UP
UP
UP

.
.
.

Cisco IOS XR Troubleshooting

OL-11138-02

4-105

Chapter 4

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

Cisco IOS XR Troubleshooting

4-106

OL-11138-02