Transcription
Troubleshooting Catalyst 4K and 6KBRKRST-3067BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public2
AgendaTroubleshooting Catalyst 4500 Architecture Interface Packet DropsCatalyst 4500-E Series Switches High CPU IOS and IOS-XE Crashes IOS-XE LicensesCatalyst 4500 Series SwitchesBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public3
Architecture OverviewShared PacketMemory Intelligent SupervisorsCPUTCAMs1‒ Supervisor Engine 7-E, 6-E, 6L-E, V-10GE, V, IV,NFL2‒ II-Plus-10GE, II-Plus, II-Plus Switching ASICsPacket ProcessorForwarding Engine‒ Packet Processor‒ Forwarding EngineSupervisor Specialised HardwareLine Card‒ TCAMs for ACLs, QoS, L3 forwardingStub ASICs‒ NetFlow (NFL) for statistics gathering Transparent Line CardsFront Panel Ports‒ Wire-rate, oversubscribed, PoE‒ 10/100, 10/100/1000, GE, 10GESupervisor and Blocking Line Card Block DiagramBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public4
Next Generation Supervisors Supervisor 6L-E Supervisor 6-E Supervisor 7-EBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public5
Troubleshooting Interface Drops24Gbps Packet drops in transmit directionPacket ProcessingEngine46xx3GE Series Linecard‒ TX queue is fullStub ASICs‒ Oversubscription‒ Pause frames receivedFront Panel Ports Packet drops in receive direction‒ Minimal buffer on receiving queue48GbpsPacket ProcessingEngine47xx6GE Series LinecardStub ASICsFrontPanel Ports24GbpsBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public6
Troubleshooting Interface DropsTX Queue is FullSUP6-E# show interfaces g2/47 counters detail begin DropsPortTx-Drops-Queue-1 Tx-Drops-Queue-2 Tx-Drops-Queue-3 Tx-Drops-Queue-4Gi2/470000PortTx-Drops-Queue-5 Tx-Drops-Queue-6 Tx-Drops-Queue-7 Tx-Drops-Queue-8Gi2/4700037748571SUP6-E# show interfaces g2/47 counters detail begin DropsPortTx-Drops-Queue-1 Tx-Drops-Queue-2 Tx-Drops-Queue-3 Tx-Drops-Queue-4Gi2/47PortTx-Drops-Queue-5 Tx-Drops-Queue-6 Tx-Drops-Queue-7 Tx-Drops-Queue-8Gi2/4700037874327Default queue 8 when QoS isdisabled Next steps‒ Consider implementing a QoS policy to share traffic amongst queues‒ Increment default output queue-limit of 40 packets to allow for more bufferspaceBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public7
Troubleshooting Interface DropsPeer is Sending Pause FramesSUP6-E# show interfaces g2/47 counters detail begin sGi2/4701300PauseFramesDrop0SUP6-E#show platform software interface g2/47 all inc BusySwitch Phyport Gi2/7 BusyStatus : ON, PauseStatus : OFFBusyBitActivityCount: 474Ignore Subport Busy: FalseBusyBit is activated when received pause framesexceed threshold.%C4K HWPORTMAN-4-BLOCKEDTXQUEUE: Blocked transmit queue HwTxQId7 on Switch Phyport Gi2/47,count 116Error log displayed when BusyBit is setBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public8
Troubleshooting Interface DropsPeer is Sending Pause Frames Continued Next steps‒ Disable flow control receive on the local interface‒ Disable flow control send on the peerBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public9
Troubleshooting Interface DropsMinimal buffer on receiving stub ASIC (applicable to linecards with oversubscriptionratio) Next steps‒ Enable flow control send on the local interface‒ Enable flow control receive on the peerSup6-E# show0 inputSup6-E# show0 inputinterfaceerrors, 0interfaceerrors, 0gi1/13CRC, 0gi1/13CRC, 0 include overrunframe, 86432 overrun, 0 ignored include overrunframe, 206658 overrun, 0 ignoredSup6-E# show interface gi1/13 counter all begin i1/1320665800PauseFramesDrop0Sup6-E# show platform software interface g1/13 stub stat in OverrunOverrunPackets: 206658(look for Rx Stats)BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public10
Troubleshooting High CPUFunctions of the 4500 CPU Sends and receives control plane trafficShared PacketMemory‒ STP, CDP, PAgP, VTP, DTP, routing protocolsSupervisor 6-ETCAMsCPU Program dynamic entries into hardwarePacket Processor‒ ACLs, CEF entries Manage access to the switchForwardingEngineTx QueueMemory‒ Telnet, SSHLine Card Manage system componentsStub ASICs‒ Fan tray, power supply, PoEFront Panel PortsBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public11
Troubleshooting High CPUSoftware-forwarded Data Traffic Packets copied to CPU but originally switched in hardware‒ Host mac address learning Packets punted to CPU for processing‒ Routing updates, BPDUS, flood of traffic Packets sent to the CPU for forwarding‒ AppleTalk, IPXBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public12
Troubleshooting High CPUUsage on IOS ThreadsDual coreSUP7-E# show processes cpu sorted detailCore 0: CPU utilization for five seconds: 4%; one minute: 2%; five minutes: 2%ProcessCore 1: CPU utilization for five seconds: 6%; one minute: 3%; five minutes: 2%PIDT C TIDRuntime(ms) Invoked uSecs 311131416 06.70605 A 4.39062 4.12207 0iosd9433L 1 11383 9848964669930 07.11A 3.223.000iosd9433L 1 943319612056644042 06.22A 5.445.110iosd9433L 1 11386 8291863000.00A 0.000.000iosd71I555758787502 01.11R 1.001.000Cat4k Mgmt HiPri52I42215762152734 00.33R 0.330.330IDB Work72I10334451988579 00.33R 0.330.330Cat4k Mgmt LoPri89I413200.00R 0.000.000ExecSUP6-E#sh proc cpu sortedCPU utilization for five seconds: 6%/0%; one minute: 6%; five minutes: 6%PID Runtime(ms)InvokeduSecs5Sec1Min5Min TTY Process511229497287703481401 3.75% 3.78% 3.79%0 Cat4k Mgmt LoPri502556152 16464011155 2.47% 2.49% 2.47%0 Cat4k Mgmt HiPri920380320 0.07% 0.00% 0.00%0 Ethchnl11120142140 0.07% 0.00% 0.00%0 Exec3820443803165 0.07% 0.06% 0.07%0 IDB Work9944600106342419 0.07% 0.04% 0.05%0 CDP ProtocolBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public13
Troubleshooting High CPUUsage on platform dependent threads These processes use CPU under the Cat4K Mgmt HiPri and Cat4KMgmt LoPri HiPri: within CPU targetLoPri: exceeds CPU targetSUP6-E#show platform health%CPU%CPURunTimeMaxPriority Average %CPU TotalTarget Actual Target ActualFgBg 5Sec Min Hour CPURkiosObflMan0.500.0040 100 500000 0:15GalChassisVp-review3.000.091033 100 500000 5:34S2w-JobEventSchedule 10.000.00100 100 500000 0:00Stub-JobEventSchedul 10.000.50105 100 500000 13:58Lj-poll1.000.0120 100 500000 1:18StatValueMan Update1.000.0110 100 500000 2:18Pim-review0.100.0010 100 500000 0:18Ebm-host-review1.000.0080 100 500000 0:05Ebm-host-util-review1.000.00100 100 500000 0:00Ebm-port-review0.100.0010 100 500000 0:01Protocol-aging-revie0.200.0020 100 500000 0:00EbmHostRedundancyMan2.000.00200 100 500000 0:00Acl-Flattener1.000.00100 100 500000 0:00IrmFibThrottler Thro2.000.0070 100 500000 0:26BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public14
Troubleshooting High CPUTraffic in the CPU Queue Events that trigger packets to be sent to the CPU queueSUP7-E# show platform cpu packet statisticsPackets Dropped In Processing OverallTotal5 sec avg 1 min avg 5 min avg 1 hour avg-------------------- --------- --------- --------- ---------368028480000Packets Dropped In Processing by CPU eventEventTotal5 sec avg 1 min avg 5 min avg 1 hour avg----------------- -------------------- --------- --------- --------- ---------Sa Miss367783320000Input Acl Fwd190000Input ACl Copy244970000 Next stepsMac flap?‒ Identify the event that is dropping packets so that we know what type ofpacket to look for in further CPU debuggingBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public15
Troubleshooting High CPU - ToolsIn-Built CPU Sniffer Packets that are punted to CPU are sent to a circular buffer Buffer retains 1024 packets at a time and does not occupy CPU cyclesSUP6-E# debug platform packet all bufferplatform packet debugging is onSUP6-E# show platform cpu packet bufferedIngress interface of packetTotal Received Packets Buffered: 1024------------------------------------Index 0:3 days 23:23:18:54927 - RxVlan: 1006, RxPort: Gi1/1Priority: Normal, Tag: No Tag, Event: 11, Flags: 0x40, Size: 64Eth: Src 00:00:0B:00:00:00 Dst 00:22:90:E0:D6:FF Type/Len 0x0800Ip: ver:IpVersion4 len:24 tos:0 totLen:46 id:0 fragOffset:0 ttl:64 proto:tcpsrc: 10.10.10.100 dst: 172.16.100.100 hasIpOptions firstFragment lastFragmentRemaining data:0: 0x0 0x64 0x0 0x64 0x0 0x0 0x0 0x0 0x0 0x0Source/destination address10: 0x0 0x0 0x50 0x0 0x0 0x0 0x8A 0x37 0x0 0x020: 0x0 0x1 0xB5 0x77 0x6A 0x7EBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public16
Troubleshooting High CPU - ToolsSPAN Capture for CPU Monitor CPU bound traffic both in send and receive directionSUP6-E# show running-config include monitormonitor session 1 source cpu rxmonitor session 1 destination interface Gi1/48SUP6-E# show monitor session 1Session 1--------Type: Local SessionSource Ports:RX Only: CPUDestination Ports: Gi1/48Encapsulation: NativeIngress: DisabledLearning: DisabledConnect laptop with Wireshark Next steps‒ Is there a pattern and is this legitimate traffic?BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public17
IOS Process Crash Reasons for a crash‒ Bus error, forced by software, address errors, watchdog timeouts All crashes generate a crashinfo‒ Present in crashinfo or platform crashdump------------------ show platform crashdump -----------------Last powerfail: 03/09/2006 02:10:24Current time: 02/19/2007 16:09:59Verify time of outageLast reload status: 00008800 038D0000Last crash: 02/19/2007 15:44:40Build: 12.2(31)SG ENTSERVICESTracebacks that need to be decodedbuildversion addr: 12288034pc 112088A4 lr 11208824 msr 20029030 vector 00000300cr 20000042 ctr 11207418 xer 00000000r0 8000FBCE r1 13FFE680 r2 0000C000 r3 13FFE6C8 r4 13FFE620 r5 00000002 r6 00000000 r7 000000FFr8 FFFFFFFF r9 00000000 r10 00000002 r11 00000008BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public18
IOS-XE Process CrashIOS-XE vs. IOS IOS-XE kernel is Linux Runs several different processes‒ IOSd, FFM, HA, licensing IOS runs as one process in a single unprotected memory space IOS-XE crash infrastructure collects crashinfo files for any failingprocessBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public19
IOS-XE Process Crash Files generated upon a crash: system kernel info, crashinfo file andcore file By default, crashinfo files are stored in the crashinfo directory (135MBEnables generation of processpartition of bootflash)core dump fileSup7-E(config)# exception coredump Generation of core files requires configurationSUP7-E#show exception informationException configuration informationCoredump file - disabled,compressedMaximum number of filesCore - 10 file(s)Process crashinfo - 10 file(s)Configured storage devices1 - crashinfo:2 - not assigned3 - not assignedDump protocol - not configuredBRKRST-3067Default – 10 crashinfo files and 10core files saved(not supported) 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public20
IOS-XE Licensing (CSL)Feature Set LicenseFeature IpbaseLanbase IpbaseLanbaseIPbaseLanbaseLanbaseBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public21
IOS-XE Licensing (CSL) Permanent license is node-locked License is for a chassis UDI (Universal Device Identifier), butstored on Supervisor bootflash License synced to hot standby supervisor No Product Activation Key (PAK) is generated for customerBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public22
License Show CommandsSup7e# show license allLicense Store: Primary License Storage1StoreIndex: 0 Feature: internal service Version: 1.0 License Type: EvaluationEvaluation period left: 23 hours 59 minutesLicense State: Active, Not in Use, EULA2accepted StoreIndex: 2 Feature: entservices Version: 1.0License Type: PermanentLicense State: Active, In UsePermanent node locked licenseLicense Store: Dynamic License StorageStoreIndex: 0 Feature: entservices Version: 1.0License Type: EvaluationEvaluation total period: 8 weeks 4 daysEvaluation period left: 4 weeks 3 daysLicense State: Inactive Temporary LicenseBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public23
Troubleshooting License InstallationLicense must be installed and operational The system will boot up with default Lanbase license level if nolicenses are installedSup7e# dir bootflash:44268 Jan 4 2011 21:46:41 7slot ent FOX1418GEW0 20110103155106655.licSup7e#license install bootflash:7slot ent FOX1418GEW0 20110103155106655.licInstalling licenses from "bootflash:7slot ent FOX1418GEW0 s.Successful:Supported1/1 licenses were successfully installed0/1 licenses were existing licenses0/1 licenses were failed to installBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public24
Troubleshooting License Installation Is the license operational?Sup7e#show license allLicense Store: Primary License StorageStoreIndex: 2 Feature: entservices Version: 1.0License Type: PermanentLicense State: Active, Not in UseLicense Count: Non-CountedLicense Priority: Medium Reboot required to make the license operationalSup7e# show versionCisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSAL-M), Version03.01.00.SG RELEASE SOFTWARE (fc4)License Information for 'WS-X45-SUP7-E'License Level: lanbaseType: Default. No valid license foundNext reboot license Level: entservicesBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public25
Troubleshooting License InstallationLicense Bootlevel: HA Upgrade Standby supervisor will always boot to the same license level as thatof Active supervisor If running SSO and Active and Standby are running different licenselevels, need to manually set level.Sup7e(config)#license boot level entservicesBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public26
Troubleshooting Catalyst 6500Series Switches
AgendaTroubleshooting Catalyst 6500 Sup720 Architecture Sup720 Layer 2 and Layer 3 Unicast Troubleshooting High CPU utilisation Virtual Switch System (VSS) Troubleshooting Sup2T Architecture Sup2T Layer 3 Unicast TroubleshootingBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public28
Supervisor 720/PFC3 ArchitectureLayer2 Control-planeE.g., LACP, BPDU andhardware programmingLayer3 Control-planeE.g., OSPF, BGP, SNMPReplication engineE.g., Multicast, SPANL3/L4 forwardingSFP /SFP GETXACECounterMSFC 3FlashDRAMFlashDRAMRPCPU1 GbpsSPCPU1 GbpsSwitch Fabric16 x 20G TracesTraces # 1 to 16NetFlowPort ASICQoSAdjFIBACLTCAM TCAM TCAM TCAML3/4 EngineMET20 GbpsEOBCFabric InterfaceandReplication EngineDBUSRBUSL2 forwardingL2 EnginePFC3L2 CAM (64K)16 Gbps BusIntegrated SwitchFabricBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public29
L2 Packet Flow Troubleshooting Check the L2 forwarding engine counters Verify the channels used in the flowMETPortPortASICASICFabricInterface &ReplicationEnginePortASICLayer 2Layer 2EngineEngineL3/4DFC3 EnginePortASICFabricInterface &ReplicationEngine4 x 1x10GE port ASICWS-X6704Module 8METSwitch FabricMETFabricInterface &ReplicationEnginePortASICBRKRST-3067PortASICLayer 2Layer 2EngineEngineL3/4DFC3 Engine 2013 Cisco and/or its affiliates. All rights reserved.FabricInterface &ReplicationEnginePortPortASICASICWS-X6748Module 7MET4 x 12xGE port ASICCisco Public30
Layer 2 Learning and Forwarding Layer 2 forwarding is based on {VLAN, MAC} pairs‒ Entries are stored based on result of hash done on MAC and VLAN MAC learning is done per PFC or DFC‒ Each PFC/DFC maintains separate L2 CAM table PFC and DFCs age entries independently‒ Refreshing of entries based on “seeing” traffic from specific host‒ New learns on one forwarding engine communicated to other engines via MACSync process MAC table size:‒ 64K entries on PFC3A / 3B / 3BXL and DFC3A / 3B / 3BXL‒ 96K entries on PFC3C / 3CXL and DFC3C / 3CXLBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public31
L2 Unicast TrafficTe8/1Te8/3Te8/3Host27.0.1.2Te8/1R2 Verify that ARP entry is present for both hostsCat6KHost17.0.1.1BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco eVlan700Gig7/3Hardware Addr0011.bced.e400TypeARPAGig8/2Age (min)-Hardware Addr000b.fca2.fe0aGig7/5Cat6K#show ip arp 7.0.1.2Protocol AddressInternet 7.0.1.2Age (min)-Gig8/3Cat6K#show ip arp 7.0.1.1Protocol AddressInternet 7.0.1.1Po1Po2Po1R132
L2 Unicast TrafficTe8/3Te8/3Host27.0.1.2Te8/1Verify the interface that both mac-addresses are learnt onTe8/1R2Cat6K#show mac-address-table address 000b.fca2.fe0a vlan 700Legend: * - primary entry; age - seconds since last seenvlanmac addresstypelearnageports------ ---------------- -------- ----- ---------- --------------------Module 7[FE 1]:* 700 000b.fca2.fe0aModule 7[FE 2]:* 700 Two Forwarding EnginesPo2ports------ ---------------- -------- ----- ---------- -------------------Module 8[FE 1]:* 700 0011.bced.e400dynamic Yes170Po1Module 8[FE 2]:* 700 T-3067Gig7/4ageGig8/1learnGig7/3typeGig8/2mac addressGig7/5vlanGig8/3Cat6K#show mac-address-table address 0011.bced.e400 vlan 700Legend: * - primary entry; age - seconds since last seen 2013 Cisco and/or its affiliates. All rights reserved.Cisco PublicPo1R133
L2 Unicast Traffic 2013 Cisco and/or its affiliates. All rights reserved.Te8/3Po1Cisco 1.1BRKRST-3067Te8/3Cat6K#show mac-address-table address 000b.fca2.fe0a vlan 700 allLegend: * - primary entry; age - seconds since last seenvlanmac addresstypelearnageports------ ---------------- -------- ----- ---------- -------------------Module 1:700 000b.fca2.fe0adynamic Yes170Po2Active Supervisor:700 000b.fca2.fe0adynamic Yes170Po2Standby Supervisor:700 000b.fca2.fe0adynamic Yes170Po2Module 7[FE 1]:* 700 000b.fca2.fe0adynamic Yes50Po2Module 7[FE 2]:* 700 000b.fca2.fe0adynamic Yes170Po2Module 8[FE 1]:700 000b.fca2.fe0adynamic Yes170Po2Module 8[FE 2]:700 000b.fca2.fe0adynamic Yes170Po2Gig8/3PrimaryEntryHost27.0.1.2Te8/1Verify that there is an entry for all fowarding engines (FE)Te8/1R2Po2Po1R134
Check load balancing configuration.Use ingress Module number incommand in case per-module loadbalancing is configured (SXH imagesand later)Te8/3Te8/3Host27.0.1.2Te8/1R2Te8/1L2 Unicast TrafficR1#show etherchannel load-balance module 1EtherChannel Load-Balancing Configuration:Po1dst-ipmpls label-ipEtherChannel Load-Balancing Addresses Used Per-Protocol:Gig7/4Gig8/1R1#show etherchannel load-balance hash-result interface po1 ip 7.0.1.2Gig7/3MPLS: Label or IPGig8/2IPv6: Destination IP addressGig7/5IPv4: Destination IP addressMode is “dst-ip”. Only use dest IP as argument.Prior to 12.2(33)SXH, use test etherchannel loadbalance (same arguments) on the SPGig8/3Non-IP: Destination MAC addressCat6KPo2Computed RBH: 0x1Would select Gi8/1 of Po1Link selected is Gi8/1 in Po1 of R1 for trafficto 7.0.1.2 leaving R1Host17.0.1.1BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco PublicPo1R135
L3 Packet Flow Troubleshooting Check L3/L4 tables Confirm HW and SW forwarding tables are synchronisedMETPortPortASICASICFabricInterface &ReplicationEngineLayer 2Layer 2EngineEngineL3/4DFC3 EnginePortPortASICASICFabricInterface &ReplicationEngine4 x 1x10GE port ASICWS-X6704Module 8METSwitch FabricMETFabricInterface &ReplicationEnginePortASICBRKRST-3067PortASICLayer 2Layer 2EngineEngineL3/4DFC3 EngineFabricInterface &ReplicationEnginePortASIC 2013 Cisco and/or its affiliates. All rights reserved.PortASICWS-X6748Module 7MET4 x 12xGE port ASICCisco Public36
L3 Unicast TrafficTe8/3SW3328, type #show ip route 9.0.1.0 include viaKnown via "eigrp 700", distance 90, metricRedistributing via eigrp 700* 7.2.1.2, from 7.2.1.2, 00:21:58 ago, via7.5.1.2, from 7.5.1.2, 00:21:58 ago, via7.4.1.2, from 7.4.1.2, 00:21:58 ago, via7.3.1.2, from 7.3.1.2, 00:21:58 ago, via7.1.1.2, from 7.1.1.2, 00:21:58 ago, viaTe8/1Host29.0.1.2Te8/1R2Equal Cost Routes to thedestination prefixCat6KNext hop used for SW based CEF(SW forwarding data path)Host18.0.1.1BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco PublicGig7/3Gig7/4Gig8/1Next hop used for HW based CEF(HW forwarding path). Note: “0” is used for bothsrc and dest L4 port numbers as test flow wasICMP echoGig8/2Check which link between R1 andCat6K is chosen.HWGig7/5R1#show mls cef exact-route 8.0.1.1 0 9.0.1.2 0Interface: Vl705, Next Hop: 7.5.1.2, Vlan: 705, Destination Mac:0050.f0f8.7400Gig8/3R1#show ip cef exact-route 8.0.1.1 9.0.1.28.0.1.1- 9.0.1.2: Vlan701 (next hop 7.1.1.2)R137
FIB/Adjacency TablesL3 FIB Table Programming Flowshow ip route (RIB)Verify Layer 3IOS Routing Table (RP)Verify Layer 2 rewriteIOS ARP Cache Table (RP)show ip cefIOS FIB Table (RP)IOS Adjacency Table (RP)IOS FIB Table (SP/DFC)IOS Adjacency Table (SP/DFC)MLS FIB Table (SP/DFC)MLS Adjacency Table (SP/DFC)remote commandmodule mod showip cefshow mls cef lookup ipaddress mod BRKRST-3067show ip arpshow ip cefadjacencyremote commandmodule mod show adjacencydetailshow mls cef adjacencyentry 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public38
L3 Unicast TrafficTe8/3Te8/3Cat6K#show ip route 9.0.1.0 i viaKnown via "eigrp 700", distance 90, metric 3072, type internalRedistributing via eigrp 700* 7.7.1.2, from 7.7.1.2, 00:07:33 ago, via TenGigabitEthernet8/37.6.1.2, from 7.6.1.2, 00:07:33 ago, via Cat6K#show vlan internal usageHost18.0.1.1BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco PublicGig7/3Gig7/4Gig8/1[snip]L3 Interface map internally to a “1-port”VLANGig8/21091 TenGigabitEthernet8/1Gig7/51090 TenGigabitEthernet8/3Gig8/3[snip]R139
L3 Unicast #show ip cef 9.0.1.2Te8/1R2nexthop 7.6.1.2 TenGigabitEthernet8/1nexthop 7.7.1.2 TenGigabitEthernet8/3Cat6K#show ip cef exact-route 8.0.1.1 9.0.1.28.0.1.1 - 9.0.1.2 IP adj out of TenGigabitEthernet8/1, addr 7.6.1.2Cat6K#show ip cef adjacency TenGigabitEthernet 8/1 7.6.1.27.6.1.2/32Cat6Kattached to TenGigabitEthernet8/19.0.0.0/8HWAdjacencyTe8/1, 000f.f8e4.d000 (Hash: 007F)Te8/3, 000f.f8e4.d000 (Hash: 7F80)Gig7/4108749 9.0.0.0/8Gig8/1PrefixGig7/3IndexGig8/2Codes: decap - Decapsulation, - Push LabelGig7/5Cat6K#show mls cef lookup 9.0.1.2 mod 7Gig8/3nexthop 7.6.1.2 TenGigabitEthernet8/1Cat6K#show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 module 7Interface: Te8/3, Next Hop: 7.7.1.2, Vlan: 1090, Destination Mac: 000f.f8e4.d000Host18.0.1.1BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco PublicR140
L3 Unicast TrafficCat6K#show adjacency ten 8/3 7.7.1.2 detailProtocol Aggregate HW adjacency statistics(SW collects it from all DFC/PFC’s forall prefixes linked to this adjacency)2001 packets, 228114 bytesepoch 0Rewrite information(Dmac Smac 0800): verify it is conformwith next hop rewrite infosourced in sev-epoch 774Encap length 14000FF8E4D0000050F0F874000800ARPTo get HW adjacency statisticfor this prefix on this moduleCat6K#show mls cef lookup 9.0.1.2 detail mod 7Codes: M - mask entry, V - value entry, A - adjacency index, P - priority bitD - full don't switch, m - load balancing modnumber, B - BGP Bucket selV0 - Vlan 0,C0 - don't comp bit 0,V1 - Vlan 1,C1 - don't comp bit 1RVTEN - RPF Vlan table enable, RVTSEL - RPF Vlan table selectFormat: IPV4 DA - (8 xtag vpn pi cr recirc tos prefix)Format: IPV4 SA - (9 xtag vpn pi cr recirc prefix)M(108749 ): E 1 FFF0 0 0 0255.0.0.0V(108749 ): 8 1 00 0 0 09.0.0.0BRKRST-3067Start adjacency pointer is 294933, 14 1 15adjacencies linked to the prefix(A:294933 ,P:1,D:0,m:14,B:0) 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public41
L3 Unicast TrafficCat6K#show mls cef adjacency entry 294933 to 294947 mod 7Index: 294933smac: 0050.f0f8.7400, dmac: 000f.f8e4.d000mtu: 9234, vlan: 1091, dindex: 0x0, l3rw vld: 1packets: 0, bytes: 0Index: 294947smac: 0050.f0f8.7400, dmac: 000f.f8e4.d000mtu: 9234, vlan: 1090, dindex: 0x0, l3rw vld: 115 HW adjacencies linked to thisprefix: which one is really used ?packets: 0, bytes: 0Cat6K#show mls cef adjacency entry 294933 to 294947 mod 7 i ackets: 0, bytes: 0packets: 0, bytes: 0packets: 0, bytes: 0Based on the packet counts, we see that the 4th adjacency(entry 294936) is being used.packets: 2001, bytes: 236118packets: 0, bytes: 0packets: 0, bytes: 0Verify that the rewrite information is correct for theadjacency.packets: 0, bytes: 0Cat6K#show mls cef adjacency entry 294936 detail mod 7Index: 294936smac: 0050.f0f8.7400, dmac: 000f.f8e4.d000 Note counter is cleared here; counter is clearedwhen adjacency is read.packets: 0, bytes: 0BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public42
Troubleshooting High CPUComponents InvolvedRP: show ibcRP: show process cpuUplink portsMSFC 3FlashRP: show ip trafficRP: show interfacesDRAMRPCPUFlashSPCPUDRAM1 GbpsInbandCPort ASICC1 GbpsInbandSup720SP: show ibcSP: show process cpuBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public43
High CPU Utilisation - ProcessProcess: ARP Input Caused by ARP flooding Static route configured with next-hop interface instead of IPIncrementing atvery high rateCat6K#show ip traffic begin ARPARP statistics:Rcvd: 6512 requests, 2092 replies, 0 reverse, 0 otherSent: 258 requests, 707 replies (0 proxy), 0 reverseDrop due to input queue full: 20 snip Cat6K#show interfaces include line protocol rateVlan501 is up, line protocol is up5 minute input rate 23013521 bits/sec, 2535 packets/sec5 minute output rate 0 bits/sec, 0 packets/secLook for abnormal input rateVLAN SVI is Virtual and counter givesonly the amount/rate of processswitched or control-pane trafficBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public44
High CPU Utilisation Identify if it is process driven or interruptsTotal CPU usage (Process Interrupt)CPU usage due to InterruptDUT# show process cpuCPU utilization for five seconds: 99%/90%; one minute: 9%; five minutes: 8%PID Runtime(ms) Invoked uSecs5Sec1Min5Min TTY Process27208881819.12% 1.11% 0.23% 18 Virtual Exec Next Steps‒ Process: recurring events, control plane process etc.‒ Interrupts: incorrect switching path, system exceeding hardware resourcesBRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public45
High CPU Utilisation - ProcessProcess: IP Input Caused by traffic that needs to be process switched or sent to theCPU Common Causes and Next Steps‒ Broadcast Storm‒ Traffic with IP-Options enabledFind the source host and isolate. Consider per-portbroadcast storm-controlFind the source host and isolate.Consider HW Rate-limiters‒ Traffic to which ICMP redirect or Unreachable required e.g., TTL 1, ACLDeny‒ Traffic that needs further CPU processing e.g., ACL LoggingConsider Optimised ACL Logging(OAL)BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco PublicUnder VLAN SVIsconfigure:no ip unreachablesno ip redirectsno ip proxy-arpConsider HW Ratelimiters46
High CPU Utilisation - ProcessProcess: Exec and Virtual ExecResponsible for tty lines(console, auxiliary)Cat6K#show process cpu include CPU Virtual ExecCPU utilization for five seconds: 30%/0%; one minute: 8%; five minutes: 5%PID Runtime(ms)InvokeduSecs5Sec1Min5Min TTY Process32721941402 29.00% 2.12% 1.89%0 Exec5418014431240.00% 0.00% 0.00%1 Virtual ExecHigh CPU when too many messagessent to console/vtyResponsible for vty lines (telnet,SSH) Next steps‒ Check if any debug is enabled via “show debug”. Issue “undebug all” if it isnot needed‒ Disable logging by “no logging console” or “no logging terminal”BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Cisco Public47
High CPU Utilisation - TrafficAnalyse Input Buffer Use when an input queue is oversubscribedCat6K#show buffers assignedHeaderDataArea Pool Rcnt46FDBC14 8029784 Small146FE0010 802CBC4 Small1. . .Size oneNoneCat6K#show buffers input-interface vlan 100 dumpBuffer information for RxQ3 buffer at 0x378B3BCdata area 0x7C05EF0, refcount 1, next 0x0, flags 0x200linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1if input 0x46C7C68 (Vlan100), if output 0x0 (None)inputtime 2d03h (elapsed 00:00:01.024)outputtime 00:00:00.000 (elapsed never), oqnumber 65535datagramstart 0x7C05F36, datagramsize 62, maximum size 2196mac start 0x7C05F36, addr start 0x7C05F36, info start 0x0network start 0x7C05F44, transport start 0x7C05F58, caller pc 0x6C1564source: 137.34.219.3, destination: 224.0.0.2, id: 0x0000, ttl: 1,TOS: 192 prot: 17, source port 1985, destination port 1985BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved.Find the interface that's holdingmost of the buffersPacket detailsCisco Public48
High CPU Utilisation - TrafficDebug Netdr Capture Capture packets being received and sent by RP to buffer space Does not introduce extra CPU processingCat6K#debug netdr cap srcindextxvlan cr BRKRST-3067(
BRKRST-3067 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 3 Agenda Troubleshooting Catalyst 4500 Catalyst 4500-E Series Switches
