Try our training for free.

Gain instant access to our entire IT training library for 1 week. Train anytime on your desktop, tablet, or mobile devices.

This Cisco video training with Jeremy Cioara covers troubleshooting Cisco networks, including topics such as IOS tools, VLANs and spanning trees, router performance issues, and more....
This Cisco video training with Jeremy Cioara covers troubleshooting Cisco networks, including topics such as IOS tools, VLANs and spanning trees, router performance issues, and more.

Related area of expertise:
  • Cisco networking level 2

Are you ready to run a Cisco network?  You will be, once you pass your TSHOOT exam. TSHOOT is the final step for earning Cisco's CCNP certification.  Employers trust that CCNP certified staff have the vital, problem-solving skills their network needs.

With tech guru Jeremy Cioara in the virtual chair next to you, you'll get the training you need super-fast, and you'll love every minute of it!  His TSHOOT video course is 80-90% hands-on, and Jeremy's filled it with tons of unscripted, real-world troubleshooting demonstrations.

By the time you're done watching, you'll be ready for the TSHOOT exam and actively troubleshooting your own network.
1. TSHOOT: Setting Your Expectations (16 min)
2. General TSHOOT: The Troubleshooting State of Mind (28 min)
3. General TSHOOT: Troubleshooting Before You're Treading Water - Proactive Steps (18 min)
4. General TSHOOT: Troubleshooting Before You're Treading Water - Proactive Steps, Part 2 (39 min)
5. General TSHOOT: IOS Tools to Monitor and Maintain the Network (27 min)
6. General TSHOOT: IOS Tools to Monitor and Maintain the Network, Part 2 (56 min)
7. Switch TSHOOT: VLANs and Spanning Tree Concept Review (19 min)
8. Switch TSHOOT: VLANs and Spanning Tree (30 min)
9. Switch TSHOOT: VLANs and Spanning Tree, Part 2 (28 min)
10. Switch TSHOOT: L3 Switching and Redundancy Protocols Concept Review (21 min)
11. Switch TSHOOT: L3 Switching and Redundancy Protocols (36 min)
12. Switch TSHOOT: L3 Switching and Redundancy Protocols, Part 2 (27 min)
13. Route TSHOOT: L3 Connectivity and EIGRP Concept Review (23 min)
14. Route TSHOOT: L3 Connectivity and EIGRP (48 min)
15. Route TSHOOT: L3 Connectivity and EIGRP, Part 2 (37 min)
16. Route TSHOOT: L3 Connectivity and EIGRP, Part 3 (19 min)
17. Route TSHOOT: OSPF and Route Redistribution Concept Review (23 min)
18. Route TSHOOT: OSPF and Route Redistribution (41 min)
19. Route TSHOOT: OSPF and Route Redistribution, Part 2 (29 min)
20. Route TSHOOT: BGP Concept Review (18 min)
21. Route TSHOOT: BGP (26 min)
22. Route TSHOOT: Router Performance Issues Concept Review (28 min)
23. Route TSHOOT: Router Performance Issues (43 min)
24. Security TSHOOT: Access List Concept Review (17 min)
25. Security TSHOOT: Access List Chaos (62 min)
26. IPv6 TSHOOT: IPv6 and IPv6 Routing Protocols (21 min)

TSHOOT: Setting Your Expectations

General TSHOOT: The Troubleshooting State of Mind

General TSHOOT: Troubleshooting Before You're Treading Water - Proactive Steps

General TSHOOT: Troubleshooting Before You're Treading Water - Proactive Steps, Part 2

General TSHOOT: IOS Tools to Monitor and Maintain the Network

General TSHOOT: IOS Tools to Monitor and Maintain the Network, Part 2

00:00:00

Alright, let's move into IOS tools to monitor and maintain the network part two. I got be honest with you. After I finished the last nugget, essentially, part one of this. I felt a little weird; I finished the recording, and I was like well I guess there it is; I sat there and I said to myself something feels weird. Not necessarily wrong or anything like that. I'm like

00:00:26

what is it. I was like what?; that is cool information that's interesting, and then it hit me, I was like, you know what I didn't really say, well here is a practical scenario you know, it was like, I was given cool tools. Like you can filter a show command in here; you can use this to pipe the output for a TFTP server; all of those kinds of things, and I sat there, and then I realized, I was like, I didn't give, I didn't feel fulfilled, like how it's fulfilling to me, when I create this brilliant real world scenario, I said alright how do you solve all this puzzle, and then I said here's how we do it, and then I put all the pieces together again, and I was like oh right that was awesome.

00:01:06

The last nugget we really didn't do that, you can filter doing this or you could do this, and then I sat, and I kind of comforted myself well look at the drama I go through just to convey information. I comforted myself and I said Jeremy, we're not doing real world scenarios at this point, but solving problems, like the rest of all the t-shoot series is all about, right now we are just looking at tools, kind of, take it this way, you know you out in your garage and you go check it out. This is a field screw

00:01:37

driver see? It can screw in this Phillips screw, and you wow that's cool, and you go like let's move on; It's interesting information, but I didn't have the, well, let's, well let us screw a Phillips screw. Let's put this on the wall over here and hold that lay there upper or something like that, so that's, so bear with me right now as I dump my feelings about this because, I started with this next one, and I'm like ok this is going to be cool too, but I'm like ah! I don't want to not have any real world solving that we're doing yet, but that's ok, we're going to be doing that as soon as get into the troubleshooting of t-shoot series and get into all the different scenarios that we're going to see. So for now, you're like, yeah let's move on and some

00:02:26

of you are probably like I'm going to fast forwarding, so ok there we go. Let's get into what we're talking about here; let's talk about how do you check resources on your router, the comment from somebody saying the internet seems slow it would be good to just do a resource check and see if my router is dropping packets; it's over loaded processing memory all those of kinds of things. We'll get into monitoring with span and rspan, and

00:02:50

finally I kind of did the miscellaneous grab bag over here; these are all, except for EEM, they're all kind of one liners if you will. Things you can do to help monitor your network more efficiently; just like the previous nugget I'm going to batch in the line interface pretty much through all the nugget areas. First off

00:03:12

let's go into checking resources, again someone comes to you and says things are running slow and that could encompass a whole gamma of troubleshooting and issues; but one of the things that should be able to do is to check the resources on the devices on the path of the packet flow; couple of ways that we can do that, we can jump into, I'll jump into my CME router first, let's so the same show command we did last time show process cpu. The processor on your device is going to be one of the key

00:03:43

ingredients to finding out how much, I don't know, cycles are being eaten and if the processor of a router is the cause of bottleneck; then if you shoot up too high in process utilization the router can't keep up and starts dropping packets and things and feels slow, things slow down. So what the most useful line

00:04:02

is, and I know we filter this last time so we can see what processors where consuming resources and what not and so on, but one of the most useful lines of it is this one right here. CPU utilization for five seconds, one minute and five minutes; now one thing that I have always accepted and I never really question was this five seconds right here it says zero per cent slash zero percent; let me see if I can get it to, there we go, so now I've got another one, one percent slash zero per cent; and for years when I was at Cisco I would like ok it's probably somewhere between one per cent and zero per cent, damn, I just kind of figure it was an average of the those two values, but that's not true. These

00:04:42

two values actually measure different things, the first number is just processor utilization based on, scratch that, let's back up, this one over here it's just processor utilization eaten up by packets being forwarded through this router; this one is the value of processor utilization for packets in processes running through this router. Now you're

00:05:08

probably going say that again, say it slower, rewind, no, check this out. This is what we have, we have our router right here, and our router is doing all kinds of things day in and day out. Let's say we've got two interfaces fast Ethernet one and fast Ethernet zero over here; packets slowing through this router just the fact that packets are coming into this router and passing through and being processed by that router is going to increase the processor utilization; there's a piece of the processor that is processing those packets as they come through; now low end routers, and when I say low end, I'm talking about the 2800 series, the 2900 series, you say well that's not low end business router, well compared to like a CRS1 in a service provider it is low end, it's just a consumer grade or enterprise grade router, mid-size business that people kind of put all over the place; those kinds of routers use the processor for both processors running on the router and packet forwarding. You know take processes running

00:06:10

on the router, what is that, well when you ssh into a router there's got to be something on that router handling the generation of encryption keys something that's handling the processing of every character type, you know, the show commands that you are entering, when you have to dig in and show commands and gather that information out of the router; that's all processes that are running on the router and that are happening right here in the processes. It has nothing to do with packets going through

00:06:41

the router over by itself; this first number is a total of all the processes running on the router and how much processor cycles is being eaten by packets forwarding. Again what I was trying to convey, I feel like I'm jumping all around here, is as packets go through the same processor on this mid-line routers handle the processes running in your router and the packet forwarding.

00:07:06

Now the packet forwarding is not actually assigned to any one of these processes; let me show you I've got a connection up here to my UC520 which is actually running my network connection right now; I'm going to do a show process cpu one here; ok, well not much going on, let's get him talking, I'm not doing anything, I've got a VPN connection, see if I can find it, open here to a remote server and you can see I just have, I just have this msi file; let's just this 22 MB file or so, I'm just going to drag and drop that to my desktop, so I can start a transfer going from that; right now packets are flowing through my UC520, I'm gonna do a show process cpu, and we see this increase to 2 per cent slash 3 per cent; let me hit the upper again, if it goes up, 3 per cent and 2 per cent, so as packets are going through the router, and, I need to copy that again, as it's going through this router this is reflected; so this is good enough; 3 per cent represents the total of all the packet processing and the processes running on this router. One per cent represents just

00:08:18

the packets that are processing on this router. So when I copy that 22 MB file packets start flowing through my UC500, my, I can't write, UC500, and you see flowing and that increased my processing utilization by one per cent. Now there are also processes running on that router; it's processing the show commands as I am hitting the up arrow; my ssh session, all of this and all of that, and they shot up to three per cent. Now the packets

00:08:46

themselves are not seen in any one of these processes. Meaning the packets are processed less, if you will, Cisco does not identify the packet processing by any one process, so this is how they kind of differentiate between the two. Does that make sense?

00:09:01

So if I wanted to say, ok, how much is just the processes running on my router, not the packet processing, how much is the ssh, the shows commands, you know all the other stuff that my router does. How much is that consuming it's just a simple math problem; is my style of math, three minus one it's two, so I would say ok the actual processor utilization by these processes, all this stuff right here, it's two per cent, if I were to filter this down, I would be able to add up and there would be about two per cent of utilization if I added all these things together that was being used eating up the processor. Does that make sense I feel like I was talking

00:09:38

to myself totally inefficient describing that but that makes sense. So that's how we can check out the processor; now let me do a little sidebar on that; this UC500 that I have has a built-in switch; it's using the same hardware that a normal Cisco switch does, so I'm gonna go, and actually have a, folder here, that's going, I'm gonna pull out my old CBT nuggets BSCI series because these files are huge; you can see I've got 229 MB there, 218, so I'm going to take these guys right here, and I'm going to copy them, now this is going across my LAN, I'm going to copy them to my desktop, so you can see I'm copying 448 MB going about 16, 15ish MB per second or about eighty or so, I'm doing the math in my head, MB per second, I gotta stop talking, I'm gonna shoot over here to my router my UC520 and do a show process cpu; check that out it's zero per cent, even no, even though my file transfer is going way faster than I could across my VPN connection, it just completed right there, but still the processors remain at zero per cent. The reason for that is because those packets were not

00:10:53

routed; they were running through the switch engine or essentially the x86. The application specific circuitry inside of that UC520 that is powering the switch now off loads all of that packet processing, so it doesn't hit the processor at all, so the reason I bring this up is that if you're on a switch, you know any Cisco switch; you know a 2900 series, a 750, a 6500 and you do a show process cpu and you see the switch hovering around 40 or 50 per cent something is wrong man; something is very wrong because your processor on the switch should almost never go up unless you are doing some really weird stuff on it, or some really advanced features on the IOS because Cisco designs those switches to where ASICS handles of the packet processing, so when you do the process show on the switch, you're only seeing things that are hitting the processor meaning processes that are running on the switch are handling all the packet processing just when I showed you that 400 or so MB copying to my desktop, is all handling, I am deleting it in the background, it's all being handled by the ASICS inside of the switch. So again your output may vary based

00:12:06

on the kind of the device that you are on. Alright the next command you can use, to add to our resources, is who memory on the router; it's a very simple command. Showmemory it's going to, it's going to divide the memory into two major pieces: processor and IO. When you are looking at these two this

00:12:28

is used for the router processes as again all these processes that are running on the router; this is used for input output processes or in plain English the packets going through your router are using these memory buffers. Now due to just looking at this, I can't sit here and go, oh yeah, we're looking at, yeah that's not good, yeah, no, this is really low, you can't do that because on this your mileage may vary; it depends on what processes you are actually running on the router, the only true, well if it says free, zero, that's probably a problem, but the only real way to know whether there's an issue with the memory on your router is by establishing a baseline, you can do that, it's monitoring software, for instance, I know I've mentioned this many times I used prtg for all of my monitoring graphs generating. This is information I'm gathering from one

00:13:23

of the routers that I watch from prtg, and you can see from right here, let me just kind of center that guy in there; this is watching the free processor memory. This is one of the graphs that I'm looking at. Now this is only the last twenty hours; you're looking

00:13:38

like wow you are all over the board buddy; well not really look at the site over here in kilobytes; we are essentially at 43 MB of free memory that we're hovering between going up and down falling down just a few kilobytes as time goes on; better view to look at two days, and you would say oh it's a little more even right there or scan across thirty days and you really a good feel that on average this router over the last thirty days, except right there we got a little outage window, this router over the last thirty days hovers around 30 MB of memory free, so you might set an alarm in your monitoring system that would say ok if we ever go below thirty MB shoot me an email, sent me a text message, do whatever your monitoring system does because that's normal. Now what would tell us if the router starts loosing

00:14:30

memory; well couple of things, one could be running a protocol and someone is performing a denial of service attack by flooding your table with all kinds of entries, ally weeded up. BGP and we'll say BGP can bury our router; for instance that little router I showed you right there; that's a little 855 router don't run BGP on that guy, at least, not with a full internet router table, you'll just watch the router melt. So again the

00:15:00

amount of routes that are in your running table or you could have a process in the router that is leaking memory. Now you know if you've had Microsoft for any amount of times you probably know what a memory loss is, nothing against Microsoft, it just happens, a process goes bad and that is why nearly 99 per cent of the problems are solved on Windows by rebooting because it solves all the memory leaks until all leaks out again, and that, you know when you get everything running slow and you have to reboot it again. So what would cause leaks on a router, typically

00:15:30

an IOS bug to where Cisco really release a t-train of software or something like that; there's a process that leaks memory again to solve it upgrade the IOS or set you router to reboot every x number of days or weeks to reset the memory back to default.

00:15:46

Now I say all this but one of the things I've got to mention is that if your router runs out of memory, some of you may notice, what happens? Bad things, bad things, the router crashes; and at that point, once the router crashes is inaccessible, this is not like Windows where you go oh yeah start, restart and then go from there. This

00:16:08

could be a router that's in Istanbul where ever that is, and you don't know how to get there somebody has to go flip a power switch on a router; that's not good, so it's definitely something worth monitoring in your routers at any time. It's one of the standard sensors that are creating

00:16:24

in almost every monitoring system that's out there. So oh one other thing if you're watching syslog; you see a message, something like this: mal failed. What's that suppose to mean; memory allocation fail; it means run, run to the router as fast as you can or the switch, the memory has leaked so far that the processes are asking for memory and the router is now generating syslog messages saying I couldn't give that process memory, and it's only a matter of time once you see those before your whole router or switch locks up because as it's on its way out. So that's your, you can see

00:17:04

from that level your disaster status on the message. So those are messages to watch out for; so now next let's talk about the interfaces themselves looking at interface statistics; how do you know if something is wrong. Let's just bring a show interface

00:17:22

command, and look at one of them, we'll go to ones that actually have packets going thru it. Let's do a show ip interface brief, alright I'm going to do a show interface fa zero slash zero right there, and we can see a bunch of packets going thru this guy. So when you do show interface;

00:17:44

a couple of things are looking for that signal trouble. Number one is, let's see, let me filter this out, includes, let's do a drop, and there we go, I'm looking at the interface just filtering it down, and you can see here I've got zero input errors; that's a good sign. Input errors typically, well I'll group this one, we've

00:18:10

got twelve output errors; input errors and output errors typically signal a few things. It can be cabling, could be a bad network card, you could be going by interference that's scrambling the data on the line, and all that's going to do is give you a ton of drops and slow down the connection. Now if it's voice over

00:18:29

IP it's easy to detect because your voice will be breaking up, all kinds of audios, but with data connection, data uses TCP for the most part, so it naturally recovers those drops, it just runs slower. So if you see a bunch of input output errors then investigate that; the other big one that can cause these types of errors are duplex mismatch, so if you have a duplex mismatch that's going on you going from half duplex to full duplex things like that; you'll see a bunch of errors coming in, so that's also signaled by collisions right here, see we've got collisions in another one in the field, and I sort of lifted past here, late collision, if you see a bunch of late collisions on the interface that is almost a sure fire sign of a duplex mismatch.

00:19:17

A late collision is when a collision happens past a, oh forget it, it's a, I'm sure if you google it you can find it, it doesn't really matter; essentially when you look at the old hub world, and you look at the architecture behind it CSMA CD and not that is specific to hubs, Ethernet uses that, the collision detection algorithm essentially listens to see if everything is clear, and if everything is clear then it sends its packet.

00:19:47

Well based on the timers and all that kind of stuff, if two devices are listening at the same time then you know this is how a collision happens in the hub world if two devices are listening at the same time and both go to send; just the way the timers work the collisions will almost always happen, I'm gonna throw this out there, I think it's right but I could be off, say the first sixty four bytes or, it's either sixty four or thirty two bytes of the frame, it's always going to happen there because of the timers that they've created on CSMA CD. So what

00:20:20

does that have to do with what we're talking about, a late collision, no, right here, a late collision happens, is if it's after, if the collision happens after the sixty four bytes or thirty two bytes or whatever the normal collision factor is. Late collision

00:20:36

means, well, the timers are just weird here you're not following CSMA CD standards almost always happens if you've got a duplex mismatch. So if you see a bunch of late collisions then immediately, bing in your mind, duplex mismatch. Let's investigate; so input

00:20:53

output errors those are typically cabling issues, late collisions let's also talk about the drops, let's see where we are, he we are. Input queue you can see right here we've got the input queue dot 75 slash zero that means how big is it maximum size it, how many drops have you had, that's what I'm talking about; that's the input drops. Now there are two different drop statistics

00:21:17

that you can gather there's the output drops and there's the input drops. Input drops are typically related to processor utilization meaning as packets are coming into the router right here; your processor is so busy it does not have the time to handle them, so your router just starts dropping packets as they come into the router that's almost always an input drop. Output drop in

00:21:43

the other hand is a normal thing to occur because typically a router is converting between link types, and I know my diagram is getting a little messy, so typically you might have a Gigabit Ethernet connection coming in going at a thousand MB per second going out on a cable modem or a T1 line or whatever; we're dropping it down to 1.5 Megabits per second, so this is like a given output drops are going to happen. If you've got a computer here it will

00:22:11

initially try to send at whatever maximum bandwidth it is, it's going to build that TCP window, and it's going to flood the router and this is going to immediately flood and start dropping packets. Those are all considered output drops, once the memory buffer inside here that are waiting to go out that T1 line has filled with packets you've reach the end, and it's going to start dropping as it goes out. If you see a bunch of output drops don't immediately

00:22:32

panic because that's normal that's normal because routers are dealing with bandwidth mismatches all the time, fine. Last thing I want to say on these statistics. Right here, you might look at this and say I see twelve output errors, reason for alarm, probably not; if I look at my statistics and I see you know fifteen packets out and there's twelve output errors ok that's a good reason for alarm; you just don't, means you just brought the router up, and you're getting a bunch, percentage wise of output errors, and I look at this and I see what ninety one million packets output with that many bytes in errors, I pad myself in the back, and I'm like good job Jeremy you've got some good cabling there; that's some phenomenal statistic rate to do that, so things are always relative always look at the number, if that number were a thousand output errors, I would probably, well, I know I would look at and say that is totally fine then relative to the number of total packets that I have.

00:23:30

A thousand output errors is totally normal. Ok, last two commands I want to toss at you, and I'll jump back over to my CME router. For this is the show inventory and show diag. Show inventory allows you to see all of the different models and makes of the cards that are installed inside of this router which is great if you're looking for a serial number, you can see the serial numbers for each one, you're looking for exactly what model of card is installed. You might say oh no I lost the card what model

00:24:03

was that, well you go that was a VIC two dash two FXS. Here's the chassis that I'm running; here's WIC card that's installed inside of there, so you can put one of those guys in google and find a replacement part for it that you can purchase to replace that. You can do a show diag and that would give you more detailed

00:24:23

information about each one of those. You can see slot zero, you can see our main board of the router. I mean typically you are typing this when you are on the phone with TAC and you go give me the serial number of motherboard we need to make sure that blah, blah, blah, whatever they're trying to have you do, and if you scroll down it will actually give you statistics for every single device or card that's inside of here, so that's a little daughter card that is PVDM that is voice modules that are inside of this V2801. So this gives you statistics and detail part firmware information about each and every one of those cards that are inside of there; so great way to get a list of those. Ok, let

00:25:02

me clear some of this off. I need some space to talk about the next one. I'm even going to take off my button, goodbye, so let's take a look at span and rspan. Now span stands for switch port analyzer; I know a lot of you think about spanning tree, but span is actually a switch port analyzer, and what it allows you to do is sniff packet traffic on a switch; as many of you know, when you install switch versus a hub, a switch divides all of the ports into their own collision domain, and all that means is everybody kind of has their own fabric to talk across; so if this computer talks to that server through the switch then this conversation will be isolated from all the other devices that are going on that switch right now. Now I don't know if

00:25:55

I mentioned this but I actually really got into wireshark, about a year ago, I gotta learn this hardcore let's go after it, and I bought a book by a Laura, Laura somebody very known gal in the wireshark ethereal world; she wrote a book called wireshark official certification manual which it goes through kind of giving tips; you know learning the information from that; and one of the things that she says when she opens the book in the introduction is most people save packet sniffing for wireshark until the last possible phase of troubleshooting, and she said I propose that it should be the first phase of troubleshooting, and I read that I looked, I scratched my head and went no, no, no, no you don't bust out wireshark as your first phase of troubleshooting, and I can say a year later, I still disagree with that statement, you know right, but you probably thought I was going to agree right, no, I still don't bust wireshark as my first phase of troubleshooting, but I will say, now that I know a lot more about it, and a lot of the filtering options and things like that, I do pull it a lot sooner than I used to do. Now on a switch, you have, let's say

00:27:08

right here; it's reporting communication issues with this server. You've done your basic troubleshooting, and you go ok I need to go see the packets; I need to see what's happening; well on a switch is going to allow you to see those packets, let's say this is your monitoring statement station running a wireshark; it's going to allow you to see those packets unless you enable span; and this enablement is so simple I just didn't even want to setup a whole switch topology and all that it's very very simple. You just go into, go into global communication mode in

00:27:43

that switch, that IOS emulator, so switch and config and you type bin monitor session and then you give it a session number, oh, you gotta spell it right. Now depending on the size of switch that you have; it can support five sessions or ten monitor sessions; it can support a lot of monitor sessions, a lot of the cycles switches support that, so I would say monitor session, let's say session one, and the session number just identifies this instance of span, so you can say I am monitoring this and then another instance is monitoring over here, this one. So you first identify your

00:28:26

source; I'll say source, interface, running out of room, hold on let me move this over, right there; source, interface, we'll do fast Ethernet zero slash one which connects to the server right here; I want to be able to see everything that gets sent to that server, and then I go back to, and I got my config, pops right back up, and I say monitor session one destination, interface, and then I would say whatever interface my ethereal or wireshark is connected to, it looks like about four over right? So fast Ethernet zero slash four right? and what that does now is pipe all the information that is sent to that server out to this interface as well, so you can see it all happening in wireshark.

00:29:17

Now you can also dance with this, you can see it allows you to monitor like the send side or the receive side or the end receive; you can also do a change of ports; you can say I want to see fast Ethernet zero one through ten; the range syntax differs based upon what device you are on. So fast file all kinds of

00:29:37

different things, these sources are coming to this destination and that will get you going, so that is setting up just basic monitoring on a switch. Now the trouble is, and it's funny, because there's this guy, and when I first introduced him to rspan, and he's like oh this saves me so much pain because he just knew about span which is monitoring within your own switch; rspan stands for remote switch analyzer, and it allows you to monitor traffic that is on another switch; so let's say there's another switch right here is connected to we'll say another switch via a trunk link that is sending traffic, and you want to monitor this port. Well the

00:30:19

trouble is you've got to monitor this port station plugged in here, and you want a monitor this guy over here, and it's funny, when I talk about this guy over here, it's like oh, I always had to move my monitor station and plug it in different IT rooms; and if that's all you know then you're probably like hey that's what I gotta do; but once you know rspan man I'm telling you, your chair gets so much warmer because you just get to sit there all day. You don't have to move, so what you can do is enable

00:30:42

rspan. I'm gonna clear a few of these things off, there you go, and then let's just identify again, we've got our target up here, change colors, we want to monitor this guy up on switch two; and then we've got our source right here I guess our monitor I'll just this guy right here on switch one. Now when you're

00:31:04

setting up rspan you need to create a dedicated VLAN just for rspan, just for rspan traffic, it's, well let me show you the config, and we'll talk about it. So let me start where my target is up here on switch two; let's say I'm on switch two on config mode, let's save some typing time, copy that to my clipboard, so I am config mode on switch two; let's create the VLAN, we'll create VLAN fifty, and I'll say that will be my rspan VLAN, and now I'm just going to name it, naming it doesn't have to be rspan, it can be whatever you want, it just has to, I'm just identifying it to myself. Now underneath VLAN fifty

00:31:51

I have to identify it to the switch, so it knows that is the one I'm going to use for networking monitoring by typing in the remote span command; now once I've done that the switch now knows that VLAN is just for rspan traffic that will it whatever we can sniff on switch two and go across that; again we have to allow VLAN fifty on the trunk of course; we'll cross that trunk and be received over here on switch one, so now I've identified the VLAN that's going to be used, and now I go back to my monitor session commands.

00:32:23

I can do monitor, let's do session two, since we did session one last time, and I'll say the source interface, this is on switch two, is fast Ethernet zero slash twenty four, it looks like he's at the far end there. So I'm monitoring that guy, and then the destination will be remote VLAN fifty, so what does that tell the switch, it says take all the traffic on, fast Ethernet zero dash 24, this guy here, into watch, and send it to VLAN fifty which it goes VLAN fifty that's a remote span end VLAN, so we will pipe that across the trunk. Now I know

00:33:10

some of you are already thinking, so I will concur with your thoughts, be warned, be careful because this can easily swamp a trunk link if you put too much; I mean think about it if you do monitor session two source interface fast Ethernet zero slash one through twenty four, you monitor all those ports, and then pipe them out VLAN fifty, this trunk link is going to die because then all of the traffic that is in one through twenty four, out that trunk link, even if it's a higher pass thru link; you run a major risk by doing that. So be careful of the ramifications

00:33:44

if you have a high bandwidth center using rspan; now you have to configure the other side, let's see if I can squeeze right here between these two. Let's do a switch one is now receiving so, what I'll do, if only I could, well I guess we can do this on Cisco switches if we use notepad; on switch one we have to do the same configuration that we have on switch two; we have to create the VLAN fifty, now vlan rspan identify and all that, but now when we're on switch one, let's go back over here to switch one in config mode; now on switch one I need to setup the monitor session, just slightly different, monitor, session, and then again I'll say two, even, I know some of you might be thinking, this number does not have to be the same between switches; you can use session three, session four, session one, whatever you want on session one; it is a locally significant number, so it does not have to be the same. So I'll say monitor session two, and we'll say

00:34:45

the destination, I'm going to start with the destination, interface will be fast Ethernet zero slash four, my monitor workstation, I'm going to pipe it out to there, oh, I think I forgot interface, so destination, interface fast Ethernet zero four, and then I'm going to type in some of you know where this is going thinking, monitor session two, the source, what will the source be, remote VLAN fifty you got it, so you see how this works, it's going to grab the traffic from VLAN fifty which is the rspan VLAN, and then pipe it out to the monitor port, the monitor interface on fast Ethernet zero slash four, pretty sweet, so both monitoring on the local switch and monitoring across the network, you can go across your whole campus with rspan, again with the same warning, be careful, you know about the trunk links, you don't want to kill them using span and rspan.

00:35:41

Alright, I'll once again wipe the screen clean here that is what span and rspan are all about, so let's now jump to this final little gambit here. Now first thing I want to mention is I could talk for probably thirty minutes on syslog, and hour plus, I would say, maybe four hours on SNMP, and hour or so on netflow and unlimited values on EEM and when I said unlimited it would be with most reaches, and documentation, because EEM documentation is very scripting orienting. So what

00:36:17

I'm going to give is then again the fly by view of this, so to introduce the tool, so you're able to see it. I'm sure you've seen before, but let's make sure you see them all right here. Syslog allows you to pipe the output that is commonly logged on your console's comm port via syslog to an outside syslog server; now many of you have connected to the console port and have seen the messages that are coming across and let me just back out here I'm in the console. Here's the syslog message, it's good

00:36:52

to gather those in one place. Quick and easy way to do it, go to google download the kiwi syslog it's a nice syslog server for windows, you know pick your favorite syslog there is Splunk; all kinds of free ones that are out there, and ones that you can pay excessively for, and go in your Cisco devices and do logging and type in the IP address of your syslog server; that's I would say all the configuration right there. Now again we can

00:37:21

talk on the ramifications of filtering that output seeing which ones need to be included, all that kind of stuff, but essentially logging such and such, and anytime you see a message that happens; see right there that message right there. It says now it just

00:37:36

started logging that's cool. Just started logging out to syslog coming out on the CLI; now that IP address doesn't actually exist but that will not receive those messages. I can choose the level of the messages being sent into this syslog server by typing in logging trap and then typing the level and above. Then again

00:37:56

this goes into the syslog messages, I typically want to see warnings and below; some people want to see errors and below; some people want to see debugging and below again you gotta have a lot of space in your system. In your syslog server and that can impact

00:38:10

your router processor because it's going to start piping that data out there; but that sets the level that is sent off to the syslog server, so that allows the router to log centrally all those messages. SNMP, SNMP is the life blood of your monitoring; simple network management protocol operates on a pull model; meaning if I have a router right here and my monitoring station right here. I'll use prtg, it's funny I it so often people might

00:38:40

be thinking are you thinking about a kick back or something, no, I even asked him if they would, but I still use them anyway, but I've got prtg right here; I'm connection to the router every x amount of interval. You know I can go here and say I want to monitor the bandwidth usage of fast Ethernet zero slash zero every three seconds or every ten seconds or whatever. Again

00:39:07

it's a pull model, so the monitoring system goes to the router and pulls that data down, and that's when I brought up the free memory, in this case, this is being checked every five minutes, on this every day, every five minutes going out and checking the free memory on this pull model basis; again the more pull the more resources you're going to consume, but the more you could say accurate, or specialized results you're going to get, so to configure a device to support SNMP, you go to the config of the device and type SNMP server, and then again I want to give you the basics but I want to show you, it's quite a bit, you want to. Quite a bit information that you can put

00:39:59

on here. I'm going to go SNMP server community; I'll define my community stream. Now this should be a very cryptic very secure, I have seen people that have community strings, that no joke, look like this, including all kinds of weird characters, simply because if you have access to that string then you can sabotage the router. Let me back up on that, let me explain; let me say

00:40:22

my community is cisco one; it's question mark, notice, right here, you are given the option of read only and read write. I would suggest have a very very very strong reason to do so only enable read only access via snmp to your router, again, moods managing system should only need read access for gathering or pulling that data from the router. They're not changing anything

00:40:50

from the router. If you have read write access in your router, we have defined cisco one as a read write community string, and somebody figured out that string all security is gone in your router, all security all of it. Actually, there is a, go on google

00:41:05

when you're bored and just type in cisco read write snmp hack or hacking commands or something like that. There, you, can send an snmp string, if you have write access to the router, again any value on that router can be changed with snmp if you have a read write access; you can change the enable password, you could change the ssh password, you could change the telnet password, you can change the user accounts, add user accounts, delete user accounts, you can do everything essentially all security is gone from your router if somebody gets that. That's one of the reasons I really

00:41:40

prefer kiwi cat tools for all of my configuration backups because it actually telnets or sshs into the device to pull the configuration whereas a lot of other monitoring systems, I have seen, require you have read write access to the router, so you can pull the running config and do comparisons. I mean, now, am I saying never,

00:42:00

never, never, ever enable read write, yes. Again I would say, it's very rare to do so, but there are some monitoring systems that when they detect something going wrong that they can make proactive changes, and you can see on this that you can attach access lists to say exactly what devices have read write access to this via IP addresses; likewise you can also use snmp version three which adds authentication to snmp. Again all I'm showing here is the basics of it, but this,

00:42:32

once you have this setup, you would now go to your monitoring system, plug in the IP address of your router and say this is the read only community string, and you can start monitoring read only different messages; that are what counts here. Where you get to the management information basically, again, snmp we talk on for a long time about. So that is the core of it,

00:42:52

so one other command that I can show you; I'm going to do snmp server, well it's actually right up here, this guy right here ifindex; it's a question mark, persists; I would suggest typing that in you router. What that does is it says the inner face index numbers, these are the kind of behind scenes numbers that are assigned by the IOS that identifies the interfaces to snmp will always stay the same. There are some

00:43:22

IOS versions, there's some times that if you reboot your router, where the interface indexes will change, and at that point your monitoring system will say; ah! your interface is down, I can't find it anymore, so this will ensure that that counter, that identifier for the interface, always stays there. The next one

00:43:42

on the list is netflow; netflow is really close to snmp in that it allows you to monitor the different devices, aspects of your router, but netflow is focused on the traffic flows going in and out of your router and switch, or whatever device you are enabling netflow on. You can get some really cool monitoring

00:44:06

statistics using netflow; basically, what it does is when a flow, when you say flow, this guy it goes to a website, and let's say it's downloading information; that is a flow of traffic is identified by its port number, by how long it's been there, by the amount of traffic that's been sent on that, so can generate a netflow report, you can even do that on your router without needing a monitoring system, a netflow report that says, well show me the top five senders by IP address, so again excuse, somebody comes and says the internet is running really slow, you got netflow running, you go to your monitoring system, show me the top five and then you see that this guy has transferred 50 GBits, not GBits, but Gigabytes of data over the last say five hours; he's just sucking up bandwidth, so you go over and find out what that guy is doing, why are you downloading so much on our corporate time; again it helps you really identify wholes in your network and the traffic flows that are going through it. So to enable netflow, again, a lot of monitoring systems

00:45:08

supported, go into your router, and you want to go under the interface, you want to go under fast Ethernet zero slash zero, and under the interface where you want to monitor and you type ip flow, and you can see we've got ingress or egress, so let's say fast Ethernet zero slash zero connects to the, let's say right here, I'm thinking as I, I'm thinking in and out, x right here, turn in on ingress network will monitor things as it's coming things this way. You really have to think about it because if you think

00:45:48

about it if you monitor ingress this way, you're really just going to see IP addresses and how much bandwidth those public IP addresses are consuming, and you can enable it in both directions in both interfaces there's no problem, but it is a, it can consume resources so, I'm going to say IP flow ingress enter, that's it, at this point netflow is enabled on my router, and I can go out and I can start using some netflow commands, well show me, let's do who IP cache flow, and then you can start identifying all the flows, again going through your router, and there's not much going on right here that is coming here, and it has identified one flow, coming in this router, it's just a little lab router, but then again, if you want to go in the router that's great, but then you want to configure this to be monitored. Now netflow,

00:46:36

unlike snmp, it's not a pull system, meaning, the monitoring system does not every x number of minutes pull it and then pull the information down. It's a push model meaning as the router collects enough data; it exports that data to the monitoring system who then can put in on the graph, so the router proactively pushes it on whatever amount of time that you specify. So the

00:47:00

way that configure that is you go in and type ip flow export because you are exporting this data,, and I would say, well first off, I would say version, you can see there's two well three versions: one, five and nine. The big, I would say, most people now a days will use either five or nine. There's just little

00:47:19

changes between each one of the versions, the key is that you have to match the version number on your monitoring software or else it won't work, so let's say I'm using version 9. I would then go in and say ip flow export destination; and then type in the IP address of my monitoring system and say a ten dot one dot one dot one. Now it's going

00:47:40

to also ask for a port. Now on most these there's a default port but notice on this one there is no default port, and that is one of the weird things about netflow. When the netflow standard was created they never picked a port to be the default. So it's

00:47:56

just pick one, you pick one that's not in use, you know a lot of people like using, and I've seen a lot of people use nine nine nine and a number, so nine nine nine and one. Then the key is that on your monitoring system you set to receive on that same UDP port. This is all UDP traffic based, so again you can

00:48:15

get into all kinds of detail on this and specify intervals; how often it's gonna be checking all those kinds of things but that's just the basics of netflow to, in showing you that this tool will let you monitor things. Alright, the last one I want to talk about is EEM, embedded event manage; probably the most intriguing one of all them that are on the screen, but the one that suck your life away, spend the most time with. EEM is literally a scripting language for your

00:48:49

router; if there is anything that the IOS cannot do that you wished it did then you'd probably write an EEM script for it. I'll give you some examples, maybe you want, maybe you want the IOS to send you a message any time somebody types the IP address command under an interface; you know changes an IP address, well that's not normal IOS, it doesn't log an IP address change on an interface, it would log like a shutdown event, but not an IP address change, so you would have to create an EEM script that did that, or like this, let me grab a router in here; you know cisco routers tell you anytime somebody leaves the configuration, for instance, if I hit control Z right here, you can say oh I now know somebody left, and if I was using centralized cisco logs, I would be like oh! Somebody just configured my router, let me see what is going on there, but what if you wanted it to send you a message when somebody went into configuration mode.

00:49:52

In my opinion that would be a lot more useful because now you can catch somebody in the act rather than waiting to have them make changes in your router and then exit the configuration mode before you're actually notified about it; that would be something that you would create an EEM script to do. Now this is the one

00:50:09

I'm not going to dive into by any means, I will be completely honest guys EEM is by far my weakest point out of all these tools simply because the IOS does what I need it to do, oh, I haven't ran into a situation yet where I needed a script to go beyond that, and I'll also say anytime somebody says scripting there's a little person inside of me that goes and crawls up into a tiny ball and shrivels. I actually went to college for computer science

00:50:38

and years and years later left college, you know, and went to work for Pizza Hut, I kid you not because I so despised programming, I thought computers must not be my thing, I must not be good at this and thankfully within two days I realized, Pizza was not my thing either, so by the grace of god himself I ended up in a Networking ilk through a long chain of events, but anytime somebody brings up scripting I just kind of, I say there's people that do that, so let me actually give you a direction to go with this. If you are really interested in getting EEM

00:51:17

going there's a whole EEM scripting community on Cisco's website, as a matter of fact, you can either google that thing I highlighted EEM scripting or here is the URL right here; just type that bad boy in your web browser, and you are on your way, but just to give you an idea of some of the scripts. This is a community

00:51:36

where they post common scripts like let's grab network management, you can see in here we've got a tclsh script to do a DNS search in the IOS, we've got script logs; all outgoing snmp traps to a log file, so it kind of converts snmp to syslog if you will. Again, it sets up a monitoring

00:51:57

that reports packet lost for WAN links, files and so on; you see kind of what this is; it's just a world, a whole world of going beyond, as a matter of fact, where did I see it, user interface, check this out. Somebody has created a Lotto eem script to allow a router to run a lotto gambling game on cisco IOS. I would say that there's a special place in

00:52:24

cisco purgatory for people that do this on production routers don't don't do it. I mean you can, I mean look at this, you can send twitter messages from the IOS, so, I mean, this is, I know when you see that you're like that's pretty cool, somebody actually liked it they're ready to do, four stars. There's a lot of you

00:52:41

can do out of box from what the IOS allows you to do; that's what eem is all about, just going beyond typical IOS commands to a scripting language where you can do just about anything. Well, as we wrap things up I want to give you a little taste of what's it's like to be in the real world of troubleshooting.

00:53:03

I want to show you where I was last night literally I get a call in the morning from a friend of mine, he actually, he kind of does the same thing I do except on the development Windows side of things, and he calls me any time he has a network issue, and I call him every time I want somebody to type some code. We kind

00:53:19

of help each other out and partner together. He calls me up and says dude, I just landed this massive contract you got to help me, and I go what do you mean, and he goes their network is just a mess, and then I go yeah ok, I understand, I understand. I had no idea, I'm going to show you a picture; this is one of their branch offices. Now there are many branch offices this

00:53:41

is one of them. I am not kidding. You see those pictures on the internet, and you're like c'mon somebody obviously was like. Yeah, whatever, but it happened, and I was there, and I was like you're kidding me, so first you go and say look here's a cisco router right there; well that's just a manage router that was providing internet connectivity that, they didn't even touch that router. What I thought was hilarious once I regain composure

00:54:09

walking in this room is the fact that they have cable management, they actually have cable management in this rack, and it's not even being used. It's like up here is like someone said I think I'll use some cable management today and run these two cables, no, I did that, I actually did that was the tenants that were in place. I actually had my brother with me, and I'm like dude

00:54:30

you got to take a picture of this. I gotta show people, but just to give you an idea this is why these skills are so necessary is because this exists, and it's just not messy, and I'm not talking, this is one thing, yeah we can spend a day on that and get it all pretty do some cable management in your cabinet all that kind of stuff, but what I'm talking about is; this which is a little cisco switch goes to this which is a Juniper firewall, how did that get in the network and then it goes down to well you can't see it, it's kind of tugged behind here, a DSL modem, I kid you not, this is a real office with real people working in it, hundreds of them, that are all going through an action tech DSL modem for internet access, so when it comes to troubleshooting these skills are so necessary, you need to be able to do all these things; assuming you don't have Juniper, I mean I actually said that, assuming that you don't have a DSL modem, you should be able to read or redirect output to different places, test connectivity and all the different methods check the resources on your devices make sure processor memory is within check, span or rspan to do some port monitoring could get scary in these kinds of networks, and then why don't we walk through syslog and snmp and netflow and eem. All tools that you can add to your

Switch TSHOOT: VLANs and Spanning Tree Concept Review

Switch TSHOOT: VLANs and Spanning Tree

Switch TSHOOT: VLANs and Spanning Tree, Part 2

Switch TSHOOT: L3 Switching and Redundancy Protocols Concept Review

Switch TSHOOT: L3 Switching and Redundancy Protocols

Switch TSHOOT: L3 Switching and Redundancy Protocols, Part 2

Route TSHOOT: L3 Connectivity and EIGRP Concept Review

Route TSHOOT: L3 Connectivity and EIGRP

Route TSHOOT: L3 Connectivity and EIGRP, Part 2

Route TSHOOT: L3 Connectivity and EIGRP, Part 3

Route TSHOOT: OSPF and Route Redistribution Concept Review

Route TSHOOT: OSPF and Route Redistribution

Route TSHOOT: OSPF and Route Redistribution, Part 2

Route TSHOOT: BGP Concept Review

Route TSHOOT: BGP

Route TSHOOT: Router Performance Issues Concept Review

Route TSHOOT: Router Performance Issues

Security TSHOOT: Access List Concept Review

Security TSHOOT: Access List Chaos

IPv6 TSHOOT: IPv6 and IPv6 Routing Protocols

Please help us improve by sharing your feedback on training courses and videos. For customer service questions, please contact our support team. The views expressed in comments reflect those of the author and not of CBT Nuggets. We reserve the right to remove comments that do not adhere to our community standards.

comments powered by Disqus
Intermediate 13 hrs 26 videos

COURSE RATING

Basic Plan Features


Speed Control
Included in this course
Play videos at a faster or slower pace.

Bookmarks
Included in this course
Pick up where you left off watching a video.

Notes
Included in this course
Jot down information to refer back to at a later time.

Closed Captions
Included in this course
Follow what the trainers are saying with ease.

NuggetLab
Files/materials that supplement the video training

Premium Plan Features


Practice Exams
These practice tests help you review your knowledge and prepare you for exams.

Virtual Lab
Use a virtual environment to reinforce what you are learning and get hands-on experience.

Offline Training
Included in this course
Our mobile apps offer the ability to download videos and train anytime, anywhere offline.

Accountability Coaching
Included in this course
Develop and maintain a study plan with assistance from coaches.
Jeremy Cioara
Nugget trainer since 2003