00:00:00 - OK. It's time to get into the actual technical material. But
00:00:05 - before we get into the troubleshooting itself, for each one of
00:00:09 - these concepts, I'm going to do a little bit of review. The first
00:00:12 - one of these is going to be focused on VLANs and Spanning-Tree.
00:00:15 - So we're going to do a brief concept review of each one of these,
00:00:18 - you can see VLANs and Spanning-Tree, and then talk about just
00:00:21 - some key troubleshooting commands that can help you out along
00:00:24 - the way.
00:00:25 - Let's start off talking about VLANs. VLANs are still, to this
00:00:30 - day, one of my favorite things to talk about, especially when
00:00:33 - you get to the CCNA person who comes in and is a little kind
00:00:38 - of fuzzy as to what a VLAN even is. The fun part is everybody
00:00:41 - has heard of it. It's like subnetting. Everybody has heard of
00:00:44 - subnetting. They kind of have that nervous fear of it. It always
00:00:49 - goes like this. I ask a class, "OK. How many of you have heard
00:00:54 - of VLANs?" and almost every hand goes up. They're, "Oh, I've
00:00:57 - heard. Oh yes, we use it. We use VLANs. VLANs are everywhere."
00:01:00 - It's kind of like this kind of popcorn response. I go, "Yes,
00:01:03 - yes, yes. Well, let me tell you what it is." And I just go to
00:01:07 - the board and I just draw a simple little, there is a switch.
00:01:11 - You bought it at Best Buy for 15 bucks. Actually, a little more
00:01:15 - than that if it supports VLANs. But it's a four-port switch,
00:01:19 - and if I break it into VLANs, let's say I put the two ports on
00:01:22 - the left into one VLAN and the two ports on the right on another
00:01:25 - VLAN, it's as if I took that switch over my knee and went, "Aah,
00:01:28 - snap!" and broke it in half, and everybody goes, "Oooh!" And
00:01:31 - now I've got two working switches. That's what a VLAN is. It
00:01:35 - actually totally breaks apart the different pieces of the network.
00:01:38 - And everybody, "Oh." I'm telling you that that's one of those
00:01:41 - things that just resonates with everybody. They get it. And then
00:01:43 - I expand out and I say, "OK. Well, a VLAN, when you do that,
00:01:48 - as if they're in their own logical group, a broadcast within
00:01:51 - a VLAN stays in a VLAN. They've got their own IT subnet." And
00:01:54 - of course, you start expanding on all these terms because these
00:01:57 - are CCNA people. You don't want to melt them. But here I'm talking
00:02:00 - to you at the end of the TSHOOT series, or into the TSHOOT series,
00:02:04 - I should say, at the end of the CCNP track, and you guys got
00:02:08 - this to where this is what VLANs are all about. You can put security
00:02:11 - boundaries. I mean what network doesn't use VLANs? You know what
00:02:16 - they're all about. So VLANs can transcend switches through trunk
00:02:21 - ports, which is a Cisco word. Every other vendor calls them a
00:02:25 - tagged port, and I like that word better because it reflects
00:02:28 - what they do, which is they keep the 802.1Q
00:02:32 - tagged on there, which is the only industry-standard tagging
00:02:36 - language, ISL is dead, and that tag stays on there. And so if
00:02:40 - this guy sends a broadcast, that's how all these other switches
00:02:43 - know, "Oh, you're sending a broadcast. It belongs to the red
00:02:46 - VLAN. I got it," and that goes to all the right ports. You can
00:02:49 - have up to 4,096 VLANs, although not every switch supports that
00:02:54 - many active VLANs. They should all support those VLAN numbers.
00:02:59 - VTP, which is the VLAN Trunking Protocol, should have been called
00:03:04 - the VLAN Replication Protocol. Cisco is really trying to steer
00:03:08 - away from this nowadays because the best practice says you should
00:03:11 - not create VLANs on switches where they don't belong. Whereas
00:03:14 - VTP replicates all your VLANs to all the different switches in
00:03:18 - the organization, whether it belongs there or not. Now
00:03:21 - speaking of best practices, everything is changed in recent years,
00:03:25 - as you probably have seen, and no longer is it considered a good
00:03:28 - practice to have VLANs spanning the entire organization. It used
00:03:32 - to be, let's say this was company let me put
00:03:38 - my pen on the right Layer here this is company A. It used to
00:03:42 - be that you could have VLAN 10 and VLAN 10 would go campus-wide.
00:03:45 - No matter where you were in that campus, if you were in VLAN
00:03:49 - 10, you were on the same subnet. It works well for things like
00:03:51 - wireless, for voice over IP, like span the entire campus. It
00:03:55 - was great. Nowadays, Cisco says you can't do that or you shouldn't
00:03:59 - do that. Nowadays, Cisco says you should be using something called
00:04:03 - local VLANs. And that's where VLANs are constrained to a wiring
00:04:07 - closet, to where, let's say this is building A and you have this
00:04:11 - wiring closet with a couple Layer of three switches, a couple
00:04:14 - access Layer switches and then your clients. VLAN 10 should stay
00:04:18 - within this block. It should not move across these, and the main
00:04:22 - reason for that is because Cisco now recommends Layer 3 everywhere.
00:04:26 - You've got Layer 3 from your distribution Layer switches to your
00:04:30 - core Layer, so that's a point-to-point link. You've got Layer
00:04:33 - 3 between, well, if you can do it, between your distribution
00:04:37 - Layer and your access Layer even to where literally, you've got
00:04:41 - point-to-point LAN we'll call them LAN links everywhere because
00:04:45 - now you can use routing protocols rather than Spanning-Tree to
00:04:49 - converge. And routing protocols can always be faster than Spanning-Tree
00:04:53 - could. So that moves you into this local VLAN design to where
00:04:58 - VLANs are now constrained into the wiring closet.
00:05:03 - Doing these reviews are kind of funny because I'm thinking, "How
00:05:07 - can I explain everything about VLANs in five minutes or less?"
00:05:11 - I mean the good news is I know I'm preaching to the choir. You
00:05:14 - guys know what VLANs are all about. So let me talk a little bit
00:05:18 - about some key troubleshooting commands kind of a mindset when
00:05:22 - you're troubleshooting VLANs. The beauty of VLANs is it's primarily
00:05:26 - Layer 2 and below. So when you're thinking about things, you're
00:05:29 - doing commands like "show mac-address-table" to see what MAC
00:05:33 - addresses have been learned on a port, what VLAN they are associated
00:05:36 - with. You're doing a "show vlan" which shows all of the VLANs
00:05:40 - that are created on the switch and what ports have been assigned
00:05:45 - to the specific VLANs. I'm telling you, when I was troubleshooting
00:05:48 - VLANs, that's one of the first commands I type. Give me a list
00:05:51 - of the VLANs because it's one thing to have the port assigned
00:05:53 - to it and it's another thing for the VLAN to actually exist.
00:05:57 - I can't tell you how many times I'm troubleshooting something,
00:06:00 - a computer has no connectivity, and I don't actually have physical
00:06:03 - access to the switch so I can't see the amber light blinking
00:06:07 - above the port, but I'm going, "What's the deal? What's the deal?"
00:06:10 - and I find out that this port has been made a member of a VLAN
00:06:15 - that actually has not been added to that switch. And the weird
00:06:18 - thing is the port just disappears. I mean it doesn't show up.
00:06:23 - When you do "show vlan," it's not listed there. It's in the running
00:06:26 - config, sure, but if the VLAN doesn't exist, then it says that
00:06:30 - the Layer 2 fabric that the computer sits on is gone, that the
00:06:35 - little blanket that it's sitting on has been whisked out from
00:06:39 - underneath so it can't talk to anything. So the "show vlan" command
00:06:42 - shows both of those things: the VLAN has been created and that
00:06:46 - the port has been assigned. "Show interface switchboard," this
00:06:50 - one is really useful for finding out trunk ports, for example,
00:06:56 - especially you want to find out which ports are configured in
00:06:59 - that dynamic mode, that mode that we all love to hate, the dynamic
00:07:03 - mode where it flips between access and trunk that will show which
00:07:05 - ones are dynamic mode. It will show trunk ports. It will show
00:07:08 - what ports are going to cross there. A matter of fact, let me
00:07:11 - jump on here real quick. For trunk ports, I love this one, "show
00:07:15 - interface trunk" allows you to see which interfaces
00:07:20 - are configured as trunk ports as well as what VLANs they are
00:07:24 - affording. So if for instance, if you have chosen to restrict
00:07:27 - what VLANs go across the trunk interface, which is a good practice,
00:07:31 - you're able to see that information right here. So maybe a VLAN
00:07:35 - is not working as you would expect on a switch. You want to make
00:07:38 - sure that it's been added to the trunk port on both sides and
00:07:41 - that's a great command to do it right there. This is a little
00:07:44 - known command and I've actually only used this once or twice
00:07:47 - in the real world, but it's very handy when you do: traceroute
00:07:53 - MAC. You guys know what a traceroute is, right, to where I'm
00:07:56 - sitting at a PC and I want to find out what routers I'm going
00:07:59 - through on the way to Google.com. You can actually do a trace
00:08:02 - and it will show you every hop. Well, traceroute MAC is something
00:08:05 - that's supported on Cisco switches, and I emphasize that because
00:08:09 - it relies on the Cisco Discovery Protocol. If I have a bunch
00:08:13 - of switches daisy chained together and I want to know what switches
00:08:18 - I'm going through to reach a specific MAC address, that's where
00:08:22 - this comes in. Pretty sweet, huh? So that's a great way to kind
00:08:26 - of do a trace through your Layer 2 infrastructure. Again, the
00:08:29 - only requirement is that all those switches in that chain have
00:08:32 - CDP enabled. Now I know everybody is like, "Well, isn't CDP
00:08:37 - a bad security practice?"
00:08:39 - I don't know. I've got to be honest. If you read Cisco documentation,
00:08:45 - you'll find one document that's saying, "Yes, turn off CDP. You
00:08:48 - don't want people finding out about CDP blah, blah, blah." But
00:08:52 - then you'll find another document that says, "Oh yes, leave CDP
00:08:54 - on. It's not a bad deal." And a matter of fact, nowadays, it's
00:08:58 - recommended to turn it on on any interface that's connecting
00:09:01 - to an IP phone because it's useful for negotiating power requirements
00:09:04 - and transmitting voice VLAN information. I mean there is a lot
00:09:07 - that CDP can do. So I will be honest, I totally leave it on everywhere
00:09:13 - in the company. It's just it's too handy of a protocol to turn
00:09:16 - off. But that being said, you may end up with a security auditor
00:09:20 - coming in saying, "Hey, CDP, bad practice." Whatever. So do what
00:09:24 - your organization's security requirements and policies have told
00:09:28 - you to. So
00:09:29 - now let's move into my fly-by review of Spanning-Tree, probably
00:09:33 - one of the toughest concepts for the entry level person to get.
00:09:38 - And I always tell people, again, I talk about how do I teach
00:09:42 - this at the CCNA level. When I get to Spanning-Tree, I am head
00:09:46 - honest. I say this is one of the toughest concepts for me to
00:09:49 - teach, and I always emphasize it's not because the concept is
00:09:53 - complex. I mean it kind of is, but I mean if you think about
00:09:57 - it, Spanning-Tree, it's so simple. You're going, "What's up with
00:10:01 - the tree?" Well, that's the whole concept, right? All Spanning-Tree
00:10:04 - does is find, "Oh, looks like there is a redundant link that
00:10:08 - could cause a broadcast storm. Boom. Let's drop a tree on it."
00:10:13 - That's Spanning-Tree and you've got this little gap here now.
00:10:15 - It breaks the link. It brings that continuity to where you don't
00:10:18 - have the loop in the network. That is Spanning-Tree in all its
00:10:21 - glory. And I always say that's why it's the toughest concept
00:10:24 - for me to teach is because that's all there is to it, but now
00:10:27 - we're going to talk about that for hours because this is not
00:10:31 - what your networks look like. You don't have a network with two
00:10:34 - switches connected with a redundant link. You have a network
00:10:37 - that has 50 switches, and this one connects to this one and this
00:10:40 - one connects to this one and that one connects to that one and
00:10:41 - that one loops back here and this one connects over here. And
00:10:44 - then you have to say, "OK. Well, how does Spanning-Tree stop
00:10:46 - that? Well, why did it block that link? Why did it choose not
00:10:49 - to block that link?" And then people go, "Oh, OK. That's why
00:10:52 - it's so complex." And I kind of, I will be dead honest with you,
00:10:56 - I hate teaching Spanning-Tree just because it's so full of just,
00:11:00 - like, boring details and, "Oh, here is what a BPDU is. And I
00:11:05 - know some of you are, like, "Well, BPDUs are important." Totally.
00:11:07 - I totally bond with you. BPDUs are important. It's Rapid Spanning-Tree.
00:11:12 - It is important. Root bridges are important but it's just concept
00:11:16 - after concept after concept that it's all focused on just blocking
00:11:19 - that link and it's just because our networks are so complex,
00:11:23 - we need all of these complex concepts for it. One of the big
00:11:28 - things you want to remember about Spanning-Tree is that there
00:11:31 - are multiple flavors of it. And I don't know if this is real,
00:11:35 - it would freak me out if it is, but these flavors of Pringles,
00:11:38 - I know there is, like, every flavor of Pringle under the sun.
00:11:40 - Matter of fact, I got Seven-Layer Taco Dip Pringles not too long
00:11:44 - ago. They were really gross. But this is actually Soft-Shell
00:11:48 - Crab Pringles, Grilled Shrimp Pringles, and Seaweed Pringles.
00:11:53 - That would freak me out to see that in somebody's cabinet. But
00:11:58 - nonetheless, three flavors of Spanning-Tree. The original Spanning-Tree
00:12:02 - protocol, great at blocking loops but extremely slow by today's
00:12:07 - standards, up to 50 seconds to reconverge just because the link
00:12:12 - went down to find a new way to the root bridge. So Spanning-Tree,
00:12:16 - it is still alive and well and in networks everywhere, but I
00:12:20 - would say if it's in your production network, like data center,
00:12:23 - get it out. You want to move over to Rapid Spanning-Tree. Rapid
00:12:28 - Spanning-Tree, I would say fairly new in terms of the switch
00:12:33 - world. It's kind of funny. I was doing this indiscernible where
00:12:37 - I actually pulled the switch out of a customer network that we
00:12:40 - were just replacing. It was CatOS. It was just old. And I threw
00:12:44 - it up on my desk and I was looking at it, and I kind of blinked
00:12:47 - twice because I looked at the label and it actually said, right
00:12:50 - on it, I mean seriously, it was a 48-port it's still sitting
00:12:54 - in my garage CatOS switch. It was like, I want to say 2948.
00:12:58 - And right on the lower right-hand corner, it said 10/100/1000
00:13:04 - Ethernet. And I'm sitting there looking at it, and I go, "Seriously?
00:13:07 - That's a Gigabit switch? 48 ports of Gigabit?" And I sat there
00:13:10 - scratching my head. I powered up. Sure enough, it's Gigabit.
00:13:13 - And I'm thinking, "Now wait a second. That thing is running CatOS?
00:13:18 - Gigabit Ethernet?" I'm, like, "Has Gigabit Ethernet been out
00:13:20 - that long? I guess it has." And that's the thing, is I started
00:13:24 - thinking about why do we not run Rapid Spanning-Tree everywhere?
00:13:28 - Well, we still have 10/100 switches from a decade ago, I mean
00:13:32 - literally, 10 years ago, that are running just fine in people's
00:13:34 - network. They don't support Rapid Spanning-Tree but they're working
00:13:38 - fine. And to really run Rapid Spanning-Tree and get all the benefits,
00:13:41 - you got to run Rapid Spanning-Tree everywhere. So Rapid Spanning-Tree,
00:13:45 - the great thing about it is that it remembers the block link,
00:13:49 - whereas Spanning-Tree, let's say we've got our typical Spanning-Tree
00:13:51 - network right here, redundant link. This is the root bridge.
00:13:55 - This is our redundant link. Well, Spanning-Tree forgets about
00:13:58 - this, so if one of these links dies, it has to kind of rediscover
00:14:01 - that and put it through the blocking, listening, learning, and
00:14:04 - all of those process. Whereas Rapid Spanning-Tree goes, OK, I
00:14:07 - blocked that link but I'm marking it as a redundant backup link.
00:14:11 - And should something terrible happen and I lose my primary, I
00:14:15 - immediately know that that is a backup and I'm going to make
00:14:18 - it active. So it makes Rapid Spanning-Tree much faster than the
00:14:21 - original. Multiple Spanning-Tree
00:14:25 - is a way of grouping Spanning-Tree instances. You may know that
00:14:29 - Cisco does not actually run Spanning-Tree and Rapid Spanning-Tree.
00:14:33 - They run per
00:14:36 - PV Spanning-Tree and
00:14:40 - PV Rapid Spanning-Tree, which, you remember, stands for Per-VLAN.
00:14:43 - Well, that's great until you have 50 VLANs and your switch is
00:14:46 - just dying because it's got a Spanning-Tree instance on each
00:14:49 - one of those. Multiple Spanning-Tree allows you to group Spanning-Tree
00:14:52 - instances together to where, let's say I've got that was odd.
00:14:58 - Everything froze for a moment. I've got my typical Spanning-Tree
00:15:01 - instance right here. I've got 50 VLANs. Well, I can say one instance
00:15:07 - of Spanning-Tree is running for 25 VLANs and that's the root
00:15:10 - bridge for that. And then one instance of Spanning-Tree is running
00:15:14 - for the other 25 VLANs,
00:15:17 - and that's going to be the root bridge for that. So I run essentially
00:15:20 - two instances of Spanning-Tree for my 50 VLANs instead of 50
00:15:25 - instances all doing pretty much the same thing. So now let's
00:15:31 - blend all that into the key troubleshooting for Spanning-Tree.
00:15:36 - First off, let me just set the TSHOOT exam aside for a moment
00:15:39 - and talk about real world. If you have a Spanning-Tree loop in
00:15:44 - your network,
00:15:46 - people go, "What do you do?" You do what we all do. You run nil.
00:15:49 - You run. I mean Spanning-Tree is one of those bad-day scenarios
00:15:54 - where I mean the switch lights are blinking like mad. You can't
00:15:57 - access them via Telnet, sometimes SSH, because they're frozen.
00:16:01 - The processor is pegged. Switches are going down and rebooting
00:16:04 - computers. I mean the network is useless. Your servers are down.
00:16:07 - Computers are down. It's not good. You run into the server room
00:16:11 - and you just start unplugging cables, really focusing on your
00:16:14 - inner switch links first and then focusing on the PCs, if you
00:16:18 - believe that's where a loop is coming from. What I usually do
00:16:20 - is start unplugging one major switch at a time and find out where
00:16:25 - the madness stops to where I can at least tell what switch. I
00:16:28 - mean sometimes the chassis-based switches are huge with a lot
00:16:32 - of connections, but at least it will focus me on which switch
00:16:34 - then I focus on the module then I say, "OK. Here is the cable
00:16:37 - that's doing it. Let's trace this and find out who caused this
00:16:41 - whole thing." And unfortunately, it's usually something that
00:16:44 - you plugged in a couple of weeks ago and totally forgot about
00:16:46 - and someone did a no-shut somewhere at some point. That aside,
00:16:50 - they're not going to have, of course, a complete network outage
00:16:53 - on the TSHOOT exam because you have to be able to access the
00:16:56 - devices. So what you'll want to do is first off, check out "show
00:17:01 - spanning-tree. "Show spanning-tree" is your quick view of the
00:17:05 - Spanning-Tree network. It will show you what interface statuses
00:17:08 - are, what your current bridge ID is, what interfaces are blocked,
00:17:13 - what are forwarding, designated ports, backup ports, all those
00:17:16 - kind of things. It will tell you who the root bridge is in the
00:17:20 - network, the priority, just a ton of information, and it will
00:17:23 - show it for every single VLAN. This one is, I don't use this
00:17:28 - too often but it's good, "show spanning-tree interface detail."
00:17:31 - And this is, by the way, a variable to where you put in whatever
00:17:34 - interface you want to focus on. That will show you the actual
00:17:37 - cost of that interface. It will show you again the bridge ID,
00:17:43 - the root bridge ID, is this the interface that helps you reach
00:17:47 - the root, all the kind of how many BPDUs you're setting on that
00:17:50 - port, just a lot of kind of nitty-gritty troubleshooting that
00:17:54 - you can get. This
00:17:57 - one, I kind of threw this one in here, not directly related to
00:18:01 - Spanning-Tree, but if the switches are accessible and you suspect
00:18:04 - it's Spanning-Tree loop, do a "show process cpu." Spanning-Tree
00:18:08 - loops don't always take out the entire network, although they
00:18:11 - can, depending on the severity of the loop. So if you're looking
00:18:14 - at your CPU processes and you're at 90% and you're typically
00:18:18 - running around 5-10%,
00:18:21 - usually a Spanning-Tree issue at that point. I mean if you're
00:18:24 - just staying pegged, there is a loop in the network. And remember,
00:18:27 - when you have a loop, I mean when you've got a couple of switches
00:18:30 - I'll just draw up a quick scenario here and you get a loop in
00:18:34 - the network, you have immortal packets.
00:18:37 - Everybody thinks, "Oh, well, they'll die eventually because of
00:18:40 - the TTL." No. TTL is a Layer 3 field in the header. The only
00:18:45 - thing that decrements the TTL, the Time to Live, is a router.
00:18:49 - And if there are no routers in the picture, then you've got a
00:18:52 - packet that will live forever and ever, looping around and around
00:18:55 - and around, taking the network out. So again, when you do that
00:18:57 - "show process cpu", if it's just steady, pegged, a lot of times,
00:19:02 - you can go, "OK. Something is up and it's probably Spanning-Tree."
00:19:06 - So it's kind of like, "Well, what's that command for? Hmm." Yes.
00:19:09 - Go through and literally start shutting down again, I would start
00:19:15 - focusing on do a quick "show cdp neighbors," see what's ports
00:19:18 - are connected to what if you don't have a network diagram. Shut
00:19:21 - down the interfaces that are connecting neighbors and see if
00:19:24 - the processor suddenly drops because you may just strike gold
00:19:29 - and hit the port that is looping.
00:19:33 - Well, that should get our blood flowing on VLANs and Spanning-Tree,
00:19:36 - looking at the concept review to some of the key troubleshooting
00:19:39 - aspects for each one of these technologies. So let's now move
00:19:42 - in to the real troubleshooting scenarios. I hope this has been
00:19:46 - informative for you, and I'd like to thank you for viewing.