FGT Troubleshooting Methodologies

0:00Hello and welcome. My name is Keith Barker. And of course, we would love it if everything just worked as expected, as it's supposed to, every single time. But the reality is, in the real world that we live in, that's not always the case. So in this set of videos, I'd like to walk you through some troubleshooting methodologies as they apply to networks in general and also specifically to 48 firewalls. So I'm glad that you've joined me for this journey, and I'll see you, my friend, in the very next video.

0:01So one of our first steps is to define the problem. And I would also add to that, define the problem before we ever look at or touch at the network devices. So if we're working with a customer, for example, either here or here, and we have access at the CLI and or the GUI to the FortiGates or whatever management tool we're using to manage those FortiGates, we want to define the problem before we start looking at the configs. Because if we have an ambiguous or missing problem definition, we're going to waste a lot of time in troubleshooting. So here are some things we'd want to get based on that fact-finding.

0:33We'd want to learn what exactly is failing. And that may also require, as we interact with users and other people, bringing it down to the level that's appropriate for our audience. For example, if we have a secretary or somebody that's not familiar with, you know, DNS and all those other protocols that are being used, as they go out to the Internet, they just may say, well, the network is down or the Internet's down or I can't get to the Internet. So we might want to clarify, you know, what they're attempting to do. And again, also asking, when's the last time that this worked? Another great question to clarify is, who is this affecting?

1:05So if there's a couple people on the team and this is person one here and this is person two here, we want to confirm is, you know, all the people in that same location on the same subnet, for example, are they all having the same problem? And then another great question is to ask, how often is it having a problem? Is it always broken or is it intermittent? And if they say it's intermittent, that's a little tougher because it's harder to verify consistently the problem. But if it's always, you know, the problem is always there, that's great news because we can clarify that, confirm that, recreate the symptom, and then start our troubleshooting.

1:37So as we're defining the problem, let me add some cautions as well. Number one, if we have a false positive where we, you know, think something's an issue, but it really isn't, that's going to take our troubleshooting in a misdirection. For example, maybe we're not allowing, based on policy, ping requests from clients out to the public internet. And as a result, they try a ping or we try a ping from here and it's not successful and I think, oh, there's a problem with IP connectivity when it really could just be a policy. Or we could be doing a trace from these devices.

2:08And that also can cause problems because trace is not done the same. Another challenge is the scope. Now, if we don't correctly identify the scope regarding who it's affecting, once again, we could be troubleshooting in the wrong direction because if the entire subnet is having a problem, it's very likely going to be something that connects those devices to the rest of the network. It could be a switch here or it could be the firewall here or the firewall here if they're all trying to get up to the internet and they can't. And that's a much different situation than just one device that can't access some resource. So for our scenario, let's imagine that we have these two devices here.

2:38I don't know their addresses. We'll look at those in just a moment. They're both on the 10.10.0 subnet. I think this guy's at .102 and the Linux device is at 1.0.1. We can verify that here in a moment. And let's imagine we get a call from the Windows user who indicated that they can't get to any of their traditional websites on the internet. Now, we say traditional. It's like, what does that mean? You might want to narrow that scope and say, okay, which website, as an example, can you not get to? And then we want to replicate that and verify that. So with this topology in mind, let's go to this PC right here.

3:10And let's just validate the problem that the user is talking about. So here's this Windows client. And let's go and bring up a command prompt. And let's do an ipconfig. So there's this IP address, 10.10.0.102 with a 24-bit mask, default gateway is 10.10.0.52. And that's the network interface it's using as part of the topology. All right, let me minimize that. And let's open up a browser. So here at the browser, it looks like it's trying to load a whole bunch of different tabs. Let's go ahead and close some of those.

3:41And let's just go to fast.com as an example. As I mentioned, that's a critical app. And sure enough, loading, loading, loading. And it is not working. All right. So we've now defined that this is one of the issues in measurable terms that the client is experiencing. They cannot get to the Internet. Let me go ahead and close that. So regarding exactly what's failing, we confirmed that it's Internet access to fast.com. And we have other websites that also are not working. But it would be nice to just do one at a time. By focusing on just one site at a time, we're keeping our symptoms identified and narrow in scope.

4:15So we can solve for that. And then we can take a look at the other websites if there's other websites as well. As far as who is affecting, let's see whether or not it's affecting this Linux device, which is a .101. So here is that Linux device. Let's open up a command prompt. And let's do an ifconfig on this Linux device. And its network interface we're using for the lab is this one right here with the IP address of 10.10.0.101. If we do a route, there we go. So its default gateway is 10.10.0.52. That looks good. But let's also just bring up a browser and verify whether or not this client can get out to fast.com.

4:49Again, the same kind of a test, same website. So we'll go to fast.com. Press Enter. And sure enough, that is working. So fast.com is provided by Netflix to simulate Netflix traffic. So that means that this client, this device, is able to access this website while the other one isn't. So as far as who is this affecting, it's just the client .102 here on the 10.10.0 subnet. And as far as always or intermittent, I can just tell you it's always.

5:19That's not because I put the bug in there or put the problem in there, or maybe it is. Anyway, it's always that case. This client just cannot get to any websites, including fast.com, while the Linux device has no problem. So the benefits of defining the problem and doing it exactly, as far as what's failing and the scope, is going to help us as we continue moving forward with the troubleshooting of this problem. So now that we have clearly defined these symptoms, the problem that the customer is having, we've also identified the scope. It's just this one PC. In the next video, we'll take the next logical step.

5:51And I'll see you in the next video for that next step.

0:00So now that we've identified the problem, that is this device right here on the 10.10.0 network at dot one zero two, and this user cannot access fast.com. We're using that as an example of a resource that normally is accessible and used by this user that they can't get to. And in conjunction with that, we'd also want to ask the user when's the last time it did work, which is a really important question to ask. And now we're going to do the next step is to classify the failure. So here are some of the questions we want to confirm as we do our troubleshooting regarding the traffic from this user that's trying to go to fast.com. In this example, is that traffic reaching the FortiGate? Because if for whatever reason that traffic never leaves the network adapter on this computer and makes it over to the FortiGate, it's not a firewall problem.

0:44Number two, if it is reaching the FortiGate, is that traffic being routed correctly? And I'm going to put that in as routing tables on the FortiGate because if the packet shows up at the firewall and the firewall says, oh, based on this destination address, I'm going to route it this way or that way or the other way. If it's not going in this direction, let me go ahead and put this in green. If it's not going this direction here, leading over to firewall one, that traffic is not going to make it out to the public internet. So this firewall didn't have, for example, a default route or didn't have a route out to the public internet using firewall one as the next hop here, that would be a problem. And then the third question, assuming it is showing up at the firewall and we have routing in place, is the firewall denying it or implicitly denying it and not forwarding that traffic? So I'm going to write that out as being denied at the firewall. And that could be based on any of the policies or security profiles associated with policies that either firewall two or firewall one have in place. So let's go ahead and answer these questions. Now, if we have access to this device, or if the user is willing to work with us, we could issue some commands here, but we could also issue commands here at firewall two or firewall one, or we could use GUI interfaces to communicate with those devices, or we could use Forta Manager to look at the routing information and do tests from these devices as well. So let's start over here at the

2:01client because I have a remote desktop to this client. Let's start here and ask the question, is the traffic from this device reaching the 48? So here at that Windows client, let's go and bring up a command prompt. So this is our IP address. There's our default gateway of 10.10.0.52, which I believe is the FortiGate. We'll confirm that here in a little bit. Let's also go ahead and just do a ping to our default gateway. That just means we have connectivity between us, the Windows client and the default gateway. So we'll do a ping to 10.10.0.52. And that is working.

2:34We also on this Windows computer could do an ARP-A and look at the ARP cache. And that's showing us here on this interface that this is the MAC address associated with the 10.10.0.52. So if we looked at the firewall, we could confirm that this MAC address A and E5 is associated with the interface on the firewall that has the IP address 10.10.0.52. And down here below with the IP address of 1.3, I'm using that for remote access into this Windows computer. And there's no default gateway set up off of that interface. So all the internet-based traffic is going to be coming from the 10.10.0.102 address off this lab interface. And if we use the upper key a couple of times and do an IP config again, we currently have 10.10.0.52 as our default gateway. Fantastic.

3:17So from a routing perspective from the client, it has firewall 2 as its default gateway. That looks good. Let's look at the routing tables on firewall 2 and firewall 1 to make sure that their routing tables are both appropriate for forwarding traffic from 10.10.0 going out to the public internet and also being able to forward traffic back to that subnet 10.10.0. Because firewall 2 is directly connected, but firewall 1 is going to need to have a route that says how to get to the 10.10.0 network. So let's take a look at the routing tables. So here is a CLI. I've got an SSH session over to firewall 2. I'm using secure CRT for that. And here at firewall 2, let's take a look at the routing table. So we use a get router info, routing table, question mark, and let's go ahead and ask for all the complete routing tables being used by firewall 2. So here we have a default route. Fantastic. It's being learned via OSPF. Fantastic. The next hop is going to be 10.12.0.51, which based on our topology is the next interface over at firewall 1. We can also verify with an execute ping that we can reach 10.12.0.51. And sure enough, we can. Fantastic. And this also shows in the routing table that we are directly connected to the 10.10.0 network. Fantastic. So the routing table on firewall 2 looks great. Let's take a look at firewall 1. So here's firewall 1. Let me go

4:35ahead and copy that command. So I'll go back. I'll do a get router info, routing table all, go back to firewall 1, paste that in and press enter. So on firewall 1, we also have a default route. And the next hop is 23.1.2.1. Let's take a look at our topology. So in our topology, 23.1.2.1 is this service provider A. So it looks like we've got connectivity out through service provider A. We have a route, a default route using service provider A. And let's also verify we can ping our service provider. So we'll do an execute ping at 23.1.2.1. And that looks good. Also here on firewall 1, let's take a look at the route back to 10.10.0. Because if this firewall is sending traffic back to our client or on the 10.10.0 network, it needs to have a route. So it has an OSPF on route. Great, great, great. And the next hop is 10.12.0.52, which is an interface over on firewall 2, which looks great. So another thing we could do here is we could try a ping or even a trace from this Windows computer or from firewall 2 to get to the 23.1.2.1. But because this Linux device has no problem getting out to the public internet, it's very likely not a routing issue. It's going to be something else. So just to verify that from this Windows computer, let's do a quick ping out to 23.1.2.1 just to verify we have connectivity. And we can also do a trace as well.

5:55So back here at PC3, the Windows computer, if we do a ping to 23.1.2.1, that looks good. Also, it's showing a TTL of 253. And that's for the responses that came back, which implies that ISP is a couple hops away. Going through firewall 2 and firewall 1, we can also do a trace. So on a Windows computer, it's called trace RT-D. Don't bother doing name resolution. And let's go ahead and go to 23.1.2.1. And that will also confirm the path we're using. So our first hop was 10.10.0.52, which is firewall 2. Next hop was 10.12.0.51, which is firewall 1.

6:32And then the last hop here, the last device was the service provider itself at 23.1.2.1. So as we went through our checklist here, the traffic is reaching the FortiGate. The routing tables on the FortiGate are correct based on what we looked at. And now the question is, is that traffic being denied at a firewall? And for that test, let's go ahead and go back to fast.com again or attempt that. So that is currently not working. And as far as matching on that, we could look at the logs here on firewall 2 and firewall 1. We could also do a packet capture right here on the computer itself. Or if we have a span port set up, we could do a packet capture off the network to see whether or not the traffic that's trying to go to fast.com is going there or not. And again, this is before we're even looking at the policies here on firewall 2 and firewall 1. So let's go over to this computer here. I've got a remote desktop open to it. Let's go ahead and run Wireshark right here and capture the traffic as it's going in and out of the network interface card. And then we'll attempt to go to fast.com and look at the results.

7:29All right. So on this computer, I'm going to go ahead and minimize the command prompt there. Let's launch Wireshark. There's a shortcut here on the desktop for it. And our network adapter is this one right here, the VLAN10 interface. I'll go ahead and start a capture. Okay. So I see maybe what the problem is, but now we have this capture running. Let me minimize that. And let's go ahead and open up a browser. So currently this browser has a bunch of default tabs trying to load. Let me close them just one at a time here. And let's open up a new tab and let's try to go to fast.com. That's our test that we're using to verify whether or not it works.

8:06Sure enough, fast.com's DNS address could not be... Oh, look at that. It's telling us that the name resolution from fast.com didn't happen on this computer. So if we go back to the command prompt, here we can type in nslookup for name resolution and try to resolve fast.com. And survey says, yeah, DNS request timed out. Here's the DNS server we're trying to use. It's trying again and again. I'll go ahead and do a control C. Also because we're capturing that, let's take a look. So here's our Wireshark. I'll go ahead and stop the capture and let's take a look at DNS.

8:42So I'm going to put a display filter just for DNS up in the top left there. And look at that. So it's asking about a lot, including fast.com. So here at the command prompt, we do the ifconfig and then we do a slash display DNS. It says, Keith, you're not on Linux. Okay. So let's hit the up arrow key and do an ipconfig display DNS. And here's what it knows about. Effectively, it has no quad A records, no IPv6. It knows about itself. That's the loop address for IPv4. And then it also has this one here, but it has no other names, including fast.com that's resolved, which would imply that this client is not correctly doing DNS name resolution.

9:20So if we go back to our capture, I'm going to stop it. And we do a display filter for just DNS up in the top left there. And if we scroll down, it has a ton of standard queries and there is fast.com right there. However, if we go in and say, you know, instead of looking for a query, go ahead and I'm going to just click and drag on response, drag that up to the display filter and say selected. And I'm going to say response equals true and press center. Here's the only response that we got. And it came from 8, 8, 8, 8, 8, coming back to 10, 10, 0, 1, 0, 2.

9:51And there's a response for google.com with the appropriate IP address associated with google.com, but none of the name resolution worked. So if we go back to the command prompt in an ipconfig dash all or slash all, either way works with the current version of the windows. Currently it's using a DNS server of 6, 7, 8, 3. However, based on our NS lookup that we just did, that's not the right DNS server, or at least it's not responding with DNS answers back to the client. So if we use the PPP that we talked about earlier, or the RAP, which I personally like the most, as far as routing goes, that was okay. Uh, address translation. Hmm. How do we verify that?

10:27You know, we could do if it's just DNS issue, uh, we could go ahead and reach a device based on its IP address. That would confirm that address translation is all working. So let's try that real quick as well. So back at the client, let's do a trace route. So in windows is T R A C E R T dash D for don't bother doing reverse lookups because name resolution is not working anyway. And let's go out to 8, 8, 8, 8, and press enter. And I'll do a control C that's enough. So here's showing going through firewall two and then firewall one, and then the service provider a or ISP a, and then other hops in the path towards 8, 8, 8, 8. So as far as reaching the public internet or getting off of our local network, that is working. So this implies a few things. Number one, IP connectivity wise, we have access out to the public internet to also because it's just the name resolution is failing. It's probably just a name resolution issue local here on the windows PC. And again, we can see that with IP config slash or dash all, and there's the DNS server it's trying to use. And with NS lookup, we confirmed it can't do name resolution. So as far as permissions go, the ICMP traffic that was used by trace route on this windows computer that was allowed out address translation worked because the replies came back.

11:38So as far as what this likely is, it's likely the wrong DNS server that this single client is attempting to use. Now, another element that I also want to chat about is that if we have a user that has some issue getting up to the internet or some resource, and they claim is the firewall. Another really important tool that we have available to us is to look at the firewall and to identify existing flows to show that, Hey, this flow of traffic is going through the firewall. It's something else outside of the firewall itself. So even though we have narrowed this down in this scenario, that is very likely a DNS issue on this computer. I'd also like to walk you through examples of confirming traffic flow for sessions going through a firewall, and we get to do that in the next video. So I'll see you there in just a moment.

0:00In this video, I'd like to walk you through a couple examples of being able to prove traffic flow through the firewall. So one of the benefits of looking at these sessions is that if the traffic is being allowed through, and then the reply traffic eventually comes back via the stateful database, via the stateful nature of the firewall, that effectively shows that the traffic is going through the firewall, that the firewall is not denying it. Now we could look at the logs and look for denies, but a lot of the implicit denies at the end of the FortiGate policies don't have logging enabled by default. So if we're looking for just the deny statement, we may not find it because of the implicit deny that is perhaps matched. However, if we show sessions from this address going out to the servers and so forth in the firewall, that can help verify that traffic is actually being forwarded correctly by the firewall.

0:45So a few questions asked. Number one, does a session exist for when a client is trying to go out to a resource? Number two, is that traffic, is that session being offloaded? In many vendors implementation, they call it the slow path versus the fast path, where the initial packets flow and they get processed by the central CPU. And then for remaining packets in that session, they're offloaded to a dedicated processor for that purpose. And that's also a good question to ask because if we are doing debugs, which we'll do in the next video, and we're looking at specific traffic flows, we might not see all that traffic if it's being offloaded to another processor. Another question to ask is, is the traffic asymmetric? Meaning, is it coming in through one path? In this topology, there's only one path in and out from this PC going through firewall two to firewall one after the public internet. But if we had another firewall or another routing device, we could have traffic that's going out one path and coming in another. For example, we have a router here that's also connected to the 10.10.0 network. And the reply traffic, if it's coming back in this path or this direction, that could cause a problem as well, because the firewall two is not seeing the reply paths coming back in. Also, this could indicate that we have a routing issue if that wasn't our intent.

1:53And normally, if traffic is going out through firewall two, we want the reply traffic to come back in to the client via firewall two. So from this client, let's generate a ping stream that we can actually look at as far as session flow. And then we'll take a look at a couple different options for looking at the session information. So back at our Windows client here in Wireshark, and fantastic, we have traffic showing there. I'll minimize that. Let's go back to the command prompt and let's do a ping slash question mark. And I want to go ahead and do a dash T. That'll just keep on pinging. So we'll do a ping dash T and let's go to 8888. That's a Google DNS server.

2:28And that's working. All right. So that's running. Now, as far as validating and verifying session information, there's a few ways of doing it. Number one, we could use the GUI. That would certainly do it. Number two, we could use the CLI. So let's take a look at both those examples of verifying session information. So let's start off on the GUI here. So this is firewall one. So here on firewall one, we'll go to FortiView sessions. And sure enough, here we have some pings. So it's showing here that we have some pings going on that are coming from 10.10.0.102. And then we have a whole bunch of other DNS requests that have gone out and continue to go out from that client. So that is one way to validate that we have traffic flowing through the FortiGate.

3:05Another option would be go down to log and report and go to forward traffic. And again, this is on firewall one that we're doing this. And here we could say, you know what? I want to go ahead and filter or anything that's not and accept. And that should show us the denies. And sure enough, we have a bunch of denies, but it's all from 10.12.0.52. It's not from our clients on the 10.10.0 subnet. So based on this happening, I also probably want to look into why that's happening. But one troubleshooting step at a time, we're focused on the client on the 10.10.0 network.

3:35And currently with the source addresses here, we're not denying on that source address of 10.10 anything. It's all 10.12. So now that we've taken a look at the FortiView sessions here from the GUI, and I just want to confirm I can still see the pink, sure enough, right there, and it's continuing to grow. Fantastic. Let's also take a look at the CLI, how we could verify a session. So instead of just looking at every single possible session that may be going through the firewall, this is firewall one, we could do a diagnosis session filter, and then put in the details regarding the interfaces involved, the ports involved, et cetera, et cetera, et cetera. Or if you just want to press enter here, it's going to show you that currently based on the filter, it's any VDOM, any source interface, any destination interface, et cetera, et cetera, et cetera. So in my small environment, I could do a diagnosis session list, press enter. This is pages and pages and pages. So for example, right here, we have our client at 10.10.0.1.0.2 with a source port of 62,000 and change, trying to go out to 6783. And because it's in the session table, that traffic was forwarded out there to the well-known port for DNS of 53. And so once again, this is another option we can use to help verify traffic is actually flowing through the firewall.

4:47So if we wanted to filter the output a little bit, what we could do is add, for example, the protocol. So I'll add in PROTO space, question mark, and I'm going to go ahead and use 01. So protocol number one is ICMP, we'll press enter. And now if we do the command diagnose session list and press enter, it'll just show us the ICMP-based sessions. And here's the information regarding that session traffic from 10.10.0.1.0.2, that's our Windows client, going out to 8888 with those continuous ping requests. And back at the GUI, we can do similar treatment. We can just right-click, for example, on the ping here, filter by application, exact match of ping, and that would show us that information. So effectively we're seeing here that ping session, that ping conversation is happening between our Windows client and the server at 8888, the Google DNS server. Also, if we scroll here to the right, it indicates that that session has not been offloaded to an NPU.

5:36So if we remove the filter and show all the sessions and then go to the right, we can notice that some of the traffic has been offloaded to the NPU. For example, this traffic right here in this session, which is coming from 10.12.0.52, another device in our topology, going out to this address, and that's our firewall at 10.12.0.52, that's actually firewall two, that's interacting with FortiGuard. So whether you want to look at the sessions here, or you want to use the CLI, whichever is most convenient or most handy, these are two different ways of looking at the session information on the FortiGate itself.