| technology | programming - David Chapman
20 Essential Linux Network Monitoring and Configuration Commands
When it comes to network monitoring and configuration commands, there is no shortage of them for Linux. Whereas, with other operating systems there are typically less commands and tools for troubleshooting networks.
Having so many Linux commands at your disposal can be overwhelming. That's why understanding them and when to use them is crucial. It'll save you time and confusion — and make troubleshooting Linux environments easier. Let's take a look at 20 must-know commands for any Linux admin.
Nmap is a great 3rd-party port scanner that allows you to scan remote endpoint ports. This can be extremely useful for tasks like verifying your firewall configuration is correct or identifying ports that should be open or not. Depending on the endpoint it may even be able to detect what operating system it is running. It also has a plugin system that allows you to run custom scripts so the use cases for it are pretty vast.
Netstat is a tool that comes with most Linux distributions and has a few use cases. For starters it will show you connection tables, which allows you to see which outgoing connections your system is making and also which incoming connections are established. Ever wonder if that service or daemon you started such as apache is actually listening on the right ports or any ports? You can use netstat to view that kind of information to help confirm whether the ports are being listened on.
Iftop, as its name implies, does something similar to the top command. Instead of processes though it does this for network connections. What iftop displays is the list of top connections ranked by throughput, so that you can see many of the top offenders chewing through your available bandwidth or throughput. What is really great about it is it shows the throughput in both directions of each top offender.
The route command falls more under the configuration category. The default route is typically the only route that is set under most cases but sometimes in the event that the Linux instance in question is a router, it may contain multiple interfaces and routes to each of them. In any case, the route command will allow you to print and change the route table.
When it comes to throughput or bandwidth testing, iperf is the de facto tool for the job. It supports a multitude of modes and can even test the line for latency and jitter. The client and server is typically wrapped into one executable so you can have it installed and on the command line indicate which end is the client and which is the server. Typically for bandwidth testing, you'll choose TCP mode for the testing. On the other hand, UDP is better for packet loss, jitter and latency testing just due to the nature of the command.
Tcpdump is an amazing tool but simplistic at the same time. At the simplest level, it allows you to capture traffic on an endpoint and typically either display to the screen or output to a file to review later or parse in another tool. It is extremely useful in viewing traffic to help validate traffic is arriving or departing as expected as well as the metadata on the packets to try to detect errors.
It can be a little overwhelming at times reading the output of tcpdump if you are not using the right filters or know what you're looking at but with a little training and experience it can take you a long way.
Ipset is a tool that works alongside the Linux ipfilter Firewall framework. It allows you to create groupings of IPs. Without this, when creating firewall rules related to multiple subnets, each one of them would have to be a separate rule but with ipsec, groupings can be created to ease the management of this and make it more transparent. Better yet, names are used which if planned appropriately can help with reading and understanding the rules over viewing subnets and needing to remember what they correlate to.
Earlier we mentioned tcpdump. When we talk about Wireshark, we typically refer to a gui version of tcpdump. Previously it went by the name Ethereal but around 2006 it changed its name. In any case, it is an excellent tool for visualizing tcpdump outputs or capturing your own in a gui interface. It is also very extensible and has many extensions to aid in visualization that go above and beyond what tcpdump provides.
For example, you can drill into TCP streams or filter by them. In other cases you can decode VoIP streams to listen to the audio and verify the quality. It also does a great job at understanding most protocols and providing you the metadata of them in easy to read formats. Tshark is the CLI companion to Wireshark. It operates a little like tcpdump except it carries a lot of the logic with it on filtering and extra command line options.
In the old days, when telnet was popular in use, you could use it to connect to a remote system to validate a port was operational and send raw commands to it. In modern times the telnet protocol is no longer in common use and many times systems do not come installed with telnet at all. Telnet had its own protocol though and was not actually a raw socket connection tool. Netcat is the tool of choice for this as it can operate completely raw and be used to do things like connect to an SMTP server and attempt to test commands. On the other hand, it’s also great to just connect to a port like ssh/22 or mysql/3306 just to see if it responds.
When dealing with path issues on a network, whether they are internet or internal network or a combination of both, the traceroute tool is one of the oldest tools for this. It operates by sending a specially crafted packet to the destination but modifying the Time To Live or TTL of the packet to determine each hop of the path and its latency. It can be used to find unexpected hops/paths internally or loops that may be happening on the Internet.
Much like traceroute, mtr can find hops on the internet between your endpoint and a remote one but where mtr is great is that it is like a continuous traceroute, showing latency for each hop. The traceroute may have looked good but how does it look over time? Running continuously also allows you to catch transient or intermittent routing issues that may be popping up.
Arp is a command that is named after a lower OSI layer protocol named ARP, which stands for address resolution protocol. it allows you to view existing arp lookups and also add or purge arp records. Often when an endpoint is fully firewalled, if you're on the same LAN segment and subnet is to view the arp table to see if it responds. If it’s communicating with the network it will typically respond to arp so the upstream gateway knows its address.
Nslookup is one of the original name server lookup tools. It allows you to specify a name server and list the types of records you want to query from it.
Dig does similar things to nslookup but depending on your need, it is a little tweaked and may meet your needs better. For starters it displays the query and answer as well as the Time To Live (TTL) on that answer. This can be helpful in just understanding when a DNS query TTL/cache will timeout.
DNS servers typically have this locked down but if allowed you can pull a full zone transfer from a DNS server and dig allows for use of this AXFR protocol to do that. Keep in mind, pulling a full zone is usually restricted by ACL so many public name servers do not allow it. But if you are the systems administrator you can grant yourself access.
Host is another tool for the same thing that nslookup and dig use. You would wonder why we have so many tools for DNS queries but each caters to a different audience. Host is a nice and simple tool. You simply run the host command with the endpoint you want to query and provide the name of the nameserver it queried and the response. Alternatively, you can point it at an alternate name server.
To understand the whois command, it is important to understand domain name registration. When one purchases a domain name from a registrar, record of that gets placed in a whois database so that it can be looked up. For example, if you need the technical contact for a domain to let them know spam is originating from their domain it can be listed. Often though, domain owners opt to register anonymously, so it is not very helpful. IT is still very good at checking to see if a domain is even registered and some metadata about it such as when it was purchased and when it was renewed last.
Ping is one of the most essential tools and oldest tools for basic troubleshooting. It exists on nearly every operating system from Linux, Windows, routers, and even Layer 3 switches. Nearly everything with a TCP/IP stack on it. It serves to perform essential network latency and line condition tests. With ping, our first inclination is to check whether all of the pings are returning or whether there are drops or timeouts. Those can indicate a networking issue somewhere. From there, the latency reported is key. Pinging on the same LAN should equate to 1 or sub 1ms latency whereas pinging across the globe may be many hundred milliseconds.
In Linux, it runs a continuous ping by default with a very low payload — but you can also increase the payload and under certain conditions do a ping flood to help test the line to see when packets drop or if the network can keep up. It is extremely versatile for basic network troubleshooting.
Nmcli is a command line tool for managing the NetworkManager. This is a subsystem that stores persistent data regarding the networking and its interfaces so that upon boot they can be brought up as expected. You may think this is trivial but it helps aid in situations where new NICs are added and ensuring the appropriate NICs are assigned the correct IP addresses. Settings as to whether an interface is DHCP or static and its DNS resolver settings.
Ip is a tool that deals with ip addressing on Linux machines. The original command ifconfig is still around but ip has been greatly expanded and in some cases, distributions no longer come with ifconfig by default. While ifconfig has been tried and true for decades on viewing and setting interface settings it has not kept up with the items over the years and the ip command does help fill some of those gaps and is a modern version of it. With this tool, ip addresses of the local interfaces can be viewed, modified. The routing table can be viewed with it and interfaces can be brought up and down. Even the arp table can be viewed with it so it rolls quite a bit of functionality into it.
Tracepath and traceroute do very similar things. The main difference though is that traceroute requires root while tracepath does not. This can be very advantageous when you are on a box and do not have root access. Tracepath is very MTU (Maximum Transmission Unit) conscious and tries to determine the MTU that works end to end. Sometimes, particularly over VPN tunnels, the MTU along the way can get clamped down a little from the default LAN 1500 to 1392 or lower. Tracepath attempts to detect this along the way which can be very useful.