Ibm com developerworks linux

Learn Linux 101: Basic network troubleshooting

When things go wrong

By Ian Shields
Updated May 26, 2021 | Published May 7, 2020

Overview

In this tutorial, learn to troubleshoot networking issues on your Linux client system. Learn to:

• Manually configure network interfaces, including viewing and changing the configuration of network interfaces using iproute2.
• Manually configure routing, including viewing and changing routing tables and setting the default route using iproute2.
• Debug problems associated with the network configuration.
• Recognize the legacy net-tools commands.

Networking in Linux

In today’s world, computer networking enables information sharing, research, and commerce across the country or across the world. Research Kenyan national parks. Sure. Buy a Swiss train ticket. No problem. Email someone down the street or on another continent. See photos from outer space. Computer networking makes all this and much more possible. But what happens when things don’t work?

About this series

This series of tutorials helps you learn Linux system administration tasks. You can also use the material in these tutorials to prepare for the Linux Professional Institute’s LPIC-1: Linux Server Professional Certification exams.

See “Learn Linux, 101: A roadmap for LPIC-1” for a description of and link to each tutorial in this series. The roadmap is in progress. This tutorial reflects the Version 5.0 objectives as updated on October 29, 2018. As tutorials are completed, they will be added to the roadmap.

This tutorial helps you prepare for Objective 109.3 in Topic 109 of the Linux Administrator (LPIC-1) exam 101. The objective has a weight of 4. This tutorial reflects the Version 5.0 objectives as updated on October 29, 2018.

Prerequisites

To get the most from the tutorials in this series, you need a basic knowledge of Linux and a working Linux system on which you can practice the commands that are covered in this tutorial. For this tutorial, you will also need one or more network connections. Sometimes, different versions of a program format output differently. So, your results might not always look exactly like the listings and figures that are shown here.

The examples in this tutorial come from Fedora 31, and Ubuntu 18.04.3 LTS.

Networking tool sets

For many users, networking configuration just happens automatically when you plug in a network cable or provide some kind of login credentials to a wifi network. In our tutorial, “Learn Linux 101: Persistent network configuration“, I showed you basic network configuration using graphical tools when you need to do some manual configuration. In this tutorial, I will show you the command line tools that you use for these tasks and for diagnosing network problems. For the LPI certification, you need to use and understand the iproute2 set of networking commands. These were developed to better integrate with the 2.2 kernel and have been around for a long time now. The older net-tools suite is still in use as many older Linux people know them and have them ingrained. Despite their modern shortcomings they still perform many useful functions so you will probably work with people who don’t know the newer tools. As early as 2009, there was a discussion about either removing net-tools altogether or not including it with distributions. This discussion is ongoing.

On some systems, such as Fedora, the iproute2 tools are installed from the iproute (no 2) package, while on Ubuntu the package is called iproute2. Listing 1 shows how to check which packages are installed or available using the dnf command on Fedora or the apt command on Ubuntu.

Listing 1. Checking for networking tools packages

The iproute or iproute2 packages install a number of binaries which you can list using the dnf, rpm, or dpkg tools. Listing 2 shows how to list the installed binaries on Ubuntu.

Listing 2. Binary files installed by iproute2

This tutorial focuses on the ip command for configuration of interfaces and routes. Other tools that you learn about in this tutorial come from the following packages:

• hostname
• traceroute or inetutils-traceroute
• iputils or iputils-ping or iputils-tracepath
• nmap-ncat or netcat-openbsd

The names and contents of these packages differ between rpm and deb based systems. So, this is just a general guide to where you might look for these commands.

Introduction to using the ip command

Before I start changing things using ip , I’ll show you how to view information about your existing network.

The ip command has the following general form:

ip [ OPTIONS ] OBJECT

Invoked without options, you will get brief help for the command as shown in Listing 3.

Listing 3. Help for the ip command

Otherwise, you need to specify some networking object to operate on. Examples include link, address, route, or neighbor. Most objects can be written in full form or in an abbreviated form. If you are in doubt about an abbreviation, the shortest unambiguous form usually works.

The command is the action to be performed on the object. Commands vary somewhat by object type. You can usually add, delete, and show (or list) objects but not all objects support all of these commands. The help for an object tells you the supported commands. For example, use ip link help or ip neigh help to get help for links or neighbors. Listing 4 shows the help for neighbors, with n being a sufficient abbreviation for neigh . Note that neighbor or neighbour will also work.

Listing 4. Help for the ip neighbor object

If you invoke ip OBJECT for some object, you will get a summary of information for the available objects of that type. Continuing with the neighbor example you might see something as shown in Listing 5.

Listing 5. Display neighbors using ip command

This information comes from the Neighbor Discovery Cache (NDC). Reachable locations become stale if there is no traffic for 30 seconds.

Detailed man pages for the ip command

You can find more detailed help for using the ip command with various objects in the man or info pages using a command such as man ip-neighbour or info ip-link . The available pages are:

• ip-address
• ip-addrlabel
• ip-l2tp
• ip-link
• ip-mad‐dress
• ip-monitor
• ip-mroute
• ip-neighbour
• ip-netns
• ip-ntable
• ip-route
• ip-rule
• ip-tcp_metrics
• ip-token
• ip-tunnel

Note that abbreviations and alternate spellings such as neighbor for neighbour are not supported for displaying these extended manual pages and also there are no individual commands with these names.

Viewing interfaces, addresses, and routes using the ip command

In order to transmit packets over an interface or link, you need a link, at least one IP address associated with the link, and a routing table that tells you which link to transmit packets over to a particular destination. This is a simplistic view of basic networking, but it will serve for learning more about the ip command. Listing 6 shows a basic example from my Ubuntu system.

Listing 6. Displaying link information using the ip command

This system has a loopback link (lo) and two Ethernet links (enp2s0 and enp4s0). The local loopback link and the link enp2s0 are both up while the link enp4s0 is down (does not show as being UP). The status of LOWER_UP indicates that the underlying media, such as an Ethernet cable, is connected.

Notice that this information does not show any traffic statistics for the links. The ip command has a number of options, including -s (or -stats , or -statistics ) to display statistics, or -br (or -brief ) to display only basic information. You can also restrict information to IP version 6 ( -6 option) or IP version 4 ( -4 option). These options may not apply to all objects. Find out more about the available options using the man or info pages ( man ip or info ip ). Listing 7 shows two examples of these options.

Listing 7. Using options with the ip command

The link information does not tell you whether there is an IPv4 or IPv6 address associated with any of the links.

Listing 8 shows how to use the ip command with the addr object to see the IP addresses associated with your links. Notice that link enp4s0 does not have any IP address associated with it. This link is currently configured to acquire addresses using Dynamic Host Configuration Protocol (DHCP).

Listing 8. Using ip addr

As with other forms of the ip command you can use options to change output. For example, -br for brief output or -4 or -6 to limit output to IPv4 or IPv6 respectively. Listing 9 shows two examples.

Listing 9. Limiting ip addr output

So far you have seen local loopback and Ethernet links. There are several other link types you may see, including wifi and bridge links. Listing 10 shows a brief IPv4 output for a Fedora system with a loopback address, an Ethernet link, a wifi link, and a bridge link for the benefit of possible virtual machines on my system.

Listing 10. Additional link types

Another object you need to get traffic in and out of your system is a route. Use ip with the route object to display routes. If you do not see any v6 information, run the command again with the -6 option as shown in Listing 11.

Listing 11. Displaying route information using ip route

Looking at the first line of output you see a default IPv4 route, which will be used for all version 4 traffic unless a more specific route is found. Traffic will be sent and received over the Ethernet link, enp2s0. The proto field indicates that this route was set up when the address was assigned from a DHCP server. Finally, the metric is a measure that is used to give one route preference over alternate routes. The lower the metric the more the route is preferred.

You also see an entry for the IPv4 subnet 192.168.1.0/24. This also uses the Ethernet link enp2s0. The route was added by the kernel during initialization, and traffic sent over this route will have a source IP address of this host, namely 192.168.1.24.

This example was a fairly simple one with one active network link. On my Fedora system with both an Ethernet and a wifi link active, the route information is a little more complex. The version 4 information is shown in Listing 12.

Listing 12. Displaying route information with multiple links

In this case, there are default routes over both Ethernet (enp9s0) and wifi (wlp8s0u2). The Ethernet link has a metric of 100 while the wifi link has a higher metric of 600. So, the Ethernet link is preferred if nothing else overrides the routing decision. The wifi link is on a guest network (subnet 192.168.3.0/24) that has limited access to resources on the main network (subnet 192.168.3.0/24). Traffic destined for the 192.168.3.0/24 subnet will be sent over the wifi adapter with a source address of 192.168.3.21 while traffic for the 192.168.3.0/24 subnet as well as less specific traffic will use the Ethernet adapter and have the source address as 192.168.1.25.

Given the above routing information, you might wonder why traffic from 192.168.1.25 to itself would appear to be routed out through the Ethernet adapter rather than just being looped inside the host. And a similar question for 192.168.3.21 and the wifi adapter. And how is traffic to 127.0.0.1 routed? The answer is that the kernel maintains a local routing table for high-priority routing of broadcast and loopback traffic. Use the ip get command to see more details about a particular route. Add oif for output interface if you want to check the route over a specific link. Several illustrations are in Listing 13.

Listing 13. Using ip with the get commands

This quick introduction to displaying link, address, and route information does not cover all of the many options of the ip command that are available. Refer back to the list of more detailed man pages that I listed in the Detailed man pages for the ip command section.

Activate or deactivate interfaces using the ip command

Before you start configuring interfaces, it’s handy to know how to activate or deactivate them. Making changes to an already active interface is generally not a good idea! Listing 14 shows how to deactivate an interface and then reactivate it using the ip command to set the interface up or down . You need administrative authority to change the status of devices such as links.

Listing 14. Deactivating and activating an interface using ip

Saving and restoring interfaces and routes

It is also a good idea to know how to save and possibly restore the state of your network and routes. The ip command can save current information to a binary file using the save option. Use showdump to show what is in such a binary file and restore to restore it. Needless to say, you will need administrator privilege for restore operations. Listing 15 shows how to dump information for an Ethernet link and for the route information, and how to show the contents of the saved binary files.

Listing 15. Saving and restoring using ip

Configure network interfaces and routes

You can use the ip command to configure links, addresses, or routes. Changes made using the ip command are not persistent. So, you will need to make the changes after each boot or else use persistent network configuration tools as described in our tutorial, “Learn Linux 101: Persistent network configuration“. Using ip is a great way to experiment and check out your planned changes.

The ip command has a limited amount of configuration ability with physical links such as Ethernet or wifi links. You can alter flags such as multicast for example and you can add or delete IP addresses. However, you can use ip to configure a large variety of virtual links, such as bridges, VLANs, or channel bonds. You can use ip link help to find the various link types, and add a type to find the parameters applicable to a particular link type. Listing 16 shows the end of the output for ip link help and the output for ip help bond.

Listing 16. Parameters applicable to a bond link

Now I will show you how to add an IP address to an existing link and how to update the routing table. For simplicity, I will restrict the discussion to IPv4 addresses, but the same general techniques work for IPv6 as well.

Recall that my Fedora system has a wifi adapter with an associated IP address of 192.168.3.21. The IPv4 address and route information is shown in Listing 17.

Listing 17. IPv4 address and route information for link wlp8s0u2

Suppose I want to change the address from 192.168.3.21 to 192.168.3.30 and also change the metric to 550. It is not possible to change the metric using the IP command. So, one way to do this is to first add the new address and new metric and then delete the original address.

First, I’ll add an address and then see what I accomplished as shown in Listing 18. Note that you need administration authority to add an address or do several of the other things I will do.

Listing 18. Using ip to add an IP address

So far, so good, but I forgot to add a broadcast address and a metric. Listing 19 shows how to delete the address that I just added and add it again with a broadcast address and a metric. Note that brd is an adequate abbreviation for broadcast and the + tells ip to derive the broadcast address from the IP address by setting or resetting the host bits of the interface prefix. You can use either + or — for this purpose or you can spell out the broadcast address in full dotted-quad format.

Listing 19. Delete an address and add a broadcast addresses using ip

Note that the new address has been added as a secondary address. There is no way to swap a secondary address for a primary address. Depending on the settings of certain sysctl variables secondary IPv4 addresses may be deleted or promoted when the primary address is deleted. This is not a problem for IPv6 addresses where secondaries are promoted if the primary is deleted. The default values may vary by Linux distribution. On the Ubuntu system, I am using the sysctl value related to promoting secondaries are shown in Listing 20.

Listing 20. Sysctl variables for promoting secondary IPv4 addresses

Before I delete the original 192.168.3.21 address, I’ll check the existing routes to help me configure the necessary routes for the new address correctly. Two of my IPv4 route entries involve the wlp8s0u2 wifi adapter as shown in Listing 21. So, I will need routes like these each with metric 550 after I delete the 192.168.3.21 address.

Listing 21. IPv4 routes

Now I will delete 192.168.3.21 and show you how the address I added is promoted to secondary. In Listing 22, you see that the new address was promoted to primary and a link scope route was automatically added with metric 550, derived from the value I specified when I added the address.

Listing 22. Deleting the primary IPv4 addresses

These changes do not update the default route over the wlp8s0u2 which still has the original metric of 600. You cannot change the metric on a route using the ip command. You have to delete the route and add it again. The addition does not inherit the metric value from the address so you need to specify it explicitly as shown in Listing 23.

Listing 23. Updating the default route

So now I have accomplished the changes I set out to illustrate. Remember that changes made using the ip command are transient and will be lost at the next reboot.

Debugging network configuration problems

This section explains how to debug network configuration problems.

Hostname

You may want to verify your hostname value as part of your debugging. While the hostname command is listed in the LPI objectives for this tutorial I’ll simply refer you to our tutorial “Learn Linux 101: Persistent network configuration“, where it and the associated hostnamectl command are both covered.

Link connectivity

I already showed you how to use the ip command to display the link status. Assuming the status shows that the link is up, you might need to verify that you have connectivity over the link. My first check usually involves the ping command which sends an ICMP ECHO request to another host, or even to your own host. Use either the -4 or -6 option to force using either IPv4 or IPv6 if you are looking to check a particular connection type. On most systems, ping6 is an alternate command equivalent to ping -6 . Use the -c option to limit the number of echo requests that are sent, otherwise ping will continue transmitting packets until you terminate it using Ctrl+C. Recall that the normal kernel routing will send traffic destined to a host address on your own system will use the local loopback link. Listing 24 shows some examples of this.

Listing 24. Using ping on the local loopback link

Other, perhaps more common, uses for ping are to check connectivity to a gateway, to another device on your corporate or local network, to a public DNS server or to an arbitrary host on the Internet. In general, start with the local interface and then move further away. Listing 25 shows how to ping my IPv4 gateway, a neighbor on my local network which has both version 4 and version 6 entries in my /etc/hosts file, and the Google public DNS server (8.8.8.8). I have used the -q (for quiet) option to just produce summary output.

Listing 25. Reaching out with ping

Looking at these examples, it is not evident which interface was used to transmit and receive the ping data. Listing 26 shows how to use the -I (for interface) option with ping to force use of my wifi interface (wlp8s0u2). Note how the attempt to ping attic4 without specifying -4 or -6 results in a message saying “Network is unreachable”. In this case, it means that my wifi adapter is not set up to handle version 6 traffic and that’s what ping was trying to use. If you see this, you may want to add -4 or -6 to check whether you may have one or the other kind of connectivity only.

Listing 26. Pinging over a specific interface

There are a number of other options that you can use to control the way ping works. Be aware that the options that you select may affect your ability to get a response.

Routes and paths

If you can’t reach a host using ping , you might try either traceroute or tracepath . Both are designed to probe a possible path to a destination. They have some similarities, but tracepath is newer and designed to not require privileged authority and also tracepath does not have as many options. Note that not all traceroute options require privilege, but some do. As with ping , each has a -4 or -6 option for IPv4 or IPv6. Some systems also have traceroute6 and tracepath6 with the obvious meanings. You can specify an interface with traceroute using the -i (not lower case as compared to ping) option, but this is not possible with tracepath . Both commands have a -m option to limit the number of hops that will be tried. By default, tracepath does not show IP addresses, use the -n option for just numeric addresses or the -b option to see both. Listing 27 shows some basic usage of tracepath .

Listing 27. Using tracepath to nearby nodes

Listing 28 shows similar information from tracepath . In this case, I also use the -i option to check the path over the wifi adapter rather than the default Ethernet adapter.

Listing 28. Using traceroute to nearby nodes

When you try to probe further, either of these commands may fail to find a path. Use the -M option of traceroute to try different methods. Traditionally these are udp (default) or icmp (ICMP). Because either or both of these may be blocked by firewalls, a newer option, tcp , is available on some implementations. You can also use -I instead of -M icmp , and -T instead of -M tcp .

Listing 29 shows a comparison of tracepath and traceroute options to find a path from my system to lpi.org. In this case, only tracepath with the -T option succeeded.

Listing 29. Comparing tracepath and traceroute

You can find more about other available options for both these commands in the man or info pages.

As you saw above, it can be challenging to verify a path from your host to an arbitrary Internet host. Usually these tools work well in parts of the network under your or your organization’s control. You also saw that using Transmission Control Protocol (TCP) for probes did work in one case where both User Datagram Protocol (UDP) and Internet Control Message Protocol (ICMP) failed. Using TCP effectively attempts to start a conversation that is not typically blocked, such as a browser or perhaps a Secure Shell (SSH) session. If there is no listener on the port, the session is rejected by the target host. If there is a listener, your host receives a positive session-setup response and immediately cancels the session before normal data transfer begins.

Sockets and ss

The ss command is part of the iproute2 package. Use it to dump socket statistics. If used without parameters, it dumps a list of open non-listening sockets (for example, TCP/UNIX/UDP) that have established connections. This can be quite long. You can use grep for simple filtering, in which case it is useful to add the -o (one line) option to get multiline output on a single line. Use the -t or -u options for TCP or UDP sockets respectively and the -l option for listening sockets. The -s option shows you a short summary of sockets. Listing 30 shows a few examples.

Listing 30. Using the ss command

The ss command also has the ability to construct quite elaborate state filters as part of the command. Listing 31 shows how to filter established TCP sockets using the SSH port, either as a source port or a destination port. It shows an example of four SSH sessions, two originating at my Ubuntu host (192.168.1.24) to my Fedora host (192.168.1.25 and 192.168.3.30) and two in the reverse direction, one V4 and one v6.

Listing 31. Using ss state filters

Note that the man page for ss says “Please take a look at the official documentation for details regarding filters.” Such details do not seem to exist in the official documentation packages as of the time of writing, although the examples in the man page are a very useful start.

Another socket tool that you can use is netcat . There are at least two versions and the name is sometimes nc or ncat . This is somewhat similar to the cat command in that you can use it to transmit text, command output, or files to another host. It operates in connect or listen mode. The listener does not have to be another netcat instance. In Listing 32, I have shown how to use ncat on my Fedora system to connect to the IBM web server on port 80. I typed in two lines to request the HTTP headers and you see the beginning of the response.

Listing 32. Using ncat (netcat) to retrieve HTTP headers

You can find more information about the various modes and options in the man pages or in examples online.

Legacy net-tools commands

Before iproute2 there was net-tools. The net-tools package is still in use, a testament to its usefulness. Table 1 shows some of the commands I have used in this tutorial and the legacy command you could use instead. Use the man pages to explore other options of these commands.

Table 1. Comparison of new and legacy commands

New command	Legacy command
ip a	ifconfig -a
ip link set enp9s0 down	ifconfig enp9s0 down
ip link set enp9s0 up	ifconfig enp9s0 up
ip addr add 192.168.3.30/24	ifconfig add 192.168.3.30 dev wlp8s0u2 ifconfig wlp8s0u2 netmask 255.255.255.0
ip r	route
ip route add default via 192.168.3.1	route add default gw 192.168.3.1
ip neigh	arp -a
ss	netstat

Conclusion

This concludes your introduction to Topic 109.3: Basic network troubleshooting.

Источник