Linux bridge no arp

Proper isolation of a Linux bridge

Vincent Bernat April 12, 2017

When configuring a Linux bridge, use the following commands to enforce isolation:

A network bridge (also commonly called a “switch”) brings several Ethernet segments together. It is a common element in most infrastructures. Linux provides its own implementation.

A typical use of a Linux bridge is shown below. The hypervisor is running three virtual hosts. Each virtual host is attached to the br0 bridge (represented by the horizontal segment). The hypervisor has two physical network interfaces:

• eth0 is attached to a public network providing various services for the virtual hosts (DHCP, DNS, NTP, routers to Internet, …). It is also part of the br0 bridge.
• eth1 is attached to an infrastructure network providing various services to the hypervisor (DNS, NTP, configuration management, routers to Internet, …). It is not part of the br0 bridge.

The main expectation of such a setup is that while the virtual hosts should be able to use resources from the public network, they should not be able to access resources from the infrastructure network (including resources hosted on the hypervisor itself, like a SSH server). In other words, we expect total isolation between the green domain and the purple one.

That’s not the case. From any virtual host:

There are two main factors of this behavior:

1. A bridge can accept IP traffic. This is a useful feature if you want Linux to act as a bridge and provide some IP services to bridge users (a DHCP relay or a default gateway). This is usually done by configuring the IP address on the bridge device: ip addr add 192.0.2.2/25 dev br0 .
2. An interface doesn’t need an IP address to process incoming IP traffic. Additionally, by default, Linux accepts to answer ARP requests independently from the incoming interface.

Bridge processing#

After turning an incoming Ethernet frame into a socket buffer, the network driver transfers the buffer to the netif_receive_skb() function. The following actions are executed:

1. evaluate the XDP receive hooks,
2. copy the frame to any registered global or per-device taps (e.g. tcpdump ),
3. evaluate the ingress policy (configured with tc ),
4. hand over the frame to the device-specific receive handler, if any,
5. hand over the frame to a global or device-specific protocol handler (e.g. IPv4, ARP , IPv6).

1

A frame can be forcibly routed (L3) instead of bridged (L2) by “brouting” the packet. This action can be triggered using ebtables .

For a bridged interface, the kernel has configured a device-specific receive handler, br_handle_frame() . This function won’t allow any additional processing in the context of the incoming interface, except for STP and LLDP frames, or if “brouting” is enabled. 1 Therefore, the protocol handlers are never executed in this case.

After a few additional checks, Linux will decide if the frame has to be locally delivered:

In this case, the frame is passed to the br_pass_frame_up() function. A VLAN-related check is optionally performed. The socket buffer is attached to the bridge interface ( br0 ) instead of the physical interface ( eth0 ). It is evaluated by Netfilter and sent back to netif_receive_skb() . It will go through the four steps a second time.

IPv4 processing#

When a device doesn’t have a protocol-independent receive handler, a protocol-specific handler will be used:

Therefore, if the Ethernet type of the incoming frame is 0x800 , the socket buffer is handled by ip_rcv() . Among other things, the three following steps will happen:

• If the frame destination address is not the MAC address of the incoming interface, not a multicast one, and not a broadcast one, the frame is dropped (“not for us”).
• Netfilter gets a chance to evaluate the packet (in a PREROUTING chain).
• The routing subsystem will decide the destination of the packet in ip_route_input_slow() : is it a local packet, should it be forwarded, should it be dropped, should it be encapsulated? Notably, the reverse-path filtering is done during this evaluation in fib_validate_source() .

Reverse-path filtering (also known as uRPF, or unicast reverse-path forwarding, RFC 3704) enables Linux to reject traffic on interfaces which it should never have originated: the source address is looked up in the routing tables and if the outgoing interface is different from the current incoming one, the packet is rejected.

ARP processing#

When the Ethernet type of the incoming frame is 0x806 , the socket buffer is handled by arp_rcv() .

• As with IPv4, if the frame is not for us, it is dropped.
• If the incoming device has the NOARP flag, the frame is dropped.
• Netfilter gets a chance to evaluate the packet (configuration is done with arptables ).
• For an ARP request, the values of arp_ignore and arp_filter may trigger a drop of the packet.

IPv6 processing#

When the Ethernet type of the incoming frame is 0x86dd , the socket buffer is handled by ipv6_rcv() .

For IPv6, reverse-path filtering needs to be implemented with Netfilter, using the rpfilter match.

• As with IPv4, if the frame is not for us, it is dropped.
• If IPv6 is disabled on the interface, the packet is dropped.
• Netfilter gets a chance to evaluate the packet (in a PREROUTING chain).
• The routing subsystem will decide the destination of the packet. However, unlike IPv4, there is no reverse-path filtering. 2

Workarounds#

There are various methods to fix the situation.

We can completely ignore the bridged interfaces: as long as they are attached to the bridge, they cannot process any upper layer protocol (IPv4, IPv6, ARP ). Therefore, we can focus on filtering incoming traffic from br0 .

Protocol-independent workarounds#

The four following fixes will indistinctly drop IPv4, ARP , and IPv6 packets.

Using VLAN-aware bridge#

With a kernel older than Linux 4.3, you’ll have to use the following command instead of ip link :

Linux 3.9 introduced the ability to use VLAN filtering on bridge ports. This can be used to prevent any local traffic: 3

This is the most efficient method since the frame is dropped directly in br_pass_frame_up() .

Using XDP receive hooks#

It’s also possible to drop the bridged frame early after it has been re-delivered to netif_receive_skb() by br_pass_frame_up() . The XDP hooks attached to an interface are evaluated before taps, ingress policy, and handlers. A very simple xdp_drop_all.c program would look like this:

It can be compiled and installed with:

The above program is so simple it could be written without a compiler:

However, ip link requires an ELF object.

This requires Linux 4.14 and this is the second most efficient method but a bit inconvenient as it needs a compiler. 4

Using ingress policy#

The ingress policy of an interface is evaluated just a bit later but before any handler. Therefore, the following commands will ensure no local delivery happens:

This is the third most efficient method but is more convenient than the previous one for a very tiny additional overhead.

Since Linux 4.2, it is possible to use Netfilter to setup the ingress policy with the same efficiency. This can be done through nftables:

Using ebtables #

Just before re-delivering the frame to netif_receive_skb() , Netfilter gets a chance to issue a decision. It’s easy to configure it to drop the frame:

However, to the best of my knowledge, this part of Netfilter is known to be inefficient.

Using namespaces#

Isolation can also be obtained by moving all the bridged interfaces into a dedicated network namespace and configure the bridge inside this namespace:

The frame will still wander a bit inside the IP stack, wasting some CPU cycles and increasing the possible attack surface. But ultimately, it will be dropped.

Protocol-dependent workarounds#

Unless you require multiple layers of security, if one of the previous workarounds is already applied, there is no need to apply one of the protocol-dependent fix below. It’s still interesting to know them because it is not uncommon to already have them in place.

arptables can also drop the packet quite early:

Because the check on the target MAC address is quite loose, they don’t even need to guess the MAC address:

If the br_netfilter module is loaded, net.bridge.bridge-nf-call-ipatbles sysctl has to be set to 0. Otherwise, you also need to use the physdev match to not drop IPv4 packets going through the bridge.

The earliest place to drop an IPv4 packet is with Netfilter: 5

If Netfilter is disabled, another possibility is to enable strict reverse-path filtering for the interface. In this case, since there is no IP address configured on the interface, the packet will be dropped during the route lookup:

Another option is the use of a dedicated routing rule. Compared to the reverse-path filtering option, the packet will be dropped a bit earlier, still during the route lookup.

Linux provides a way to completely disable IPv6 on a given interface. The packet will be dropped as the very first step of the IPv6 handler:

As with IPv4, it’s possible to use Netfilter or a dedicated routing rule.

About the example#

If you didn’t get a reply, you could still have issues with IP processing. Add a static neighbor entry before checking the next step:

To check if IP processing is enabled, check the bridge host’s network statistics:

If the counters are increasing, it is processing incoming IP packets.

About MACVLAN interfaces#

When source-address learning is not needed, bridges can be replaced by MACVLAN or MACVTAP interfaces. Have a look at the following article for more details: “Bridge vs Macvlan.”

In this case, the packet processing is different and asymmetric. When a frame exits the MACVLAN interface, it is directly transmitted to the appropriate interface (either another MACVLAN interface or the lower device). In this direction, communication with the host is not possible.

However, when a frame enters the lower device, if the destination MAC is not one of the sub-interfaces, the frame is processed by the host using the appropriate protocol handler (ARP , IPv4, IPv6). Therefore, isolation can only be obtained by using the protocol-dependent methods.

A frame can be forcibly routed (L3) instead of bridged (L2) by “brouting” the packet. This action can be triggered using ebtables . ↩︎

For IPv6, reverse-path filtering needs to be implemented with Netfilter, using the rpfilter match. ↩︎

With a kernel older than Linux 4.3, you’ll have to use the following command instead of ip link :

The above program is so simple it could be written without a compiler:

However, ip link requires an ELF object. ↩︎

Источник

• BridgeNetworkConnectionsProxyArp

Introduction

This document describes a means to bridge IP traffic between two interfaces using Proxy ARP, /32 host routes and standard Linux routing/forwarding. For a Layer 2 solution — an ethernet bridge — refer to BridgeNetworkConnections.

The benefit of using an IP / Layer 3 solution is that it will work when the outward-facing interface is a wireless ethernet client, without using WDS and without resorting to NAT. Simple Layer 2 bridging does not work in this case due to the vagaries of wireless AP client behaviour; WDS may help but AP support can be patchy.

Using Proxy ARP permits the bridged clients to be part of the existing network and supports bidirectional traffic, e.g. for a server or printer. DHCP and mDNS will also work using the appropriate helpers.

One scenario is connecting a wired network to a wireless LAN using a host that has both wifi (wlan0) and ethernet (eth0) interfaces. A specific example is using a Raspberry Pi with a USB wifi adapter to connect a wired-ethernet printer to the WLAN. No static configuration is required other than the wifi SSID and authentication for the Pi — the Pi and printer acquire DHCP addresses from the existing DHCP server, and the printer continues to be reachable via mDNS and otherwise operates as if it were patched to a wired port on the main network.

The term ‘bridge’ in this document refers to the host doing the proxy-arp and routing between the two networks, though keep in mind there is no layer 2 bridging involved. The ‘outside’ network is the existing network that hosts the DHCP server, gateway router, etc. The ‘inside’ network is the one with the hosts that need to be bridged and made to appear to be on the outside network.

Summary

Proxy ARP is a technique by which a device on a given network answers the ARP queries for a network address that is not on that network, that is to make the hosts on one network appear to be logically part of a different physical network.

The bridge host will proxy ARP requests from the inside network to the outside, and respond to ARPs from the outside network on behalf of inside hosts. Linux will only do this for hosts that are known via the routing table, so a /32 host route must be created pointing to the inside host (one for each inside host). The route is also required for IP forwarding to work, i.e. when IP traffic arrives after the ARP process has completed.

As an example, to manually configure and test this out where the primary LAN has a network address of 10.42.0.0/24:

1. configure an inside client with a static IP of 10.42.0.11/24
2. on the bridge:
3. ping from the inside host to an outside host, and examine the ARP table:

Note that no IP address is required on the bridge’s inside ethernet interface for proxy ARP to work (though see below re. DHCP relay).

If you run tcpdump on the bridge’s ethernet and wlan interfaces, you’ll see the ARP request from the inside host being proxied to the outside interface, with the ARP source being the bridge’s outside-facing interface’s MAC address. The ARP table on the inside hosts will show the bridge’s inside interface MAC for all outside hosts, and similarly for outside hosts the MAC for all inside hosts will be the bridge’s outside interface MAC.

DHCP Relay

DHCP is a Layer 2 protocol and can’t traverse the Layer 3 ‘bridge’ so we use the dhcp-helper utility to listen for DHCP requests from the inside network and relay them to the DHCP server on the outside network. The relayed DHCP request will contain the IP address of the bridge’s inside interface in order for the DHCP server to know which network to allocate an IP address for the new client from. We want the DHCP allocation to be from the same network as the outside LAN, the simplest implementation is to reflect the outside interface’s IP address on to the inside interface. i.e. the bridge’s inside and outside interfaces will have the same IP address; though the inside interface’s network will be a host /32 netmask so as not to upset routing.

Alternatively the bridge inside interface can be assigned a static IP from the outside network, in which case the post-up step in /etc/network/interfaces configuration below can be omitted.

Automating the Process

parprouted is designed to monitor the ARP table and both proxy ARP requests and install matching /32 host routes. Running parprouted with the inside and outside interfaces handles the ARP and routing completely automatically. Note that the kernel’s proxy ARP mechanism (/proc/sys/net/ipv4/conf/all/proxy_arp) is not required.

parprouted does not handle packet forwarding nor DHCP or mDNS, so these features need to be enabled separately.

Installing the software

The following assumes wlan0 is the already-configured outside interface, eth0 the unconfigured inside interface. Tested on Debian Wheezy 7.8 on 2015-08-02, on armv6l (Raspberry Pi v1).

Edit /etc/network/interfaces to configure the interfaces:

Edit /etc/sysctl.d/local.conf to enable IP forwarding:

Enable DHCP relay: /etc/default/dhcp-helper

Edit /etc/avahi/avahi-daemon.conf to enable mDNS relaying:

Reboot, and hosts connected to the bridge’s ethernet should acquire a DHCP address and have full IP connectivity!

Known Issue with RTL8188CUS USB Wifi Adapter

Description of issue: When using an RTL8188CUS based wifi device, bridge connectivity is very unreliable and most ARP requests to the bridge’s wlan address fail. This issue has minimal impact on a host that is configured as a normal client and primarily sending traffic but has a significant impact when the host is acting as a bridge.

Источник