Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

GRE or IP-in-IP tunnel is up but nothing passes: protocol 47 or 4, rp_filter and routing on Linux

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A Linux gre or ipip interface can show UP without the far end receiving anything. An interface that shows UP and passes nothing means packets stop somewhere between the two tunnel endpoints or just after they arrive. The fastest way to find where is to capture the outer protocol (47 for GRE, 4 for IP-in-IP) on the physical interface at both ends while you send test traffic.

The scope is Linux ip tunnel and ip link tunnels. MTU and TCP stalls are a different fault.

What the error means

There is no error message for this fault: the interface looks healthy and traffic does not arrive.

GRE (RFC 2784, March 2000) is carried directly in IP. The RFC says "The IPv4 protocol 47 [RFC1700] is used when GRE packets are encapsulated in IPv4." The GRE header itself holds a Protocol Type field, an optional checksum and a version, and nothing in it tells the sender whether the far end is receiving. IP-in-IP (RFC 2003, October 1996) sets the outer IP Protocol field to 4. Neither uses TCP or UDP, so there are no port numbers. A rule that allows "UDP 500" or "TCP 443" does not let either through, and a firewall must match the IP protocol number.

The Linux ip-tunnel(8) page lists the IPv4 modes as ipip, sit, isatap, vti and gre. A tunnel is defined by a mode, a remote address and a local address, and the man page says local "must be an address on another interface of this host".

On receipt the kernel decapsulates the packet and treats the inner packet as arriving on the tunnel interface. That is where the Linux-specific causes sit: reverse-path filtering applies to the inner source address, and the routing table has to send the reply back into the tunnel.

Some routers add a keepalive. The Cisco SD-WAN note on troubleshooting GRE interfaces (updated 2 November 2021) shows Up Down with keepalives on and Up Up with them off, and says that transmitted packets with no rise in the receive counter "points to a remote issue". That note is specific to that platform. The ip-tunnel(8) page lists no keepalive option.

Likely causes

Cause What you see How to confirm
Protocol 47 or 4 dropped on the path (upstream firewall, cloud security group, provider) Outer packets leave host A; none arrive at host B Capture ip proto 47 (or 4) on both outer interfaces
Host firewall at either end drops the protocol Packets arrive on the outer interface but not on the tunnel interface Firewall counters; capture on both interfaces
Mode or address mismatch (gre against ipip, local and remote swapped or wrong) A sends protocol 47, B expects 4 (or the reverse); or packets arrive from an unexpected address Compare ip tunnel show on both ends and the protocol seen in the capture
Strict rp_filter drops the decapsulated packet Inner packets show on the tunnel interface with the request seen, no reply sent sysctl values and ip route get for the inner source
No route into the tunnel, or no return route Pings to the far tunnel address work; the far network does not ip route get and ip route show on both ends
Option mismatch (key, checksum, sequence) or a low fixed TTL Outer packets arrive, no inner packets appear ip -d link show on both ends; the ttl value
NAT on the path Works one way or intermittently Compare source address seen at B with host A's address

Prove the cause before changing anything

These read-only commands are taken from the man pages and vendor documents listed under Sources. They have not been run or lab-tested.

Example (illustrative): host gw-a (outer address 203.0.113.10, LAN 10.10.1.0/24) and host gw-b (outer address 198.51.100.20, LAN 10.20.0.0/24) have a GRE tunnel gre1, with tunnel addresses 172.31.255.1/30 and 172.31.255.2/30. The outer interface on both is eth0.

1. Read the tunnel definition on both ends.

ip tunnel show
ip -d link show gre1
ip -s link show gre1

ip tunnel show lists tunnels. The -d option outputs more detailed information and -s adds statistics. Check three things: the mode is the same on both ends; local on one end is remote on the other; and any key, checksum or sequence options are set the same on both. In the statistics, a TX count that rises on gw-a and an RX count that does not on gw-b places the fault between them.

2. Capture the outer protocol at both ends while you ping. Send a ping from gw-a's tunnel address to gw-b's tunnel address, and capture on each outer interface:

tcpdump -n -i eth0 'ip proto 47'

ip proto protocol is the pcap-filter primitive for an IPv4 packet of that protocol type, and the protocol can be a number. Use ip proto 4 for IP-in-IP. The -n option stops name lookups and -i selects the interface. Read the result like this:

  • Packets leave gw-a and none appear on gw-b's eth0: the packets are dropped, rewritten or routed away on the path. Look at the firewall, security group or provider between the endpoints.
  • Packets appear on gw-b's eth0 but the tunnel interface shows nothing: look at gw-b's firewall, then at mode, address or option mismatch.
  • Protocol 47 leaves gw-a but gw-b is built for protocol 4 (or the reverse): the modes differ.
  • Packets arrive on gw-b's eth0 and inner packets appear on gre1, but no reply leaves: the cause is rp_filter or routing at gw-b.

3. Capture on the tunnel interface.

tcpdump -n -i gre1

If you see the inner request arriving and no reply, the outer path is fine and the fault is on the receiving host.

4. Read the firewall counters. For iptables, the manual documents -L to list rules, -n for numeric output and -v to show interface names and the packet and byte counters:

iptables -L -n -v --line-numbers

For nftables, the wiki documents nft list ruleset. A rule that matches the IP protocol uses ip protocol 47 or meta l4proto 47, and iptables accepts -p as a number or a name from /etc/protocols. Look for a rule or a default policy that drops protocol 47 or 4 on the INPUT chain at the receiving host, and for counters that rise while you ping. The kernel also tracks GRE in connection tracking: the nf_conntrack-sysctl page lists nf_conntrack_gre_timeout (default 30 seconds) and nf_conntrack_gre_timeout_stream (default 180 seconds). The kernel page does not say how a ruleset treats the return direction, so judge that from the counters.

5. Check reverse-path filtering.

sysctl net.ipv4.conf.all.rp_filter
sysctl net.ipv4.conf.gre1.rp_filter

The kernel documentation for rp_filter says: 0 is no source validation, 1 is strict mode, 2 is loose mode. In strict mode "each incoming packet is tested against the FIB and if the interface is not the best reverse path the packet check will fail. By default failed packets are discarded." The maximum of the all and per-interface values is used, the default is 0, and "some distributions enable it in startup scripts".

Then ask the kernel which interface it would use to reach the inner source address:

ip route get 10.10.1.5

Run it on gw-b with an address from the far LAN. ip route get prints a single route "exactly as the kernel sees it". If the result names eth0 while the packets arrive on gre1, strict mode will discard them. If it names gre1, reverse-path filtering is not the cause.

6. Read the routes.

ip route show

Each end needs a route to the far LAN through the tunnel interface (or the far tunnel address). A route missing at either end breaks the reply path, even when the outer packets pass.

Fix

Protocol 47 or 4 is filtered on the path

Ask the owner of the filter to allow IP protocol 47 (GRE) or 4 (IP-in-IP) between the two outer addresses, in both directions. For a cloud security group or a firewall that lists protocols, choose the protocol by number rather than TCP or UDP. Where a NAT device sits on the path, the Cisco SD-WAN note says "Sometimes, because of network address translation (NAT), GRE Keepalives can be dropped".

Risk: a firewall change affects other traffic. Restrict the rule to the two outer addresses and to this protocol, and remove it with the same change record if it does not help.

Host firewall drops the protocol

On an iptables host, a narrow rule inserted before the drop is enough, for example on gw-b:

iptables -I INPUT 1 -p 47 -s 203.0.113.10 -d 198.51.100.20 -j ACCEPT

-I inserts at the given position, -p takes a protocol number, and -s, -d and -j select the source, destination and target. On nftables, add a rule with the matcher ip protocol 47 (or meta l4proto 47) and the accept statement in the input chain. Make the same change at the other end for the opposite direction. Use -p 4 for IP-in-IP.

Risk: a rule inserted at the wrong position can change what is allowed for other traffic, and a rule typed on a live host is lost on restart unless you save it the way your distribution does. Back out with iptables -D INPUT <line number>; the documented form is -D chain rulenum.

Mode, address or option mismatch

Make the two ends mirror images. ip tunnel change is the documented way to alter an existing tunnel, for example to correct remote. Match the mode, local, remote, ttl and any key, checksum or sequence options.

Disruptive step: changing or recreating a tunnel interrupts everything it carries and may drop the routes pointing at it. Do it over a path that does not use the tunnel, such as the console or a second link, and record the old settings from ip -d link show first so you can restore them.

Strict rp_filter drops the packet

The preferred fix is to correct the routing so the best path back to the inner source is the tunnel interface. If the design needs asymmetric paths, the kernel documentation says "If using asymmetric routing or other complicated routing, then loose mode is recommended." Set loose mode on the tunnel interface:

sysctl -w net.ipv4.conf.gre1.rp_filter=2

Remember that the larger of the all and per-interface values applies. If net.ipv4.conf.all.rp_filter is 1, setting the interface to 0 still leaves strict mode in force; setting the interface to 2 works because 2 is larger. To keep the change after restart, put the line in the sysctl configuration file your system loads (sysctl -p loads a named file).

Risk: loose mode weakens the spoofed-source protection that strict mode provides. Limit the change to the tunnel interface, and back out with sysctl -w net.ipv4.conf.gre1.rp_filter=1 if that was the previous value.

Route missing

ip route add 10.20.0.0/24 dev gre1

Run the mirrored command on the other host. Where the far side is a router, use via and its tunnel address.

Risk: a wrong prefix can send unrelated traffic into the tunnel. Remove it with ip route del and the same arguments.

Confirm the fix

  1. Ping the far tunnel address from each end. The capture on eth0 at both ends now shows protocol 47 (or 4) in both directions.
  2. Capture on gre1 at the receiving host and see both the request and the reply.
  3. Ping a host on the far LAN from a host on the near LAN.
  4. ip -s link show gre1 shows RX and TX counters rising together.
  5. For a router with keepalives, the interface state reads Up Up with keepalives on.

What to watch afterwards

  • Keep the firewall rule for protocol 47 or 4 in the permanent ruleset, not only in the running one.
  • Alert on a tunnel interface whose RX counter has stopped rising while TX still rises. That is the signature the Cisco note describes, and it applies to any tunnel.
  • If large transfers stall while small pings work, suspect MTU rather than the causes above.

Risks and limits

  • The man pages contain no sample output, so what to look for is described in words, apart from the Cisco SD-WAN samples that apply only to that platform.
  • The ip-tunnel(8) page lists no keepalive option. A keepalive state on a router is vendor behaviour and differs by platform.
  • The netfilter documentation cited here gives the matching syntax and the connection-tracking timeouts. It does not state how a particular ruleset treats the return direction of GRE, so use the counters, not an assumption.
  • No commands are given for Cisco IOS XE, Junos or MikroTik; use that vendor's documentation for the router end.

Sources