About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.
If new connections work after a firewall cluster fails over, but connections that were already open hang or reset, the standby unit most likely took over without the session table of the unit that failed. A stateful firewall only passes mid-stream packets for sessions it knows about. Either synchronisation was off, it was not working, or the entries were never loaded into the kernel on takeover.
No vendor documents a single error line for this fault. You see it as client-side resets and timeouts on SSH, database, RDP and file-transfer sessions at the moment of the failover. The sections below cover a Linux conntrackd pair, a FortiGate cluster and a Palo Alto pair.
What the error means
A stateful firewall keeps a table of the connections it has seen and decides what to do with later packets by looking them up. When the active unit fails, the standby becomes active. Whether it knows the existing connections depends on whether the two units were replicating that table.
The three vendors covered here describe the same idea in different words:
- Linux conntrackd. The conntrack-tools manual describes
conntrackdas the daemon that can "synchronize the connection tracking state table between several firewalls in High Availability (HA) scenarios". Each node keeps an internal cache (what it is filtering) and an external cache (what the other node is filtering). On takeover the external cache is committed into the kernel table. - FortiGate. Fortinet documents that "when session-pickup is enabled, the FGCP synchronizes the primary unit's TCP session table to all cluster units". If it is left disabled, "the cluster doesn't track sessions, and active sessions must be restarted or resumed after a failover."
- Palo Alto Networks. HA synchronises sessions over the HA2 link, with an exception: PAN-OS documents that "High availability (HA) isn't supported for decrypted sessions", so decrypted SSL Forward Proxy, SSL Inbound Inspection and SSH Proxy sessions are not resumed after a failover.
Why a missing entry kills a session rather than just slowing it is not spelled out by a vendor for failover. The conntrack-tools manual gives the closest statement, when it explains deleting an entry by hand with conntrack -D: this "allows you to block TCP traffic if" you have "a stateful rule-set that drops traffic in INVALID state" or you set /proc/sys/net/netfilter/nf_conntrack_tcp_loose to zero. A standby that never learned the session is in the same position as one where the entry was deleted. That is an inference from the manual, not a documented failover case.
Even with synchronisation working, nobody promises every session survives. Fortinet says: "Under ideal conditions, all TCP sessions should be resumed. However, this is not guaranteed, and under less than ideal conditions, some TCP sessions may need to be restarted."
Likely causes
| Cause | What you see | How to confirm |
|---|---|---|
| Session sync is switched off | Every established session breaks at every failover; new sessions are fine | FortiGate: Session pickup setting. Linux: no conntrackd process, or no Sync section in its configuration |
| Sync is configured but nothing replicates | Standby shows no foreign entries before the failover | Linux: conntrackd -s shows external cache of 0 on the standby. Palo Alto: no received sync messages on the passive unit |
| Entries replicated but not loaded on takeover | Standby external cache is full, but sessions still die | Linux: no committing external cache line in the conntrackd log at the failover; HA manager does not run primary-backup.sh |
| Sync messages being lost | Some sessions survive, some do not; sequence tracking reports lost packets | Linux: conntrackd -s network shows Packets lost |
| Sessions the vendor does not synchronise | Only UDP, ICMP or decrypted sessions break | FortiGate: UDP and ICMP are not synchronised by default. Palo Alto: decrypted sessions are not synchronised |
| Rule set not well formed | Mid-stream packets dropped on the new active unit even though entries exist | Linux: conntrack-tools requires a "well-formed stateful rule-set"; check the INVALID-state handling |
Prove the cause before changing anything
These commands are taken from vendor documentation and have not been run or lab-tested. Everything in this section only reads state.
Linux: conntrackd with keepalived
On the active node and the standby, as root:
conntrackd -s
conntrackd -s network
conntrackd -s cache
The manual shows the general output as an internal and an external block, each with current active connections. Its rule for reading it: "If the replication works fine, conntrackd -s displays the active's internal cache should display the same number of entries than the backup's external cache and vice-versa."
- Good: the active node's internal count and the standby's external count are close to equal, and move together.
- Bad: the standby's external count stays at 0 or far below the active node's internal count.
conntrackd -s network prints sequence-tracking statistics. The manual's sample shows Packets lost: 42726 as an example of a problem to look for. The manual's troubleshooting answer for lost packets is to raise McastRcvSocketBuffer where the cause is buffer overrun on the multicast sender or receiver.
To see the entries themselves, the conntrackd(8) page gives conntrackd -i ("Dump the internal cache, i.e. show local states") and conntrackd -e ("Dump the external cache, i.e. show foreign states").
Next read the log from the last real or test failover. The manual says that after a failover the log should show lines like:
[notice] committing external cache
[notice] Committed 1545 new entries
If there is no such pair at the takeover time, entries were not injected into the kernel. The manual's example places the log at /var/log/conntrackd.log. Your LogFile setting decides where yours is.
Finally, confirm that your HA manager calls the transition script. The manual states: "The HA manager invokes this script when a transition happens", referring to primary-backup.sh, and says it is for the HA manager (keepalived, in its examples) to call it on becoming active, becoming backup or entering a fault state. Check that the script exists in /etc/conntrackd/ and that keepalived runs it on each transition. The keepalived configuration synopsis documents notify_master, notify_backup and notify_fault as "scripts run on each state transition".
FortiGate
Fortinet's page gives the GUI path. Go to System > HA, select the primary FortiGate and click Edit. Under Cluster Settings, look at the Session pickup setting. Close the dialog without saving. In the CLI the same setting is session-pickup under config system ha.
The same page says UDP, ICMP and broadcast sessions are separate: "By default, the FGCP does not maintain a session table for UDP, ICMP, or broadcast packets, even when session pickup is enabled."
Palo Alto Networks
A third-party note (Johannes Weber, 13 June 2018, on PAN-OS 7.1.14) warns that the green HA2 indicator "is always 'Up' as long as the HA interface is up. It does NOT indicate whether the session sync is working or not." It says to verify the session count on the passive unit, or run this command on the passive unit a few times:
show high-availability state-synchronization
and check that the received counters (session setup, session teardown, session update) increase. A passive unit that shows 0 received messages and a session count of 0 is not holding the active unit's sessions. This is a blog from an older PAN-OS release; the command itself is the point, not the exact output layout on your version.
Example (illustrative): two Linux firewalls, fw-a (active) and fw-b (standby), protect 10.0.20.0/24. A nightly database job from 10.0.10.15 to 10.0.20.30 breaks whenever fw-a is rebooted. On fw-a, conntrackd -s shows 4,210 current active connections in the internal cache. On fw-b the external cache shows 0. The conntrackd process on fw-b is running but its dedicated sync interface is configured with the wrong address, so nothing arrives. Replication, not takeover, is the fault.
Fix
Work from the cause you proved, one change at a time.
Sync switched off or not working
FortiGate. Enable it:
config system ha
set session-pickup enable
end
Risk: this is a change to the HA cluster configuration on a production firewall. Fortinet's page does not describe how it affects a running cluster, so make the change in a maintenance window and keep the previous value written down so you can set it back. To also synchronise UDP and ICMP, Fortinet says to enable session-pickup first and then set session-pickup-connectionless enable.
Linux. Fix the replication path: the manual requires "a dedicated link" between the firewalls, which it calls "mandatory for security reasons", a working HA manager, and a configuration file on every replica. Check the dedicated-link addresses and interface in the Sync section of /etc/conntrackd/conntrackd.conf. The manual describes ft-fw as "based on a reliable protocol that performs message tracking" and notrack as "a best effort replication protocol, ie. unreliable protocol", and prefers FT-FW over UDP, or multicast. The conntrackd.conf(5) page lists the modes and the socket buffer settings.
After a node rejoins the cluster, conntrackd -n requests a resync with the other node (FT-FW and NOTRACK modes only). Risk: the manual warns about a race. The daemon's control socket must be open before -n is run, otherwise the node can end up knowing none of the other's connection states.
Entries replicated but not committed
Wire the HA manager to the script. In keepalived this means the notify_master, notify_backup and notify_fault hooks calling primary-backup.sh, as the manual describes. Then test.
A manual commit exists: conntrackd -c ("Commit external cache to conntrack table"). Risk: use it only on the node that is already carrying the traffic, and only to recover during an incident. It injects foreign entries into the live kernel table, and the conntrackd.conf page notes that CommitTimeout set too low can make established TCP entries with no traffic hang (an SSH connection without keepalive is its example).
Lost sync messages
Follow the manual's own answers: raise the multicast socket buffers for lost packets, and raise SocketBufferSize and SocketBufferSizeMaxGrown when the log reports the maximum netlink socket buffer reached.
Sessions the vendor does not synchronise
If it is only decrypted sessions on a Palo Alto pair, this is documented behaviour, not a fault. The page says the firewall decrypts new sessions that start after the failover based on your decryption policy, so clients must reconnect. If it is only UDP or ICMP on a FortiGate, use session-pickup-connectionless as above.
Rule set not well formed
The conntrack-tools manual says a "well-formed stateful rule-set" is required "otherwise you are likely to experience problems during the fail-over", and links a worked iptables example. Compare your rules against it, particularly where INVALID packets are handled.
Confirm the fix
- Start a long-lived test session through the cluster, such as an interactive SSH session or a large file copy between test hosts. Use test machines, never a production database job.
- Before the failover, check the replication counters again: on Linux, the standby's external cache count matches the active node's internal cache; on a Palo Alto pair, the passive unit's received counters are increasing.
- Disruptive step: trigger a failover during a maintenance window (stop keepalived on the active node, or use the vendor's suspend or failover function). Traffic through the active unit will be interrupted for the failover time, and any session that is not synchronised will break. Back out by failing back the same way.
- The test session should pause briefly and then continue. On Linux the conntrackd log on the new active node should show
committing external cacheand aCommitted N new entriesline. - Repeat in the other direction, and judge by a sample of sessions, since resumption is not guaranteed.
What to watch afterwards
- Compare the active node's internal cache count with the standby's external count on a schedule, and alert on a persistent gap.
- Alert on a rising
Packets lostinconntrackd -s network. - On Palo Alto, watch the passive unit's received sync counters and session count, not the HA2 indicator.
- After every firmware or configuration change on either unit, rerun the controlled failover test.
Read next
Related articles you may want next.
- TroubleshootingConnections Half-Work Through Two Gateways: Proving Asymmetric RoutingTwo gateways or a second NIC and connections reset or hang? Prove the return path with captures on both legs, then fix routing, not the firewall's state checks.Read article
- TroubleshootingSmall Requests Work but Large Transfers Hang: Proving a Path MTU Black HolePing works but large transfers or TLS stall? Prove a PMTU black hole with DF pings and captures, then fix it by allowing ICMP, clamping MSS or lowering MTU.Read article
- TroubleshootingNSX Logs Late TCP Packets Against the Wrong Firewall RuleInterpret NSX 3.x/4.x stateful DFW flow expiry and late TCP packets before changing a rule that allowed the original connection.Read article
Risks and limits
- Fortinet's and Palo Alto's pages describe the setting and the exception, but print no failing log line for an unsynchronised failover. The cause is read from counters and settings, not from a vendor-published drop message.
- The conntrack-tools manual is copyrighted 2008 to 2020 and its sample log lines are dated 2008 and 2011. The command options come from the current man pages.
- The FortiOS commands are from the 7.2.8 documentation. Other FortiOS releases may differ. The Palo Alto command example is from PAN-OS 7.1.14 in a 2018 third-party note, and the decryption statement is from the PAN-OS 11.1 documentation updated 13 March 2026.
- Cisco and Juniper firewalls are not covered.
- A failure that also happens without any failover is a routing or path fault, not session synchronisation.
Sources
- conntrack-tools user manual — Pablo Neira Ayuso, netfilter.org, copyright 2008-2020, accessed 17 November 2026
- conntrackd(8) — Arch Linux manual pages, date not shown, accessed 17 November 2026
- conntrackd.conf(5) — Arch Linux manual pages, date not shown, accessed 17 November 2026
- Configuration Synopsis — Keepalived project, date not shown, accessed 17 November 2026
- Session pickup (FortiOS 7.2.8 Administration Guide) — Fortinet Document Library, date not shown, accessed 17 November 2026
- Decryption Basics (PAN-OS 11.1) — Palo Alto Networks, updated 13 March 2026, accessed 17 November 2026
- Notes regarding Palo Alto HA2 Session Sync — Weberblog.net (third-party), 13 June 2018, accessed 17 November 2026
This article was originally published on .