Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Failover cluster event 1135: cluster node was removed from the active failover cluster membership

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

Event 1135 means that one node of a Windows failover cluster missed enough heartbeat replies that the other nodes marked it down and dropped it from active membership. Microsoft's node-removal page says the event is logged on all nodes except the one removed. Microsoft's causes are path problems between the nodes: packet loss, delay, a blocked port, security software or a configuration fault on the node itself.

Start with which nodes name which, then read the cluster log for the same minute. Raising the heartbeat thresholds can make the event go away, but Microsoft says it "just masks the problem."

What the error means

Microsoft's event 1135 page gives the message as:

Cluster node 'NODE A' was removed from the active failover cluster membership. The Cluster service on this node may have stopped.
This could also be due to the node having lost communication with other active nodes in the failover cluster.
Run the Validate a Configuration wizard to check your network configuration.
If the condition persists, check for hardware or software errors related to the network adapters on this node.
Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

Each node sends heartbeat packets over the cluster networks and expects a reply. A missed reply is a lost heartbeat. Microsoft states that "By default, Cluster nodes have a limit of five failures in 5 seconds before the connection is marked down." If every route to a node is down, the node is removed, and on that node the Cluster service is terminated and restarted so it can try to rejoin.

The heartbeat is a UDP unicast packet on port 3343. Two settings control tolerance: the delay between heartbeats (SameSubnetDelay, CrossSubnetDelay) and the threshold, which is the number of missed heartbeats (SameSubnetThreshold, CrossSubnetThreshold). Tolerance is delay multiplied by threshold. Microsoft's own examples: a 2-second delay with a threshold of 10 gives 20 seconds.

The defaults are not the same on every Microsoft page. The node-removal page gives 5 for both thresholds and a range of 3 to 10. The threshold page says Windows Server sets SameSubnetThreshold to 10. The Azure SQL page lists SameSubnetThreshold as 20 for Windows Server 2019, 10 for 2016, and 5 for 2008 to 2012 R2. Read your own values before comparing anything.

Likely causes

Cause What you see How to confirm
Network hardware or path failure Several nodes report the same node at the same time Network trace from both nodes; the pattern across System logs
Packet loss at the adapter or virtual adapter Receive discards climb Counter Network Interface\Packets Received Discarded above zero
Port 3343 UDP blocked by a firewall, switch or router Repeated removals from first start Firewall rule check; the port table in Microsoft's page
IPv6 neighbour discovery rules disabled Heartbeat uses IPv6 by default The two Core Networking rules in Windows Firewall
Network profile changing between Domain and Public Removal coincides with a profile change NetworkProfile Operational log on the removed node
IPsec policy refresh All traffic blocked for a short time Timing against a Group Policy refresh
Antivirus scanning cluster paths Removal after a scan starts Exclusion list
Old NIC driver or firmware, or NIC teaming on the heartbeat NIC Lost heartbeats with no other change Driver versions; teaming status
Latency across sites with tight thresholds Removal during WAN congestion Validation network latency results

Prove the cause before changing anything

1. Read the pattern. Collect every 1135 from the System log on each node. PowerShell 5.1 documents LogName, Id and StartTime as hashtable keys:

$Date = (Get-Date).AddDays(-2)
Get-WinEvent -FilterHashtable @{ LogName='System'; Id=1135; StartTime=$Date } | Format-List TimeCreated, Message

Microsoft describes three patterns. If all nodes report the same node, look at that node's network. If a two-site cluster shows each site naming all of the other site, the link between sites failed. If the names form no pattern, "the network channels between the nodes are choked".

2. Read the cluster log for the same minutes. The cmdlet Get-ClusterLog creates a log for each node. -TimeSpan is in minutes, -Destination . copies the files to the current folder, and the timestamps are in GMT unless you add -UseLocalTime.

Get-ClusterLog -Destination . -TimeSpan 30 -UseLocalTime

This writes files on every node and does not change cluster settings. Microsoft's sample log shows what to search for: a line that a remote endpoint is unreachable, a line Marking Route ... as down, lines saying all routes to a node are down, and route history lines with Heartbeats counter/threshold: 5/5. The cmdlet cannot be run remotely without CredSSP.

3. Read the thresholds.

get-cluster | fl *subnet*

4. Check receive discards. In Performance Monitor add Network Interface\Packets Received Discarded. Microsoft says the counter is cumulative until reboot, and any value above zero suggests the receive buffer needs raising.

5. Check the firewall rule.

netsh advfirewall firewall show rule name="Failover Clusters (UDP-In)" verbose

6. Check profiles. On the removed node, read Applications and Services Logs\Microsoft\Windows\NetworkProfile\Operational for a profile change at the same time.

7. Run the network validation tests. Microsoft recommends the Validate a Configuration wizard. Risk: the Azure SQL page warns that running a validation report with the storage option can trigger a disk reset or surprise-removed event. Select only the network tests in the wizard on a production cluster, and do it in a maintenance window.

Fix

Network path. Repair the failing cable, switch, firewall rule or WAN link. For a link between sites, Microsoft says to verify connectivity with the provider.

Ports. Microsoft's table lists UDP 3343 for the Cluster Service (TCP 3343 during a node join), TCP 135 for RPC, UDP 137, UDP and TCP 464, and TCP 445 for SMB. Risk: a firewall change affects all traffic that rule governs. Record the rule before editing; back out by restoring it.

IPv6 neighbour discovery. Enable the inbound and outbound rules "Core Networking - Neighbor Discovery Advertisement" and "Core Networking - Neighbor Discovery Solicitation" if IPv6 is on and they are off. Risk: same firewall caution. Back out by disabling them again.

Antivirus. Microsoft lists exclusions: the file share witness path, %Systemroot%\Cluster, and, for Hyper-V clusters, the virtual machine directories, mms.exe, Vmwp.exe and C:\Clusterstorage. Risk: exclusions lower scanning coverage on those paths. To back out, remove the exclusions you added. Microsoft also says "Just disabling the antivirus software is insufficient in most cases" because the filter driver stays loaded, and that disabling it leaves the server unprotected. Do not uninstall security software on a production node without the owner's decision.

Drivers and firmware. Update NIC drivers and firmware one node at a time. Disruptive step: the node's network interrupts during the change. Fail roles to another node first, and back out by rolling back the driver version.

VMware vmxnet3 adapters. Microsoft's steps raise Small Rx Buffers (default 512, maximum 8192) and Rx Ring #1 Size (default 1024, maximum 4096) gradually. Disruptive step: adapter property changes can interrupt the adapter. Change one node at a time, and back out by returning the values.

Thresholds. Do this last. Record get-cluster | fl *subnet* first, then set the values:

(get-cluster).CrossSubnetThreshold = 40
(get-cluster).SameSubnetThreshold = 40

The value 40 is the one the Azure SQL page recommends for Windows Server 2012 and later (the page's command is shown for Windows Server 2012 to 2019). The page says the change is immediate and needs no restart, and that same-subnet values should not exceed cross-subnet values. Disruptive step: a larger tolerance delays detection of a real failure. At a 1-second delay, a threshold of 40 means 40 seconds before recovery starts. Back out by setting the recorded values. Microsoft's pages disagree on the ceiling: one gives 3 to 10, another recommends 40 for Windows Server 2012 and later, so check the value your version accepts and use a smaller value if it does not accept 40. For a SQL Server availability group, the Azure SQL page also says lease timeout must stay below 2 × SameSubnetThreshold × SameSubnetDelay.

IPsec refresh. Microsoft says all security associations are torn down during a policy update and traffic is blocked while they are renegotiated. Removing that delay is a network and security decision.

Confirm the fix

Collect System-log 1135 events from every node over a period that includes the time the fault used to occur, such as a backup window. Run Get-ClusterLog for the same window and confirm there are no Marking Route ... as down lines. Compare Packets Received Discarded against a reading from before the change. A cumulative counter must stop rising, not return to zero.

Example (illustrative): a three-node cluster sql01, sql02 and sql03 in the CONTOSO domain, with a heartbeat delay of 1000 ms and a threshold of 5. Tolerance is 1000 ms × 5 = 5,000 ms, or 5 seconds. At 02:14 sql01 and sql02 each log 1135 for sql03. The cluster log on sql01 shows the route to sql03 marked down, and Performance Monitor on sql03 shows Packets Received Discarded rising during the 02:00 backup. A stall of 8 seconds ends the heartbeat run at 5 seconds. Raising the threshold to 10 gives 10 seconds and hides it, but the stall remains. Raising the vNIC receive buffers addresses it.

What to watch afterwards

  • Alert on event 1135 from any node, and on Event ID 1177 and 1069, which Microsoft lists as symptoms that can accompany it.
  • Track Packets Received Discarded by node, and compare after each driver or host change.
  • Re-read get-cluster | fl *subnet* after any cluster upgrade, since defaults differ by version.
  • Keep a record of the thresholds you changed and why.

Risks and limits

These commands have not been run or lab-tested.

Microsoft's pages on this event are dated 2026-02-12 but contain older material: the node-removal page uses Windows Server 2008 R2 node names, and the event page lists Windows Server 2016 to 2022 and Azure Stack HCI. Threshold defaults differ between pages, so this topic does not state one. The sample cluster log in Microsoft's page comes from 2014. Changes to firewall rules, adapter settings and thresholds affect cluster stability while they are made. The topic does not cover loss of quorum or forced starts.

Sources