NSX Traceflow Says Delivered but the Application Still Fails: What the Trace Proves

A Delivered observation in NSX Traceflow means that a marked test packet built by NSX Manager was reported as delivered at a particular endpoint in the trace. Which endpoint that is has to be checked. When the last observation names the destination VM's logical port, the trace shows that the packet crossed the NSX components it recorded and reached that port. When it names a bridge or another element, it does not show that, as explained below.

Even in the best case, Delivered does not show that the destination operating system received anything:

• The NSX 4.2 guide says the injected packet "is never actually delivered to the destination guest VM".
• Broadcom KB 387914 says the guest operating systems of the sender and receiver do not see Traceflow packets.

(Traceflow, NSX 4.2; KB 387914)

So a green trace with a broken application narrows the question. It does not answer it. This article shows how to read a trace result that already exists, what it leaves unproven, and how to interpret guest evidence that has already been collected.

This article is about interpreting existing evidence. It does not cover running a trace, running a connection test, enabling In-band Network Telemetry (INT) or changing MAC learning. Each of these either sends traffic or changes configuration, so each belongs in your change process.

What a Traceflow result is

Traceflow is not a ping between two guest network stacks:

• NSX Manager builds a packet with the headers you choose, marks it, and injects it at the source port.
• Each NSX entity on the path then reports how it handled that packet, on input and on output.
• The guide states that this tells you "whether issues occur when receiving a packet or when forwarding the packet."

(Traceflow, NSX 4.2)

The result view has two parts, a graphical map and a table of observations:

• The first row has the observation type Injected.
• The table can be filtered to All, Delivered or Dropped.
• If there are dropped observations, the Dropped filter is applied by default.

(Perform a Traceflow, NSX 4.2)

Two consequences follow:

• The trace answers only for the packet that was described. It uses the protocol, ports and TCP flags entered in the form, and SYN is the default flag for TCP. A trace to port 443 says nothing about port 1433. A SYN-only trace says nothing about the rest of the session.
• The trace follows one synthetic packet in one direction. A second trace with source and destination swapped is a separate synthetic packet, with its own headers and flags. It can add limited supporting evidence about that packet. It does not show the real reply, such as the SYN-ACK of an actual connection, or the stateful exchange that follows. This is a reading of the documented scope, not a Broadcom statement.

What Delivered does and does not show

Use this table before anyone says "the network is fine".

The guide also limits where a destination can be:

• It can be any device in the NSX overlay or underlay, but not north of an NSX Edge node.
• It must be on the same subnet or reachable through NSX distributed logical routers.

So a trace cannot vouch for anything beyond the Edge, such as a physical firewall, a load balancer or the internet path.

Before you start

• The existing trace result. Use the result view if it is still open, or a retained export, screenshot or table supplied by whoever ran the trace. This article does not assume NSX Manager keeps old results for later retrieval. If no result can be produced, record that as a gap. Do not start a new trace under this procedure.
• The failing transaction, written down: source VM, vNIC and IP address; destination IP address; protocol; source and destination ports, if known; time of failure; and the exact error the application or user saw.
• Any guest evidence already collected near the failure time, such as listener output or a connection-test result, with the time it was taken and the host it came from.
• The NSX version. This article is scoped to NSX 4.2.x.

Version and scope boundaries

From KB 387914 and the NSX 4.2 guide:

• HCX extended networks: Traceflow does not work with them, so a trace on such a network is not evidence either way.
• VLAN-backed segments:
• Before NSX-T 3.2.0.1, Traceflow covered overlay networks only.
• From 3.2.0.1, a VLAN trace needs INT enabled. Otherwise it fails with "Traceflow on VLAN logical port InternalLogicalPort/ requires INT (In-band Network Telemetry) to be enabled."
• VLAN Traceflow supports only TCP, UDP and ICMP, and it is not supported on Edge nodes.
• If the injected packet traverses an Edge node, the result is incomplete and shows only the ESXi observations.
• Federation: Traceflow in Federation starts at NSX 3.2.1.
• Rate limit: the NSX 4.2 guide gives a Traceflow rate limit of 10 requests per second.

If your VLAN trace shows only ESXi observations, compare it with the path you expect. If that path includes an Edge node or another element that should have reported an observation, treat the trace as incomplete. If the expected path stays on ESXi hosts, a host-only result can be complete.

Step 1: check that the trace matches the failure

Compare the trace's packet with the failing transaction, field by field.

The guide notes that the form fills in the VM's IP and MAC addresses when VMware Tools is installed. Otherwise they were typed in by hand, and a typed address is a common way for a trace to test something other than the real flow. (Perform a Traceflow, NSX 4.2)

If any field does not match, the trace does not answer your question. Record which field differs. If some fields are unknown, record the match as incomplete. A matching trace, if wanted, is a new request through your change process.

Step 2: read the observations, not only the final status

Filter to All and read every row from Injected to the last observation.

• Identify the final endpoint. Record exactly what the last Delivered observation names. Conclude "reached the destination VM's port" only if it names that port. If it names a bridge, an Edge uplink or anything else, use the matching row of the table above.
• Note the firewall observations. Record which distributed firewall or gateway firewall rule each observation reports, if shown. An allow rule means NSX allowed this packet as described, and nothing more.
• Count the Delivered observations. One is expected for unicast, and several for multicast and broadcast. For unicast, the guide says the packet can show as replicated when the logical switch does not know the destination tunnel endpoint (TEP), and as flooded when the VDS does not know the destination virtual switch port.

The multiple-observations warning

Broadcom KB 445868 (environment NSX 4.2.x) covers this warning:

Traceflow round has multiple physical received observations. Please check whether the underlay switch floods packets. Your hypervisor may be in nested environment.

The KB ties the warning to a segment with MAC Learning and Unknown Unicast Flooding enabled:

• When MAC learning is enabled, the entry for a destination MAC address on the uplink ages out after 10 minutes without traffic from that address.
• Until the address is learnt again, traffic to it is flooded to every port in the broadcast domain.
• That flooding is why the trace shows extra hops.

(KB 445868)

Treat the warning as a limit on the evidence, not as the cause of the application failure. Extra observations make it harder to say which path real traffic takes, but they do not show that it failed.

The KB says the default segment profile has MAC learning disabled. MAC learning is often enabled on purpose, for example for nested hypervisors, which the warning text itself mentions. Changing the profile changes forwarding for every port on the segment. Do not change it as part of this diagnosis. Raise it with the segment owner.

Step 3: interpret guest evidence that already exists

Guest evidence answers questions the trace cannot. The commands below are shown so that you can recognise and read their output.

• Listener queries (Step 3a) only read state.
• Connection tests (Step 3b) send real traffic. Interpret one that has already been run. Any new test is a separate, approved activity, and it is not required to complete this procedure.

The Windows references are the Microsoft Learn NetTCPIP pages, as served in the Windows Server 2025 documentation view. The Linux reference is the upstream ss(8) manual snapshot named in the version scope. Check the equivalent documentation for your guest's own release. None of these commands was run for this article.

3a: was a TCP listener present?

Typical queries for TCP port 1433:

Get-NetTCPConnection -State Listen -LocalPort 1433

ss -ltn 'sport = :1433'

The ss options are -l for listening sockets, -t for TCP and -n for numeric output. Adding -p shows the process (ss(8); Get-NetTCPConnection).

How to read a result:

• A listener is shown. Note its local address:
• A listener bound only to 127.0.0.1 (or ::1) accepts loopback connections only.
• A listener bound to a specific address accepts connections to that address.
• A wildcard binding (0.0.0.0 or ::) covers that address family on the host.
• Compare the bound address with the destination address in Step 1.
• The query completed successfully with no output. No TCP listener on that port was visible to that query, on that host, at that time. On Linux, that is within the network namespace the query ran in. The ss manual documents -N for switching to another named namespace, so a listener elsewhere would not appear.
• The query failed, or ran on a different host or at a different time. That is an evidence gap, not proof that the service is stopped.
• The service uses UDP. These TCP queries cannot show a UDP listener. Record a gap.

3b: what does an existing connection test show?

A typical Windows test is Test-NetConnection -ComputerName <address> -Port <port> -InformationLevel Detailed (Test-NetConnection). If one has already been run and its output is available:

• TcpTestSucceeded : True. A TCP connection from that guest's own stack to that address and port completed at that time. It does not prove that the earlier failing connection would have completed, or that the application layer works.
• False. That test did not complete a TCP connection. The cause may be in the guest, the network or the service. Use it together with Steps 1 to 3a, not on its own.
• The detailed output also shows the source address and next hop the guest chose. That helps when the guest has more than one interface, and it gives you the source address to check in Step 1.

Reading the combinations

These are candidates to investigate next. None of them is a verdict.

Worked example (hypothetical)

A web VM cannot reach its database VM on TCP 1433. The application logged a connection timeout at 09:12.

1. The trace. A retained trace result from 09:30 shows Injected, distributed firewall observations with an allow rule on both hosts, and a final Delivered observation naming the database VM's logical port. Source VM, vNIC, both IP addresses, TCP, destination port 1433 and SYN match the failure. The application log does not record the source port, so the match is labelled incomplete for source port.
2. Observations. There is one Delivered observation and no multiple-observations warning.
3. Listener. The database administrator supplies Get-NetTCPConnection output taken at 09:35 on the database VM. It shows a TCP listener on 1433 bound to 127.0.0.1 only.

What the evidence supports:

• At 09:30, NSX forwarded a synthetic SYN to the database VM's port, and the rules on the observed path allowed it.
• At 09:35, the service was listening on loopback only. A listener bound to 127.0.0.1 does not accept connections arriving at the server's network address. This supports a listener-binding explanation, if the same service had the same binding at 09:12.

What remains unknown:

• Whether the binding was the same at 09:12.
• Whether a rule matching on source port behaves differently for the real traffic.
• The real reply path.

Next owner: the database owner, to confirm the service's binding history around 09:12. The NSX evidence goes on the ticket as recorded, with its limits. It is not a claim that every network cause has been excluded.

Troubleshooting

KB 387914 lists where to look when the Traceflow request itself fails:

• NSX Manager: /var/log/policy/policy.log and /var/log/proton/nsxapi.log
• ESXi host: /var/run/log/vmkernel.log and /var/run/log/nsx-syslog.log
• Edge node: /var/log/syslog

Reading these logs is a read-only task, but it needs administrative access to those systems.

Acceptance checks

1. The trace was compared with the failure field by field, including both ports and the address family. Each mismatch or unknown field is recorded, and the match is labelled complete or incomplete.
2. The final Delivered endpoint is recorded exactly. Any bridge, Edge or multiple-observation result, or any VLAN-path limit, is recorded as a limit on the evidence.
3. Each piece of guest evidence is recorded with its host, time, protocol and, on Linux, namespace. Failed or mismatched queries are recorded as gaps.
4. The ticket states what the evidence supports, what remains unknown and which owner inspects next. An unresolved outcome is an acceptable result.
5. Any new trace or connection test was requested through your change process, and its result is attached. This procedure itself requires neither.

Risk notes

• This procedure changes nothing and sends nothing. It interprets results that already exist.
• Connection tests send real traffic. Treat any new test as an approved activity with the application owner's agreement.
• Running a new trace injects packets. Broadcom describes Traceflow as injecting specially crafted packets into the data flow. KB 387914 also lists an issue where Edge nodes show as failed in the NSX UI because of Traceflow logging.
• INT and MAC learning are configuration changes. Enabling INT is done through an NSX API call, and Broadcom publishes performance considerations for it. A MAC discovery profile applies to every port on the segment. Neither should be changed to make a diagnosis easier.

These are source-documented behaviours. Expert review and a lab check against your own NSX design are still needed before any configuration change in production.

Learning notes

• Traceflow injects a test packet and records how NSX handled it. Guests never see that packet.
• Delivered is an observation at a named endpoint. Check that the endpoint is the destination VM's port before concluding the packet reached it.
• A trace answers for its exact packet, in one direction, at one time. A swapped trace is another synthetic packet, not the real reply.
• A listener query that returns nothing is evidence only for that protocol, host, namespace and time.
• Extra observations under MAC learning and flooding limit the evidence. They are not a diagnosis.

References

Related articles

• Design and troubleshoot VMware networking: VMkernel, vSwitch, uplink and MTU checks. NSX overlays are out of its scope.
• Application health checks that prove a usable service
• Asymmetric routing and stateful firewall drops