About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.
When every line in a web server's access log carries the load balancer's address instead of the visitor's, the server is doing what it was built to do: it reports the other end of the TCP connection, and that end is the load balancer. The visitor's address reaches the server only if the load balancer passes it on, either in an HTTP header such as X-Forwarded-For or in a PROXY protocol line at the start of the connection. The server must also be told to trust that value from the load balancer and from nothing else. Find out which method the load balancer uses, set the server to read the same one, then limit trust to your own proxy addresses.
What the symptom means
A web server's idea of "the client" is the peer address of the socket it accepted. Behind a load balancer, that peer is the load balancer. The default combined log format of nginx starts with $remote_addr, so the log shows the load balancer on every line. The same address feeds anything else that keys on the client: rate limits, allow lists and geographic rules.
The HAProxy manual puts it this way: "Since HAProxy works in reverse-proxy mode, the servers see its IP address as their client address."
There are two ways to carry the real address across the extra hop.
An HTTP header. X-Forwarded-For is the widely used, non-standard header. RFC 7239 (June 2014) standardises the Forwarded header, for example Forwarded: for=192.0.2.60;proto=http;by=203.0.113.43. HAProxy adds the header with option forwardfor. The manual warns that the header "is always appended at the end of the existing header list", so "the server must be configured to always use the last occurrence of this header only", because a client may already have sent one.
The PROXY protocol. The load balancer writes one line before any other data on each new connection. Version 1 is human-readable, for example PROXY TCP4 192.168.0.1 192.168.0.11 56324 443, followed by a carriage return and line feed. The specification says the receiver "MUST be configured to only receive the protocol" and "MUST not try to guess whether the protocol header is present or not", and that it "SHOULD ensure proper access filtering so that only trusted proxies are allowed to use this protocol". HAProxy documents it as informing the server about the client's address "whatever the upper layer protocol", which is why it suits traffic the load balancer does not decrypt.
Either way, the receiving server has to be switched on to use the value. In nginx that is the realip module, which the documentation says "is not built by default" and needs --with-http_realip_module. In Apache it is mod_remoteip.
Likely causes
| Cause | What you see | How to confirm |
|---|---|---|
The load balancer sends nothing (HAProxy without option forwardfor, or a TCP-mode listener with no PROXY line) |
Logs show the load balancer. No X-Forwarded-For header reaches the server |
Log $http_x_forwarded_for temporarily; read the load balancer configuration |
| The server is not set to read the header or PROXY line | The header is present but the logged address is still the load balancer | nginx -V for the realip module, or httpd -M for remoteip_module; search the configuration for the trust directives |
| The trusted-proxy list does not contain the address the server actually sees | Logs show a different internal address, for example a second proxy or a translated address | Compare the logged address with the set_real_ip_from or RemoteIPTrustedProxy entries |
| Several proxies in a chain, and recursive search is off | Logs show the nearest outer proxy (for example a CDN), not the visitor | Read real_ip_recursive, and the full header value |
| The two ends disagree about the PROXY protocol | Connections fail or logs show a garbled request. nginx logs broken header |
Check send-proxy on the load balancer against proxy_protocol on the server listener |
| The server trusts the header from anyone | Logged addresses are not the real visitors. A tester can choose their own | Send a request with a made-up header and read the log |
Prove the cause before changing anything
Start with read-only checks.
1. Read the load balancer's configuration. For HAProxy, look in the frontend or backend for option forwardfor (header) or send-proxy / send-proxy-v2 on the server line (PROXY protocol). haproxy -c -f <file> checks a configuration and exits; the management guide says it exits "before trying to bind" and returns zero if all is well.
2. Read what the web server is set to do.
nginx -V
nginx -T
nginx -V prints the configure parameters; look for --with-http_realip_module. nginx -T tests the configuration and dumps it; search the output for set_real_ip_from, real_ip_header and proxy_protocol. The dump holds the whole configuration, which can include file paths and credentials, so do not paste it into a ticket or chat without checking it first. For Apache, httpd -M lists loaded modules (look for remoteip_module), and httpd -S prints the parsed virtual hosts.
3. Send a test request that carries a made-up header.
curl -s -H 'X-Forwarded-For: 203.0.113.200' https://app.example.com/
-H adds an extra header to the request, and -s makes curl silent: no progress meter and no error messages. Then look for that request in the access log. A good result is the visitor's real address, with 203.0.113.200 appearing only in the raw header. A bad result is 203.0.113.200 as the logged address, which means the server trusts a client-supplied value. The same test, sent from a machine that is not a trusted proxy, should never change the logged address.
4. See the header itself, if you need to. nginx exposes any request header as $http_ followed by the header name in lower case with dashes turned to underscores, so $http_x_forwarded_for is the header. The module also keeps the original socket peer in $realip_remote_addr. Adding both to a temporary log format shows exactly what arrived.
Disruptive step: adding a log_format and access_log line needs nginx -t followed by nginx -s reload. The reload starts new workers with the new configuration and gracefully shuts down the old ones. To back out, remove the lines and reload again.
Fix
Disruptive step: every change below edits the live configuration of a load balancer or web server. Test the syntax first (haproxy -c -f <file>, nginx -t, or httpd -t), apply it with a reload, and keep a copy of the previous file. To back out, restore the copy and reload again.
Load balancer sends the header
For HAProxy in HTTP mode, put option forwardfor in the frontend or backend. The manual lets you exclude a network with except, rename the header with header, and use if-none only "in perfectly trusted environment", because it might otherwise cause "a security issue if headers reaching HAProxy are under the control of the end-user".
Risk: this edit changes the headers every backend request carries once HAProxy reloads. To back out, remove the option and reload again.
nginx reads the header
set_real_ip_from 192.0.2.10;
real_ip_header X-Forwarded-For;
real_ip_recursive on;
set_real_ip_from defines the trusted addresses "known to send correct replacement addresses". It accepts an address, a CIDR range, or unix:. real_ip_header defaults to X-Real-IP, so setting it is not optional if your load balancer sends X-Forwarded-For. The documentation says the address and port "should be specified according to RFC 3986". It does not mention the Forwarded header, so do not assume a Forwarded: for=... value is accepted; test it.
Risk: a wrong set_real_ip_from list lets a client who is not a trusted proxy choose the address nginx logs and uses for access rules. To back out, remove the three lines and reload.
Apache reads the header
RemoteIPHeader X-Forwarded-For
RemoteIPTrustedProxy 192.0.2.10
The documentation processes a list of addresses "in Right-to-Left order" and stops "when a given useragent IP address is not trusted to present the preceding IP address". For private ranges (10/8, 172.16/12, 192.168/16, 169.254/16, 127/8) use RemoteIPInternalProxy, because RemoteIPTrustedProxy leaves such addresses in the header and the documentation says private addresses are evaluated only when internal proxies are registered. The overridden address then feeds Require ip, %a logging and mod_status; the original connection address stays available as %{c}a.
Risk: the replaced address feeds Require ip rules, so a wrong trusted-proxy list can open or close access for the wrong clients. To back out, remove the two lines and reload gracefully.
PROXY protocol on both ends
On the load balancer, add send-proxy or send-proxy-v2 to the server line. The manual notes that with this option set, health checks also send the PROXY line unless an explicit port or addr is used. On nginx:
listen 80 proxy_protocol;
set_real_ip_from 192.0.2.10;
real_ip_header proxy_protocol;
nginx supports version 2 of the protocol since 1.13.11. In Apache 2.4.31 and newer, RemoteIPProxyProtocol On makes the upstream client send the header "every time it opens a connection or the connection will be aborted", unless it is listed in RemoteIPProxyProtocolExceptions. In Apache the setting applies per address and port, not per name-based virtual host. In nginx the proxy_protocol parameter applies to every connection accepted on that listen port.
Disruptive step: switching the PROXY protocol on at one end only drops traffic. nginx logs broken header for a connection that does not start with a PROXY line, and Apache aborts it. Change both ends in the same window. To back out, remove the setting from both ends.
Example (illustrative): the visitor is 198.51.100.77. The request passes through a CDN at 203.0.113.9 and then a load balancer at 192.0.2.10 before reaching web01. The header arriving at web01 is X-Forwarded-For: 198.51.100.77, 203.0.113.9, and the socket peer is 192.0.2.10. With set_real_ip_from 192.0.2.10; set_real_ip_from 203.0.113.9; and real_ip_recursive off, nginx replaces the peer with "the last address sent", which is 203.0.113.9: the CDN. With real_ip_recursive on it picks "the last non-trusted address", which is 198.51.100.77. Leaving out the second set_real_ip_from line gives the same wrong answer as turning recursion off.
Confirm the fix
- Repeat the curl test from a machine outside the proxy chain. The access log shows that machine's real address, and a made-up
X-Forwarded-Forvalue does not change it. - Compare the logged address with a known source, such as the address your own workstation shows as its public one.
- Check that rules keyed on the client address (allow lists, rate limits) now act on the visitor and not on the load balancer. In Apache,
Require iprules use the overridden address. - For PROXY protocol, confirm that the load balancer's health checks still pass, because they send the line too.
What to watch afterwards
- Keep the trusted list in step with the load balancer: a new load balancer address, a CDN range change or a second proxy tier will bring the symptom back.
- Watch for
broken headerentries in the nginx error log after any load balancer change. - Never open the trusted list wider than your own proxies to fix a short-term problem. Anyone inside the list can name any address they like.
- If the server is also reachable directly (for example for monitoring), keep that path separate. The PROXY specification says port sharing between public and private access is prevented on purpose.
Read next
Related articles you may want next.
- HostingLoad-balancer health checks that mark a working application as downWhy wrong health URLs, TLS settings or deep checks cause false negatives—and how to design probes that match real user paths in steady-state hosted operations.Read article
- TroubleshootingHAProxy 503 Service Unavailable: No server is available to handle this requestHAProxy 503: No server is available to handle this request. Read the log flags and server status to find a failed health check, backend rule or queue.Read article
- TroubleshootingConnections drop after a few idle minutes through a firewall or load balancer: idle timers and TCP keepaliveTCP sessions die after the same idle time through a firewall or load balancer. Find the idle timer and set keepalive below it on Linux, nginx or SSH.Read article
Risks and limits
- These commands have not been run or lab-tested. They are taken from vendor documentation and source files listed below.
- RFC 7239 says the
Forwardedheader "cannot be relied upon to be correct", as every node on the way, including the client, can alter it. The part of the chain before your first trusted proxy is only as reliable as that proxy. - Apache documents that it is "trivial for the remote useragent to impersonate another useragent", so trust only intermediate hosts you control.
- Cloud load balancers have their own switches for client address handling. They are not described here.
- nginx logs
broken headerwhen a connection to aproxy_protocollistener does not begin with a valid PROXY line, as the nginx 1.30.6 source shows. The exact wording of other servers' messages is not covered.
Sources
- RFC 7239: Forwarded HTTP Extension — IETF, June 2014, accessed 18 November 2026
- The PROXY protocol, versions 1 and 2 — HAProxy Technologies, revision 2020/03/05, accessed 18 November 2026
- HAProxy 3.0 Configuration Manual — HAProxy, dated 2026/10/14, accessed 18 November 2026
- HAProxy 3.0 Management Guide — HAProxy, no date shown, accessed 18 November 2026
- nginx: ngx_http_realip_module — nginx documentation, no date shown, accessed 18 November 2026
- nginx: ngx_http_core_module (listen, embedded variables) — nginx documentation, no date shown, accessed 18 November 2026
- nginx: ngx_http_log_module — nginx documentation, no date shown, accessed 18 November 2026
- nginx: command-line parameters — nginx documentation, no date shown, accessed 18 November 2026
- nginx 1.30.6 source: ngx_proxy_protocol.c — nginx project, tag release-1.30.6, accessed 18 November 2026
- nginx 1.30.6 source: ngx_http_request.c — nginx project, tag release-1.30.6, accessed 18 November 2026
- nginx 1.30.6 source: ngx_http_log_module.c — nginx project, tag release-1.30.6, accessed 18 November 2026
- nginx: download — nginx.org, lists stable 1.30.6 and mainline 1.31.8, accessed 18 November 2026
- Apache HTTP Server project: download — Apache Software Foundation, shows 2.4.71 released 2026-11-11, accessed 18 November 2026
- Apache HTTP Server 2.4: mod_remoteip — Apache Software Foundation, copyright 2026, accessed 18 November 2026
- Apache HTTP Server 2.4: httpd command-line options — Apache Software Foundation, no date shown, accessed 18 November 2026
- curl man page — curl project, describes curl 8.23.1, accessed 18 November 2026
This article was originally published on .