Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

nginx 502 Bad Gateway: connect() failed (111: Connection refused) while connecting to upstream

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

When nginx shows "502 Bad Gateway", it sat between the visitor and a backend (an application server, PHP-FPM, another web server) and could not get a usable answer from that backend. The nginx error log records why on one line, and that line decides the fix. One such line is connect() failed (111: Connection refused) while connecting to upstream, which means nothing is listening at the address nginx was told to use. The same 502 also comes from an unreadable socket, a backend that closes early, an oversized response header, or an upstream group with no usable server left.

What the error means

nginx answers 502 when an error occurs while it connects to a backend, sends the request or reads the response header, and it has no other server to try. In the nginx 1.30.6 source the default status for an upstream failure is NGX_HTTP_BAD_GATEWAY (502); a timeout becomes 504. The visitor's page reads "502 Bad Gateway".

By default proxy_next_upstream is error timeout. A connection error therefore makes nginx try the next server in the group, if there is one, before it gives up. If an upstream block has several servers, you may see one error line per attempt.

The error line has a fixed shape. The source builds it as the message, then while followed by what nginx was doing, then client, server, request, upstream and host fields. The "doing" part is connecting to upstream, sending request to upstream or reading response header from upstream. The parenthesised pair is the operating system's error number and text, which come from the C library. For reference, the Linux kernel headers define 111 as ECONNREFUSED, 110 as ETIMEDOUT, 13 as EACCES, 2 as ENOENT and 104 as ECONNRESET.

Likely causes

Cause What you see (error log line, start of line) How to confirm
Nothing listens at the address and port connect() failed (111: Connection refused) while connecting to upstream ss -ltnp shows no listener on that port
Wrong host name, port or IP version in proxy_pass or fastcgi_pass connect() failed (111: Connection refused) for some requests only The name resolves to several addresses, and the backend listens on one of them
Unix socket missing connect() to unix:/path failed (2: No such file or directory) while connecting to upstream ls -l on the path
Unix socket not readable or writable by the nginx worker connect() to unix:/path failed (13: Permission denied) while connecting to upstream Compare the socket owner, group and mode with the user directive
Unix socket queue full connect() to unix:/path failed (11: Resource temporarily unavailable) while connecting to upstream A comment in the nginx 1.30.6 source (ngx_event_connect.c) says Linux returns EAGAIN instead of ECONNREFUSED for unix sockets when the listen queue is full; then check the PHP-FPM or application backlog and workers
Backend closes the connection before sending a header upstream prematurely closed connection while reading response header from upstream or recv() failed (104: Connection reset by peer) The backend's own log; crash or kill
Response header larger than the buffer upstream sent too big header while reading response header from upstream Header size against proxy_buffer_size
Every server in the group is marked unavailable no live upstreams while connecting to upstream max_fails and fail_timeout in the upstream block

Prove the cause before changing anything

All of these are read-only.

1. Read the log line for a failing request. The path is whatever error_log says (nginx -T prints the configuration and lets you find it). The nginx documentation's default is error_log logs/error.log error;. Match the start of the line to the table.

2. Read the proxy settings.

nginx -T

Find the proxy_pass, fastcgi_pass or upstream block for the failing site. Note the address, the port or the socket path, and the user directive in the main context.

3. Check what is listening.

ss -ltnp

-l shows listening sockets, -t TCP, -n numeric addresses, -p the processes using sockets. The man page does not say whether the process column needs extra privileges, so run it as an administrator. No line for the backend port, or only a different address (for example 127.0.0.1:8080 while nginx tries [::1]:8080), explains a refused connection.

4. Talk to the backend without nginx.

curl -sI http://127.0.0.1:8080/
curl -sI --unix-socket /run/app/app.sock http://localhost/

-I fetches headers only and --unix-socket connects through a Unix socket. If these work, the backend answers outside nginx, so look at nginx's address, permissions or timing next. This is a reasoning step, not a statement from a manual. A FastCGI socket such as PHP-FPM's does not speak HTTP, so use ls -l on it instead.

5. Check the process. systemctl status <unit> shows whether the service is running and its last log lines.

Fix

Disruptive step: every configuration change below alters live proxy behaviour. Run nginx -t, apply with nginx -s reload, and keep a copy of the previous file. To back out, restore the copy and reload again.

Backend not listening

Start it, or correct the port. Disruptive step: starting or restarting an application service interrupts its current users. Do it in a quiet window and have the previous configuration ready to restore.

Wrong address, or IPv4 against IPv6

The nginx documentation says that if a name in proxy_pass resolves to several addresses, "all of them will be used in a round-robin fashion". A name such as localhost that resolves to both an IPv6 and an IPv4 address, with a backend listening on one only, can therefore log a refused connection for the address nothing listens on. With the default proxy_next_upstream error timeout, nginx passes the request to the next address, so the request can succeed on the other address while the error line is still logged. That follows from the documented behaviour and has not been tested here. Use an explicit literal address: proxy_pass http://127.0.0.1:8080;.

Risk: changing proxy_pass redirects live traffic once nginx reloads, and a wrong address turns every request into a 502. To back out, restore the previous line and reload.

Socket permission

The PHP manual says of PHP-FPM: "In Linux, read/write permissions must be set in order to allow connections from a web server", and the defaults are that "user and group are set as the running user, mode is set to 0660". In the pool, set listen.owner, listen.group and listen.mode so the nginx worker user can open the socket. The worker user is the user directive in nginx.

Disruptive step: reloading PHP-FPM or nginx applies the change. Keep a copy of the pool file, and put it back and reload if requests fail.

Queue full or backend too slow

For PHP-FPM, the listen.backlog default on Linux is 511. A full queue is a capacity problem, not a path problem; see "PHP-FPM Has Reached pm.max_children: Measure Queueing Before Adding Workers".

Backend closes early

Read the backend's own log first. Causes include a crash, a killed worker, or a request limit. nginx only reports that the connection ended.

Header too big

proxy_buffer_size defaults to one memory page ("4k|8k"), and "if it exceeds the buffer size, the response is considered invalid". Raise it, or shrink the cookies or headers the application sends.

Risk: a larger proxy_buffer_size uses more memory for every proxied connection. To back out, restore the previous value and reload.

No live upstreams

By default, max_fails is 1 and fail_timeout is 10 seconds: one failed attempt takes the server out of rotation for 10 seconds. The nginx upstream documentation adds: "If there is only a single server in a group, max_fails, fail_timeout and slow_start parameters are ignored, and such a server will never be considered unavailable." From that sentence it follows, untested here, that a one-server upstream group does not produce this state. If every server in a group of two or more fails at once, nginx logs no live upstreams until one returns. The fix is the backend, not nginx; raising max_fails hides the fault.

Example (illustrative): web01 proxies to an application on 127.0.0.1:8080 and a customer reports 502s that began after a package update. The error log shows connect() failed (111: Connection refused) while connecting to upstream for every request. ss -ltnp shows nothing on port 8080, and systemctl status shows the application stopped after the update. After the service is started, curl -sI http://127.0.0.1:8080/ returns headers and the site recovers.

Confirm the fix

  1. The same curl -sI against the backend directly returns a header.
  2. A request through nginx returns the normal status, and no new connect() failed lines appear in the error log.
  3. Test a request that exercises the application, not only the home page.
  4. If you changed socket permissions, repeat the request as the nginx worker user's path (a real browser request is the test), not as an administrator.

What to watch afterwards

  • Alert on the count of 502 responses and on upstream errors in the error log, not on user reports.
  • Make the service restart on failure, and monitor the listener, so a crashed backend is noticed before visitors do.
  • After any package update to the web server or the application runtime, check the socket path, owner and mode.
  • If the backend sits behind a load balancer, check its own health checks as well.

Risks and limits

  • These commands have not been run or lab-tested. They are taken from the nginx, PHP and Linux documentation and from the nginx source listed below.
  • The wording of log lines is from the nginx 1.30.6 source. Other versions may differ slightly.
  • The text after the error number comes from the operating system. The Linux man-pages page describes the same codes in slightly different words, for example "Connection reset" for ECONNRESET.
  • Mandatory access control systems such as SELinux and AppArmor can also deny a connection or socket access. Their messages are not covered here.
  • The exact rendering of a Unix socket path in the log line (unix:/path) was not confirmed in the source files listed.
  • The IPv4 and IPv6 round-robin explanation is derived from the documentation and has not been tested.

Sources