Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

nginx 504 Gateway Time-out: upstream timed out (110: Connection timed out) while reading response header

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A 504 from nginx means the backend it proxies to did not respond inside one of nginx's timers. The error log says which phase ran out of time: upstream timed out (110: Connection timed out) while connecting to upstream, while sending request to upstream or while reading response header from upstream. Each phase has its own directive and, in the nginx documentation, a default of 60 seconds. Raising the timer is sometimes right, but only after you know whether the backend is slow for a legitimate reason or stuck, because a longer timer on a stuck backend just makes the visitor wait longer for the same failure.

What the error means

In the nginx 1.30.6 source, an upstream timeout is logged at the error level as upstream timed out, with the operating system's ETIMEDOUT appended as (110: Connection timed out) and the phase appended after while. nginx then finishes the request with NGX_HTTP_GATEWAY_TIME_OUT, status 504, and the page reads "504 Gateway Time-out". The Linux kernel headers define ETIMEDOUT as 110.

Three timers apply while nginx is talking to the proxied server, and each has a FastCGI twin:

Phase in the log Proxy directive FastCGI directive Default What the documentation says it measures
while connecting to upstream proxy_connect_timeout fastcgi_connect_timeout 60s "a timeout for establishing a connection"; "cannot usually exceed 75 seconds"
while sending request to upstream proxy_send_timeout fastcgi_send_timeout 60s The time between two successive write operations, "not for the transmission of the whole request"
while reading response header from upstream proxy_read_timeout fastcgi_read_timeout 60s The time between two successive read operations, "not for the transmission of the whole response"

The read timer applies once the connection is made and the request is sent. It does not limit the total time. A response that trickles data every 30 seconds never trips it. A backend that sends nothing for 61 seconds does.

Likely causes

Cause What you see How to confirm
A request that legitimately takes longer than 60 seconds (report, export, large query) 504 after a fixed 60 seconds on one URL $upstream_header_time is near 60 for that URL only
A stuck or overloaded backend (database lock, saturated worker pool, external call with no timeout) 504s on many URLs at busy times, normal otherwise Header time rises with load. The backend's own logs show the same requests running long
Backend host not reachable (firewall drop, host down) while connecting to upstream $upstream_connect_time near 60, or curl --connect-timeout to the backend fails
Backend not reading the request while sending request to upstream Large uploads or a backend that has stopped reading
The retry multiplied the wait Total request time is a multiple of 60 seconds $upstream_addr lists several addresses separated by commas
A PHP script with no time limit Script runs on after nginx gave up request_terminate_timeout is 0 (off) and max_execution_time is large or 0 (its default for web requests is 30 seconds)

Prove the cause before changing anything

1. Read the error log line. The phase after while selects the row in the table above. Find the log path with nginx -T, which dumps the configuration.

2. Time the backend directly.

curl -s --max-time 120 -o /tmp/response.out -w '%{time_connect} %{time_starttransfer} %{time_total}\n' http://192.0.2.30:8080/reports/monthly

time_connect is the TCP connection time, time_starttransfer is the time to the first byte, and time_total is the whole operation. --max-time limits the transfer in seconds, and -o writes the body to a scratch file (here /tmp/response.out, which you can delete afterwards) so the timing line stays readable. If time_starttransfer is well above 60, the backend is slow, regardless of nginx. If it is fast, the delay is between nginx and the backend.

3. Add timing to the access log. nginx exposes $request_time, $upstream_connect_time, $upstream_header_time, $upstream_response_time, $upstream_addr and $upstream_status. Several servers or tries are separated by commas and colons. A line such as $request_time $upstream_connect_time $upstream_header_time $upstream_addr $upstream_status placed in a temporary log_format shows where the time went.

Disruptive step: adding the log format needs nginx -t and then nginx -s reload, which starts new workers and gracefully shuts down the old ones. To back out, remove the line and reload again.

4. Look at the backend. Its own log and process list show whether the request is working or waiting. For PHP-FPM, if request_slowlog_timeout is set (default 0, off), it dumps a backtrace of a slow request to the slowlog file, which names the function that is waiting.

Fix

Disruptive step: every change below (timers, retry limits) alters live proxy behaviour after a reload. Run nginx -t first and keep a copy of the previous file. To back out, restore the copy and reload again.

Slow request: fix first, then raise the timer for that location only

If the work is legitimately long, move it out of the request (a queue, a background job and a status page). If it must stay in the request, raise the timer only where it is needed:

upstream app {
    server 192.0.2.30:8080;
    server 192.0.2.31:8080;
}

location /reports/ {
    proxy_pass http://app;
    proxy_read_timeout 120s;
    proxy_next_upstream off;
}

proxy_read_timeout is allowed in the http, server and location contexts. Keep the default elsewhere so that a stuck backend still fails in 60 seconds. proxy_next_upstream off; stops the retry on the second server described under retry multiplication, which suits a slow report that should not run twice. The addresses are documentation addresses.

Disruptive step: changing the configuration and running nginx -s reload. Test with nginx -t first. To back out, restore the previous file and reload.

Retry multiplication

By default proxy_next_upstream is error timeout, and proxy_next_upstream_tries and proxy_next_upstream_timeout are both 0, meaning no limit. A timeout therefore passes the request to the next server in the group. The documentation adds that "non-idempotent" methods (POST, LOCK, PATCH) are not passed on once sent. If a GET to a two-server group times out on both, the visitor waits about 2 x 60 = 120 seconds before the 504. That total is derived from the documented defaults, not measured. To cap it, set proxy_next_upstream_timeout or proxy_next_upstream_tries, or, if retries after a timeout are harmful, take timeout out of proxy_next_upstream (for example proxy_next_upstream error;) or set it to off to stop all retries.

Connect timeout

Check routing and firewalls between nginx and the backend, and that the backend host is up. The nginx documentation says proxy_connect_timeout "cannot usually exceed 75 seconds"; it does not say what happens when you set a longer value, so treat a setting above 75 seconds as unproven.

Risk: a longer connect timer keeps visitors waiting longer on an unreachable backend. To back out, restore the previous value and reload.

Send timeout

A backend that does not read the request body may be stuck or waiting on its own limits. Look at the backend's logs before changing proxy_send_timeout.

PHP backends

The PHP manual states max_execution_time defaults to 30 seconds for web requests and 0 on the command line. PHP-FPM's request_terminate_timeout defaults to 0, which means off, and is described as the option to use "when the 'max_execution_time' ini option does not stop script execution for some reason". The PHP manual's file-upload page notes that max_execution_time "only affects the execution time of the script itself": time spent on database queries and the file upload process is not counted. The PHP configuration page adds: "On non Windows systems, the maximum execution time is not affected by system calls, stream operations etc." On a Linux server, a script waiting on a database or a system call can therefore outlive max_execution_time, and it outlives request_terminate_timeout only when that option is left at 0.

Example (illustrative): web01 runs nginx in front of an application at 192.0.2.30:8080 with two servers in the upstream group. A monthly report at /reports/monthly takes 90 seconds. Users get a 504 after about 120 seconds. The error log reads upstream timed out (110: Connection timed out) while reading response header from upstream, twice for the same request. Direct curl timing shows time_starttransfer of 91 seconds. The log's $upstream_addr shows two addresses, so the first 60-second timeout was followed by a retry on the second server and a second 60-second timeout: 2 x 60 = 120 seconds before the 504. The report is moved to a background job. Until then, a location /reports/ block with proxy_read_timeout 120s; and proxy_next_upstream off; lets the 90-second report finish on the first server, because 90 seconds is below 120 seconds and no retry follows.

Confirm the fix

  1. The test URL returns 200 and $upstream_header_time in the log is below the timer you set, with room to spare.
  2. No new upstream timed out lines appear in the error log for the URL.
  3. Other URLs still fail in 60 seconds if the backend hangs. A long timer applied to one location should not have spread.
  4. Check the backend load during the test. A longer timer on a backend that is already saturated moves the queue, not the problem.

What to watch afterwards

  • Log $upstream_header_time and alert on the 95th percentile, not on 504 counts after the fact.
  • Keep a list of locations with raised timers and the reason for each.
  • Check the backend's own limits (max_execution_time, database statement timeouts) so that the backend gives up before nginx does and returns a useful error.

Risks and limits

  • These commands have not been run or lab-tested. They are taken from nginx, PHP and curl documentation and from the nginx source listed below.
  • A longer proxy_read_timeout holds a worker connection open for longer. Many slow requests can use all connections.
  • Defaults are as documented for nginx on nginx.org. A distribution's configuration or an application's own include files may set different values.
  • The 120-second example total is derived from defaults, and real retry behaviour depends on max_fails and the group's other settings.
  • Timers in front of nginx (load balancers, CDNs, browsers) are separate and may expire first. They are not covered.

Sources