Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

HAProxy 503 Service Unavailable: No server is available to handle this request

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A browser page headed "503 Service Unavailable" with the line "No server is available to handle this request." is HAProxy's own error page. HAProxy sends it when a request arrives but there is nowhere to send it: every server in the backend is marked down, no backend was selected for the request, a server is held out by an administrator, or the request waited in a queue until it expired. The HAProxy log line for that request names the backend, the server (or <NOSRV>) and a four-letter termination state, and those two fields narrow the cause in a minute. Read them first, then ask the running process for the server status.

What the error means

The HAProxy configuration manual lists the cases in which HAProxy generates a 503 itself: "when no server was available to handle the request, or in response to monitoring requests which match the "monitor fail" condition". The same manual describes other errors it generates on its own: 502 for an empty, invalid or incomplete server response, and 504 when "the response timeout strikes before the server responds".

The built-in page for this error, in the HAProxy v3.0.0 source, is:

HTTP/1.1 503 Service Unavailable
Content-length: 107
Cache-Control: no-cache
Content-Type: text/html

<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

If your page looks different, the configuration may use an errorfile 503 line with a custom page, or the 503 may have come from the application behind HAProxy and been passed through. In both cases the HAProxy log still records the request, so the steps still apply.

Four situations produce the 503:

  1. No usable server. By default a server without the check keyword is "always considered available". With check, a server is taken out after a number of consecutive failed health checks. The default is 3 (fall), checked every 2000 ms (inter). A server returns after 2 successes (rise). When all servers are out, every request gets the 503.
  2. No backend selected. The manual says that if no use_backend rule matches and there is no default_backend, then "in case of a frontend, no server is used and a 503 service unavailable response is returned". A backend name written as a plain word is checked when the configuration loads, and "an error is reported if the specified backend does not exist", so a plain-name typo stops HAProxy loading. A rule that never matches (a wrong host name or path) loads fine.
  3. An administrator state. The management guide describes a server in maintenance as receiving no traffic and no health checks (set server ... state maint, disable server).
  4. Queue expiry. When a server's maxconn is reached, requests wait. "If the timeout is reached, it is considered that the request will almost never be served, so it is dropped and a 503 error is returned to the client."

Likely causes

Cause What you see How to confirm
Every server fails its health check, or is in maintenance 503 on all requests to that backend at once, with <NOSRV> as the server and the SC flags (SC--). Servers show DOWN or MAINT Status and check_status columns of show stat. L4CON, L7STS or L7TOUT in check_status
Health check is stricter than the application The application answers real users but not the check URL. Servers flap or stay down option httpchk uses OPTIONS / unless told otherwise. Only 2xx and 3xx count as valid
No use_backend matched and there is no default_backend 503 only for some host names or paths. Others work Read the frontend rules against the request's Host header and path
Servers held in maintenance Status MAINT. Nothing in the log about checks status column of show stat; srv_admin_state in show servers state
Queue expired Peaks of 503s under load; the server is up. Log shows the sQ flag Flags in the log; maxconn on the server line; timeout queue
HAProxy picked a server and the connection failed 503 with a real server name and the SC flags (connection refused) or sC flags (connect timeout) Flags and server name in the log. The server name separates this case from "no server"

Prove the cause before changing anything

All of these are read-only.

1. Confirm HAProxy produced the page.

curl -si https://app.example.com/

-s silences the progress meter and -i (--show-headers) shows the response headers with the body. The built-in page carries the exact body above.

2. Read the log line for a failing request. With option httplog, the HTTP log format is %ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r. Take three things from a 503 line:

  • %b/%s: the backend and the server. The manual says <NOSRV> is shown instead of a server name when the request was aborted before reaching a server.
  • %TR/%Tw/%Tc/%Tr/%Ta: a value of -1 means that phase never happened. A large Tc means HAProxy waited to connect. A large Tw means it waited in a queue.
  • %tsc: the termination state. Read the first two letters against the manual's table: SC means the server or a device in front of it "explicitly refused the TCP connection", sC means the "timeout connect" struck, and sQ means the request "spent too much time in queue and has been expired".

The manual's own example for SC-- is a 503 where "the connection to the server failed ('SC--') after 4 attempts of 2 seconds". There the server is named (px-http/srv1) and the timers read 3183/-1/-1/-1/11215: Tc is -1, only TR and Ta are positive, so Tc does not separate the cases. The manual's table gives SC only as a refused connection. The v3.0.0 source shows a second way to get the same two letters: assign_server() in backend.c returns SRV_STATUS_NOSRV when the backend's total weight (lbprm.tot_weight) is 0, and the code that follows sets the error type STRM_ET_CONN_ERR. default_srv_error() in stream.c and http_return_srv_error() in http_ana.c turn that type into a 503 with SF_ERR_SRVCL and SF_FINST_C, which the log shows as SC. No server was chosen, so the line shows <NOSRV>. Read the server and the flags together:

  • <NOSRV> and SC--: HAProxy had no server to try. Check health, maintenance and weight.
  • A real server name and SC: a server was tried and refused the connection. See "Connection fails". Only the server name separates the two cases.

3. Ask the running process. If the configuration has a stats socket, the management guide shows the pattern for the runtime commands:

echo "show stat" | socat stdio /var/run/haproxy.sock
echo "show servers state" | socat stdio /var/run/haproxy.sock

The socket path is whatever your stats socket line says. In the show stat CSV, read pxname, svname, status, check_status, last_chk and lastchg (seconds since the last up/down change). The check_status codes include L4CON ("layer 1-4 connection problem, for example "Connection refused" (tcp rst) or "No route to host" (icmp)"), L7STS ("layer 7 response error, for example HTTP 5xx") and L7TOUT ("layer 7 (HTTP/SMTP) timeout").

4. Check the configuration without loading it. haproxy -c -f <file> exits "before trying to bind", with a zero status when the configuration is valid.

Fix

Disruptive step: every configuration change below (health check, default_backend, maxconn, timeout queue) changes live traffic handling once HAProxy reloads. Check with haproxy -c -f <file> first and keep a copy of the previous file. To back out, restore the copy and reload again.

Servers fail their health check

Test the exact request the check sends from the HAProxy host with curl. If the application answers OPTIONS / with 401 or 404, point the check at a URL that returns 200:

backend app
    option httpchk GET /healthz
    server app1 192.0.2.21:8080 check
    server app2 192.0.2.22:8080 check

option httpchk <method> <uri> <version>: the version defaults to HTTP/1.0, and the manual notes that the Host field is mandatory in HTTP/1.1 and needs http-check send. Keep the check cheap, and make sure the URL returns 2xx or 3xx while the application is healthy.

Disruptive step: changing the configuration file and reloading HAProxy replaces the running configuration. Run haproxy -c -f <file> first. To back out, restore the previous file and reload again.

No backend selected

Add a default_backend, or correct the ACL that should have matched:

frontend www
    bind :80
    use_backend api if { path_beg /api }
    default_backend app

Disruptive step: this edit changes where every unmatched request goes once HAProxy reloads. To back out, restore the previous file and reload again.

Server stuck in maintenance

The management guide documents the runtime form: set server <backend>/<server> state [ ready | drain | maint ]. Setting ready "puts the server in normal mode". Do this only if you know why it was set to maint.

Risk: returning a server to service sends it live traffic at once. To back out, set it to drain or maint again.

Queue expiry

The manual says maxconn on a server is "very important as it can save fragile servers from going down under extreme loads". Raising it only helps if the server can take the extra concurrency. Otherwise lengthen timeout queue modestly, or add capacity. If timeout queue is not set, "the same value as the backend's connection timeout ("timeout connect") is used".

Risk: a higher maxconn sends more concurrent requests to a server that may not cope, and a longer timeout queue makes users wait longer before the 503. To back out, restore the previous values and reload.

Connection fails

Use this section only when the log line shows a real server name instead of <NOSRV>. Then SC means something refused the connection. Check that the application is listening on the address and port in the server line, and that no firewall blocks the path from the load balancer. sC means nothing answered in time, which points at routing, a firewall that drops silently, or an overloaded host.

Example (illustrative): lb01 balances app1 (192.0.2.21) and app2 (192.0.2.22) with check and no option httpchk. A change on the application makes both servers refuse connections on port 8080 for 20 seconds. With inter 2000 ms and fall 3, HAProxy marks each server down after 3 x 2000 ms = 6,000 ms, so for up to about 6 seconds requests still go to the failing servers and fail with connection errors (SC flags with a server name in the log). After that, with both servers marked down, requests get the "No server is available" page for the rest of the outage, and their log lines show <NOSRV> and SC--. After the application returns, rise 2 means two good checks, 2 x 2000 ms = 4,000 ms, before traffic resumes. These figures follow from the documented defaults and are not a measurement.

Confirm the fix

  1. show stat reports UP for the servers, with check_status of L4OK, L7OK or similar, and lastchg counting up.
  2. The curl -si test returns the application's response, not the built-in page.
  3. New log lines for the same requests show a real server name instead of <NOSRV> and a 2xx or 3xx status.
  4. Test a request that matches a different rule, such as the /api path, to prove that each route reaches a server.

What to watch afterwards

  • Alert on servers going DOWN and on backend 503 counts, not on the user report.
  • option log-health-checks makes HAProxy log health check status changes, so a flapping server shows up before an outage. It needs a configuration change and a reload.
  • Recheck maxconn and the Tw timer after traffic growth. Growing queue time is an early sign.

Risks and limits

  • These commands have not been run or lab-tested. They are taken from the HAProxy documentation and source listed below.
  • The manual's table describes SC only as an explicitly refused connection. The <NOSRV> with SC-- reading comes from the v3.0.0 source files named in Sources, not from the manual. The source read does not give the flag string for a frontend that has no use_backend match and no default_backend, so none is given here.
  • The default values quoted (fall 3, rise 2, inter 2000 ms) are for HAProxy 3.0. Configuration files with their own defaults section may differ.
  • Cloud-managed load balancers and HAProxy Enterprise have their own diagnostics.
  • The runtime set server command changes the running process. Whether the change survives a reload is not stated in the section read, so also check the configuration file.

Sources