API gateways and ingress in the order of a multi-system restore

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Do not send users through the front door until the backends behind it are the ones you mean to serve, and do not call those backends recovered while the front door still points somewhere else. The gateway is not a coat of paint on the applications. It is the policy that chooses which address receives the request. A restored gateway configuration is a restored policy. If that policy still names the failed site, you have just reopened the failed site.

The mechanics below are from NGINX's upstream module, the documentation served on nginx.org on 11 November 2026. Other gateways and ingress controllers make the same kind of choice with different syntax. This article does not translate their configuration, and it does not tell you to reload a live proxy. A reload that swaps the server list can change the destinations of new requests, and it was not performed here.

Why it matters

NIST SP 800-34 Rev. 1 says the recovery sequence for a complex system should follow the priorities from the impact analysis, and that procedures should restore components in a logical order so related systems are not hit by the outage of something they need. The gateway is related to every application it fronts. Its recovery time is the recovery time of the first user journey you are willing to open, not the time it takes to start a process called "gateway".

Two mistakes are symmetrical.

• The gateway is up, with yesterday's server list. Users arrive. NGINX will try the servers in the group, and if none of them returns a success it gives the client the result of the communication with the last server. That is a public failure, and it is also a test you did not mean to run against production names.
• The applications are up and tested from inside the recovery network, and the gateway still points at the old addresses, or it is not listening on the name users type. The internal test passes. The business is still down. NIST's reconstitution is a functionality test of the system users actually use. A test that skips the front door has not tested the system.

How it works

A group of servers is the policy

NGINX's upstream directive defines a group of servers. They may listen on different ports, and the group may mix TCP addresses with UNIX-domain sockets. By default, requests are spread by weighted round-robin. In the documentation's example, a server with weight 5 receives five of each seven requests, and two other servers receive one each.

The address of a server can be a domain name or an IP address, with an optional port. If you omit the port, the port is 80. A domain name that resolves to several addresses defines several servers at once. That single sentence is why the gateway cannot be recovered independently of name service. Whoever controls the name controls the membership of the group, unless you wrote IP addresses. If the name still has both the old site and the new site, both are in the rotation. Round-robin will give a share of users to each. You do not get a clean cutover by accident.

The resolve parameter monitors changes to the addresses of a domain name and updates the upstream configuration without a restart. The page says that before version 1.27.3 this parameter was only in the commercial subscription, that the server group must be in shared memory, and that a resolver directive is required. For a static upstream server hostname without enabled resolve, resolution occurs when configuration loads; having a capable binary alone does not enable dynamic resolution. A later DNS change does not move traffic until the configuration is loaded again. Our recommendation: know which of those two behaviours you have before the incident. A plan that says "we will repoint DNS and the gateway will follow" is true only for the configuration that actually watches the resolver.

Failure is not the same as "marked down"

NGINX can pass a failed request to another server, subject to the relevant next-upstream policy, retry/time limits and whether the request can still be retried. If none succeeds, the client receives the result of the last attempt. What counts as an unsuccessful attempt is defined by the related proxy_next_upstream family of directives, which this article does not restate. The default accounting is enough to plan around.

max_fails is the number of unsuccessful attempts, inside the fail_timeout window, after which the server is considered unavailable for a further fail_timeout. The default number of attempts is 1. The default fail_timeout is 10 seconds. Setting max_fails to 0 turns the accounting off. For a multi-server group, a counted unsuccessful attempt can therefore make a backend temporarily unavailable under the default settings. An important exception: with only one server in the group, max_fails, fail_timeout and slow_start are ignored; that server is never marked unavailable by those controls. During a recovery, backends flap. They answer, they refuse, they answer. The gateway will keep offering them to users on that cycle unless you have taken them out on purpose.

down marks a server permanently unavailable. backup marks a server that receives requests only when the primary servers are unavailable, and it cannot be combined with the hash, ip_hash or random methods. A recovery-site server left as backup does not receive users while any primary is still considered available. A primary that is up and serving the wrong copy is available. The backup will sit idle while users hit the wrong copy. A primary with counted unsuccessful attempts may be skipped after max_fails, and then the backup can be used, which may allow a retry to a backup if the configured policy permits. The user may see success, an error or extra latency; the backup still needs independent readiness checks.

Our recommendation: before the gateway is on the user path, every server in the group is either an intended, tested backend or is marked down. Do not use passive failure, or the backup flag, as the cutover mechanism. Passive failure still sends the request that discovers the failure. The backup flag waits until the primaries are unavailable, which is not the same as "we have decided the recovery site is the one".

What has to be true before the door opens

The applications behind the gateway have their own gates: identity, names, storage, secrets, and whatever queue they consume. Those are not repeated here. The gateway adds four checks of its own.

1. The server list matches the site you are serving. IP addresses are explicit. Names are only as good as the resolver the gateway uses, at the moment the configuration is loaded, unless resolve is in effect on a build that has it. Check from the gateway, not from a laptop on another network.
2. Servers you do not want are down, not merely "probably going to fail". A failed server is still tried until the failure count is reached, and then only for fail_timeout. down is the documented permanent mark.
3. A request through the gateway, not only a request to the backend's private address, returns the success you defined. NIST's functionality test is of the system. The system the user hits includes this hop. If the gateway adds a header, a timeout or a TLS certificate the backend test never saw, the private test will not catch it.
4. The certificate and the name users type are the ones on this gateway. A backend can be perfect and the browser can still refuse the door. That check is part of opening the door, not a follow-up ticket.

Only then do you send the ordinary user path at the gateway. Internal testers can use it earlier. Ordinary users are the reconstitution step, and they should be a decision with a name on it, not the side effect of a process starting.

Weighted round-robin matters when both sites are accidentally present. The documentation's weight example is five requests to one server and one each to two others. A restored configuration that leaves the old site at the default weight of 1, beside a new site at weight 1, sends about half the users to each. There is no "small" wrong server. If you meant a temporary trickle, the weight has to be a number you chose, on backends you have already accepted, not a leftover address.

Order against the rest of the estate

Our recommendation:

• Build and test backends with the gateway not yet on the public name. Give them a private way in. a name that already points at the gateway may deliver real traffic as soon as a matching listener and route become available.
• Load the gateway configuration while it is still unreachable to users, or behind a controlled maintenance response or traffic block. Confirm the upstream membership from that configuration.
• Open one application path at a time, in the business order from the impact analysis. A single gateway process often fronts many applications. Starting the process is not starting every application. Routes you are not ready to serve need an intentional maintenance or rejection response, without access to unapproved backends.
• Repoint the public name only when the path you are opening has passed the gateway test. Keep public ingress names separate from upstream backend names. Public DNS controls which gateway clients reach, subject to cached answers; an upstream hostname controls NGINX’s backend membership. Changing one does not necessarily change the other. Without dynamic upstream resolution, a backend-name change requires the appropriate configuration reload. Account for old DNS answers and existing connections during either transition.
• Leave a way back. The previous server list and DNS records are only the routing part of rollback. They are safe to restore only if the old backends remain authorised and data-consistent; new writes may make a simple reversal unsafe. NIST's recovery procedures are supposed to be stepwise. "Reload and see" without the previous list written down is how you discover you cannot return.

Health dashboards that scrape the backend directly will go green before the gateway is honest. Use the signal that matches the user path: a request that enters through the gateway. A backend scrape that skips the gateway is a different fact, useful and incomplete.

What to choose and why

These are our recommendations.

Risks and limits

• This is NGINX's upstream module, as documented on nginx.org on 11 November 2026. Envoy, a cloud load balancer, or a commercial ingress controller can behave differently. Do not copy down and max_fails into another product's configuration language.
• resolve was commercial-only before 1.27.3. Confirm the binary you will actually reload. The documentation's availability note is the warning.
• Passive failure accounting and retries are conditional. The documented one-failure/ten-second defaults do not apply to a single-server group, and retry policy limits what is retried. Those defaults are a safety net for a broken backend, not a traffic-management policy for a dual-site recovery.
• A domain name that resolves to many addresses becomes many servers. That is the module's rule. Combined with a name that still advertises both sites, it is a split of user traffic.
• No NGINX was reloaded. Timeouts, TLS and proxy_next_upstream were not restated as a procedure. The passive-health defaults above are the ones the upstream page states.
• NIST does not describe gateways. It supplies the rule that the order should protect the systems that depend on each other, and that reconstitution is a real test.

Research and verification

Drafted with AI assistance on 11 November 2026 and independently source-reviewed on 13 November 2026. No article commands or recovery operations were executed. This is educational source research, not a lab-validated production procedure.

Sources

Independently reopened 13 November 2026.

• nginx.org, Module ngx_http_upstream_module. Upstream groups, addresses and DNS, round-robin, max_fails, fail_timeout, backup, down, and resolve.
• NIST SP 800-34 Rev. 1, May 2010: PDF. Sequence of recovery, and reconstitution as a functionality test.

This article was originally published on 13 November 2026.