Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

"Temporary failure in name resolution" on a Linux server: test each layer from systemd-resolved down

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

"Temporary failure in name resolution" is the C library's text for the error code EAI_AGAIN. The getaddrinfo(3) manual says it means "the name server returned a temporary failure indication", and to try again later. In practice it means the program could not get a usable answer from any DNS server it was told to ask. The fault is in one of four places: the server's DNS settings, the local resolver service, the network path to the upstream server, or the upstream server itself.

Do not edit /etc/resolv.conf first. On a systemd-resolved system the manual recommends that it be a symbolic link to the stub file, so check which tool owns the file before editing it. Find which layer fails, then change the setting that owns it.

What the symptom means

Programs resolve names by calling getaddrinfo, which follows the order in /etc/nsswitch.conf. The glibc source holds the message text: EAI_AGAIN maps to "Temporary failure in name resolution", while EAI_NONAME maps to "Name or service not known" and EAI_NODATA to "No address associated with hostname". The getaddrinfo manual describes EAI_NONAME as "the node or service is not known". So a "temporary failure" says the lookup could not be completed. A "not known" says it was completed and the name was not found. Treat them differently.

The path of a lookup on a systemd-resolved system

The systemd-resolved manual describes the pieces:

  • systemd-resolved offers a local DNS stub on the addresses 127.0.0.53 and 127.0.0.54 on the loopback interface. The manual notes that programs should use the glibc NSS API where they can, but those that read /etc/resolv.conf are pointed at the stub.
  • /run/systemd/resolve/stub-resolv.conf "lists the 127.0.0.53 DNS stub (see above) as the only DNS server". The manual says it "should not be used directly by applications, but only through a symlink from /etc/resolv.conf", and that this mode of operation "is recommended".
  • The upstream servers come from several places: "the global settings in /etc/systemd/resolved.conf, the per-link static settings in /etc/systemd/network/*.network files ... the per-link dynamic settings received over DHCP, information provided via resolvectl(1), and any DNS server information made available by other system services".
  • If /etc/resolv.conf is something else, the manual says "systemd-resolved is consumer rather than provider of this configuration file".

So the symptom with a nameserver 127.0.0.53 line in /etc/resolv.conf has two halves. The stub can be missing or stopped. Or the stub is running but has no working upstream server to ask.

FallbackDNS= in the resolved.conf manual is "only used if no other DNS server information is known".

Timing

The resolv.conf(5) manual gives the library's defaults: timeout is "the amount of time the resolver will wait for a response from a remote name server before retrying", measured in seconds, default 5 (RES_TIMEOUT), and attempts defaults to 2. It says that timeout "may not be the total time taken by any resolver API call", so do not add the numbers up to predict how long a failing lookup takes. It also says "Up to MAXNS (currently 3, see <res_state.h>) name servers" are used.

Likely causes

Cause What you see How to confirm
No upstream DNS server configured for the link resolvectl status shows no DNS servers for the active link DHCP lease without DNS, or static config missing a nameservers entry
Upstream server unreachable (firewall, route, outage) Other traffic works, DNS queries time out dig @<upstream> name fails from the server
systemd-resolved stopped or failed /etc/resolv.conf points at 127.0.0.53 but nothing answers there systemctl status systemd-resolved; journalctl -u systemd-resolved
/etc/resolv.conf wrong, empty or replaced Wrong server addresses, or file not a link to the resolved stub ls -l /etc/resolv.conf and its contents
Name service switch has neither resolve nor dns getent fails but dig works The hosts: line of /etc/nsswitch.conf
Config applied to the wrong interface after a change Static DNS set on a name that no longer exists Interface names in the netplan or NetworkManager config against the links listed by resolvectl status
Network down altogether Nothing resolves, nothing connects Interface state and routes

Prove the cause before changing anything

Work from the application's view downwards. All commands read only.

  1. Resolve the way programs do. The getent manual says ahosts passes each key to getaddrinfo with the unspecified address family, so it uses the same path as applications:
getent ahosts example.com

If this fails, the system path is failing. The getent manual defines exit status 2 as "One or more supplied key could not be found in the database" and does not say what status a lookup that timed out returns, so read the error text and not only the number. If it prints addresses, the system path works and the problem is in the program or its own resolver settings.

  1. Ask systemd-resolved. The resolvectl manual says query resolves domain names:
resolvectl query example.com
  1. See what systemd-resolved is configured to use. The manual says status "shows the global and per-link DNS settings currently in effect":
resolvectl status

Note which interface carries your default route and which DNS servers it lists. Empty means no server is configured there.

  1. Check /etc/resolv.conf:
ls -l /etc/resolv.conf
cat /etc/resolv.conf

Expect either a link into /run/systemd/resolve/ or the content a different manager wrote. A nameserver 127.0.0.53 line with a stopped service is a failure.

  1. Check the service:
systemctl status systemd-resolved
journalctl -u systemd-resolved
  1. Ask an upstream server directly, bypassing the local resolver. The BIND documentation describes dig as a tool for interrogating DNS name servers, with dig @server name:
dig @192.0.2.53 example.com

If this works while resolvectl query fails, the fault is local. If this also fails, the server is unreachable or not answering from here.

  1. Read the hosts: line in /etc/nsswitch.conf. The nsswitch.conf(5) manual shows an example line of the form hosts: dns [!UNAVAIL=return] files; read the actual line on your host, not the example.

Example (illustrative): On web01, apt and curl both report "Temporary failure in name resolution". getent ahosts example.com fails. ls -l /etc/resolv.conf shows a link to /run/systemd/resolve/stub-resolv.conf and systemctl status systemd-resolved shows the service running. resolvectl status lists no DNS servers for the main interface, because its static network config has an address and gateway but no nameservers entry. dig @192.0.2.53 example.com from the same server works, so the network path is fine. Adding the server to the interface's config fixes the lookup.

Fix

No upstream server configured

On a Netplan-managed system, first copy the file you are about to edit to a place outside /etc/netplan, so you can restore it. Use your own file name in place of FILE:

sudo cp /etc/netplan/FILE.yaml /root/FILE.yaml.bak-YYYYMMDD

The Netplan reference documents the nameservers field with addresses and search lists:

nameservers:
  addresses: [192.0.2.53, 198.51.100.53]
  search: [example.com]

Put it under the interface's section. The Netplan documentation says netplan try applies a configuration "and automatically rolls it back if the user does not confirm the configuration within a time limit", and is useful on remote systems to avoid being locked out. It also states that if netplan try times out or is cancelled you should verify the configuration was in fact reverted.

Disruptive step: applying network configuration can briefly interrupt connectivity, including your own SSH session. Use sudo netplan try, and keep a console available. The back-out is the automatic rollback, or copying the saved file back to /etc/netplan and applying it the same way.

On NetworkManager, the nmcli manual shows nmcli con modify ABC +ipv4.dns 8.8.8.8 to add a DNS server to a connection. Use your own connection name and your own server address.

Disruptive step: this changes the stored connection profile. Applying it to a connection that is already active can drop that connection and with it your own SSH session. The nmcli manual documents nmcli connection up, and nmcli device reapply, which will "Attempt to update device with changes to the currently active connection made since it was last applied". Keep a console open. To back out, remove the entry with the - prefix, as the manual's examples do for addresses (nmcli con modify ABC -ipv4.dns <address>), then apply again the same way.

systemd-resolved stopped

Read the journal for why. Then start it:

sudo systemctl restart systemd-resolved

Disruptive step: restarting the resolver may interrupt lookups in progress for a moment. The resolved manual does not document the effect of a restart on the cache or on lookups in progress, so expect a brief interruption. There is nothing to back out; the service reads its settings again when it starts.

/etc/resolv.conf wrong

If your system uses systemd-resolved, the manual's recommended mode is for /etc/resolv.conf to be a symbolic link to /run/systemd/resolve/stub-resolv.conf. Check which tool in your distribution's network tooling owns the file before changing it. The resolved manual describes systemd-resolved as a "consumer rather than provider" of a /etc/resolv.conf that is not its own stub link, so the tool that owns the file decides its contents; change the setting in that tool.

Name service switch

If getent fails but dig works and resolvectl query works, check the hosts: line. On a systemd-resolved system the systemd-resolved manual says the glibc module nss-resolve(8) is required for the NSS resolver functions to use systemd-resolved, so the line may name resolve. Whether dns alone is enough on your host, check against your distribution's documentation, and correct the line under that guidance.

Risk: /etc/nsswitch.conf controls how every program on the host resolves names, not one service. A wrong line can break all lookups at once, including the ones used to fetch a fix. Keep a copy of the file before you edit, keep a console session open, and back out by restoring the copy.

Upstream unreachable

If dig @server fails, the fault is outside the server: firewall rules for UDP and TCP port 53, the route to the DNS server, or the DNS server itself. Test from another machine on the same network to tell them apart. A hosted environment's DNS servers can be given by DHCP, so a changed network or VLAN may need a new address.

Clear the cache after a fix

resolvectl flush-caches flushes all DNS caches that the service keeps locally, according to the manual.

Confirm the fix

  1. getent ahosts example.com prints addresses.
  2. resolvectl query example.com returns an answer.
  3. The failing program works. If it still fails, check whether it has its own DNS setting or a container with its own resolver.
  4. Reboot in a maintenance window, then repeat. Confirm the setting survives. Disruptive step: a reboot interrupts every service on the server for the time it takes to start again. There is nothing to back out; keep a console open in case the server does not return.

What to watch afterwards

  • Configure at least two DNS servers, on different hosts if you can.
  • Watch resolver errors in the journal for systemd-resolved.
  • After restoring or cloning a VM, check that interface names and DNS settings still match.
  • Record where each server's DNS comes from (DHCP or static configuration) so you can find it quickly.

Risks and limits

  • These commands have not been run or lab-tested. They come from the manual pages and documentation cited in Sources.
  • This covers systemd-resolved, Netplan and NetworkManager. Servers that use a different resolver (for example a locally run caching DNS server) need that product's tools.
  • The manual pages are for systemd 262. Older systemd versions may differ.
  • The nsswitch.conf(5) line shown in the manual is an example, not a statement of any distribution's default.
  • Changing DNS or network configuration on a remote server can cut your own access. Keep out-of-band console access when you can.
  • A fault in a public or provider DNS server is outside your control; the checks only show that it is where the failure sits.

Sources