Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

"fork: retry: Resource temporarily unavailable": the process limit, not memory

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

When a Linux shell prints fork: retry: Resource temporarily unavailable, it asked the kernel for a new process and the kernel answered EAGAIN. The fork(2) manual lists four limits under its first EAGAIN entry: the per-user process limit, the system-wide thread limit, the maximum process ID, and the cgroup process limit. Memory shortage is a different error. Find which limit was hit before you raise any of them.

What the error means

The text after the colon is the C library's text for EAGAIN. The errno(3) page gives it as "Resource temporarily unavailable". The fork: retry prefix comes from bash. In its source, bash calls fork(), and while it fails with EAGAIN bash reports fork: retry, sleeps, and tries again. The first sleep is one second and the sleep doubles after each failure, up to a maximum. This is read from the bash source, not from the bash manual, and the mirror's version is not shown. Other shells word the error differently.

The fork(2) page says EAGAIN means "A system-imposed limit on the number of threads was encountered", and lists:

  • the RLIMIT_NPROC soft resource limit, which limits the number of processes and threads for a real user ID;
  • the kernel's system-wide limit /proc/sys/kernel/threads-max;
  • the maximum number of PIDs, /proc/sys/kernel/pid_max;
  • the PID limit pids.max imposed by the cgroup process number (PIDs) controller.

A different error covers memory: fork(2) says ENOMEM means fork "failed to allocate the necessary kernel structures because memory is tight". So a server with free memory can still refuse to fork. The kernel's cgroup v2 page makes the same point: "a fork bomb is likely to exhaust the number of tasks before hitting memory restrictions".

Two details matter when you read the numbers. The getrlimit(2) page says RLIMIT_NPROC counts processes "or, more precisely on Linux, threads" for the real user ID, so a service with many threads uses the limit up quickly. It also says the limit "is not enforced for processes that have either the CAP_SYS_ADMIN or the CAP_SYS_RESOURCE capability, or run with real user ID 0". The cgroup v2 page says PIDs in the controller "refer to TIDs, process IDs as used by the kernel", so threads count there too.

Likely causes

Cause What you see How to confirm
Per-user limit (RLIMIT_NPROC, LimitNPROC=, nproc in limits.conf) One non-root account cannot start anything; root can prlimit --pid PID for a process of that account, and a count of that account's threads
systemd TasksMax= for one service (cgroup pids.max) The one service fails to start workers; the rest of the host is fine systemctl status shows its task count; systemctl show shows the limit; cgroup pids.events counts hits
Runaway process creation (a loop, a leak, a fork bomb) Task count climbs until a limit stops it ps sorted by thread count
System-wide threads-max or pid_max Every account is affected, including root Compare total threads with the two files

Prove the cause before changing anything

All of these are read-only.

1. Which account or service is full? List processes with their thread counts, largest first:

ps -eo pid,user,nlwp,args --sort=-nlwp

The ps page documents -e (all processes), -o (the output columns), nlwp ("number of lwps (threads) in the process") and --sort. A process or account with a very large total points at the cause.

2. The per-user limit. For a process that belongs to the failing account:

prlimit --pid 13134

The prlimit page documents -p, --pid and gives this form as "Display limit values for all current resources". Replace the example process ID with a real one and read the line for the maximum number of processes (the page lists -u, --nproc as "Maximum number of processes"). To count the threads that account already runs, list them with ps -L -U appuser -o pid,lwp (the ps page documents -L as "Show threads, possibly with LWP and NLWP columns" and -U as selecting by real user ID or name, which is what RLIMIT_NPROC counts). Each line below the header is one thread. Compare that count with the soft limit.

3. The service limit under systemd. If the failing work is a service:

systemctl status app.service
systemctl show --property=TasksMax --property=EffectiveTasksMax app.service

The systemd.resource-control page says TasksMax= controls the pids.max control group attribute and that "The effective configuration is reported as EffectiveTasksMax=". Its default comes from DefaultTasksMax= in systemd-system.conf, which, on the systemd-system.conf page, "Defaults to 15% of the minimum of kernel.pid_max=, kernel.threads-max= and root cgroup pids.max"; that page gives 4915 as the value with the default kernel.pid_max. The systemctl status example on the systemctl page includes a Tasks: line.

4. The cgroup counters. The kernel cgroup v2 page defines these files for a non-root cgroup: pids.max (read-write, the hard limit, default "max"), and the read-only pids.current ("The number of processes currently in the cgroup and its descendants") and pids.events, whose max entry counts how many times the cgroup's total number of processes hit the pids.max limit. Read them in the service's cgroup directory: the CGroup: line of systemctl status names its path, and the files sit under /sys/fs/cgroup/ at that path (for example cat /sys/fs/cgroup/system.slice/app.service/pids.events, where the path is the one the CGroup: line shows). A pids.events max count above zero proves the limit was hit.

5. The system-wide limits.

cat /proc/sys/kernel/threads-max /proc/sys/kernel/pid_max

The proc_sys_kernel(5) page describes threads-max as the "system-wide limit on the number of threads (tasks) that can be created" and pid_max as the value at which PIDs wrap around, which "also acts as a system-wide limit on the total number of processes and threads". Its default is 32768, and on 64-bit systems it can be set up to 2^22.

Fix

Raise a limit only when the count is legitimate. If thread counts are rising without bound, find the process in step 1 and fix that first.

A service limit (TasksMax=)

Save the current definition, including any drop-ins, under a dated name:

systemctl cat app.service > app-service-before-20261120.txt

The systemctl page says cat shows the fragment and drop-in files of units, each preceded by a comment that includes the file name.

Disruptive step: creating a drop-in with a higher TasksMax limit lets a faulty service use more of the host's shared process IDs before it stops, so the fault can spread to other services.

Create the drop-in with the limit. The page says systemctl edit creates a drop-in file and, after editing, configuration is reloaded; --drop-in= names the file (the page's own example is --drop-in=limits.conf, so the name here ends in .conf).

systemctl edit --drop-in=tasksmax-20261120.conf app.service

In the editor, add:

[Service]
TasksMax=8192

The value 8192 is an example, not a recommendation. Check the result with systemctl show --property=TasksMax app.service. A running service may need a restart for a new limit to apply; the systemd pages do not say, so check the shown value after the change, and restart only in a window where its users can tolerate it. To back out, run systemctl cat app.service, which prints the path of the drop-in you created, delete only that file, and run systemctl daemon-reload.

The systemd.exec page says LimitNPROC= limits "the number of processes from one (real) UID" cumulatively and is not enforced for a service running as root, and that "TasksMax= ... is typically a better choice than LimitNPROC=".

A per-user limit from login sessions

The limits.conf(5) page says the pam_limits module applies these limits to user login sessions, with lines of the form domain, type, item, value, and nproc is the "maximum number of processes". An example from the page is @faculty hard nproc 50.

Risk: a higher nproc for an account removes a guard against runaway processes. Save the file first: cp /etc/security/limits.conf /root/limits-conf-saved-20261120, edit it, and note that the change applies to new login sessions. To back out, run cp /root/limits-conf-saved-20261120 /etc/security/limits.conf and start a new session.

System-wide limits

Raise threads-max or pid_max only after a per-user or per-service limit has been ruled out. The proc_sys_kernel(5) page says a value written to threads-max is checked against available RAM. Record the current values first (step 5) and change them only after a per-service limit has been ruled out.

Example (illustrative): On web01, the application's start script (a bash script run by app.service) prints fork: retry: Resource temporarily unavailable in the service's log during a traffic peak, and the application stops creating new workers. Administrators who log in to web01 at the same time have no trouble, because their shells are not inside the service's cgroup. systemctl show --property=TasksMax --property=EffectiveTasksMax app.service reports 4915 for the application, and systemctl status shows its task count at the limit. The service's pids.events file shows a max count above zero. The application had been started with many threads per request. The team sets TasksMax=8192 in a drop-in, and also fixes the application's thread pool size.

Confirm the fix

  1. systemctl show --property=TasksMax app.service shows the new value.
  2. The service's task count in systemctl status stays below the limit at the next peak.
  3. The max entry in pids.events stops increasing.
  4. New shells no longer print fork: retry.

What to watch afterwards

  • Alert when a service's task count passes a set share of its limit, not only when it fails.
  • Watch the max count in pids.events for services you raised.
  • Keep systemctl cat output of important services in your records.
  • Treat a rising thread count as a fault to investigate, not a number to keep raising.

Risks and limits

  • These commands have not been run or lab-tested.
  • The retry behaviour is taken from bash source; the bash manual does not describe it.
  • The systemd pages do not state whether a running service picks up a new TasksMax= without a restart.
  • Container runtimes set their own pids limits and are not covered.

Sources

  • fork(2) — Linux man-pages 6.19, 2026-06-05, accessed 20 November 2026
  • getrlimit(2) — Linux man-pages 6.19, 2026-02-08, accessed 20 November 2026
  • errno(3) — Linux man-pages 6.19, 2026-05-28, accessed 20 November 2026
  • proc_sys_kernel(5) — Linux man-pages 6.19, 2026-02-08, accessed 20 November 2026
  • Control Group v2 — Linux kernel documentation, page shows "October, 2015", accessed 20 November 2026
  • systemd.resource-control(5) — systemd 262~devel (development manual), colophon: obtained 2026-08-04, most recent upstream commit 2026-08-03, accessed 20 November 2026
  • systemd.exec(5) — systemd 262~devel (development manual), colophon: obtained 2026-08-04, most recent upstream commit 2026-08-03, accessed 20 November 2026
  • systemd-system.conf(5) — systemd 262~devel (development manual), colophon: obtained 2026-08-04, most recent upstream commit 2026-08-03, accessed 20 November 2026
  • systemctl — systemd 262 documentation, no date shown, accessed 20 November 2026
  • limits.conf(5) — Linux-PAM manual page, 08/04/2026, accessed 20 November 2026
  • prlimit(1) — util-linux 2.43, 2026-05-24, accessed 20 November 2026
  • ps(1) — procps-ng, 2025-04-23, accessed 20 November 2026
  • bash jobs.c — unofficial GNU bash source mirror, file header shows copyright 1989-2015 and no release date, accessed 20 November 2026