Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

"Too many open files" on a Linux service: raise LimitNOFILE the right way

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

"Too many open files" is the text of the error EMFILE. It means one process tried to hold more file descriptors than its own limit allows. Files, sockets and pipes all count. For a service started by systemd, the limit that applies is the one systemd sets for that unit (LimitNOFILE=). The limits.conf file is applied by the pam_limits.so module to user login sessions, so editing /etc/security/limits.conf is not how a unit's limit is set.

Do not raise the number first. Read the limit the running process actually has, count how many descriptors it holds, and decide whether the service needs more headroom or is leaking.

What the error means

Every Linux process has a resource limit called RLIMIT_NOFILE. The getrlimit(2) manual describes it as a value one greater than the highest file descriptor number the process may use, and says that attempts to exceed it "yield the error EMFILE". The errno(3) manual gives the text: "Too many open files", "Commonly caused by exceeding the RLIMIT_NOFILE resource limit", and says it can also be caused by exceeding /proc/sys/fs/nr_open.

The limit has two values. The soft limit is what the process runs under. The hard limit is the ceiling the process may raise its own soft limit to. The /proc/<pid>/limits file shows both for a running process (proc_pid_limits(5)).

Do not confuse EMFILE with its neighbour. ENFILE, "Too many open files in system", means the whole machine reached /proc/sys/fs/file-max. The proc_sys_fs(5) manual says you may then see VFS: file-max limit <number> reached in the kernel log. That is a different fault with a different fix.

Where a service gets its limit

The systemd-system.conf manual documents DefaultLimitNOFILE= as defaulting to 1024:524288 (soft 1024, hard 524288). The same page says these are defaults for units, and that LimitNOFILE= in a unit overrides them.

The Linux-PAM limits.conf(5) manual describes a different path. The pam_limits module "applies ulimit limits ... to user login sessions", and the manual states that "all limit settings are set per login". A service that systemd starts at boot is not a login, so look at the unit and the systemd defaults, and check the running process.

What systemd's own manual says about this directive

The systemd.exec manual lists LimitNOFILE= with the note "Do not use", and then explains. Be careful raising the soft limit above 1024, because select(2) cannot work with descriptors above 1023 on Linux. The hard limit now defaults to 524288. In the manual's words, "Typically applications should increase their soft limit to the hard limit on their own", if they do not use select(2).

When the application does not raise its own limit, a LimitNOFILE= drop-in is the unit-level way.

Likely causes

Cause What you see How to confirm
Soft limit is the default 1024 and real demand is higher Errors appear under load, then clear when load drops /proc/PID/limits shows 1024 soft; descriptor count approaches it at peak
Descriptor leak in the application Count climbs steadily, never falls, errors arrive at a similar uptime Sample the count several times over hours; it only rises
limits.conf edited but the unit was not changed Edit made, service behaves as before /proc/PID/limits still shows the old value after a restart
Application sets a lower limit itself The unit allows more than the process has Compare unit property with /proc/PID/limits; read the application's own setting (for example nginx worker_rlimit_nofile)
System-wide table full "Too many open files in system" in many programs, kernel log line about file-max /proc/sys/fs/file-nr allocated count close to the maximum
Limit raised above 1024 for a program that uses select(2) Odd failures with descriptor numbers above 1023 Application documentation; systemd.exec note above

Prove the cause before changing anything

All of these are read-only.

  1. Find the process ID of the service:
systemctl show -p MainPID app.service

MainPID is documented in systemctl(1) as the service's current main process identifier, which is runtime state. Substitute the number it prints wherever <MainPID> appears below.

  1. Read the limits the process actually runs under:
cat /proc/<MainPID>/limits

The proc_pid_limits(5) manual says this file "displays the soft limit, hard limit, and units of measurement for each of the process's resource limits". Find the row for open files. Compare the soft value with the error you are seeing.

  1. Count the descriptors held. The proc_pid_fd(5) manual says /proc/<pid>/fd/ contains "one entry for each file which the process has open, named by its file descriptor". List it and count the entries:
sudo ls -l /proc/<MainPID>/fd
sudo ls /proc/<MainPID>/fd | wc -l

Use sudo when the service runs as another user: the proc_pid_fd(5) manual says access to the links in this directory is governed by a ptrace read-access check, and without permission the second command prints 0 while the error goes to stderr. MainPID covers the main process only. If the message comes from a child or worker process (nginx workers have their own limits and descriptors), take the same count on the process ID that logs the error.

Descriptors for pipes and sockets appear as links of the form type:[inode], for example socket:[2248868] in the manual's own example, so the listing also shows what kind of descriptor dominates. Many sockets point to connections; many regular files point to a log, spool or cache directory.

  1. Repeat the count several times across a busy period. A count that rises and falls with traffic is demand. A count that only rises is a leak.

  2. Cross-check from outside the process. The prlimit(1) manual says that with a process ID and no limit values it displays the current limits:

prlimit --pid <MainPID> --nofile
  1. If many unrelated programs fail, read the system-wide counters instead:
cat /proc/sys/fs/file-nr
cat /proc/sys/fs/file-max

The first file holds three numbers: allocated handles, free handles and the maximum. The proc_sys_fs(5) manual says the free-handles value is always zero (since Linux 2.6), so a zero there is not a fault. If the allocated figure is far below the maximum, the per-process limit is your problem.

Example (illustrative): app01 runs shop-api.service. At 14:05 the journal shows "Too many open files" from the service. systemctl show -p MainPID shop-api.service gives 2417. /proc/2417/limits shows a soft limit of 1,024. Listing /proc/2417/fd returns 1,021 entries, of which 1,009 are socket: links. That leaves 1,024 − 1,021 = 3 spare descriptors. Counts taken at 10:00 and 12:00 were 410 and 770, and they fall back to about 400 each night. The count follows traffic, so this is demand, not a leak. The vendor confirms the program does not use select(2), so the administrator sets soft 4,096 and hard 65,536, well below the nr_open ceiling of 1,048,576. That leaves 4,096 − 1,021 = 3,075 descriptors of headroom.

Fix

Cause 1: raise the limit for the unit

Use a drop-in file so the vendor unit is untouched:

sudo systemctl edit app.service

In the editor add:

[Service]
LimitNOFILE=4096:65536

Caution before you save: the systemd.exec manual says "Be careful when raising the soft limit above 1024, since select(2) cannot function with file descriptors above 1023 on Linux." The first number is the soft limit, so 4096:65536 raises it above 1024. Do this only for a program that does not use select(2); the application's documentation or vendor can confirm. If it does use select(2), keep the soft limit at 1024 (LimitNOFILE=1024:65536) and leave the program to raise it itself, as the manual recommends.

Risk: saving the editor changes the unit's configuration. The systemctl(1) manual says edit writes a drop-in and that after the editor exits "configuration is reloaded", equivalent to daemon-reload. The running service keeps its old limit until it is restarted, so nothing changes for live connections at this point. To back out, run sudo systemctl edit app.service again and remove the lines, or use systemctl revert as described below.

A single number, such as LimitNOFILE=65536, sets soft and hard to the same value, which raises the soft limit above 1024 without that choice.

Choose the value from your measured peak plus headroom, and keep it at or below /proc/sys/fs/nr_open, which the proc_sys_fs(5) manual says is "a ceiling on the value to which the RLIMIT_NOFILE resource limit can be raised" (default 1048576).

Disruptive step: the new limit applies only to processes started after the change, so restart the service:

sudo systemctl restart app.service

Restarting drops live connections and interrupts work in progress. Schedule it. To back out, edit the drop-in again and restore the earlier value, then restart. systemctl revert app.service also removes the drop-in, but the systemctl(1) manual says it removes every drop-in for that unit and also "any user-configured unit file that overrides a matching vendor supplied unit file" (under /etc/systemd/system and /run/systemd/system), so a full local replacement unit goes too. Check systemctl cat app.service first.

Cause 2: a leak

A higher limit only delays the failure. Find what holds the descriptors with ls -l /proc/<MainPID>/fd, and report the pattern to the application's owner. Restarting clears the count as a stopgap, with the disruption noted above. Raise the limit only as a measured buffer while the fix is built.

Cause 3: limits.conf edited

Keep the edit if you also need it for logins. Set the service limit in the unit as above. For a program started by hand from a shell, the shell's ulimit -n is the equivalent of the directive, according to the table in systemd.exec.

Cause 4: the application's own setting

nginx documents worker_rlimit_nofile as changing "the limit on the maximum number of open files (RLIMIT_NOFILE) for worker processes", used "to increase the limit without restarting the main process". Check the product's documentation for the equivalent setting.

Cause 5: the system-wide table

The proc_sys_fs(5) manual shows echo 100000 > /proc/sys/fs/file-max as an example of raising it. The figure is the manual's example, not a recommendation. Find what holds so many handles first.

Risk: this write changes a kernel setting for the whole machine, not for one service. Before writing, record the current value with cat /proc/sys/fs/file-max. To back out, write that value back the same way. The proc_sys_fs(5) manual gives the write only as a run-time example; it is not a documented way to make the value permanent, so confirm the persistent form for your distribution before relying on it.

Confirm the fix

  1. After the restart, read /proc/<new MainPID>/limits again. The main process ID has changed. The soft and hard values must match your drop-in.
  2. Run the load that failed before. The "Too many open files" message must not return.
  3. Check that the descriptor count at peak is comfortably below the new soft limit.

What to watch afterwards

  • Record the descriptor count at peak each week. A slow upward trend across weeks, with traffic flat, points to a leak.
  • Alert when the count passes a fraction of the soft limit that you choose, so you act before errors arrive.
  • After application upgrades, recheck /proc/PID/limits. A new unit file shipped by the vendor can change the picture.
  • Memory: the systemd.exec manual says descriptors are accounted like other memory, so watch the service's memory after a large increase.

Risks and limits

  • These commands have not been run or lab-tested. They come from the vendor and project manual pages listed below.
  • The documented defaults (1024:524288) are from the systemd 262 manual pages. Older systemd releases and distribution packaging can differ, so always read /proc/PID/limits on the actual host.
  • The 4,096 and 65,536 figures are examples from the illustrative scenario. Choose yours from measurement.
  • User services, run by a per-user service manager, cannot raise limits above those of that manager (systemd.exec); that case is not covered here.
  • Containers and other resource controls may impose their own ceilings; these manuals do not cover them.
  • The limits.conf documentation states what the module applies to. It does not list every way a process can be started, so the running process remains the final check.

Sources