Australian cloud for business software — built and run in Sydney since 2003

+61 2 8282 5656
support@netleverage.com
Talk to an engineer
Talk to an engineer
NetLeverage
NetLeverage

Linux OOM Killer on a VM: Find the Cause, Set Memory Limits

Technical article
Troubleshooting

Direct answer: When you suspect a Linux service was killed for lack of memory, find the record of the kill for the right boot before naming a killer: the kernel log entry (which, with the default oom_dump_tasks, includes a task table) or the systemd-oomd journal. A process can also exit on a failed allocation without either killer acting, so with no record the cause stays unresolved. Then establish which constraint applied: a cpuset, a memory policy, a memory (or swap) limit such as a cgroup memory.max, possibly on a parent slice, or the whole system. Correlate the cgroup counters with the time of the kill; one counter or a reading taken after the kill does not prove a cause. Fix the growth, or set limits from measured use. Do not disable the killer or set every important process to -1000.

Why it matters

An OOM kill on a VM looks like a service crash. Adding VM memory or protecting one process are the obvious reactions, and neither is safe without the record. Protecting one process moves the next kill to another. A kill inside a fixed cgroup limit can happen while the VM has free memory, so adding memory may change nothing, while a percentage limit grows with the VM.

How it works

Overcommit and why the killer exists

vm.overcommit_memory has three modes: 0, the default heuristic ("Obvious overcommits of address space are refused"); 1, always overcommit; and 2, no overcommit beyond swap plus a share of RAM, where in most situations a process "will receive errors on memory allocation" rather than be killed. /proc/meminfo shows CommitLimit and Committed_AS. In the default mode, granted memory may not exist when used; when the kernel cannot find or reclaim it, it kills a process.

The four contexts of a kernel OOM kill

The oom_score_adj manual page scores each candidate from 0 to 1000, roughly its share of the "allowed" memory, which depends on the context:

  • the cpuset's memory nodes, if "the memory assigned to the allocating task's cpuset" is exhausted;
  • the memory policy's nodes, if "a mempolicy's node(s)" are exhausted;
  • "that configured limit", if "a memory limit (or swap limit)" is reached;
  • "all allocatable resources", if "the entire system" is out of memory.

oom_score_adj (-1000 to +1000) is added to the score; -1000 "is equivalent to disabling OOM-killing entirely for that task".

oom_kill_allocating_task (default 0) chooses between scanning all tasks by score and killing the task that triggered the shortage.

What the kernel records

With vm.oom_dump_tasks at its default of 1, the kernel logs "a system-wide task dump (excluding kernel threads)" at each OOM kill, with pid, uid, memory sizes, swap entries, oom_score_adj and name, which helps show why the killer ran and why it chose its victim. At zero, the dump is suppressed. The exact line format is not in the sources used, so it is not quoted.

Cgroup limits, ancestors and percentages

With systemd and cgroup v2, each service runs in its own cgroup inside slices. Non-root cgroups provide:

File Meaning (kernel documentation)
memory.current Memory used by the cgroup and its descendants
memory.peak Highest usage since creation or a reset
memory.high Throttle limit; going over it "never invokes the OOM killer"
memory.max Hard limit; if usage cannot be reduced, "the OOM killer is invoked in the cgroup"
memory.min, memory.low Protection from reclaim
memory.events Counters including max, oom and oom_kill, hierarchical
memory.events.local The same counters for this cgroup only, "not hierarchical"
memory.oom.group Kill the whole group together or not at all

memory.events includes events "down the hierarchy", so a parent's counter can rise because of a child, and oom_kill counts kills "by any kind of OOM killer": a rise shows a kill, not that this unit's own MemoryMax= caused it.

systemd reports EffectiveMemoryMax= as "the most stringent limit of the unit and parent slices", capped by physical memory. MemoryMax= and MemoryHigh= may be absolute or a percentage of installed physical memory, so adding VM memory leaves an absolute limit unchanged, raises a percentage one, and may or may not change the effective limit.

systemd recommends "MemoryHigh= as the main control mechanism and use MemoryMax= as the last line of defense". The kernel warns that over-committing memory.min "may lead to constant OOMs".

With memory.oom.group set, the group is killed together, except tasks at -1000, which "are never killed".

Pressure, and systemd-oomd

Pressure Stall Information (PSI) reports time stalled waiting for memory in /proc/pressure/memory and each cgroup's memory.pressure: "some" when some tasks stalled, "full" when all non-idle tasks did (thrashing).

systemd-oomd uses cgroup v2 and PSI "to monitor and take corrective action before an OOM occurs in the kernel space" for units with ManagedOOMSwap=kill or ManagedOOMMemoryPressure=kill. It sends SIGKILL to every process of a selected cgroup, but "only descendant cgroups are eligible candidates for killing" (leaf cgroups, or those with memory.oom.group set), so the killed cgroup can differ from the configured unit; record which one died. oomctl lists monitored cgroups and pressure.

What the service manager does next

OOMPolicy= sets the unit's response to a kill: continue keeps it running; stop terminates it cleanly; kill sets memory.oom.group so all its processes are killed. With stop and kill the service ends "in the oom-kill failed state after which Restart= may apply". OOMScoreAdjust= sets the unit's oom_score_adj, -1000 to 1000.

Read-only checks

None of these were run for this article. Reading cgroup files and journals does not change them; writing to memory.peak resets it, so read only.

  1. Pick the boot: journalctl --list-boots lists boots. journalctl -k alone shows only the current boot, so for an earlier incident use journalctl -k -b <boot> with --since/--until.
  2. Find the kernel record: save the kill entry, the task table and any named memory cgroup or constraint. If no kernel entry exists for that time, do not conclude there was no kill; go to step 3.
  3. Check systemd-oomd: read its journal for the same window, for example journalctl -u systemd-oomd -b <boot>, if the service is present, and run oomctl to see which cgroups it monitors. Record the cgroup it killed.
  4. Find the unit's cgroup: the CGroup: line of systemctl status <unit>; the files sit in that path under the distribution's cgroup v2 mount point.
  5. Read the limits: systemctl show <unit> --property=MemoryHigh --property=MemoryMax --property=MemorySwapMax --property=MemoryMin --property=MemoryLow --property=EffectiveMemoryMax --property=EffectiveMemoryHigh --property=OOMPolicy --property=OOMScoreAdjust. Note whether each limit is absolute or a percentage in the unit file, and read the same properties for the parent slices.
  6. Read the counters: memory.events, memory.events.local, memory.current, memory.peak and memory.pressure in the unit's cgroup and parents; a counter means something only as a change across the incident.
  7. System-wide state: /proc/meminfo (CommitLimit, Committed_AS), /proc/pressure/memory, and sysctl vm.overcommit_memory vm.oom_kill_allocating_task vm.oom_dump_tasks vm.panic_on_oom.
  8. Protections: /proc/<pid>/oom_score_adj for important processes, and memory.min or memory.low values.
  9. Growth: record the service's memory over a workload cycle: unbounded growth or a steady need above its limit.

Record a missing file or Effective* property as unavailable on that version, not as a clean result.

Candidate causes (hypotheses to test)

Hypothesis Evidence that supports it Fix layer
The unit reached its own MemoryMax= Kernel record names the unit's memory cgroup; max, oom and oom_kill in the unit's memory.events.local rose at the incident time The unit's limit or the service's memory use
A parent slice's limit applied EffectiveMemoryMax= lower than the unit's own MemoryMax=; counters rose in the slice's memory.events.local The slice limit, or the services sharing it
A cpuset or memory policy ran out Kernel record names a cpuset or memory policy constraint The cpuset or NUMA policy configuration
Whole VM out of memory Kernel record without a cgroup limit; low memory and high pressure at the time Workload, service sizing or VM memory
One process grows until killed Same process largest in successive task dumps The application
Protections moved the kill Important processes at -1000; another service killed oom_score_adj / OOMScoreAdjust= choices
systemd-oomd selected a cgroup Entry in the systemd-oomd journal; the cgroup is a descendant of a monitored unit ManagedOOM* settings and thresholds

Decision guidance

Every row is a change needing a change record, a maintenance window and a measured baseline; limit changes can need a restart. None were executed for this article.

Decision Prerequisites Reference
Fix unbounded growth in the application Growth shown across task dumps or a workload cycle Application owner; a larger limit only delays the kill
Raise a unit's MemoryMax=, with MemoryHigh= below it Steady workload needing more than the limit; parent slice limits read; absolute or percentage form chosen deliberately systemd.resource-control(5)
Raise a parent slice's limit EffectiveMemoryMax= shows the slice is the binding limit; other services in the slice reviewed systemd.resource-control(5)
Add VM memory Kill was system-wide, or the binding limit is a percentage Capacity planning; recheck effective limits
Change OOMScoreAdjust= or OOMPolicy= Recorded reason; which process the kill moves to; restart behaviour reviewed systemd.exec(5), systemd.service(5)
Change ManagedOOM* settings Monitored cgroups from oomctl; descendants that could be selected systemd-oomd.service(8), systemd.resource-control(5)

What not to do

  • Do not set every important process to -1000. The kill moves to whatever is left, and if nothing can be killed the system cannot recover.
  • Do not change vm.overcommit_memory to fix one service; mode 2 can make allocations fail, and the application must handle that.
  • Do not set vm.panic_on_oom unless panic and failover is the design ("for failover of clustering").
  • Do not set min_free_kbytes high to keep memory free. The kernel documentation warns: "Setting this too high will OOM your machine instantly."
  • Do not set memory.min protections that add up to more than the VM's memory.

Limitations

  • Untested: no command was run for this article; lab validation is outstanding.
  • oomd.conf(5) thresholds and proc_pid_oom_score(5) are not covered.
  • Hypervisor ballooning, swap sizing, cgroup v1, systems without systemd and container runtimes are not covered.
  • An observation window without a kill is evidence for that workload and window only.

Acceptance checklist

  • [ ] For each kill, the kernel record or the systemd-oomd record is saved with boot, time, process and cgroup; where neither exists, the incident is marked unresolved.
  • [ ] The constraint is classed as unit limit, parent limit, cpuset, memory policy, system-wide or systemd-oomd, with the supporting counter changes.
  • [ ] Unit and parent limits, effective limits, OOMPolicy= and protections are recorded before and after any change, noting absolute or percentage form.
  • [ ] Any limit change uses a measured value, with MemoryHigh= as the main control.
  • [ ] No process was set to -1000 without a recorded reason.
  • [ ] After the change, memory.events.local, memory.peak and both journals were read over a stated window and workload; the result is reported for that window only.

Sources

Displayed dates as shown when accessed on 30 October 2026.

Related articles

Learning notes

  1. Start from the kill record for the right boot: kernel log or systemd-oomd journal.
  2. Four kernel contexts: cpuset, memory policy, memory limit (own or parent), whole system.
  3. memory.events is hierarchical and counts any OOM kill; use memory.events.local and changes over time.
  4. Effective limits include parent slices; percentage limits grow with VM memory, absolute ones do not.
  5. systemd-oomd monitors configured units but kills eligible descendants; record which cgroup died.