Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Docker container Exited (137): OOMKilled, a stop timeout, or a kill

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

Exited (137) in docker ps -a means the container's main process was ended by a SIGKILL. Docker's documentation says a filter on exit status 137 finds containers that were killed by SIGKILL(9). The signal says how the process died, not why. Memory is one possible cause, but a docker stop that ran out of time, a docker kill, or a Docker daemon restart give the same number. Read State.OOMKilled and the finish time before you raise a memory limit.

What the error means

The Docker reference for docker ps has a section called "Filter by exit signal". It says you can find containers that exited with status 137, meaning SIGKILL(9) killed them, using docker ps -a --filter 'exited=137'. It then lists the events that result in a 137 status:

  • the init process of the container is killed manually;
  • docker kill kills the container;
  • the Docker daemon restarts, which kills all running containers.

The docker stop reference adds a fourth. It says the main process receives SIGTERM, and after a grace period SIGKILL. The --timeout option sets the number of seconds to wait for the container to stop after the first signal; if the container does not exit after the timeout, it is forcibly killed with SIGKILL. If no default is configured for the container, the daemon determines it, which is 10 seconds for Linux containers. A container that ignores SIGTERM therefore ends as 137 after docker stop.

For memory, the Engine API definition of container state describes OOMKilled as whether a process within the container has been killed because it ran out of memory since the container was last started. Two details matter. It says "a process within this container", not necessarily the main one. And the definition covers the time "since the container was last started", so read it together with FinishedAt.

The resource constraints page explains the kernel side: if the kernel cannot free enough memory for important system functions, it throws an out-of-memory exception and starts killing processes, and any process is subject to killing. Docker adjusts the OOM priority of the Docker daemon so it is less likely to be killed, but does not adjust the priority of containers, which makes an individual container more likely to be killed than the daemon.

Likely causes

Cause What you see How to confirm
The container hit its memory limit State.OOMKilled is true, the exit code is 137 docker inspect shows OOMKilled true; docker events shows an oom event; the host kernel log names the process
The host ran out of memory Exit 137; Docker's pages do not say whether State.OOMKilled is set when the host kernel, not a container limit, ends the process The host kernel log (step 6) names the killed process; many containers may die close together
docker stop timed out Exit 137 at a time that matches a stop, deploy or restart, about the stop timeout after it began FinishedAt lines up with a deploy or an operator command; OOMKilled is false
docker kill or kill -9 on the main process Exit 137 with OOMKilled false and no stop timing Shell history, automation logs, or a script that calls docker kill
The Docker daemon restarted Every running container on the host exited 137 at the same moment Many containers share one FinishedAt; a package upgrade or manual service restart occurred then
Memory limit too low for the workload Repeated 137 on a schedule, OOMKilled true docker stats shows the container near its limit before it dies

Prove the cause before changing anything

All of these are read-only.

1. List the 137s.

docker ps -a --filter 'exited=137'

The page shows this exact form and says -a is needed to include stopped containers.

2. Read the container state.

docker inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}} {{.State.FinishedAt}}' CONTAINERNAME

Replace CONTAINERNAME with a name from step 1. The docker inspect page documents --format with a Go template. A result starting with true means a process in the container was killed for lack of memory. A result starting with false and exit code 137 means the kill came from elsewhere: a stop timeout, a kill command, a daemon restart, or a kill by the host kernel (Docker's pages do not say whether the flag is set in that case, so read the kernel log in step 6).

3. Compare finish times across containers. Run step 2 for each container that exited 137. Several containers with the same FinishedAt point to a daemon restart or host event, not one container's memory.

4. See what the daemon recorded. The Docker events reference lists kill and oom among container events and says --since takes Unix timestamps, date strings, or Go duration strings such as 10m or 1h30m:

docker events --since 24h --filter event=oom

If the command keeps waiting for new events after it prints the past ones, stop it with Ctrl+C in your terminal. The events reference says only the last 256 events are returned, so narrow the filters if the list is long. Events are a record of what the daemon saw; an empty result does not prove that no kill happened.

5. Check the container's limit and use. docker inspect CONTAINERNAME prints the whole JSON; in the Engine API the memory limit field is Memory, described as "Memory limit in bytes". For running containers:

docker stats --no-stream

docker stats shows MEM USAGE / LIMIT and MEM % for running containers, and the --no-stream option takes the first sample only. Stopped containers return no data.

6. Check the host. If OOMKilled is true, or OOMKilled is false but many containers died with memory pressure on the host, read the kernel's record using Linux OOM Killer on a VM: Find the Cause, Set Memory Limits.

Fix

If it was a memory kill (OOMKilled true). There are three honest options: give the container more memory, make the application use less, or add host memory. The Docker resource constraints page lists the options: -m or --memory sets the maximum amount of memory the container can use, with a minimum of 6m. --memory-swap sets the total memory plus swap; if --memory and --memory-swap are set to the same value, the container gets no swap. The page also says you should not try to circumvent the safeguards by setting --oom-kill-disable on a container, and that the option should only be used on containers where you have also set --memory.

The method described here applies a new limit by creating the container again with new flags. First save its settings and make sure you can start the old one:

docker inspect CONTAINERNAME > /root/CONTAINERNAME-inspect-2026-11-20.json

Disruptive step: stopping and recreating a container interrupts the service it provides. Stop the old container with docker stop CONTAINERNAME and leave it in place (stopped, not removed) while you create the new one under a different name (the old one keeps its name while it exists) with your usual start command plus the new --memory value. Back-out: stop the new container and run docker start CONTAINERNAME to bring the old container back with its original settings. Remove the old one only after the new one has run for a while.

Raise the limit by an amount you have measured. If docker stats shows steady growth that never levels off, a bigger limit may only delay the kill, which can point to an application memory leak (this is advice, not something the Docker pages state).

If docker stop timed out. The application is not exiting when it receives SIGTERM. Fix the application to handle the signal and shut down within the grace period, or set a longer timeout. The docker stop page documents -t, --timeout per command, --stop-timeout at container creation, and STOPSIGNAL or --stop-signal to change the first signal. Save the container's settings first with the docker inspect CONTAINERNAME > /root/CONTAINERNAME-inspect-2026-11-20.json command shown above. Risk: a longer timeout makes every stop and deploy slower, and a hung process still ends in SIGKILL at the end. Applying --stop-timeout means recreating the container, which interrupts its service, so follow the same steps as above: stop the old container but do not remove it, and create the new one under a different name. Back-out: stop the new container and run docker start CONTAINERNAME to bring the old container back with its original settings.

If docker kill or a script was responsible. Find and correct the automation. The page lists it as a documented cause of 137, not a fault.

If the daemon restarted. Find why: a package upgrade, an automated restart, a crash. Schedule restarts for maintenance windows. The dockerd reference describes a --live-restore option, "Enable live restore of docker when containers are still running"; read the live restore documentation before relying on it, since it has its own limits. As general advice (the Docker pages used here do not state it), a restart policy on the containers can bring them back after the daemon returns.

Example (illustrative): on app03, the container report-worker runs with --memory=512m, which is 512 × 1,048,576 = 536,870,912 bytes. Every night at about 02:30 it shows Exited (137). docker inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}} {{.State.FinishedAt}}' report-worker prints true 137 and a finish time of 02:31. Other containers on app03 are unaffected, so the daemon did not restart. docker stats --no-stream taken during the run shows MEM USAGE / LIMIT close to 512MiB. The job loads a whole report into memory. The team first changes the job to stream its output, which cuts peak memory, and raises the limit to 768m only if testing shows it is still needed.

Confirm the fix

Wait for the next scheduled run of the failing job, or reproduce the load. Check docker ps -a --filter 'exited=137' stays empty for that container, docker inspect shows OOMKilled false, and docker stats --no-stream during the peak shows headroom below the limit. For a stop timeout, run docker stop on a test container in a safe window and confirm the exit code is 0 or the application's own code, not 137.

What to watch afterwards

  • Peak MEM USAGE against LIMIT for containers with limits.
  • docker events for oom events, or your monitoring's equivalent.
  • The exit code and OOMKilled flag for each container that exits.
  • Daemon restarts that coincide with a block of 137 exits.
  • Host memory and swap; see Docker's note that swap is slower than memory but can buffer against running out.

Risks and limits

Exit code 137 is a signal outcome, not a diagnosis. OOMKilled can be true when a non-main process was killed, and it covers the time since the container last started. Increasing a limit can move the kill to the host if the host has no spare memory. Recreating a container loses anything in its writable layer that is not in a volume. These commands have not been run or lab-tested.

Sources