Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Docker no space left on device: measure /var/lib/docker before you prune

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

When Docker fails with no space left on device, the filesystem that holds Docker's data has no free blocks (or no free inodes). Find what is using the space before you remove anything. The culprit can be a single container log file, not images. docker system prune is safe only if you know what it will remove, and the volume options can delete application data.

What the error means

The text No space left on device is the Linux error ENOSPC (the Linux man-pages errno page defines it that way). Docker reports it from whatever operation failed: pulling an image layer, building, starting a container, or writing a log.

Where the data lives, per Docker's daemon documentation: the daemon persists its data in a single directory, by default /var/lib/docker on Linux. When the containerd image store is used (the page says it is the default for Docker Engine 29.0 and later on fresh installations), image contents and container snapshots are stored in /var/lib/containerd, while volumes and configs stay in /var/lib/docker. With the classic storage drivers such as overlay2 (the default for upgraded installations), everything is in /var/lib/docker. So measure both directories' filesystems on a newer host. If you moved the directory with the data-root option, measure that path instead; docker info prints it as Docker Root Dir.

Container logs can be a cause. Docker's logging page says that by default no log rotation is performed, so the log files of the default json-file driver can use a significant amount of disk space for containers that generate much output. The json-file page says it writes each container's output to files on the host. Its max-size option defaults to -1, which is unlimited, and max-file defaults to 1. A container that logs heavily and never rotates grows without bound.

Likely causes

Cause What you see How to confirm
One container's JSON log has grown very large Free space falls steadily while image counts stay flat The log path from docker inspect is many gigabytes in ls -lh
Unused images and old layers A long image history after many deploys docker system df shows a large reclaimable figure for images
Build cache A CI or build host with many builds docker system df -v and the build cache row, if the listing shows one
Stopped containers with large writable layers Many Exited containers docker system df -v lists containers with a SIZE column
Volumes holding real data The volume figure is large and its links count is above 0 docker system df -v shows volume LINKS and SIZE
Inodes exhausted, not bytes df -h shows free space but no space left continues df -i shows 100% inode use
Something outside Docker fills the same filesystem docker system df totals are small du on the filesystem root shows another large directory

Prove the cause before changing anything

All read-only.

1. Where does Docker keep its data?

docker info

Find Docker Root Dir (the reference output shows /var/lib/docker) and the storage driver line.

2. How full is the filesystem?

df -h /var/lib/docker
df -i /var/lib/docker

df -h prints sizes in powers of 1024 in a readable form. df -i reports inode information instead of block use; if the inode percentage is high while bytes are free, deleting large files will not help and the cause is many small files.

3. What does Docker think it uses?

docker system df
docker system df -v

The Docker reference says the command displays the amount of disk space used by the Docker daemon, and the example output lists TYPE, TOTAL, ACTIVE, SIZE and RECLAIMABLE for images, containers and local volumes. The verbose form lists each image with SHARED SIZE and UNIQUE SIZE, each container with its size, and each volume with its LINKS and SIZE. The page notes that network information is not shown because networks do not consume disk space.

4. Find large container logs. For each suspect container (the verbose listing names them):

docker inspect --format='{{.LogPath}}' CONTAINERNAME

The docker inspect page shows --format='{{.LogPath}}' as the way to print a container's log path. Replace CONTAINERNAME with a name from docker ps -a. Then run ls -lh on the exact path it printed. A file of tens of gigabytes explains a full disk.

5. If Docker totals are small, look outside Docker.

du -h -x -d 1 /

The du manual defines -x as skipping directories on different file systems and -d as the depth.

Fix

Work from the least risky action.

1. Make logs rotate, then deal with the existing file. The json-file page gives this daemon configuration for rotation:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

It says log options must be strings, so numbers are in quotes, and that you must restart Docker for the change to take effect for newly created containers; existing containers do not use the new configuration automatically. A per-container alternative is docker run --log-driver json-file --log-opt max-size=10m --log-opt max-file=3. The logging page also recommends the local logging driver for other situations because it performs log rotation by default; to see the daemon's current default driver, run docker info --format '{{.LoggingDriver}}', which the page shows printing json-file.

Save the current file first:

cp -p /etc/docker/daemon.json /etc/docker/daemon.json.2026-11-20.bak

Then add the keys above to /etc/docker/daemon.json, merging them with any keys already there (the dockerd reference gives /etc/docker/daemon.json as the default location on Linux). If the file does not exist, the copy above fails and there is nothing to save; create the file with the JSON above and note that no earlier file existed, because that changes the back-out below. Then check it:

dockerd --validate --config-file /etc/docker/daemon.json

The dockerd reference describes --validate as checking a configuration file without starting the daemon, and shows configuration OK for a valid file.

Disruptive step: restarting the Docker daemon. The Docker documentation for exit status 137 lists a daemon restart as an event that kills all running containers (unless live restore is enabled, which the dockerd reference describes as an option to keep containers running). Plan it in a window. Back-out: copy /etc/docker/daemon.json.2026-11-20.bak back over /etc/docker/daemon.json, run the validate command again, then restart the daemon. If /etc/docker/daemon.json did not exist before you created it, there is no .bak file; to undo, remove the new /etc/docker/daemon.json (it holds only the keys you added) and then restart the daemon.

The json-file page warns that these files are designed to be accessed exclusively by the Docker daemon and that touching them with external tools may interfere with logging. So do not truncate or delete a live log file by hand. Recreate the container so it picks up the rotation settings; the old log goes away with the old container. Select the exact container and save its settings first:

docker ps -a
docker inspect CONTAINERNAME > /root/CONTAINERNAME-inspect-2026-11-20.json

Risk: removing the old container deletes its writable layer and its log file, and they cannot be recovered. Anything the application stored inside the container, rather than in a volume or bind mount, is lost. Check the Mounts section of the saved file to see what is kept outside the container. The saved file is JSON and cannot be passed to docker run, so before you remove anything, write down your docker run command or Compose definition and compare it with the Mounts, Env and HostConfig sections of the saved file; if you cannot rebuild the container from them, do not remove it. Safer order: run docker stop CONTAINERNAME and leave the old container in place, stopped and not removed; create the new container under a different name from the rebuilt definition; remove the old one only after the service has run well for a while. Back-out while the old container still exists: stop the new container and run docker start CONTAINERNAME. Once the old container is removed, its writable layer and log cannot be recovered; recreate the container from the definition you rebuilt.

2. Prune what you can name. The Docker reference for docker system prune says it removes all stopped containers, all networks not used by at least one container, all dangling images and unused build cache; with -a it removes all unused images, and with --volumes it prunes anonymous volumes as well. Confirm the object classes before running anything:

docker system df -v

Narrower commands remove less: docker container prune (stopped containers), docker image prune (dangling images by default), and docker builder prune (build cache, with --filter until=24h to keep recent cache, and --keep-storage to keep an amount). Build cache can be rebuilt, so removing it mainly slows the next build.

Risk: pruning deletes containers, images or cache, and a deleted object cannot be recovered. A stopped container may be one you meant to restart, and a removed image must be pulled or built again, which needs registry access and may hit pull limits. Back-out: none for the deleted objects; pull or rebuild images from their sources. Run the narrowest command that covers the objects you listed, and answer the confirmation prompt only after reading it. Do not use -f, which skips the prompt.

docker builder prune --filter until=24h

3. Volumes need a person, not a command. docker volume prune by default removes only anonymous volumes, and with -a removes all unused volumes. An unused volume is one no container references, so a database volume whose container is stopped counts as unused. The docker system prune page says volumes are not removed by default, to prevent important data from being deleted if no container is currently using the volume. Take a backup of any volume you are not certain about and read Backing up a Docker host: volumes, bind mounts and what an image does not hold first.

4. Add space or move the data directory. The daemon page describes the data-root option for a different storage directory, and notes that it does not move containerd's image data. Treat that as a planned migration, not an emergency step.

Example (illustrative): on app02, deployments fail with no space left on device. df -h /var/lib/docker shows a 40 GB filesystem at 100%. docker system df shows images of 6 GB with most reclaimable. docker inspect --format='{{.LogPath}}' shop-api prints a path whose ls -lh shows 29 GB, which is 29 ÷ 40 = 72.5% of the filesystem. The service had no max-size. The team validates a daemon.json with the rotation settings, restarts Docker in a maintenance window, and recreates shop-api. Then they run docker builder prune --filter until=24h to recover build cache. No volumes are touched.

Confirm the fix

Run df -h /var/lib/docker again and confirm free space with margin. Run docker system df and compare the figures. Start the container or pull the image that failed. After a rotation change, create a container and check its log directory stays within max-size times max-file (10 MB × 3 = 30 MB in the example above, plus a small overhead).

What to watch afterwards

  • Free space and inode use on the filesystem that holds Docker's data, with an alert well before 100%.
  • docker system df trends, especially the reclaimable figure.
  • The largest container log files.
  • Build hosts: schedule build cache pruning with a time filter.

Risks and limits

The behaviour and defaults are from the Docker documentation as read on 20 November 2026 and may change with Docker versions. Pruning is a deletion that cannot be undone, and volume pruning can remove application data. Rotating logs does not recover space from existing large files. Docker's documentation does not list the causes of a full disk; the "what you see" cells in the causes table are general practice, and each one is confirmed by the command named beside it. These commands have not been run or lab-tested.

Sources