Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

/boot is full and the kernel update will not install: remove old kernels the safe way

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

When a kernel update fails because /boot is full, one cause is a pile of old kernels and their boot images that were never removed. The failing step can be the boot image build, which reports "No space left on device" (the error ENOSPC). Free the space by removing old kernel packages with the package manager, keep the one that is running, and then finish the interrupted install. Do not delete files from /boot by hand.

What the symptom means

On Debian and Ubuntu, installing a kernel package triggers update-initramfs, which builds the initial boot image and writes it to /boot. The update-initramfs manual says the script "keeps track of the existing initramfs archives in /boot". The script in the Ubuntu packaging tree writes the new image to a file named initrd.img-<version>.new, moves it into place on success, and prints "update-initramfs: failed for" followed by the file name and a return code on failure. It also keeps a backup of the previous image under names ending .dpkg-bak and .bak, made as a hard link where possible and otherwise as a copy, so /boot can hold a second copy of an older image.

The errno(3) manual gives the text: "ENOSPC No space left on device". If the build runs out of room the package's post-install step fails, and apt or dpkg reports the package as not configured.

/boot can be a separate small partition, so the root filesystem can have plenty of free space while /boot has none. That is why the general "no space left" checks on the root filesystem miss it. "Linux Says No Space Left but Bytes Are Free: Check Inodes and Quotas" covers the case where the wrong mount is being checked.

How kernels are kept and removed

apt. The apt-get manual says autoremove "is used to remove packages that were automatically installed to satisfy dependencies for other packages and are now no longer needed", and autopurge is "a shortcut for autoremove --purge". The apt.conf manual (apt 3.3.8) says APT::Protect-Kernels "tells apt autoremove that kernels are protected and defaults to true", and that APT::NeverAutoRemove::KernelCount "defaults to 2, meaning two kernels are kept. Apt will always keep the running kernel and the latest one." Older apt releases may not have these options, so read your own manual page. The apt source's shipped configuration lists versioned kernel packages (linux-.*) so that autoremove can recognise them.

DNF. The DNF configuration reference documents installonly_limit: "Number of installonly packages allowed to be installed concurrently. Defaults to 3. The minimal number of installonly packages is 2." installonlypkgs lists packages that are only ever installed, never upgraded: "Kernels in particular fall into this category". The command reference documents dnf remove --oldinstallonly: "Removes old installonly packages, keeping only latest versions and version of running kernel."

Likely causes

Cause What you see How to confirm
Many old kernels accumulated /boot at or near 100 percent; several linux-image versions or kernel RPMs installed df -h /boot and the package list in the steps that follow
/boot too small for the number of kernels kept Fills again within months of cleaning Size of /boot against the size of one kernel set
Old kernels never autoremoved Debian-family: no autoremove has been run; kernels marked as manually installed apt-get -s autoremove lists nothing, yet old kernels remain
Interrupted install left the package unconfigured apt says a package is not configured; dpkg --configure -a message The error text from apt
Leftover .bak or .new boot images Space used by files that no package lists as current Directory listing of /boot. Do not delete them by hand; the manuals give no procedure for clearing them, so ask someone who can check each file against the packages
Inodes exhausted instead of bytes "No space left on device" with free blocks df -i /boot. For the general inode and quota checks see "Linux Says No Space Left but Bytes Are Free: Check Inodes and Quotas"

Prove the cause before changing anything

  1. See which filesystem holds /boot and how full it is. The df manual says -h prints human-readable sizes, and -i lists inode information instead of block usage:
df -h /boot
df -i /boot
  1. Note the running kernel, which must never be removed. The uname manual says -r prints the kernel release:
uname -r
  1. List the installed kernels. Debian-family: the dpkg-query manual says -W lists packages matching a pattern, with the name and installed version separated by a tab:
dpkg-query -W 'linux-image-*'

The pattern also matches meta-packages such as linux-image-generic, so read the list and ignore entries that are not a kernel version. For the status of each entry, list them with dpkg-query -l 'linux-image-*'. The dpkg-query manual says the first three columns show the desired action, the package status (i installed, c config-files only, H half-installed and so on) and errors. The dpkg(1) manual defines config-files as "Only the configuration files or the postrm script and the data it needs to remove of the package exist on the system"; the ls -l /boot comparison in a later step shows what remains in /boot.

Red Hat-family: the DNF command reference documents repoquery --installonly for installed installonly packages:

dnf repoquery --installonly
  1. See what apt would remove, without removing anything. The apt-get manual describes -s as a simulation of what would happen, with no change to the system:
sudo apt-get -s autoremove
  1. Read the directory with ls -l /boot and compare file sizes with the version list. This is a look only.

Example (illustrative): web01 has a 512 MiB /boot partition. uname -r prints 6.8.0-45-generic. dpkg-query -W 'linux-image-*' lists five kernel versions. In this example each version's files take about 95 MiB, so 5 × 95 = 475 MiB is used and 512 − 475 = 37 MiB is free. A new kernel needs about 95 MiB, which is more than 37 MiB, so the install fails with "No space left on device" and the package is left unconfigured. apt-get -s autoremove lists three of the older versions as removable. The made-up sizes show the arithmetic only. Use your own numbers.

Fix

Before any kernel is removed

Take a VM snapshot or backup of the server first, because a removed kernel may not be installable again. To back out, revert to the snapshot or restore the backup.

Then keep a second bootable kernel besides the running one (uname -r). Check with ls -l /boot that it has both a kernel image and a matching boot image (file names differ between distributions), and that the bootloader menu offers it. A kernel whose install was interrupted is not a fallback. If no complete second kernel exists, remove nothing.

Debian-family

First free some space by removing old kernels, never the running one.

Disruptive step: apt-get autoremove --purge removes and purges every automatically installed package that is no longer needed, not only kernels. Read the whole list from the simulation in step 4, not only the kernel lines, and run the command only if every package on it is one you expect to lose.

sudo apt-get autoremove --purge

The apt-get manual describes --purge as using purge instead of remove, which also removes configuration files. Back-out: the snapshot or backup, or installing the package again from a repository that still offers that version, which also needs free space in /boot. Keep the running kernel and the newest one, as apt does by default.

If the old kernels are not listed (autoremove keeps what it considers needed), name one explicitly from the list you made, other than the running kernel and the newest.

Disruptive step: apt-get remove --purge removes the named kernel package and its configuration files. Back-out: the snapshot or backup.

sudo apt-get remove --purge linux-image-<old-version>

If apt itself refuses to run because of the half-installed package, apt prints the command it needs. The apt source's wording is "dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem." Space must be freed before that command can succeed. The dpkg manual documents -r, --remove ("removes everything except conffiles") and -P, --purge ("removes everything, including conffiles"), which work at the dpkg level and do not need apt.

Disruptive step: dpkg --remove on a kernel image package removes it without any of apt's checks. The apt.conf manual documents the running-kernel and latest-kernel protection for apt autoremove only, and the dpkg manual documents no equivalent, so check the package name against uname -r yourself. Back-out: the snapshot or backup.

sudo dpkg --remove linux-image-<old-version>

After freeing space, finish the interrupted install.

Disruptive step: dpkg --configure -a runs the post-install scripts of the unpacked packages, including the kernel's, and can fail again with "No space left on device", which leaves the package half-configured. The dpkg manual says it configures "all unpacked but unconfigured packages". Re-check df -h /boot first. There is no back-out for a configuration that completes; if it fails, free more space and run it again.

sudo dpkg --configure -a

Risk: apt-get -f install can add or remove other packages to repair dependencies, and the apt-get manual describes -f or --fix-broken as an attempt "to correct a system with broken dependencies in place". Read the plan it prints before you accept. Back out by declining at the prompt; once accepted, removed packages need reinstalling from the repository.

sudo apt-get -f install

Red Hat-family

List the installed kernels, then preview the removal with --assumeno ("Automatically answer no for all questions"); both are in the command reference. Compare the preview with uname -r and your second kernel. Neither command removes anything:

dnf repoquery --installonly
sudo dnf --assumeno remove --oldinstallonly

Disruptive step: sudo dnf remove --oldinstallonly removes old kernels and keeps the latest versions and the running kernel, according to the command reference. Back-out: the snapshot or backup. Reinstalling needs the repository to still offer that version and free space in /boot.

To stop it recurring, set installonly_limit in dnf.conf. The documented minimum is 2, and the reference explains that 1 is not allowed "since it complicates kernel upgrades due to protection of the running kernel from removal".

Risk: installonly_limit in dnf.conf changes how many kernels DNF keeps at the next kernel update; a lower value removes older kernels sooner and leaves fewer to boot into. Back out by restoring the previous value, which does not bring back a kernel that has already been removed.

What not to do

Do not delete files in /boot by hand to make room. Those files belong to packages, and the package database will still believe they exist. Removing the wrong image can leave the next boot without a usable kernel. If /boot is so full that even removing a package fails, mount-level repairs belong to someone who can reach the console; take a snapshot or backup first.

Make /boot bigger

If /boot is too small for your kernel policy, the lasting fix is a larger partition, which is a storage change and needs its own plan and backup. It is outside what a package command can do.

Confirm the fix

  1. df -h /boot shows comfortable free space. Leave room for at least two new kernel images, measured from your own system.
  2. Re-run the failed update. It completes without "No space left on device".
  3. dpkg-query -W 'linux-image-*' (or the DNF equivalent) lists the kernels you intended to keep, including the running one.
  4. Check the boot menu or bootloader entries match the kernels that remain. Do this before the next reboot, in a maintenance window.

What to watch afterwards

  • Alert on /boot usage, not only on the root filesystem.
  • Run autoremove (Debian-family) or set installonly_limit (Red Hat-family) so the count stays bounded.
  • After each kernel update, note the free space left in /boot and compare with the size of one kernel set.
  • Reboot into new kernels on a schedule, so you are not forced to keep several old ones.

Risks and limits

  • These commands have not been run or lab-tested. They come from the manual pages, documentation and packaging sources in the Sources list.
  • The update-initramfs behaviour (the .new, .dpkg-bak and .bak names and the failure message) is read from the Ubuntu devel packaging tree. Other releases may differ.
  • The KernelCount and Protect-Kernels options come from the apt 3.3.8 manual page and may not exist in your apt.
  • Removing a kernel while something still depends on it can break out-of-tree modules. Read the simulation output before you accept removals.
  • /boot problems on systems using other layouts (an EFI system partition, separate encrypted /boot) are not covered.

Sources