Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

"Could not get lock /var/lib/dpkg/lock-frontend": find the process holding it

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

The message "Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 2312 (unattended-upgr)" means another process already holds the package system's lock. On Debian and Ubuntu servers the holder can be the scheduled unattended upgrade or a second administrator's session. Wait for it to finish, or stop it cleanly. Do not delete the lock file.

apt itself says so. After the "Could not get lock" error it prints a notice: "Be aware that removing the lock file is not a solution and may break your system."

What the error means

apt and dpkg serialise package changes with locks, so two tools never edit the package database together. The apt source shows how the lock works. It opens the lock file (creating it if missing) and then asks the kernel for a write lock with fcntl and F_SETLK. If that fails, it asks the kernel who holds the lock and prints the process number. The name in brackets is read from the Name: line of /proc/<pid>/status.

The Linux man-pages page for F_SETLK states that "record locks are automatically released when the process terminates". Two consequences follow:

  • The lock file existing on disk does not mean it is held. The file is created by the attempt to lock it, and is normal to find.
  • A lock is never "stale" in the way a leftover pid file is. If apt reports a holder, a live process has it. If no process holds it, the next apt run succeeds.

apt takes two locks. The error text for each is in the apt source:

Message Meaning
"Could not get lock /var/lib/dpkg/lock-frontend. It is held by process N (name)" The front-end lock is held by process N. The path is the directory of the dpkg status file plus lock-frontend, so yours may differ on an unusual layout.
"Unable to acquire the dpkg frontend lock (...), is another process using it?" The follow-up message for the same condition when the front-end lock cannot be taken.
"Unable to lock the administration directory (...), is another process using it?" The same check on the second lock, named lock.
"Unable to acquire the dpkg frontend lock (...), are you root?" The lock could not be taken for another reason, which the message ties to privileges.

Why unattended upgrades hold it

Debian's apt source ships two timers. apt-daily.timer runs the "Daily apt download activities" at 06:00 and 18:00 with a random delay of up to 12 hours. apt-daily-upgrade.timer runs the "Daily apt upgrade and clean activities" at 06:00 with a random delay of up to 60 minutes. Both set Persistent=true. Your distribution may change these schedules.

The Ubuntu Server documentation describes the effect on a machine that was off: "the timer may be triggered immediately at the next startup", so unattended upgrades "may often run on system startup and thereby cause immediate activity and prevent other package operations from taking place at that time". A VM that has just been powered on or restored can meet this. The same page says the run is "still subject to the RandomizedDelaySec value", so it may start some time after boot and not at once.

Likely causes

Cause What you see How to confirm
Unattended upgrade or daily apt job running Error appears soon after boot or in the early morning; holder name is unattended-upgr, apt, apt-get or dpkg Holder name; /var/log/unattended-upgrades/unattended-upgrades.log still being written
A second administrator or tool running apt or dpkg Holder is an interactive apt, apt-get, dpkg or a configuration tool fuser -v on the lock file shows the user and command
Long download or install in progress Same holder, but it is making progress Log or terminal of the holder keeps moving
Holder is stuck Same process number for a long time with no log activity Process age and log checks in the steps that follow
Not privileged "are you root?" variant id; use sudo
Earlier run was interrupted apt says "dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem." (the apt source puts sudo in front of the command when run through sudo) The message itself

Prove the cause before changing anything

These checks only read.

  1. Read the error. It names the process number and, when it can, the process name.
  2. Ask which processes use the lock. The psmisc fuser manual says fuser "displays the PIDs of processes using the specified files", and -v shows them "in a ps-like style" with user and command:
sudo fuser -v /var/lib/dpkg/lock-frontend

Use the path from your own error message.

  1. Look at the unattended-upgrades log. The unattended-upgrades manual says all operations are logged to /var/log/unattended-upgrades/unattended-upgrades.log and dpkg's output to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log. If either is changing, the holder is working.
  2. See whether the timers fired recently:
systemctl list-timers 'apt-daily*'

The systemctl manual says list-timers lists timer units "ordered by the time they elapse next", and accepts patterns.

  1. Read the units' journal for the activity: journalctl -u apt-daily-upgrade.service (the unit name is from the apt source; -u is documented in journalctl(1)).
  2. Check how long the holder has been running. The ps(1) manual documents -p (select by process ID), -o (choose output columns) and the etime column ("elapsed time since the process was started"):
ps -o pid,etime,cmd -p 2312

Use the process number from your own error message. A holder that has run for a long time and has added nothing to either log is the stuck case in the table above. A holder with a short run time and a growing log is working.

Example (illustrative): On web01, an administrator logs in 4 minutes after the VM boots and runs sudo apt-get install nginx. apt prints "Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 2312 (unattended-upgr)". sudo fuser -v /var/lib/dpkg/lock-frontend shows root running unattended-upgr. The log file's last line is 40 seconds old and the dpkg log is growing. The VM was off overnight, so Persistent=true ran the missed upgrade at startup. The administrator waits, and tries again 6 minutes later, when the log ends with a completion line.

Fix

Waiting, with a time limit

Do nothing and retry once the holder exits. To let apt wait for you, the apt source reads a setting named DPkg::Lock::Timeout (default 0, no wait). Passing it on the command line makes apt show "Waiting for cache lock:" and retry once a second, up to that many seconds:

sudo apt-get -o DPkg::Lock::Timeout=300 install nginx

This option is documented in the source, not in the apt manual pages. In the source, the wait returns at once when the timeout is 0 or when no progress display is available, so behaviour in non-interactive runs has not been verified. Check your version's behaviour with a harmless command first.

Stopping the scheduled job

If the upgrade must stop, prefer the service manager to a signal. The apt source defines apt-daily-upgrade.service as a one-shot unit with TimeoutStopSec=900 and KillMode=process.

Disruptive step: sudo systemctl stop apt-daily-upgrade.service stops the unit, but with KillMode=process the systemd.kill manual says "only the main process itself is killed". The unit's ExecStart is /usr/lib/apt/apt.systemd.daily install, a script that starts unattended-upgrade, so the unit's main process is that script and unattended-upgrade runs as its child. The stop may therefore leave the process holding the lock running, and the lock may still be held afterwards. The same manual describes process as "not recommended!". The stop cannot be undone, and it may interrupt the unit's own script between steps. After the stop, run sudo fuser -v /var/lib/dpkg/lock-frontend again. If a process is still listed, it is still working: wait for it to finish, and use the log files above to watch it. If it has stopped writing to both logs for a long time (see the process age check above), do not signal it without reading the log first.

The unattended-upgrade manual documents that --minimal-upgrade-steps is the default and that it can be cancelled with SIGINT. No command for that is given here, because it stops a running dpkg part-way, and the only recovery is the dpkg --configure -a step that follows.

If an upgrade was interrupted, either by the stop or by another cause, apt tells you what to run.

Before running it, take a VM snapshot or backup of the server, because the command changes the package database and runs package scripts. Then confirm that no process holds the lock:

sudo fuser -v /var/lib/dpkg/lock-frontend

An empty result means nothing holds the lock. Back out of a failed run by reverting to the snapshot or restoring the backup.

Disruptive step: sudo dpkg --configure -a runs the maintainer scripts of every unpacked but unconfigured package, and what a script does is up to the package, so it may change or restart services. The dpkg manual describes --configure with -a or --pending: "all unpacked but unconfigured packages are configured". There is no back-out: configuring a package cannot be reversed by this command. If it fails, the package stays unconfigured and the error names the script that failed. Run it in a maintenance window for a production server. Then re-run the apt command, and let the next scheduled upgrade complete the rest.

sudo dpkg --configure -a

Stopping it recurring after power-on

The Ubuntu Server documentation shows how to stop the timer catching up after a shutdown: run systemctl edit on apt-daily-upgrade.timer and apt-daily.timer and add

[Timer]
Persistent=false

so that "the timer will trigger the service only on the next scheduled time".

Risk: this is a configuration change that the system manager reloads when systemctl edit finishes. The impact is that a machine that was off at the scheduled time no longer catches up on startup, so its updates wait for the next scheduled slot. To back out, run systemctl edit on each timer again and remove the Persistent=false lines, or run systemctl revert on the timer (the systemctl(1) manual says it removes the unit's drop-ins).

What not to do

Do not delete lock-frontend or lock. apt's own message says it is not a solution and may break the system. The lock is held by a running process, not by the file's presence, so deleting the file does not make the holder stop.

Confirm the fix

  1. sudo fuser -v /var/lib/dpkg/lock-frontend lists no process.
  2. Re-run the apt command. It starts without the lock error.
  3. If you stopped a job, dpkg --configure -a finishes with no errors, and apt-get has nothing broken left to fix.
  4. Run the dpkg package checks from "Checking a restored Linux server against its package records: rpm --verify, dpkg --verify and debsums" if you suspect an interrupted run damaged files.

What to watch afterwards

  • Run package changes on a schedule that avoids the 06:00 to about 07:00 window if the distribution's default timers apply.
  • After powering on or restoring a VM, give the first upgrade run time to finish before manual changes.
  • Use the DPkg::Lock::Timeout setting in automation, so a configuration-management run waits, instead of failing on the lock.
  • Keep an eye on the unattended-upgrades log for repeated failures; a run that always fails keeps starting.

Risks and limits

  • These commands have not been run or lab-tested. They come from the manual pages, apt source and Ubuntu documentation in the Sources list.
  • DPkg::Lock::Timeout and the message wording come from the apt source main branch (changelog head 3.1.12). Older apt releases may print slightly different text or lack the option.
  • Timer times and delays come from Debian's apt source; Ubuntu or your own drop-ins may differ. Read systemctl list-timers on the host.
  • The name in brackets is only the process name. It does not tell you whether the process is healthy.
  • Stopping an upgrade can leave packages half-configured. Take a snapshot or backup before intervening on a production server when time allows.
  • This covers Debian-family systems using apt and dpkg. Red Hat-family systems use different tools and messages.

Sources