Australian cloud for business software — built and run in Sydney since 2003

+61 2 8282 5656
support@netleverage.com
Talk to an engineer
Talk to an engineer
NetLeverage
NetLeverage

Getting into a Linux server you cannot log in to: rescue targets, rd.break and boot loader passwords

Technical article
Backup & Recovery

Originally published:

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Recovering a server's data is only half of a recovery. Someone also has to be able to log in to it. Access can fail even when the restored data is intact. The restored system may come from an image whose root password nobody recorded. The admin accounts may depend on a directory service that is not reachable yet. An account may have expired, or a changed disk layout may leave an entry in the filesystem table that stops the boot before any login prompt appears.

Linux has several documented ways to get a shell from the console in these situations. They differ in what they need, how much of the system they start, and how much they bypass. This guide explains the options as the documentation describes them, then covers the other side: anyone with console access can use the same features, so protecting them is part of running a server you may need to recover.

This is recovery-access planning for systems you are authorised to administer, not a password-reset procedure.

Why it matters

A lock-out tends to happen at the worst moment: during an outage, after a restore, with people waiting. Teams that have rehearsed console access know which option they need and how long it takes. Teams that have not are left searching for instructions on a console with no copy and paste, often through a remote management interface.

There is also a security dimension that is easy to miss. Several of the methods below give a root shell without any password. If a server's console can be reached by people who should not have root, through an unprotected management interface, a shared rack or a virtual machine console, those methods are open to them as well.

How it works

The levels of a systemd boot

The bootup(7) manual and systemd.special(7) describe a boot as reaching a series of targets. Two are relevant here.

rescue.target "pulls in the base system (including system mounts) and spawns a rescue shell", for administering "the system in single-user mode with all file systems mounted but with no services running, except for the most basic." The manual gives the kernel command line option "systemd.unit=rescue.target" and its short alias "1".

emergency.target "starts an emergency shell on the main console" and "does not pull in other services or mounts. It is the most minimal version of starting the system in order to acquire an interactive shell." It is selected with emergency on the kernel command line, and is also used automatically "when a file system check on a required file system fails and boot-up cannot continue." The manual adds that "depending on how emergency.target is reached, the root file system might be mounted read-only or read-write".

The kernel-command-line(7) manual lists emergency, rescue, single and their rd. forms as "compatibility and convenience options".

Both rescue and emergency mode normally ask for the root password. They use sulogin, which the sulogin(8) manual describes as the program that prompts: "Give root password for system maintenance". So these modes are the right tools when the root password is known but normal logins fail: a broken directory login, a misbehaving service, a bad filesystem table entry. They do not help when the root password itself is unknown.

When the root account is locked

The sulogin manual explains a detail that affects many modern installations, where the root account is locked and administrators use sudo: "If the root account is locked and --force is specified, no password is required." Its -e (--force) option starts a root shell without a password when the password files are damaged or the root account is locked, and the manual warns: "Only use the -e option if you are sure the console is physically protected against unauthorized access."

How a given distribution configures this varies. On a system with a locked root account, rescue and emergency modes may refuse entry entirely, or may let anyone in. Find out which before you need it.

rd.break: stopping in the initramfs

Before the real root filesystem takes over, most distributions boot into a small initial RAM filesystem (initramfs). On systems whose initramfs is built with dracut, the dracut.cmdline(7) manual documents rd.break, which will "drop to a shell at the end", and rd.break= with named breakpoints, which drops "to a shell before the defined breakpoint starts". It also documents rd.shell, which allows "dropping to a shell, if root mounting fails".

This shell runs before the installed system's login rules apply, which is why distribution documentation uses it for root password recovery. Distribution procedures typically involve remounting the real root filesystem, changing the password, and, on systems with SELinux, ensuring files are relabelled afterwards. That last step matters: a password file changed from the initramfs can end up with the wrong SELinux label, and the next boot then fails to accept logins. Follow your distribution's documented procedure rather than improvising, because the details differ between releases.

The initramfs shell cannot read an encrypted root filesystem until it is unlocked, so disk encryption remains a barrier to this route.

Debug shells

The systemd-debug-generator manual describes systemd.debug_shell and rd.systemd.debug_shell: "the debug shell service 'debug-shell.service' is pulled into the boot transaction and a debug shell will be spawned during early boot. By default, /dev/tty9 is used". It also notes that "the shell may also be turned on persistently by enabling it with systemctl".

A debug shell is useful for diagnosing a boot that hangs. It is also a root shell on a console. Never leave it enabled persistently on a production server.

The common factor: the kernel command line

Kernel command-line settings are one route to these modes, often through boot-menu editing. Some can also be activated by services or reached automatically. Available shells depend on the installed boot stack, image contents and security policy; an editable command line does not guarantee every method exists. That is the point to protect.

Matching the cause to the route

The routes above are not interchangeable. Choosing the least invasive one that fits the cause keeps the system closer to its normal state and leaves fewer things to undo. The following pairings are our reading of the documentation, not a distribution procedure.

  • The boot stops before a login prompt and drops to an emergency shell by itself. The emergency.target description covers this case: a required filesystem failed its check or could not be mounted. After a disk replacement or a restore onto different hardware, a filesystem table entry that names a device identifier which no longer exists is a typical cause. The root password, if known, gets you in, and the fix is to the configuration, not the accounts.
  • The system boots, but nobody can log in. If administrators normally authenticate through a directory service, the restored server may not reach it yet, or its machine account may be out of step with the directory. A local break-glass account avoids this entirely. Without one, evaluate the supported console recovery route and its authentication requirements before changing accounts.
  • The root password is known, but the root account is locked. This is the sulogin case described above. Behaviour depends on how the distribution invokes sulogin, so check it in advance.
  • Nobody knows the root password and no local administrator exists. This is the case for an initramfs break and the distribution's documented password recovery. It is also the case most often caused by restoring an old image whose credentials were never recorded.
  • The boot hangs without dropping to any shell. A debug shell on another console lets you look at what is waiting, then should be turned off again.

In every case, record what you found and what you changed. Lock-outs after restores tend to repeat, because the same image or the same configuration gets restored again next time. Fixing the cause in the source image, or in the build that creates it, is what stops the repeat.

Protecting console access

The protective measures are standard. What matters is that they are applied consistently and that the people who need emergency access still have it. (our recommendations, except where a source is named)

  • Protect the boot loader. GRUB supports password-protected menu editing. Its grub-mkpasswd-pbkdf2 tool will "generate PBKDF2 password hash" for use in GRUB's configuration, so the password itself is not stored in plain text. Configuration details differ between distributions; follow your distribution's documentation. A boot loader password stops casual editing of boot entries at the console.
  • Protect the firmware. A boot loader password does little if someone can boot other media. Set a firmware setup password and limit boot devices, and protect the remote management controller, whose virtual media and console features give the same power remotely.
  • Encrypt disks where the threat model warrants it. Encryption protects data at rest from anyone who boots other media, at the cost of planning for unlock during recovery. Keep a hardware-independent recovery key and usable header backup where the encryption design supports them.
  • Physically and logically restrict the console. Racks, hypervisor consoles and management networks are part of the access boundary.

What to choose and why

Before anything goes wrong

  • Keep break-glass credentials. Record the root password, or a local administrator with sudo rights that does not depend on a directory service, in a protected password vault. Rotate it, record who can retrieve it, and check that it works on restored systems.
  • Record the boot loader and firmware passwords in the same vault. A boot loader password nobody can find turns a minor lock-out into a major one.
  • Know your distribution's behaviour with a locked root account in rescue and emergency mode, and its documented root password recovery procedure.
  • Test console access during restore rehearsals. A restore test that never logs in has not tested the thing you will need first.

On the day

  1. If normal logins fail but the root password is known, use rescue mode, or emergency mode if the boot cannot reach rescue. Look at the journal and the filesystem table first: a missing disk in the filesystem table is a common cause after hardware changes.
  2. If the root password is unknown, follow your distribution's documented recovery procedure. A dracut-based procedure may use an initramfs break such as rd.break; other boot stacks differ. Plan for an SELinux relabel where applicable.
  3. If the system is encrypted, you will need the unlock method before any of this. Restoring an encrypted disk image still requires the original unlock material. A separately usable data backup can support rebuilding on a new encrypted volume; an image copy cannot bypass lost keys.
  4. Afterwards, remove any temporary boot options, change any password that was exposed or reset, and record what happened.

Risks and limits

  • Distribution differences. Initramfs tools other than dracut, such as those used on Debian and Ubuntu, have different options. This article quotes dracut and systemd documentation only.
  • SELinux and similar controls can make a well-intended password reset fail at the next boot unless labels are corrected.
  • Remote consoles add latency, keyboard layout problems and session time-outs. Practise through the interface you will actually use.
  • Nothing here was run. Options are quoted from manual pages for planning, and distribution procedures were not reviewed directly.

Sources

This article was originally published on .