Australian cloud for business software — built and run in Sydney since 2003

+61 2 8282 5656
support@netleverage.com
Talk to an engineer
Talk to an engineer
NetLeverage
NetLeverage

Out-of-band management for bare-metal recovery: virtual media, Redfish and BMC readiness

Technical article
Backup & Recovery

Originally published:

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Recovering a physical server often starts with three tasks: seeing its console, attaching recovery media and telling it to boot from that media. In a server room, that means a monitor, a keyboard and a USB stick. For servers in a data centre, a colocation rack or a branch office, it usually means the server's management controller.

Most server-class hardware includes a baseboard management controller (BMC): a small, separate computer on the motherboard, using dedicated or shared network connectivity and standby power. It can remain available when the host operating system fails, but still depends on power, firmware and a reachable management path. Vendors give their controllers different names, but many now support a common interface, the DMTF Redfish standard, as well as the older IPMI.

This guide explains the controller features that matter for recovery, as the Redfish schemas describe them, what has to be prepared in advance for them to help, and why the controller itself is a sensitive system.

Why it matters

A recovery plan for physical servers that assumes someone can stand in front of the machine has a hidden dependency: travel time, site access and staff availability. A controller that has been set up, tested and secured can reduce that dependency, while hardware faults may still require local hands. One that was never configured, has an expired password, sits on an unreachable network or runs firmware with known faults adds delay at the worst moment.

The same features that make remote recovery possible, console access, media insertion and power control, give BMC administrators substantial control over boot, console and power. Disk encryption, signed-boot policies and hardware design can still constrain what data they can read. Preparing the BMC for recovery and protecting it are the same project.

How it works

What Redfish describes

Redfish is a DMTF standard for managing servers over HTTPS with structured data. Its schemas, published at redfish.dmtf.org, define resources such as managers (the BMC), computer systems (the server it manages) and virtual media. Vendors implement subsets, so a schema property existing does not mean a given server supports it. Check the vendor's documentation for the model in front of you.

Virtual media

The VirtualMedia schema "contains properties related to the monitor and control of an instance of virtual media, such as a remote CD, DVD, or USB device. A manager for a system or device provides virtual media functionality."

Its key properties and actions, as the schema describes them:

  • Image: "the URI of the media attached to the virtual media", which "may specify an absolute URI to remote media or a relative URI to media local to the implementation." If the URI is not accessible, services "should populate the Conditions property in Status".
  • InsertMedia action: attaches remote media. The service "should reject the request if the Image property in the resource does not contain null", and "users are expected to eject media prior to inserting new media."
  • EjectMedia action: detaches the media; afterwards "inserted shall be set to false and the image name shall be cleared."
  • Inserted: "whether the media is mounted and visible to the host system."
  • WriteProtected: whether the media "prevents writing to that media". In the InsertMedia action, if the client does not specify it, "the service shall default this value to true."
  • TransferProtocolType: the protocols a service may use to fetch the image, enumerated as CIFS, FTP, HTTP, HTTPS, NFS, SCP, SFTP, TFTP and OEM.
  • TransferMethod: "Stream image file data from the source URI" or "Upload the entire image file from the source URI to the service."
  • EjectPolicy (added in v1.6.0): when media is ejected automatically, for example "after the media is used", "during a system power or reset event", "when a session is terminated", on a timer, or "Persistent".

For recovery, the practical points are these:

  • The BMC fetches the image, not your laptop. With a URI-based insert, the controller connects to the image server. That server must be reachable from the management network, which is often isolated on purpose.
  • Streaming depends on the network for the whole recovery. If the image is streamed and the connection drops, the recovery environment loses its media. The Upload method copies the image to the controller first, where the controller supports it and has space.
  • Write protection is the default for a reason. Recovery media should normally be read-only.
  • Eject policies affect multi-reboot procedures. If media is ejected on power or reset, an installer that reboots midway may lose its media.

Verifying the image server

The schema includes certificate checking for the image source. VerifyCertificate (added in v1.4.0) indicates whether the service will verify the image server's certificate against a configured collection, and the schema states: "If this property is not supported by the service, it shall be assumed to be false. This property should default to false in order to maintain compatibility with older clients."

In other words, unless someone has turned it on and loaded the right certificates, a BMC may fetch a boot image over HTTPS without confirming who served it. For media that will run with full control of a server, that is worth closing.

The schema also notes that the image Password "shall be null in responses", so it should not be readable back from the service.

One-time boot overrides

Attaching media is only half the job; the server has to boot from it. The ComputerSystem schema describes a boot override:

  • BootSourceOverrideTarget: "the source to boot the system from, overriding the normal boot order". The enumerated values include Cd, Usb, Pxe, UefiHttp, Hdd, BiosSetup, UefiShell, UefiTarget, UefiBootNext and, from v1.19.0, Recovery ("boot to a system-designated recovery process or image"). The schema adds: "Virtual devices for a target should take precedence over a physical device."
  • BootSourceOverrideEnabled: "Once for a one-time boot override, and Continuous for a remain-active-until-cancelled override. If set to Once, the value is reset to Disabled after the BootSourceOverrideTarget actions have completed successfully."
  • Both state that changes "do not alter the BIOS persistent boot order configuration".
  • BootNext maps to the UEFI one-time boot setting, and the schema notes that Continuous is not supported with UefiBootNext "because this setting is defined in UEFI as a one-time boot setting."

A one-time override is the right default for recovery: the server boots the recovery media once, then returns to its normal boot order. A continuous override left in place can make a repaired server boot back into the recovery environment.

Consoles

The Manager schema describes a GraphicalConsole "(KVM-IP) service", a SerialConsole, and a CommandShell, which "refers to an interface used to interact with the manager itself, not a dedicated console session redirected from a host operating system." It points to the ComputerSystem resource for host consoles.

A graphical console shows firmware screens, boot loader menus and the recovery environment. A serial console is lighter and easier to log, but only shows what the firmware and operating system send to the serial port, which usually needs configuring in advance.

IPMI and ipmitool

Many controllers also support IPMI, the older management protocol. The ipmitool manual describes chassis bootdev options such as pxe, disk, cdrom and bios ("Force boot into BIOS Setup"), and Serial-over-LAN through sol activate, which "requires the use of the lanplus interface". It warns about credentials: "Specifying the password as a command line option is not recommended", and offers a password file or environment variable instead.

Evaluate the enabled management protocols against the vendor's current firmware and security guidance. The presence of Redfish alone does not establish that the configured service is secure.

What the controller cannot do

A management controller gives you the console, media and power. It does not give you a recovery. The pieces that make the recovery work still have to exist elsewhere, and the plan should name each one.

  • A recovery image that suits the hardware. The rescue or installer image needs drivers for the server's storage controller and network cards, and must boot in the firmware mode the server uses (UEFI or legacy). The schema's BootSourceOverrideMode property exists because the boot mode matters to the override. An image that boots on one model may not see the disks on another.
  • A backup to restore from. Virtual media can deliver a restore tool, but the data has to come from somewhere the recovery environment can reach. The BMC-to-image-server route and the recovery operating system-to-backup-storage route are different. Plan and verify each separately; the controller need not carry restored data itself.
  • Credentials and keys. The unlock method for encrypted disks, the root or break-glass password and the controller's own login all need to be retrievable when the main systems are down.
  • A working controller. Controllers fail too: a hung web interface, a full local store, expired certificates or a firmware fault. Check the model-specific controller-reset procedure and its host effects before relying on it; a schema does not guarantee a non-disruptive reset.

The controller is the access path. Treat it as one dependency in the recovery plan, tested with the others.

What to choose and why

Prepare before anything fails

These are our recommendations:

  • Inventory every controller: address, model, firmware version, licence level (virtual media and graphical console are licensed features on some models), and who can log in.
  • Put controllers on a dedicated management network, reachable by administrators through a controlled path such as a jump host or VPN, never directly from the internet.
  • Decide where recovery images will be served from, inside the management network, and test that each controller can reach it.
  • Turn on image certificate verification where supported, and load the image server's certificate.
  • Use individual named accounts with the least privilege that the task needs, store break-glass controller credentials in the password vault, and remove default accounts or change their passwords.
  • Keep controller firmware current under the vendor's advisories, as you would for any other networked system.
  • Configure serial console redirection in firmware and in the operating system's boot settings if you want text-based consoles to work.
  • Test it. Once a quarter, attach a harmless recovery image to a non-production server, boot it once, and confirm the server returns to its normal boot order afterwards.

On the day

  1. Open the console first to see what the server is doing before changing anything.
  2. Check what media is attached and eject anything left over from earlier work.
  3. Insert the recovery image as write-protected, with an eject policy that suits the procedure.
  4. Plan the approved boot transition: confirm the selected target, the outage window, application shutdown requirements and one-time override behaviour before any reset.
  5. Afterwards, eject the media, confirm the override has cleared, and record what was done.

Risks and limits

  • The BMC is a privileged system. Anyone who controls it can boot arbitrary media, see the console and power the server off. Treat its accounts, network and firmware with the same care as the most sensitive server it manages.
  • Implementation varies. Schema properties are optional for vendors and differ between firmware releases. Test the features you rely on, on each model.
  • Licensing. Some vendors license virtual media or graphical console separately. Confirm before relying on them.
  • The Redfish specification document was not reviewed. This article quotes the schemas only. The specification describes protocol behaviour, authentication and sessions in detail.
  • Nothing here was run. Properties and options are quoted for planning.

Sources

This article was originally published on .