Australian cloud for business software — built and run in Sydney since 2003

+61 2 8282 5656
support@netleverage.com
Talk to an engineer
Talk to an engineer
NetLeverage
NetLeverage

SELinux labels after a restore: why services get "permission denied" and how relabelling works

Technical article
Backup & Recovery

Originally published:

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

On a Linux system running SELinux, access to a file depends on more than its owner, group and mode bits. Each file also carries a security context, often called a label, and the loaded policy decides which processes may use files with which labels. The selinux(8) manual says these contexts "are stored in the extended attributes of the file system", and adds: "Problems with SELinux often arise from the file system being mislabeled."

A restore can bring back every byte of a file, its ownership and its permissions, and still leave it with the wrong label or none at all. The symptom is a service that fails with "permission denied" on files that ls -l says it should be able to read. This guide explains why that happens and how to plan restores so that it does not.

Why it matters

Labelling faults are easy to misdiagnose. Administrators see a permission error, check ownership and mode, find nothing wrong, and are then tempted either to loosen file permissions or to switch SELinux off. The first changes the system's security for no benefit. The second hides the fault without fixing it, and leaves a server running in a state its owners did not choose.

One possible fault is that the restore lost labels or brought labels the target policy does not expect. Permission errors can also come from ordinary permissions, access-control lists, mount settings or service configuration. Evidence must identify the layer before any correction.

How it works

Where labels live

The SELinux Notebook describes labelling of filesystems that support extended attributes, such as ext4. It names three parts:

  1. A policy statement that identifies which filesystems use extended attributes.
  2. A "file contexts" file "that defines what the initial contexts should be for each file and directory within the filesystem".
  3. "A method to initialise the filesystem with these extended attributes", such as fixfiles and setfiles, with chcon, restorecon and restorecond also able to relabel files.

The label itself is stored in an extended attribute named security.selinux. The Notebook shows both ways of viewing it: ls -Z, and getfattr -n security.selinux.

That separation matters for recovery. The file contexts configuration says what a file should be labelled, based on its path and type. The extended attribute says what it is labelled. A data-only restore may affect the second while a system or configuration restore can also replace the policy and local rules. Both sides of that comparison need checking.

How copying and moving affect labels

The Notebook summarises the default behaviour: when a file is copied it "takes on label of new directory", and when it is moved it "retains the label of the file". The Notebook also notes that a running restorecond daemon can apply configured contexts as files are created.

Restores often behave like copies. A file written fresh by a restore program typically receives a label derived from where it is created, not the label it had on the original system, unless the program explicitly restores the original label. Files restored into a staging directory and then moved into place may keep the staging directory's label. In both cases, the restored file can end up with a label the service's policy does not allow it to read. (our reading of the Notebook summary applied to restore tools)

What backup and copy tools keep

Whether labels survive depends on whether the backup method carries extended attributes, and in particular the security.selinux attribute.

  • GNU tar documents --selinux ("Enable SELinux context support") and --xattrs ("Enable extended attributes support") as options. They are not described as defaults in the manual page, so an archive made without them should be assumed not to carry labels.
  • rsync documents --xattrs (-X) to "preserve extended attributes". Its archive mode, -a, is described as "-rlptgoD (no -A,-X,-U,-N,-H)", so it does not carry extended attributes on its own.
  • Image-level backups of a block device carry whatever is on the device, labels included, because they copy the filesystem rather than its files.
  • Copies through filesystems or protocols without extended-attribute support, such as many network shares and FAT-formatted media, cannot hold the label at all.

The general lesson is to know which category your backup falls into before you need it, and to test a restore on a system with SELinux enforcing, not only on one where it is disabled.

What a missing or wrong label looks like

The selinux(8) manual offers a specific clue: "If you see an error message containing file_t, that is usually a good indicator that you have a serious problem with file system labeling." In common policies, file_t is the type given to files with no valid label.

More usually, files carry a valid but wrong type, such as a general type inherited from a parent directory instead of the type the service expects. The service then fails with an ordinary-looking permission error, and the evidence is in the audit log. The ausearch manual gives examples for exactly this: ausearch --start today -m avc -i to "check the SELinux log for any denials today", and a broader search across avc, user_avc, selinux_err and user_selinux_err messages. An AVC denial records source and target security contexts, the denied permission and related event information. It establishes a policy denial, not automatically a labelling fault; the process domain, boolean settings or intended access rule may instead be wrong.

The matchpathcon manual describes a read-only check that complements this: it "queries the system policy and outputs the default security context associated with the filepath", noting that "identical paths can have different security contexts, depending on the file type". Comparing what matchpathcon expects with what ls -Z shows confirms a mismatch without changing anything.

Relabelling tools

The restorecon manual says the program "is primarily used to set the security context (extended attributes) on one or more files", and can also "correct inconsistent labels". Several details matter during recovery:

  • With -n, restorecon makes no changes: "don't change any file labels (passive check). To display the files whose labels would be changed, add -v."
  • It does not recurse by default: "restorecon by default does not operate recursively on directories."
  • For an existing context, the normal change is to its type. The manual's detailed options distinguish -F, which also changes user, role and range, from -U, which changes user and role but preserves range. Its introductory summary groups these options too broadly; use the detailed definitions.
  • Files with a customizable type are left alone "unless the -F option is used". Those types are listed per policy in a customizable_types file.

The fixfiles manual describes a wrapper for larger jobs. It "is primarily used to correct the security context database (extended attributes) on filesystems"; by default it "will relabel all mounted ext2, ext3, ext4, gfs2, xfs, jfs and btrfs file systems as long as they do not have a security context mount option". check and verify "print any incorrect file context labels, showing old and new context, but do not change them". fixfiles onboot "will setup the machine to relabel on the next reboot". The relabel action also prompts to clear /tmp, so it is not a neutral command to run casually.

For a whole system, selinux(8) states: "The best way to relabel the file system is to create the flag file /.autorelabel and reboot." The upstream fixfiles script shows that onboot writes this file and can add options to it. A full relabel walks every file on the labelled filesystems at boot, so on large systems it takes time and should be planned as part of the outage.

Local file-context rules

Policies label standard locations, such as web content directories, automatically. Many servers also hold data in non-standard places: a database on a separate mount, web content under a custom path. Administrators normally tell SELinux about these with semanage fcontext.

The semanage-fcontext manual explains that these definitions are "local file context modifications stored in file_contexts.local" and "have higher priority than those specified in policy modules". Its examples pair each rule with a relabel: "Remember to run restorecon after you set the file context". For instance, a rule giving everything under /web the web content type, followed by restorecon -R -v /web.

Local rules can be missing after a rebuild. If a server is rebuilt from installation media and its data restored, the default policy has no idea that a custom path should carry a special type. Relabelling then correctly applies the default labels to that path, and the service still fails. The fix is to restore the local rules first, then relabel.

The semanage-export manual describes the supported way to carry them: "semanage import and export can be used to extract the SELinux modifications from one machine and apply them to another." It also warns that import "will remove all current semanage customizations on the second machine", because the exported list begins by deleting existing ones.

Filesystems without labels

The mount(8) manual documents mount-time context options. context= "is useful when mounting filesystems that do not support extended attributes, such as a floppy or hard disk formatted with VFAT, or systems that are not normally running under SELinux". It assigns "the entire disk one security context". This is relevant when a recovery mounts restore media or a disk from another system. Fixfiles skips filesystems mounted with a context option.

What to choose and why

Prefer backups that carry labels

For SELinux systems, choose backup methods that preserve extended attributes, including security.selinux, and verify it with a test restore that checks ls -Z on representative files. Image-level backups carry labels by nature; file-level tools need the right options or features.

Back up the policy customisations

Include the output of semanage export in the server's configuration backup, together with the SELinux mode and policy type from /etc/selinux/config. Also retain custom policy modules and their source or installation packages: exporting semanage customisations is not a complete backup of every policy component. A rebuilt system needs a compatible policy and an explicit decision about whether historical labels should be preserved or regenerated.

Diagnose before relabelling

When a restored service fails:

  1. Check the audit log for AVC denials around the failure.
  2. Compare expected and actual labels with matchpathcon and ls -Z on the paths named in the denial.
  3. Run restorecon or fixfiles in check mode on the affected tree to see what would change.

Then fix in order: restore missing local file-context rules, relabel the affected paths, restart the service and confirm the denials stop.

Choose scope deliberately

Relabel the restored tree when only part of a system was restored. Schedule a full autorelabel when the whole root filesystem came from a source that did not keep labels, or when the system ran without SELinux for a period. Allow for the extra boot time.

Do not switch SELinux off as the fix

Permissive mode exists for diagnosis. The SELinux Notebook describes it as "generally used for testing", with denials still logged. If an authorised diagnostic plan uses it briefly to investigate a policy denial, record that you did and return to enforcing once labels are corrected. The selinux(8) manual also notes that booting with a non-SELinux kernel can itself cause mislabelling.

Risks and limits

  • Distribution differences. Policy types, customizable types and default paths differ between distributions and releases. Use the documentation for your system.
  • Multi-level security. Systems using MLS or multi-category ranges carry extra fields in the label; restorecon's handling of those fields depends on options. This article does not cover MLS deployments.
  • Containers and virtual machines. Container runtimes and virtualisation tools may assign per-instance categories. Restoring their storage may need tool-specific relabelling.
  • Nothing here was run. Commands are named from manual pages. Full relabels traverse the selected filesystems and can change many labels; rehearse them on a non-production copy.
  • AppArmor is different. Some distributions use AppArmor, which uses path-based profiles rather than file labels. The restore issues described here do not apply in the same way.

Sources

This article was originally published on .