Australian cloud for business software — built and run in Sydney since 2003

+61 2 8282 5656
support@netleverage.com
Talk to an engineer
Talk to an engineer
NetLeverage
NetLeverage

File names that change on the way back: Unicode normalisation and legacy encodings in backup and restore

Technical article
Backup & Recovery

Originally published:

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A restore can preserve file contents and still fail because names changed. A folder is backed up from one system and restored to another, and a few files arrive with names that look right but do not behave right. A program cannot find a file it created. A copy stops with an error saying a file already exists, though only one is visible. A name that once showed an accent now shows a question mark or a pair of odd symbols.

Two causes to investigate are Unicode normalisation and legacy encodings. The first is Unicode normalisation: the same visible name can be written in more than one sequence of characters, and different systems treat those sequences differently. The second is legacy encodings: older Linux servers may hold names that are not UTF-8 at all.

This guide explains both from the primary standards and vendor documentation, then sets out what to check before a cross-platform restore. It focuses on encoding and normalisation rather than the separate limits involving forbidden characters, case or path length.

Why it matters

File names are not just labels. Applications store them in databases, project files, playlists, links and configuration. If a restore changes the bytes of a name, even invisibly, three things can follow.

References break. A document management system, a website or a media library that recorded a path will look for the original bytes and may not find the restored file.

Collisions appear. Two names that one system kept apart may be treated as the same name on another, so one file cannot be restored beside the other.

Comparisons mislead. A sync or verification tool comparing source and restored trees byte by byte may report files as missing and new, although the user sees the same name.

How it works

Linux stores bytes, not characters

POSIX defines a filename as "A sequence of bytes consisting of 1 to {NAME_MAX} bytes used to name a file." The only bytes it rules out are these: "The bytes composing the name shall not contain the or characters." Nothing in that definition requires the bytes to be valid UTF-8, or to be in any particular Unicode form.

Linux distributions now normally use UTF-8 locales. The utf-8(7) manual page describes how, in a UTF-8 locale, "plaintext file content, filenames, and environment variables are encoded in UTF-8". That is a convention of the user environment, not a rule the file system enforces (our reading of the POSIX definition). A file system such as ext4 will normally accept a name written in Latin-1 by an old program just as readily as a UTF-8 one, and will keep both exactly as given.

One visible name, several byte sequences

Unicode often has more than one way to write the same character. Microsoft's normalisation guide gives the classic example: "Capital A with dieresis (umlaut) can be represented either as a single Unicode code point" (U+00C4) or as capital A followed by a combining dieresis (U+0041 U+0308). Both display as Ä.

The Unicode Standard Annex on normalisation, UAX #15, calls this canonical equivalence: "a fundamental equivalency between characters or sequences of characters which represent the same abstract character, and which when correctly displayed should always have the same visual appearance and behavior."

To deal with it, UAX #15 defines normalisation forms. The two that matter for file names are:

  • NFD, canonical decomposition: characters are split into a base letter plus combining marks.
  • NFC, "Canonical Decomposition, followed by Canonical Composition": characters are decomposed, then recombined into precomposed forms where they exist.

The purpose is stated in the annex summary: "When implementations keep strings in a normalized form, they can be assured that equivalent strings have a unique binary representation." If every system kept names in the same form, the problem would largely disappear. They do not.

Two further forms, NFKC and NFKD, also fold compatibility differences such as ligatures and full-width letters. UAX #15 warns that they "must not be blindly applied to arbitrary text", because they "erase many formatting distinctions". Do not apply them as a generic recovery fix: compatibility folding can merge names that users intentionally distinguished.

Linux, Windows and macOS make different choices

Linux. An ordinary Linux directory treats a name as bytes, so an NFC name and an NFD name for the same visible text are two different files and can sit side by side (our reading of the POSIX definition). ext4 offers an optional per-directory case-insensitive mode. The kernel documentation explains that its comparison "is implemented by normalizing the strings to the Canonical decomposition form, as defined by Unicode, followed by a byte per byte comparison", while staying "name-preserving on the disk". Names that are not valid UTF-8 are a special case: unless strict mode is set, ext4 "falls back to considering the entire string as an opaque byte sequence", and case-insensitive lookups do not work for that file.

Windows. NTFS stores long file names in Unicode, and Microsoft notes "that the original long file name is always preserved." Microsoft provides normalisation functions for applications, and its guide observes that normalisation "eliminates some differences but preserves case". Microsoft's path documentation says Windows file I/O treats names as an opaque sequence of WCHARs and does not require Unicode normalisation. Applications and network paths may impose additional behaviour, so test the actual route.

macOS, older volumes (HFS Plus). Apple's HFS Plus technical note describes a fixed decomposition algorithm, and explains that "The HFS Plus algorithms cannot evolve because such evolution would invalidate existing HFS Plus volumes." Apple's file system FAQ summarises the effect: "HFS+ stores the normalized form of the filename on disk to provide normalization insensitivity." In other words, whatever form a program supplied, the disk held one form.

macOS, the documented APFS model. Apple's FAQ (a retired archive page, which Apple says "may not represent best practices for current development") states: "APFS accepts only valid UTF-8 encoded filenames for creation, and preserves both case and normalization of the filename on disk in all variants." For macOS from High Sierra it adds that APFS is normalisation-insensitive, and that "Being normalization-insensitive ensures that normalization variants of a filename cannot be created in the same directory, and that a filename can be found with any of its normalization variants." The same historical FAQ describes rejection of code points unassigned in Unicode 9.0; that version-specific statement is not presented as a tested limit of current macOS.

These documented choices identify several collision risks:

  • An ordinary byte-sensitive Linux directory can hold two normalisation variants. The documented normalisation-insensitive APFS model cannot.
  • Ordinary Linux directories can hold invalid UTF-8 byte names. The cited APFS model requires valid UTF-8.
  • A name copied from an HFS Plus volume may arrive on Linux in a decomposed form that looks identical to, but differs from, the same name typed on a Linux or Windows machine.

Normalisation is stable, but not simple

Two properties from UAX #15 matter for anyone tempted to fix names in bulk. First, stability has conditions. UAX #15 says a string with no unassigned characters, normalised under one Unicode version, must remain normalised under future versions. The stronger guarantee across past and future versions is stated for its Normalization Process for Stabilized Strings, which rejects unassigned code points. Record the algorithm and supported repertoire rather than applying that stronger claim to arbitrary byte names.

Second, the forms are not closed under joining strings: "none of the Normalization Forms are closed under string concatenation." A tool joining fragments within one filename must consider the boundary: a combining character can interact with the preceding fragment. Joining normalised path components with an ordinary slash is a different case because the separator forms a boundary. This is a detail for developers, but it is one reason to use established tools rather than home-made scripts.

Legacy encodings: the older problem

Before UTF-8 became the norm, Linux and Unix systems often used 8-bit encodings such as ISO 8859-1. A filename containing non-ASCII bytes from such an encoding may be invalid as UTF-8; ASCII-only names are shared by many encodings. On a modern system those names display as replacement characters or escape sequences.

Tools deal with this differently. The rclone documentation, for example, says that "If an invalid (non-UTF8) filename is read, the invalid characters will be replaced with a quoted representation of the invalid bytes", so the stored name differs from the original. Amazon S3 keys are themselves defined as "a sequence of Unicode characters encoded in UTF-8", so a name that is not UTF-8 cannot be stored as is. Exact restoration therefore needs a supported reversible encoding or an archive preserving the original byte names. A changed transfer representation alone does not establish whether the eventual decode succeeds; verify the complete round trip.

A related fault is double encoding: UTF-8 bytes treated as Latin-1 and converted to UTF-8 again. The convmv manual discusses this case explicitly as "double encoded UTF-8" and describes an option for fixing it.

What the common tools do

Samba. smb.conf has a unix charset setting that "Specifies the charset the unix machine Samba runs on uses", with the note "Default: unix charset = UTF-8". Samba converts between that and what SMB clients use. If an old share holds Latin-1 names while Samba assumes UTF-8, Windows and Mac clients will see damaged names. Whether Samba normalises names between clients was not confirmed in the sources used here.

rsync. rsync copies names as bytes unless asked to convert. Its manual says "Rsync can convert filenames between character sets using this option" for --iconv, and warns that "rsync does not do any conversion of names in filter files", so include and exclude rules may need both forms. Character-set conversion is not the same as Unicode normalisation; the manual describes conversion between charsets only.

convmv. convmv is a long-standing tool for this exact problem. Its manual says "It just converts the filenames, not the content of the files." It can convert from a legacy encoding to UTF-8, and offers --nfc ("target files will be normalization form C") and --nfd. It "also converts the symlink target pointer in case the symlink target is being converted, too." Importantly, "By default convmv will just print what it wants to do"; a separate option is needed to rename anything. Its manual also contains strongly worded opinions about Apple's design, and some version details in it are dated, so treat its commentary on macOS as background, not as a source.

A worked example

Illustration only, not a measured case.

A design studio runs a Linux file server shared to Mac and Windows machines over SMB. Over the years, some files were copied in from an old Mac external drive formatted HFS Plus, and others were saved directly by Windows users. A project folder ends up holding two files that both display as Café menu.pdf: one with a precomposed é, from a Windows machine, and one with e plus a combining accent, from the old drive. Linux treats them as two names, so both exist.

The server is backed up nightly. Later, a manager restores the project folder onto a Mac laptop with an APFS volume, for a client meeting. Because APFS will not allow two normalisation variants of one name in the same directory, the second file cannot be created, and the restore tool reports a conflict or overwrites one file with the other, depending on the tool (our reading of the APFS statement).

The same folder also contains an older subfolder of scanned invoices created by a script on a previous server that used Latin-1. Their names are not valid UTF-8. APFS refuses them outright. On the Linux server they had always appeared slightly garbled to Mac users, and nobody had noticed why.

What would have prevented it: a check, before the restore, for names that are not valid UTF-8 and for names that collide once normalised; a decision about which form the server should hold; and, if names were converted, a test run of the conversion in its print-only mode first, followed by a check of anything that stores paths.

A planning checklist

  1. Find invalid names first. Before any cross-platform restore, list names that are not valid UTF-8. Do not assume raw byte names can be used directly on a target requiring UTF-8.
  2. Find normalisation collisions. Look for names in the same folder that become identical when normalised to NFC. These cannot coexist on APFS.
  3. Preserve existing identity first. If a migration requires a normalised form, choose one after checking application references and collisions. NFC can be a practical convention, but uniformity is not a reason to rename a working archive during recovery.
  4. Convert in print-only mode first. Review the full list of proposed renames before applying any, and keep the list.
  5. Check what stores paths. Databases, project files, links, playlists and web content may reference the old byte sequences; update or test them after any conversion.
  6. Match Samba's charset to reality. Confirm unix charset matches the encoding the names actually use.
  7. Separate recovery from renaming. Where the archive and staging filesystem support the original names, recover an isolated copy first. If conversion is unavoidable, preserve the originals and an explicit reversible mapping before applying it.
  8. Avoid blanket compatibility folding. NFKC and NFKD can collapse intentionally distinct names; use them only if an explicit application contract requires and validates that transformation.
  9. Compare names and content without merging collisions. Retain the original byte-name manifest, identify each transformation, and use normalised forms to detect ambiguity. A matching normalised name and checksum alone can conceal a missing second file.
  10. Record the decision. Note which form and encoding each share uses, for the next migration.

Risks and limits

  • Path-specific behaviour. Windows API guidance does not establish every application or SMB-client transformation. Test the deployed client, server and filesystem combination.
  • Apple source age. The APFS FAQ is a retired archive document; current macOS behaviour may differ in detail.
  • Bulk renames are risky. Renaming files changes paths that other systems depend on; reliable reversal requires a preserved original-name record and a plan for collisions or intervening changes.
  • Case and forbidden characters require separate checks; normalisation alone does not make a name portable.
  • Not covered. Bidirectional text, look-alike characters used for deception, and non-Latin scripts beyond the general principles.

Sources

This article was originally published on .