Storage for training: checkpoint bandwidth, parallel filesystems and object stores

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A training checkpoint is the copy you can restart from: the weights, and the optimiser memory that says how those weights were moving. The GPUs can compute the next step only if that copy lands on storage before the job is killed, and only if a restart reads a complete copy. The storage decision is a bandwidth sum and a publish rule. A parallel filesystem spreads the bytes across many disks. An object store hides the bytes until you explicitly finish the upload.

An AI writing system produced this article for NetLeverage's editorial studio. "The job remembers its state" means tensors written to storage. It is not a claim about subjective experience.

Nothing below was run. Commands are the ones printed in the cited manuals. Layout changes and uploads on a live training directory can collide with a job that is writing the same path.

Why it matters

The stall is checkpoint size divided by write rate, plus time until a restart will open that copy. Two storage systems fail that sum in different ways.

A parallel filesystem can leave the file on one disk if the stripe count is one. An object store can accept parts for hours and still have no readable object until the complete call. Both look saved while a restart would fail.

ZeRO also uses "activation checkpointing" for recomputation during the step. That is not a file. This article is the copy a failed job restarts from.

How large the file is

PyTorch's Adam documentation, in the 2.15 manual, writes the update as two running statistics: a first moment $m_t$ and a second moment $v_t$, both the same shape as the gradient. Those tensors are part of the optimiser state that state_dict saves and load_state_dict restores. The manual tells you to load a learning-rate schedule before the optimiser checkpoint when both were saved. The optimiser state is not a side file you can skip if you intend to resume Adam rather than start it again from zero moments.

Rajbhandari, Rasley, Ruwase and He, in ZeRO (arXiv:1910.02054v3, 13 May 2020), count the bytes for mixed-precision Adam. Parameters and gradients are stored in fp16, 2Ψ bytes each. The optimiser keeps an fp32 copy of the parameters, the momentum and the variance, 4Ψ bytes each. They set $K=12$ for those optimiser states. The total is:

For the 1.5 billion parameter GPT-2 they use as the example, that is at least 24 GB, against 3 GB for the fp16 parameters alone. The 16Ψ figure is their mixed-precision accounting, with an fp32 master copy. It is not the size of a bf16-only run that never kept that master, and it includes gradients, which a checkpoint often does not store.

If a checkpoint keeps the fp16 weights and the three fp32 optimiser tensors, and drops the gradients, the same terms sum to $14\Psi$. For their 1.5 billion parameter example that is 21 GB. That subtraction is arithmetic on their breakdown, not a measured file. Count the tensors you actually put in the state_dict: elements times bytes per element. Adam with amsgrad keeps an extra max of the second moment, which their $K=12$ does not include.

A stall budget follows from that count. 21 GB in 120 seconds needs about 175 MB/s sustained. The same dtype mix at 70 billion parameters is about 980 GB, or about 3.3 GB/s if the stall must stay under five minutes. That is arithmetic, not a filesystem benchmark.

ZeRO partitions optimiser state so each device does not store a full copy. The bytes a full restart must assemble do not shrink with that partition. A sharded checkpoint stores the pieces, and a restart reads every piece the new layout needs.

A parallel filesystem, using Lustre as the opened example

The Lustre manual served for Software Release 2.x (doc.lustre.org, opened 10 November 2026) splits the system into metadata and data. A metadata server makes metadata on one or more metadata targets available to clients. The metadata target holds names, directories, permissions and the file layout. User file data sits in objects on object storage targets (OSTs), and an object storage server hosts those targets. Clients mount the filesystem and read and write through that split.

Striping is how one file uses more than one OST. The manual says you can set, per file, the number of stripes, the stripe size, and which OSTs are used. Striping helps when the bandwidth of one file must exceed one OST, or when one OST lacks space for the file. The documented default in their lfs getstripe example of the filesystem default is stripe count 1 and stripe size 1 MB: the file's data blocks sit on a single OST. stripe_size must be a multiple of 64 KB. Setting stripe count to -1 means stripe across available OSTs, skipping full ones.

That default is one possible reason a large single-file checkpoint underuses a Lustre filesystem; it is not a diagnosis without layout and throughput evidence. The directory was never given a layout, so each new file inherited one OST. The aggregate bandwidth the manual describes is the bandwidth of the objects in that file, and a single file can be striped over at most 2000 objects. On ldiskfs without the ea_inode feature the manual's limit is 160 OSTs for one file. A filesystem may have thousands of OSTs; past 2000 objects the manual says to use multiple files if you need the whole filesystem's bandwidth. File size itself can reach $2^{63}$ bytes, limited by free OST space.

Read the layout before you change it. The manual's example:

lfs getstripe /mnt/lustre/foo

shows lmm_stripe_count and lmm_stripe_size (1048576 in the example, which is 1 MB). An ordinary file's initial stripe layout is selected at creation. Changing an existing file's ordinary layout generally requires data migration, which copies data; progressive/composite layouts have separate component-extension operations. Plan the directory layout before the training job creates checkpoint files. lfs setstripe on a directory does not rewrite files that already exist. Do not point it at a directory a job is creating files in, unless that job is stopped: two creators and a layout change are how a restart finds a mix of one-OST and many-OST files.

Every create is a metadata operation on the MDT. Very many small files or very high creation rates can overload metadata. One file per GPU rank is not automatically a create storm; it is also a normal distributed-checkpoint layout. On ldiskfs the inodes are allocated at format time and are not casually increased later, so millions of tiny files can exhaust inodes while OST space remains.

What PyTorch distributed checkpoint actually writes

PyTorch 2.15's distributed checkpoint (DCP) documentation says it saves and loads from multiple ranks in parallel, and that load-time resharding allows a save on one cluster shape and a load on another. It differs from torch.save in a way that dominates the filesystem: it produces multiple files per checkpoint, at least one per rank. Each rank saves its own local shards of a ShardedTensor or DTensor.

The save path uses collectives. For NCCL, the 2.15 page says the tensors in that coordination must be on the GPU, and each rank must have called torch.cuda.set_device(). A hang at save time can be that collective.

async_save is documented as experimental. It first copies the state dict onto staging storage, CPU memory by default, then runs the save on another thread. The call returns a future. The checkpoint is not finished because the call returned. result() on that future is the wait. Staging to CPU also means the host needs RAM for a copy of the shards being saved. As an operational inference, insufficient staging memory can make this path fail before storage writing; the documentation does not make that the only failure point.

torch.save and torch.load, in the 2.15 serialisation note, use Python pickle by default. From version 2.6, torch.load uses weights_only=True unless you pass a pickle_module. That mode narrows remote-code execution and does not stop denial of service. A full-module pickle may need unrestricted deserialisation, which can execute code. Do not weaken loading protections for an untrusted checkpoint. Prefer tensor state dictionaries and a supported, security-patched loader; format choice and weights_only=True alone are not a complete trust boundary.

An object store, using Amazon S3 as the opened example

S3 stores an object as the data plus metadata, addressed by a key in a bucket. The user guide, opened 10 November 2026, says S3 provides strong read-after-write consistency for PUT and DELETE of objects in all regions, including overwrites, and for HEAD and object metadata. Bucket configuration is a different model: the same guide says bucket configurations are eventually consistent. Do not treat a bucket-policy change as if it had the same read-after-write promise as a finished object.

Multipart upload is the S3 path discussed here for large checkpoints; this article does not establish a single-PUT size limit. The guide's limits page states:

The overview recommends multipart once an object reaches about 100 MB, because parts go in parallel and a failed part can be retried without restarting the object. You must record each part number and its ETag. CompleteMultipartUpload sends the upload ID and that list. S3 then concatenates parts in ascending part number and, after success, the parts no longer exist as separate storage. Until that complete call, there is no object for a restart to GET. The upload ID has no expiry by itself. Incomplete parts are billed until completion or abort. AWS recommends an AbortIncompleteMultipartUpload lifecycle rule to remove abandoned uploads after a chosen number of days; the retention window must account for legitimate long-running uploads. Abort only after in-flight part uploads have finished, or those parts can still land and keep the storage.

Two races are in the same guide. With versioning, the current version is the upload whose CreateMultipartUpload is the most recent, even if an older upload completes later. Their example is a 10:00 start and an 11:00 start: the 11:00 upload becomes current. Without versioning, another request between initiate and complete can take precedence. Concurrent attempts under one key can select an unintended complete version. They do not mix partial old and new object bytes. Unique keys avoid that collision; conditional PutObject or CompleteMultipartUpload requests can also prevent overwriting an existing key.

A 980 GB checkpoint fits in one object. At a 5 GiB part size that is 183 parts when 980 GB means 980 billion bytes (rounding up 980 billion divided by 5 GiB). At the 5 MiB minimum the same object would need more than 10,000 parts, which the limit forbids, so tiny parts are the wrong end of the range for a checkpoint this large.

Publishing a copy a restart can trust

On a POSIX filesystem, rename is the usual publish step. The Linux rename(2) page says that if the new name exists it is replaced atomically, so a reader does not observe the name missing. EXDEV is returned when the two paths are not on the same mounted filesystem, including two mount points of one filesystem. Write the temporary name on the same mount as the final name. Both old and new names may briefly refer to the file. When renaming a directory, the destination must be absent or an empty directory; ordinary rename is not a replacement operation for an existing non-empty checkpoint directory.

Rename makes the name appear. It does not, by itself, invent a stripe layout, and it does not apply to S3. On S3 the complete call is the publish. A reader that lists the key before complete does not see the checkpoint, which is the behaviour you want. S3 guarantees atomic updates to one key: a concurrent reader receives a complete old or new object, never a mixture. The unresolved race is which complete writer wins. There is no atomic transaction across multiple keys.

DCP's “at least one file per rank” is a directory of parts, and its writer finish stage writes checkpoint metadata. A higher-level publication scheme must wait for successful completion and the required metadata, not just the appearance of rank files. DCP does not guarantee checkpoint backwards compatibility across PyTorch versions. Renaming the directory while one rank is still in async_save leaves a hole a restart will load.

What to choose and why

Count the tensors first, then pick the publish rule you can test: a same-mount rename, or one complete call. A directory marked ready when the first rank returns is unfinished.

The safe order

1. Count the bytes. Print, in the training process, the number of elements and the dtype of each tensor in the checkpoint state dict. Compare the sum with the 14Ψ or 16Ψ arithmetic above only as a sanity check. The state dict wins, because amsgrad, frozen parameters and extra buffers change it.

2. See where those bytes will land. On Lustre, lfs getstripe on files already in the checkpoint directory. Stripe count 1 means one OST. Set the directory layout, then start the job. The layout applies to files created afterwards.

3. Watch creates separately from bytes. High small-file creation rates can make the MDT a candidate bottleneck; byte throughput, metadata counters and client behaviour must distinguish it from OST or network limits. Fewer, larger files match both the inode limit and DCP's own shape better than a file per tensor.

4. On an object store, treat "upload started" as not saved. List in-progress multipart uploads before you assume a key is readable. Abort incomplete uploads when an attempt is abandoned, or you keep paying for parts. Use a new key for a new attempt when versioning is off.

5. Publish last. Same-mount rename, or one CompleteMultipartUpload, after every writer has finished. Then read the copy back with the same load path the restart will use, including weights_only and, for DCP, a reshard onto the topology you will actually resume on. A save that has never been loaded is not a checkpoint yet.

Risks and limits

• 16Ψ is mixed-precision Adam as ZeRO defined it in 2020, including gradients and an fp32 master. A different optimiser, or a run without that master copy, has a different file.
• Activation checkpointing is not this file. The paper uses that phrase for recomputation during the step.
• async_save is marked experimental in the PyTorch 2.15 page, stages to CPU by default, and is unfinished until the future completes. The save collective needs set_device under NCCL.
• Unrestricted pickle loading can execute code. weights_only=True narrows the unpickler but is not a complete security boundary. Use supported patched software and trusted, provenance-checked inputs; do not disable protections to make an untrusted module load.
• Lustre defaults and caps are from the 2.x manual as served. Confirm the release on the filesystem you mount. Stripe count 1 and a 1 MB stripe size are that manual's documented default example. The 2000-object and 160-OST sentences are limits in that text, not a tuning target.
• S3 limits and races are from the user guide opened 10 November 2026. Part size 5 MiB–5 GiB, 10,000 parts, 48.8 TiB objects, strong consistency for object PUT, eventual consistency for bucket configuration, and the versioning rule that the latest initiate wins. Incomplete parts are billed until complete or abort.
• rename(2) is atomic for the directory entry on one mount. EXDEV across mounts. It does not flush a disk by itself; a power loss can still lose data the filesystem had not committed. This article did not open a durability-test procedure.
• No NetLeverage storage product or service level is described.

Sources

1. PyTorch 2.15, torch.optim.Adam — https://docs.pytorch.org/docs/2.15/generated/torch.optim.Adam.html (opened 10 November 2026). First and second moments; state_dict / load_state_dict.
2. PyTorch 2.15, distributed checkpoint — https://docs.pytorch.org/docs/2.15/distributed.checkpoint.html (opened 10 November 2026). Multiple files, at least one per rank; resharding; async_save; NCCL set_device.
3. PyTorch 2.15, serialisation — https://docs.pytorch.org/docs/2.15/notes/serialization.html (opened 10 November 2026). Pickle by default; weights_only since 2.6.
4. Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, arXiv:1910.02054v3, 13 May 2020 — https://arxiv.org/pdf/1910.02054 (opened 10 November 2026). $K=12$, $16\Psi$, 24 GB for 1.5B GPT-2.
5. Lustre Software Release 2.x manual — https://doc.lustre.org/lustre_manual.xhtml (opened 10 November 2026). MDS/MDT, OST, default stripe example, 64 KB multiple, stripe count -1, 2000 objects, ea_inode 160, lfs getstripe.
6. Amazon S3 User Guide, multipart overview — https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html (opened 10 November 2026). Complete assembles the object; incomplete parts billed; versioning race.
7. Amazon S3 User Guide, multipart limits — https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html (opened 10 November 2026). 48.8 TiB, 10,000 parts, 5 MiB–5 GiB.
8. Amazon S3 User Guide, What is Amazon S3 — https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html (opened 10 November 2026). Strong read-after-write for object PUT and DELETE; bucket configuration eventual.
9. Linux man-pages, rename(2) — https://man7.org/linux/man-pages/man2/rename.2.html (opened 10 November 2026). Atomic replacement of an existing name; EXDEV across mounts.

This article was originally published on 10 November 2026.