Large Uploads Stall or Restart: Multipart, Resume and Leftover Parts

Direct answer: Whether an interrupted upload can continue depends on the protocol and the tool, not the file size.

• S3: A multipart upload keeps its finished parts until the upload is completed or aborted. A tool that remembers the upload ID can continue by uploading only the missing parts. A tool that does not will start a new upload, and the old one's parts stay in the bucket, billed but invisible as an object, until something aborts them. List incomplete uploads with ListMultipartUploads, abort the ones that are really abandoned, and add an AbortIncompleteMultipartUpload lifecycle rule so leftovers are cleaned up automatically after a set number of days.
• SFTP: OpenSSH's reput/reget (or put -a/get -a) append to the existing partial file. The manual warns that if the partial copy differs from the source, the result is likely to be corrupt.
• rsync: --partial keeps an interrupted file so the next run can reuse it, and --partial-dir keeps it out of the way while it is incomplete. --append is safe only for files that are known to grow and never change.

Whichever method you use, check the finished file's integrity before you rely on it.

Why it matters

Large transfers fail more often than small ones simply because they run longer. A dropped connection, a change of route or a maintenance window on either side becomes more likely the longer the job runs. When a failure late in a ten-hour upload means another ten hours, the job can miss its window every night and never finish. If the stalls happen at the same point every time, also consider the path itself: a path MTU problem can make large transfers hang while small requests work. See the path MTU article listed under Related articles.

On S3 there is a second, quieter cost. The Amazon S3 User Guide says that after you start a multipart upload "there is no expiry; you must explicitly complete or stop the multipart upload". Until you do, "you are billed for all storage, bandwidth, and requests for this multipart upload and its associated parts". The parts do not form an object until the upload is completed; they are listed with ListMultipartUploads and ListParts, not as objects. A job that fails and restarts every night can keep adding parts that nobody sees until the bill or a storage report shows them.

Resume options that skip checks can also produce a file that looks finished but is wrong. That is worse than a failed upload, because nobody tries again.

How it works

S3 multipart upload

A multipart upload has three steps: create (initiate), upload parts, complete. The Amazon S3 User Guide sets out the rules:

Points from the multipart documentation that matter for resume:

• The upload ID is the handle. CreateMultipartUpload returns it, and every later call (upload part, list parts, complete, abort) needs it. A tool that does not save the upload ID across a crash or restart cannot resume that upload.
• Parts are independent. Parts can be uploaded in any order. If one fails, only that part is retried. Uploading a new part with the same part number overwrites the earlier part.
• Completion assembles the object. CompleteMultipartUpload takes the list of part numbers and their ETags, and S3 concatenates the parts in ascending part-number order. The CompleteMultipartUpload reference lists the errors you may see when it goes wrong:
• InvalidPart: a listed part is missing, or its ETag does not match.
• InvalidPartOrder: the list is not in ascending order.
• EntityTooSmall: a part other than the last is under 5 MB.
• NoSuchUpload: the upload ID is invalid, or the upload was already completed or aborted.

Completion can take several minutes to finalise. While it runs, S3 may already have sent 200 OK, and the reference warns that a request can still fail afterwards: the error is then embedded in the 200 response body. AWS SDKs detect this and raise an error; anything calling the API directly must parse the whole response. HTTP status alone does not tell you the upload completed. - Abort frees the parts, but check. The AbortMultipartUpload reference says that after an abort no more parts can be uploaded with that upload ID and stored parts are freed. However, part uploads that are still in progress "might or might not succeed", so it "might be necessary to abort a given multipart upload multiple times". To confirm, call ListParts and check the list is empty. The multipart overview adds that once you stop an upload you are not billed for its parts. - Part size limits object size. Because an upload can have at most 10,000 parts, the part size the tool uses caps the object size: part size × 10,000. The AWS CLI's multipart_threshold and multipart_chunksize settings, for example, control when it switches to multipart and the part size it uses (the threshold defaults to 8 MB). Check how your tool chooses part size for very large files. Do not assume that small parts scale up without limit. - Multipart is recommended for larger objects. AWS recommends considering multipart for objects of 100 MB or more, partly because over an unreliable network "you only need to retry uploading the parts that are interrupted".

Incomplete uploads and lifecycle clean-up

Two API calls show what is outstanding:

• ListMultipartUploads lists in-progress uploads in a bucket, with their key, upload ID, initiator, storage class and Initiated time. It returns up to 1,000 per request and paginates with NextKeyMarker and NextUploadIdMarker.
• ListParts lists the parts of one upload, with part number, size, ETag and last-modified time. It returns up to 1,000 per request. If the upload uses SSE-KMS or DSSE-KMS, the ListParts reference says the caller also needs kms:Decrypt on the key for the request to succeed.

A bucket lifecycle rule with the AbortIncompleteMultipartUpload action tells S3 to stop uploads that are not completed within a set number of days after initiation, and to delete their parts. The Amazon S3 User Guide recommends this as a best practice to keep storage costs down. It also states that:

• the rule applies to existing incomplete uploads as well as future ones;
• it has no effect on completed uploads;
• it does not delete objects;
• no early-delete charges apply.

A rule can cover the whole bucket or be limited to a prefix. Lifecycle actions are not instantaneous: the lifecycle documentation notes there may be a delay between eligibility and removal. The rule is a safety net, not a substitute for jobs that clean up after themselves.

One sizing point: the timer counts from initiation, not from the last part. Set the number of days longer than your longest legitimate upload, including any resume window, or the rule will abort an upload that was still going to finish.

Integrity after a resume

S3 can check what you upload. Supported checksum algorithms include CRC-64/NVME (the default), CRC-32, CRC-32C, SHA-1, SHA-256 and others. AWS-owned clients calculate a checksum and S3 validates it before storing the object. For multipart uploads there are per-part checksums and a whole-object checksum, which is either composite or full-object depending on the type and algorithm. The published article on verifying a delivered backup, listed under Related articles, explains how to read and compare these, and why a multipart ETag is not an MD5 of the file. This article does not repeat that. The point here is that a resumed upload must still end with that check.

SFTP resume (OpenSSH sftp)

In OpenSSH's sftp, reput resumes an upload and is equivalent to put -a; reget resumes a download and is equivalent to get -a. The command-line option sftp -a attempts to continue interrupted transfers instead of overwriting existing partial or complete copies. The manual's warning applies to all of these: resumption "assumes that any partial copy … matches" the source, and if it differs "the resultant file is likely to be corrupt".

In practice, appending is safe only when:

• the partial file at the destination was written by an interrupted transfer of the same source file, and
• the source has not changed since.

If either is in doubt, transfer the whole file again, or verify the result against a checksum computed at the source. The OpenSSH sftp(1) manual documents no command for obtaining a checksum of a remote file, so you need a hash computed on both sides. A manifest delivered with the file supplies the expected digest, but the receiver must still compute the file's actual digest and compare.

rsync partial transfers

The rsync manual is explicit about each option:

• Default: rsync deletes a partially transferred file if the transfer is interrupted.
• --partial keeps the partial file, "which should make a subsequent transfer of the rest of the file much faster". The partial file sits at the destination name, so a reader could see an incomplete file there.
• --partial-dir=DIR keeps partial files in DIR instead of at the destination name. The next run uses them to resume and then deletes them. If --whole-file is specified or implied, any partial-dir files found are simply removed, because rsync is not using its delta-transfer algorithm.
• --append is for files "known to be growing larger" whose existing content on the receiver matches the sender. The manual says it updates such files "without verifying any of the existing content" and can be "dangerous" if you are not sure.
• --append-verify works like --append but includes all of the file's data in the checksum verification. It is less efficient, but potentially safer. The manual still calls it dangerous unless every file is a shared, growing file. Like --append, it updates files in place and skips receiver files that are not shorter than the sender's. Before rsync 3.0.0, --append behaved like --append-verify.
• --inplace updates files in place. It implies --partial, and conflicts with --partial-dir and --delay-updates.

For backup files or archives that are rewritten rather than grown, --partial-dir without --append is the safer combination. rsync's normal delta transfer then reuses what it can, and the whole-file check still applies.

Read-only checks

None of these was run for this article. Run them from an administrative identity, not from the delivery credential, unless that credential has been given exactly these list permissions. They make list and read requests only. The S3 listing calls are ordinary requests and may be billed as such. Do not paste credentials or signed URLs into tickets.

1. Are there incomplete multipart uploads, and how old are they?

# Read-only. Needs s3:ListBucketMultipartUploads on the bucket. Paginate if more than 1,000.
aws s3api list-multipart-uploads --bucket example-delivery-bucket --prefix deliveries/

Record each upload's key, upload ID, initiator and Initiated time. Several uploads for the same key started on successive nights suggest a job that restarts instead of resuming, but are not proof. Correlate each upload ID, initiator and time with the job's logs, the tool's documented retry and abort behaviour, and who owns the upload. Allow for any pause-and-resume window the job uses. Uploads older than any legitimate transfer are candidates for abandonment until that review is done.

2. How much does an incomplete upload hold?

# Read-only. Needs s3:ListMultipartUploadParts, plus kms:Decrypt on the key if the upload uses SSE-KMS or DSSE-KMS.
# Paginate with --part-number-marker if more than 1,000 parts.
aws s3api list-parts --bucket example-delivery-bucket --key deliveries/site-a/archive-2026-11-01.tar --upload-id <upload-id-from-check-1>

Add up the part sizes to see what is stored. Note the most recent part's LastModified. An upload whose last part was written minutes ago may still be running.

If ListParts returns AccessDenied, the inspection is incomplete. It does not mean the upload is empty, and after an abort it does not prove clean-up. Grant the missing KMS permission to the administrative identity through the key and IAM policies. Do not broaden the delivery credential to run an administrative check.

3. Is a clean-up rule already in place?

# Read-only. Needs s3:GetLifecycleConfiguration. An error saying no configuration exists means there are no rules.
aws s3api get-bucket-lifecycle-configuration --bucket example-delivery-bucket

Look for a rule with AbortIncompleteMultipartUpload, its DaysAfterInitiation, its filter (whole bucket or a prefix), and whether it is Enabled. Keep a copy of the full output: any later change replaces the whole lifecycle configuration.

4. What does the transfer tool do on failure?

Read the tool's documentation and its log from a failed run:

• Does it save the upload ID and resume?
• Does it abort on failure, and does its credential allow s3:AbortMultipartUpload? See the credentials article listed under Related articles.
• Does it simply start again?
• What part size does it use?

For SFTP and rsync jobs, record the exact options: -a, reput, --partial, --partial-dir, --append, --inplace.

5. Is the destination file at an SFTP or rsync target complete?

# Read-only on each side. Compare a hash computed at the source with one computed at the destination.
sha256sum /data/exports/archive-2026-11-01.tar          # on the source host
sha256sum /srv/incoming/archive-2026-11-01.tar          # on the destination host, if you have shell access there

Reading a very large file takes time and disk I/O on a production host, so schedule it. If you have no shell access at the destination, use whatever verification the provider documents, or have the receiver compute the digest and compare it with a delivered manifest.

Interpreting the results

Decision guidance

Make changes under change control. Aborting an upload and changing a lifecycle configuration cannot be undone for the data involved.

Decision 1: Abort abandoned multipart uploads

For each upload from Check 1 that is older than any legitimate job, has no recent part activity (Check 2), has been matched to its producer through logs and ownership, and whose producer is confirmed not to resume it:

# Changes data: deletes this upload's parts. Cannot be undone. Needs s3:AbortMultipartUpload.
aws s3api abort-multipart-upload --bucket example-delivery-bucket --key deliveries/site-a/archive-2026-11-01.tar --upload-id <upload-id>

Abort one upload ID at a time from your reviewed list. Do not script "abort everything in the bucket", which could stop an upload that another job or another team is running. After each abort, run ListParts for that upload ID. Per the AbortMultipartUpload reference, repeat the abort if parts were still arriving, until listing shows no parts or the upload no longer exists.

• Verification: ListMultipartUploads no longer shows the upload. The completed objects under the prefix are unaffected: abort does not touch completed objects.
• Rollback: None. The parts are deleted. The only recovery is to upload the file again, so be sure the upload is abandoned before aborting it.

Decision 2: Add or adjust an AbortIncompleteMultipartUpload rule

Choose DaysAfterInitiation longer than your longest legitimate upload, including resume windows. Scope the rule to the delivery prefix if other users of the bucket have different needs. Then:

1. Merge. The PutBucketLifecycleConfiguration reference says writing a configuration overwrites the existing one. Merge the new rule into the existing configuration saved in Check 3. Do not write a file containing only the new rule.
2. Review. Check the merged document with the bucket owner. Expiration rules elsewhere in it delete objects.
3. Apply it with put-bucket-lifecycle-configuration.

• Verification: get-bucket-lifecycle-configuration shows every previous rule plus the new one. After the rule's period has passed (allowing for lifecycle delay), ListMultipartUploads shows no uploads older than the threshold.
• Rollback: Re-apply the saved configuration from Check 3. Uploads already aborted by the rule cannot be restored.

Decision 3: Make the S3 upload resume instead of restart

Use a tool or SDK feature that records the upload ID and the completed parts, and continues the same upload after an interruption. Check its documentation for this explicitly.

S3 assembles whatever parts were uploaded for that ID, and valid part checksums do not show the parts came from the same version of the source. Reuse saved parts only when the source is provably unchanged: a stable snapshot or export, or a source identity check (size, modification time and a recorded digest). Tie the saved state to the bucket, key, upload ID, source revision and part layout with ETags. If source identity cannot be established, start a fresh upload of a stable source and abort the obsolete one deliberately (Decision 1). After completion, compare the object's checksum with an expected value computed from that source; S3's validation of the transmitted parts alone is not enough. Give its credential s3:ListMultipartUploadParts if it lists parts to resume, and s3:AbortMultipartUpload if it should clean up when it gives up. Choose a part size that keeps the largest expected file under 10,000 parts and every part except the last at least 5 MiB. Keep checksum validation enabled.

• Verification: After a deliberately interrupted test upload of a non-production file, the next run uploads only the missing parts (its log or ListParts shows the earlier parts reused) and completes. The object's checksum matches the expected value computed from the source, as described in the published verification article. A test that changes the source between runs is refused reuse and starts a fresh upload. No incomplete upload is left for that key.
• Rollback: Revert the tool or configuration change. Then abort any upload the test left behind (Decision 1).

Decision 4: Make SFTP and rsync resume safely

• rsync: prefer --partial-dir=.rsync-partial (a relative directory, which rsync creates and removes as needed, and excludes from the transfer) over --partial. Avoid --append unless the files only ever grow. If you must append, use --append-verify, which carries the same restriction to shared, growing files, works in place (and so conflicts with --partial-dir), and leaves the partial at the final name. To keep partials hidden while appending, transfer into a staging directory and publish by rename.
• SFTP: use reput/put -a only for a partial that you know came from an interrupted transfer of the same unchanged source. Otherwise upload the whole file again. Upload under a temporary name and rename it once complete, so that readers and downstream jobs never pick up a partial file. Agree the naming convention with whoever consumes the files.

Whichever route, compare a source-side hash with one computed at the destination (Check 5), or deliver a checksum manifest against which the receiver computes and compares the actual digest.

• Verification: After an interrupted test transfer, the next run completes and the destination hash matches the source. No partial file is visible under the final name during the transfer.
• Rollback: Restore the previous options. Delete any partial or temporary files the test created at the destination, after confirming they belong to the test.

What not to do

• Do not abort every incomplete upload in a bucket by script. Review each one and confirm its producer will not resume it.
• Do not take HTTP 200 or a bare existence check as proof of completion. Verify the object's checksum.
• Do not set a lifecycle abort period shorter than your longest upload.
• Do not write a lifecycle configuration containing only the new rule. That removes the bucket's other rules.
• Do not use --append or reput on files that may have changed since the partial copy was written.
• Do not count a file as delivered because its size matches. Compare checksums.
• Do not widen the delivery credential to s3:* to let a tool clean up. Grant the specific multipart action it needs.

Limitations

• AWS and named tools only: The article covers Amazon S3 general purpose buckets, OpenSSH sftp and rsync, from their current documentation. Other clients (graphical SFTP tools, backup software, SDK transfer managers) implement resume differently; use their documentation.
• S3-compatible services: These may differ in part limits, listing behaviour and lifecycle support.
• Out of scope: Throughput tuning and measured transfer speeds are out of scope, and no figures are given.
• No lab testing: None of the commands was run for this article.

Acceptance checklist

• [ ] Incomplete multipart uploads on the delivery bucket were listed with age and size (Checks 1–2).
• [ ] Each aborted upload was matched to its producer, confirmed not to be resumed, reviewed individually and confirmed gone; completed objects were untouched (Decision 1).
• [ ] Resume reuses saved parts only for a provably unchanged source; completion outcomes are established by checksum, not HTTP status (Decision 3).
• [ ] An AbortIncompleteMultipartUpload rule exists, longer than the longest legitimate upload, merged with the existing lifecycle rules (Decision 2).
• [ ] The transfer tool's failure behaviour is known: resume, abort or restart (Check 4).
• [ ] Interrupted test transfers resume and finish with a matching checksum (Decisions 3–4).
• [ ] Readers never see partial files under the final name.

References

All accessed 2 November 2026.

• AWS, Amazon S3 User Guide, "Amazon S3 multipart upload limits": maximum object size, parts, part numbers, part sizes, list limits. https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html
• AWS, Amazon S3 User Guide, "Uploading and copying objects using multipart upload": three steps, upload ID, independent parts and retry, same part number overwrites, no expiry until complete or stop, billing for parts, no billing after stop, 100 MB recommendation, lifecycle recommendation. https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
• AWS, Amazon S3 User Guide, "Configuring a bucket lifecycle configuration to delete incomplete multipart uploads": AbortIncompleteMultipartUpload, DaysAfterInitiation, applies to existing uploads, no effect on objects or completed uploads, no early-delete charges, prefix scope. https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html
• AWS, Amazon S3 User Guide, "Managing the lifecycle of objects": delay between eligibility and action; list before changing rules. https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html
• AWS, Amazon S3 User Guide, "Lifecycle expiration general considerations": billing during lifecycle delay. https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html
• AWS, Amazon S3 User Guide, "Checking object integrity in Amazon S3": supported algorithms, CRC64NVME default, client and server checksum validation, multipart checksum types. https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
• AWS, Amazon S3 API Reference, "ListMultipartUploads": fields including Initiated, 1,000 limit, pagination markers. https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html
• AWS, Amazon S3 API Reference, "ListParts": part fields, 1,000 limit, part-number-marker, kms:Decrypt required for SSE-KMS/DSSE-KMS uploads. https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html
• AWS, Amazon S3 API Reference, "AbortMultipartUpload": parts freed, in-progress parts may succeed, repeat abort, confirm with ListParts. https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html
• AWS, Amazon S3 API Reference, "CompleteMultipartUpload": ascending concatenation, parts list with ETags, InvalidPart, InvalidPartOrder, EntityTooSmall, NoSuchUpload (upload may have been aborted or completed), completion may take minutes, errors embedded in a 200 OK response. https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html
• AWS, Amazon S3 API Reference, "PutBucketLifecycleConfiguration": a new configuration overwrites the existing one; retained details must be included. https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html
• AWS CLI, "AWS CLI S3 Configuration": multipart_threshold (default 8MB) and multipart_chunksize. https://docs.aws.amazon.com/cli/latest/topic/s3-config.html
• OpenBSD manual, sftp(1): -a, get -a/put -a, reget/reput, corruption warning when partial copies differ. https://man.openbsd.org/sftp.1
• rsync project, rsync(1) manual: default deletion of partial files, --partial, --partial-dir, --whole-file interaction, --append, --append-verify (shared growing files, in place, skips receiver files not shorter), --inplace. https://download.samba.org/pub/rsync/rsync.1

Related articles

• Verify a Backup Delivered to S3 or SFTP Before Recovery

• Scheduled Jobs After Migration: Prevent Duplicate Work

• S3 Credentials for Automated Delivery: Least Privilege and Rotation

• Small Requests Work but Large Transfers Hang: Proving a Path MTU Black Hole
• SFTP Host Key Changed: Verify It and Rotate Keys Safely

Learning notes

1. An S3 multipart upload never expires on its own. Its parts are billed until it is completed or aborted, and they are not visible as objects.
2. Resume needs the upload ID. A tool that does not keep it starts again and leaves the old parts behind.
3. An AbortIncompleteMultipartUpload rule cleans up automatically. Its timer counts from initiation, so set it longer than your longest real upload.
4. SFTP reput and rsync --append trust the partial copy. Use them only when it certainly came from the same unchanged source.
5. A resumed transfer still ends with a checksum comparison against the source. A matching size, a 200 response or an object that exists is not proof.
6. Saved multipart parts are reusable only when the source has not changed since they were uploaded.