Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

S3 backup uploads fail with RequestTimeTooSkewed: the server clock, not the bucket

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

RequestTimeTooSkewed is an HTTP 403 from S3 whose documented meaning is "The difference between the request time and the server's time is too large." Every signed S3 request carries the sender's time. If the backup host's clock is far from the service's, S3 rejects the request before it looks at your permissions. Bucket policy, keys and KMS settings are not the problem, so changing them wastes time and can weaken access.

Fix the clock on the machine that signs the request, prove the offset first, and then confirm with a restore rather than a green upload.

What the error means

Requests to AWS are signed with Signature Version 4. The signature covers a UTC time stamp. According to Elements of an AWS API request signature, including the date and time "helps prevent third parties from intercepting your request and resubmitting it later". The time stamp "must be in UTC" in the format YYYYMMDDTHHMMSSZ, and you can send it in an x-amz-date header, a date header, or an X-Amz-Date query parameter.

Because the time stamp comes from the signer's clock, a host whose clock is wrong signs every request with the wrong time. The service compares it with its own clock. The SigV4 overview says: "In most cases, a request must reach AWS within five minutes of the time stamp in the request. Otherwise, AWS denies the request."

The S3 error list (Error) gives RequestTimeTooSkewed its description above with "HTTP Status Code: 403 Forbidden". It states no number of minutes. AWS documents five minutes for SigV4 requests "in most cases". MinIO, an S3-compatible service, sets the constant globalMaxSkewTime in its source code to "15 minutes skew allowed"; that governs MinIO, not AWS. Different services set their own limit, so a skew of a few minutes might pass on one and fail on another. A skew of hours or days fails on all of them.

The message wording on the wire may differ from the table description. Search your job log for the code RequestTimeTooSkewed, not for a whole sentence.

Other signing errors look similar. In the IAM troubleshooting page, a request time in the future gives Signature not yet current and an expired one gives Signature expired (troubleshooting SigV4). That page covers AWS signing generally; it does not say which S3 operations return which text.

Likely causes

Cause What you see How to confirm
Time service stopped or never started Clock drifts; errors begin after days or weeks timedatectl shows the system clock not synchronised; chronyc tracking shows Not synchronised
Machine restored or resumed after being off First backup after the restore fails; clock far behind Offset roughly equals the time the machine was down
Time service refuses a large correction Service running but the clock stays wrong Windows: MaxPosPhaseCorrection and MaxNegPhaseCorrection limits; chrony: maxchange
NTP traffic blocked (UDP 123) Time source unreachable chronyc sources or w32tm /query /peers shows no reachable peer
Clock set by hand, wrong zone or hour Local time looks right but UTC is hours out date -u against a trusted reference
Signer is a different system A script, gateway or container host, not the backup server Check the host that actually holds the credentials and signs

Prove the cause before changing anything

These commands are taken from the vendor documentation and have not been run. They only read.

On Linux with systemd, timedatectl status "show[s] current settings of the system clock and RTC, including whether network time synchronization is active" (timedatectl(1)); its example output includes System clock synchronized: yes and NTP service: active. Print the UTC time yourself with date -u and compare it with a trusted source.

If the host runs chrony (chronyc reference):

chronyc tracking
chronyc sources

tracking reports System time ("the current offset between the NTP clock and system clock") and Leap status, which can be "Normal, Insert second, Delete second or Not synchronised". sources lists the time sources chronyd is using.

On Windows (Windows Time service tools):

w32tm /query /status
w32tm /query /source
w32tm /query /peers
w32tm /stripchart /computer:<time-source>

/status displays the W32Time status (the page's /status /verbose sample includes Phase Offset and Last Successful Sync Time), /source the time source, and /stripchart shows the offset between this computer and another one.

Also read the limits the service works within. On Windows, w32tm steps the clock at once if the offset exceeds MaxAllowedPhaseOffset (300 seconds on domain members, 1 on stand-alone machines). It then applies MaxPosPhaseCorrection and MaxNegPhaseCorrection: on stand-alone machines these default to 54,000 seconds (15 hours), and the page says that for a larger change "it logs an event instead" of correcting. Domain controllers default to 172,800 seconds (48 hours); domain members default to 0xFFFFFFFF, meaning the service always corrects. Check with w32tm /query /configuration.

Example (illustrative): the backup host BK-LNX01 is restored from a three-day-old image after a storage failure. The first nightly job fails with RequestTimeTooSkewed. date -u shows a time just over 3 days behind a trusted clock. chronyc tracking reports Leap status : Not synchronised, and chronyc sources lists no usable source because a firewall rule blocks UDP 123 after the restore. The cause is the clock, and behind it the blocked time traffic. No bucket setting is involved.

Fix

Work from the cause, not the error.

1. Restore time synchronisation. Allow the NTP traffic (Windows W32Time uses UDP port 123 and "can only use UDP 123 as the source port"). On systemd hosts, timedatectl set-ntp true enables network time synchronisation (shown in the man page examples). Confirm the service is actually running and has a reachable source.

2. Bring a far-out clock back. By default chronyd corrects by slewing: "the system clock might be so far adrift that this slewing process would take a very long time". The chronyc manual describes makestep: "It tells chronyd to cancel any remaining correction that was being slewed and jump the system clock by the equivalent amount, making it correct immediately." It also warns: "BE WARNED: Certain software will be seriously affected by such jumps in the system time."

chronyc makestep

Risky step: a clock jump. Stop the backup job and any software sensitive to time moving backwards or forwards (databases, schedulers, replication, anything writing time-ordered logs) before you step the clock. A host that was a restored copy of production may also now carry production's jobs; do not step it while those can run. The chrony configuration page notes that "on most systems it is desirable to step the system clock only on boot, before starting programs that rely on time advancing monotonically forwards". Also read the configuration: a maxchange directive makes chronyd ignore, or even exit on, offsets larger than its limit ("A syslog message will be generated when an offset is ignored or it causes the exit").

3. Windows. w32tm /resync tells the computer to "resynchronize its clock as soon as possible"; add /rediscover to redetect the network configuration and sources first. If the offset is beyond the correction limit above (a stand-alone host more than 15 hours out), the service logs an event instead of fixing it. The page gives no step-by-step for that case, and warns not to change the registry values it lists. A practical step is to set the clock by hand to the correct time first; Microsoft does not describe this step on that page, and a manual change carries the same clock-jump risks as step 2.

4. A source of wrong time that is not the host. If the signer is a container, appliance or gateway, fix that system's time source. Follow that vendor's own documentation for containers and appliances.

Confirm the fix

  1. Offsets are small: chronyc tracking shows Leap status : Normal and a small System time, or w32tm /stripchart shows a small NtpOffset.
  2. The next backup uploads without RequestTimeTooSkewed.
  3. A successful upload is not proof. Download one recent backup object back from the bucket and compare its checksum with the source, then restore a sample into a scratch location and open it with the application that owns the data.

What to watch afterwards

  • Alert on time synchronisation state and offset on every backup host, not only on job failure. A stopped time service produces slow drift that fails only weeks later.
  • After any restore, resume or clone of a backup host, check the clock before its first scheduled run.
  • Record whether any other component signs requests with the same credentials; they need the same watch.
  • Recurrence signal: RequestTimeTooSkewed appearing again with a different host in the log, or with only certain jobs (those started by a different scheduler).

Risks and limits

  • These commands have not been run or lab-tested. They are taken from the documentation named in Sources.
  • Conflicting guidance on tolerance: the AWS SigV4 overview says "within five minutes ... in most cases"; the S3 error list gives no figure; MinIO's source allows 15 minutes. No S3-specific figure is documented by AWS, so none is claimed here.
  • Time stamp errors can share a symptom with other signing errors (SignatureDoesNotMatch). The S3 presigned URL guide lists an unsynchronised system clock among the causes of SignatureDoesNotMatch (using presigned URLs).
  • Stepping a clock backwards or forwards can disturb other software; test on a non-production host first.
  • Third-party S3-compatible services set their own tolerance; check theirs.

Sources