Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Postfix mail stuck in the queue as deferred: read the reason

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

Mail in Postfix's deferred queue is not lost. It failed with a temporary error, and Postfix will try again until the message is older than maximal_queue_lifetime (default 5 days). The reason for each failure is recorded: it appears in parentheses in postqueue -p, and after status=deferred in the mail log.

Read that reason before you change anything. It points to a different fix each time (DNS, a blocked port, a remote server throttling you, a TLS or authentication problem), and flushing the queue repeatedly makes things worse, not better.

What the symptom means

Postfix keeps several queues. The QSHAPE documentation describes them: mail arrives in the incoming queue, moves to active while Postfix is trying to deliver it, and lands in deferred when an attempt fails with a temporary error. A fourth queue, hold, holds mail on purpose.

How often Postfix retries is documented. The qmgr manual says "mail that cannot be delivered upon the first attempt is deferred. The time interval between delivery attempts is doubled after each attempt" (exponential backoff). The QSHAPE page adds that each deferred message gets a cool-off time of "at least $minimal_backoff_time and at most $maximal_backoff_time", and "the next retry time is set by doubling the message's age in the queue, and adjusting up or down to lie within the limits." The postconf(5) defaults:

Parameter Default Meaning
minimal_backoff_time 300s Minimal time between attempts for a deferred message
maximal_backoff_time 4000s Maximal time between attempts
queue_run_delay 300s Time between deferred queue scans
maximal_queue_lifetime 5d After this, a message that keeps failing with a temporary error is treated as undeliverable
bounce_queue_lifetime 5d Same limit for bounce messages
delay_warning_time 0h Time after which the sender gets a delay notice; 0h means the feature is off

Two consequences follow. A message that has failed for hours is retried roughly every hour (the cap of 4000 seconds is 4000 ÷ 60 = about 67 minutes), so it will not clear on its own the minute you fix the fault. And with delay_warning_time at 0h, nobody is told when a message is delayed; the first notice the sender receives may be the bounce after five days.

Likely causes

Cause Reason text you may see How to confirm
DNS cannot resolve the recipient domain's MX Name service error for name=... type=MX: Host not found, try again Check DNS from the mail server itself; see the FAQ's entry on this message
Outbound port 25 is blocked or the remote host is unreachable A connection failure to the remote mail server; the reason names its address and port Test a TCP connection to port 25 of the remote host from the mail server
The remote server is throttling or deferring you A 4xx reply from the other side, such as Gmail's 421 4.7.28 rate limit The reason shows the remote's own text; the same domain dominates the queue
Lost connection during the message transfer lost connection after DATA or timed out while sending end of data Repeats consistently for some destinations; the FAQ suggests path MTU discovery or a firewall
A relay or smarthost refuses you A 4xx or 5xx from your own relay, for example a login or policy failure The relay's text is in the reason
The queue is large because of bounces to forged senders Many messages from <>, huge deferred count, many unknown destinations qshape deferred shows hundreds of unrelated domains

Prove the cause before changing anything

All of these only read. Run them on the mail server, as a user allowed to read the queue.

1. List the queue with the reasons. postqueue -p produces "a traditional sendmail-style queue listing" in which "each queue entry shows the queue file ID, message size, arrival time, sender, and the recipients that still need to be delivered. If mail could not be delivered upon the last attempt, the reason for failure is shown." The queue ID can be followed by a status character: * means the message is in the active queue, ! means it is on hold, and # means it is forced to expire.

postqueue -p

For scripting, postqueue -j produces the same listing as JSON lines (Postfix 3.1 and later).

2. See the destinations and ages together. qshape "helps the administrator understand the Postfix queue message distribution in time and by sender domain or recipient domain". It reads the queue directories, so "it must run as the superuser or the mail_owner". Rows are domains; columns are age buckets, the first being 0 to 5 minutes by default and each following bucket twice as long as the last, the final bucket having no limit.

qshape deferred

The QSHAPE documentation's examples show a healthy queue as mostly empty, and a deferred queue full of dictionary-attack bounces as hundreds of unrelated domains. One domain with most of the count points at that destination. Many unrelated domains point at your own side (DNS, firewall, relay).

3. Read the log lines for one message. Take a queue ID from step 1 and search the mail log for it. The Postfix FAQ shows the form, with the reason in parentheses:

postfix/smtp[2349]: 74FBF30501: to=<recip@recip.domain> relay=none, delay=3944, status=deferred (Name service error for name=recip.domain type=MX: Host not found, try again)

Find where your system writes the mail log. The Postfix manual's examples use /var/log/maillog, and Postfix 3.4 and later can also write its own file named by maillog_file.

4. Look at one message. postcat -q QUEUEID prints the queue file in readable form; the -q option "search[es] the Postfix queue for the named files instead of taking the names literally". By default it shows the envelope and content ("as if the options -beh were specified"). Read the sender, recipient and headers to check whether this is mail you expected to send.

Example (illustrative): On mail01, postqueue -p shows 240 messages, nearly all with a reason that describes a connection time-out and a recipient at example.net. qshape deferred shows a single row, example.net, with the count growing in the 40 to 160 minute buckets. The log lines show relay=mx.example.net[198.51.100.20]:25 with a time-out reason. Only one destination is affected, which suggests the remote end or a path to it, not your DNS. A TCP test to 198.51.100.20 on port 25 from mail01 times out, and the same test from another network connects, so the problem is between mail01 and that host. A firewall or provider rule is the first thing to check, and "Outbound mail times out after a move: the new host cannot reach port 25" explains how to prove it.

Fix

Fix the cause the reason points to. Then, and only then, ask Postfix to retry.

  • DNS. The Postfix FAQ notes that "the result of nslookup for the hostname may be irrelevant. Postfix is required to look up the MX record first", that the check is invalid if run as a privileged user (Postfix lookups can fail because of wrong permissions on system files such as /etc/resolv.conf), and that a chrooted SMTP client needs files inside the queue directory. Test the MX record first, from the mail server, as the lookup Postfix makes.
  • Blocked port 25 or an unreachable host. Follow "Outbound mail times out after a move: the new host cannot reach port 25" and, where the provider blocks port 25, send through an authenticated relay.
  • A remote server throttling you. Slow the sending and fix the cause on your side; see "Gmail 421 4.7.28: Gmail has detected an unusual rate of email originating from your IP address". The Postfix parameters are default_destination_rate_delay and the per-transport form transport_destination_rate_delay, with the postconf(5) note that "the delay is enforced by the queue manager. The delay timer state does not survive postfix reload or postfix stop."
  • Rejected by your relay or by the receiver. Authentication problems are covered in "Application Email Rejected or Junked After a Server Move: SPF, DKIM, DMARC and PTR". If your own server refuses relays, see "Postfix 554 5.7.1 Relay access denied: who may send through the server".
  • A queue of bounces to forged senders. QSHAPE's advice is "recipient validation is strongly recommended through use of the local_recipient_maps and relay_recipient_maps parameters". That is a configuration project in its own right, not a quick fix.

Ask Postfix to try again

Disruptive step: postqueue -f "flush[es] the queue: attempt to deliver all queued mail". The manual warns: "flushing undeliverable mail frequently will result in poor delivery performance of all other mail." Run it once, after the fix, not on a loop. The narrower forms are postqueue -i QUEUEID, which "schedule[s] immediate delivery of deferred mail with the specified queue ID", and postqueue -s site, which schedules delivery of all mail queued for the named site (it must be eligible for the fast flush service). Prefer these when only one destination was affected. There is nothing to back out except stopping the flush; the messages stay in the queue if delivery fails again.

Hold, requeue or delete (use sparingly)

postsuper does maintenance on the queue and is restricted to the superuser.

Queue IDs are reused. The postsuper(1) manual warns: "Postfix queue IDs are reused" (always in older releases, and in current ones when enable_long_queue_ids is no), and "there is a very small possibility that postsuper deletes the wrong message file when it is executed while the Postfix mail system is delivering mail." So, before each of the commands below: re-list the queue with postqueue -p immediately beforehand, confirm that the ID still shows the sender and recipient you expect, act on one queue ID at a time, and never use ALL.

  • postsuper -h QUEUEID puts mail "on hold" so that no attempt is made to deliver it. The manual notes that while mail is on hold it does not expire, and postsuper -H QUEUEID releases it.

    Risk: a held message is not retried and does not expire until you release it, so held mail can sit unnoticed. Note the queue IDs. To back out, release them with postsuper -H QUEUEID.

  • postsuper -r QUEUEID requeues the message into the maildrop queue, where it is copied to a new queue file. The manual lists ways its handling differs from a new submission, so use it for a real reason and not as a retry button.

    Risk: a requeued message gets a new queue file and is handled as the manual describes for requeued mail, which differs from the original. The old queue ID no longer applies, and there is no step to restore the original file.

  • Disruptive step: postsuper -d QUEUEID deletes the named message from the queue. There is no way back: the message is removed from the queue and will never be delivered by this server. Write down the queue ID, sender and recipient first, and tell the sender if the mail mattered.

Confirm the fix

  1. After the single flush, postqueue -p shows the count falling, and qshape deferred shows the affected destination emptying.
  2. The mail log shows status=sent for messages that were deferred. Check one message by queue ID from start to finish.
  3. The sender or the recipient confirms they received a test message sent after the fix.
  4. Watch the queue for one retry cycle (at least queue_run_delay, 300 seconds by default) to confirm that new mail does not defer again.

What to watch afterwards

  • Record the size of the deferred queue and the age of the oldest message, and alert when either grows, instead of waiting for users to complain.
  • Consider setting delay_warning_time (off at 0h by default) if senders should hear about delays sooner than the bounce after five days.
  • Do not shorten maximal_queue_lifetime as a way of clearing the queue. It makes the bounce arrive sooner; it does not fix delivery.

Risks and limits

  • These commands have not been run or lab-tested. They are taken from the Postfix documentation listed below.
  • Reason texts differ by remote server and by Postfix version; the FAQ example is one real shape, not a list. The causes table is a guide to where to look, not a complete catalogue.
  • qshape and postsuper need the superuser. postqueue -p and postqueue -f are the unprivileged user interface.
  • The -j JSON listing and some status characters depend on the Postfix version, as marked on the manual pages.
  • A deferred message that reaches maximal_queue_lifetime is bounced; the bounce can itself be deferred.

Sources