Retries Can Multiply the Original Failure

Every failed job went back to the queue for another attempt 60 seconds later. There was no terminal condition and no meaningful failure classification. A malformed request received the same treatment as a short provider outage.

During long outages, roughly 50,000 jobs could sit in the system and keep trying for days. The queue backend filled, workers spent their time reproducing known failures, and the original dependency problem became sustained load on our own application.

The retry loop was intended to make work reliable. It made failure durable without making success more likely.

A retry is another production request

Retry configuration often looks like waiting:

try
wait 60 seconds
try again

The waiting is cheap when the queue stores a delayed message. The next attempt is not. It reserves a worker, loads application state, queries databases, deserializes payloads, acquires locks, writes logs, and calls the dependency again.

At the scale from the supplied incident, the first-order projection is uncomfortable:

50,000 jobs
× 1 attempt per minute
× 1,440 minutes per day
= 72,000,000 scheduled attempts per day

That is a theoretical upper bound if capacity can execute every scheduled attempt. It is not a retained production measurement. Worker limits, execution time, congestion, and queue ordering can reduce completed attempts while increasing backlog age.

The DBG-08 fixture keeps that distinction explicit. It calculates 72 million scheduled opportunities, not 72 million observed remote requests.

Even when only a fraction execute, every attempt competes with new work and with retries which might succeed. A retry policy is therefore also a load policy.

Classify before scheduling another attempt

The old loop asked one question:

Did this attempt fail?

A useful policy needs at least two:

Can the same operation succeed later without changing its input?
Is repeating it safe?

Those answers separate common failure classes:

“Transient” is not a synonym for “exception.” A parse error does not become valid after a minute. A missing required field does not heal overnight. Expired credentials need configuration work, not persistence from the queue.

Unknown outcomes deserve their own state. If a remote write may have succeeded before the response was lost, retrying can duplicate the effect. The previous article established that a timeout stops local waiting, not remote work. Retry policy must consume that result rather than flatten it into a generic failure.

Backoff changes pressure, not correctness

Increasing delay gives a dependency time to recover and reduces how often the application repeats a failure. It does not make a malformed request retryable or a non-idempotent write safe.

The DBG-08 fixture uses this present-day teaching schedule:

attempt 1: immediately
attempt 2: after       60 seconds
attempt 3: after      300 seconds
attempt 4: after    1,500 seconds
attempt 5: after    7,500 seconds
attempt 6: after   37,500 seconds
then: terminal failure

The five delays grow by a factor of five. The schedule is executable and finite, but it is not the historic repair schedule and it is not a production recommendation.

Current Laravel documentation allows a backoff array for queued work and uses successive values for later retries. It also supports maximum attempts or an expiry time . Framework mechanics make the policy expressible. They do not choose the failure classes, factor, maximum elapsed time, or safe side effects.

Backoff usually needs jitter when many jobs fail together. Without it, 50,000 jobs delayed for the same duration become 50,000 jobs eligible together. The fixture omits jitter deliberately so its schedule remains deterministic. A production design should record the jitter rule and its maximum effect on the deadline.

Provider instructions can override a local curve. A valid retry window from a rate-limit response is evidence about when capacity may return. It still needs a local maximum; a provider should not own unlimited use of our queue.

Finite attempts create a decision point

“Keep trying until it works” avoids deciding what failure means. It also means the job can outlive the customer need, the source data, credentials, code version, or business deadline which justified it.

A finite policy forces an answer:

after the last attempt:
    mark the operation failed
    preserve the last useful error
    record the final attempt time
    notify the right owner when action is possible
    release queue capacity

Laravel currently passes the final exception to a job’s failed method and represents attempt exhaustion explicitly. That hook is useful only if terminal state survives outside the worker.

The old system’s queue messages were effectively fire-and-forget. There was no durable intent record from which work could be selected again later. If a job was removed, rebuilding it safely was difficult. Keeping the same job alive became the path of least resistance.

That constraint explains the design without defending it. Queue retention was standing in for application state.

The queue is transport, not the record of intent

A durable operation record can be small:

operation identity
operation kind
input reference or immutable payload
status
attempt count
next eligible time
last failure class
last error summary
remote identity, when known
created, updated, and terminal times

The queue message then means:

attempt operation 42 now

It does not mean:

operation 42 exists only while this message survives

This separation allows a long outage to end the current retry series without forgetting the business request. A scheduler can select eligible failed or deferred operations 48 hours later, after a configuration repair, or after an operator decision. The new message carries the same operation identity and rechecks whether work is still required.

Forty-eight hours is an example of deferred selection, not a supplied schedule. The actual date should follow the business deadline and dependency evidence.

Durable intent also makes deletion less frightening. In the supplied incident, the accumulated retry jobs were eventually removed and replaced with a finite backoff policy. With an application record, clearing transport backlog does not erase which operations still need a decision.

This is not an argument to build a workflow engine for every email. Some work is cheap, reconstructible, and harmless to lose. The stronger record becomes valuable when the effect matters, the outage can be long, input may change, or operators need to decide between retry, repair, and abandonment.

Stop failures which input cannot repair

A malformed request should usually reach terminal failure on its first classified attempt. Repeating it hides the useful error behind queue noise.

The terminal record should retain why it cannot succeed:

class: invalid-input
field: destination postal code
decision: permanent failure
next action: correct the source data and submit a new operation

Do not mutate the queued payload silently until it passes. That produces an effect different from the one requested and makes the repair impossible to audit.

There are exceptions. A validation failure can be transient when validation depends on reference data which is known to be incomplete and will be refreshed. The classification then belongs to that named dependency:

unsupported country in immutable product rules -> terminal
catalogue generation not active yet           -> deferred

The exception type alone may not encode that distinction. Classify at the boundary which understands both the failure and the operation.

Retry only the smallest safe boundary

If a job performs five steps and the last remote call fails, retrying the entire job may repeat the first four.

load source
create local row
reserve funds
submit remote write
persist response

A safe retry needs one of three things:

• earlier steps are pure or idempotent under the same operation identity;
• durable state lets the job resume from the first incomplete transition; or
• compensation returns the operation to a state where the whole sequence is safe to run again.

Without that proof, “the queue retries the job” is an implementation detail which changes business behavior.

This article owns the classification and bounded lifecycle of one queued operation. A later article will examine what happens when an HTTP client, queue job, and workflow each apply their own retry count. Three attempts at one layer can become nine or 27 remote calls when layers multiply. That is a different failure from the 50,000-job backlog here.

Enforcement can require a decision, not prove it

A retained codebase later added a static-analysis rule for queued work. A recognized job must declare:

• a positive execution timeout;
• a positive attempt limit or retry expiry; and
• positive backoff when the job contains recognized external I/O.

That rule prevents a new job from silently inheriting unbounded worker defaults. It does not know whether the backoff is appropriate, whether the failure is transient, whether the payload is durable, or whether the effect is safe to repeat.

Use enforcement to make omissions visible. Use behavioral tests to prove the policy:

malformed input -> one attempt, terminal
transient outage -> increasing delay, then success
long outage -> finite attempts, terminal
unknown remote result -> reconciliation, no blind retry
terminal operation -> later queue delivery performs no effect

The DBG-08 fixture executes the first four policy shapes as deterministic state transitions. It does not run Laravel or a real queue. It cannot prove worker timing, Redis capacity, provider recovery, or production throughput.

Operate retries as accumulated risk

Queue depth alone cannot distinguish valuable new work from repeated failure. Useful retry observations include:

pending operations by failure class
attempts per completed effect
oldest next-eligible operation
operations at terminal failure
unknown remote outcomes awaiting reconciliation
attempt rate by dependency
backlog age by original creation time

The original creation time matters because every retry creates a fresh execution timestamp. A job attempted today may represent a customer request which stopped being useful three days ago.

An emergency response also needs a safe way to pause one failure class, quarantine malformed work, or postpone a dependency without stopping unrelated queues. Deleting 50,000 jobs is sometimes the necessary containment. It should be followed by a decision about which business operations were erased from transport and which still exist durably.

The incident did not show that retries are bad. It showed that an unclassified, unbounded retry is not recovery. It is a request generator whose stopping condition is somebody else’s return to health.

Retry when unchanged input can plausibly succeed later, repeating is safe, and the next attempt still serves a live operation. Increase the delay, stop permanently, and keep the reason for the work somewhere more durable than the message which happens to carry it today.