Retry Budgets Multiply Across Layers

An HTTP connector allows three attempts. The queue job calling it also allows three attempts.

When the remote service keeps failing, the job does not make three calls. It makes nine:

job attempt 1 -> connector calls 1, 2, 3
job attempt 2 -> connector calls 4, 5, 6
job attempt 3 -> connector calls 7, 8, 9

Both settings look modest in isolation. The remote system receives their product.

Count total attempts before naming the policy

Configuration vocabulary is treacherous here. Some libraries use tries to mean total attempts. Others use retries to mean additional attempts after the first.

Write the arithmetic in total attempts:

maximum remote calls
    = workflow operations
    × job attempts per operation
    × connector attempts per job

With two workflow operations, three job attempts, and three connector attempts, the ceiling is eighteen calls.

This is only the call count. If each connector attempt waits thirty seconds, the elapsed-time budget can also exceed the job timeout. If each call has a side effect, the duplication risk grows with the same nesting.

An end-to-end budget should therefore name calls, elapsed time, and side-effect safety. “Three retries” names none of them clearly.

Give each layer a different question

Several layers may retry safely, but only when they own different failure classes.

The connector can handle a connection refused before any request left the process. It cannot decide whether a booking should still exist tomorrow.

The job can recover from a worker crash or a database deadlock. It cannot assume that a timed-out remote mutation did nothing.

The workflow can schedule a later reconciliation after a prolonged outage. That is a new decision recorded in business state, not another invisible spin through the same call stack.

Transport retry depends on request semantics

The retained HTTP policy distinguishes naturally idempotent reads, explicitly idempotent mutations, and mutations declared non-retryable.

That distinction matters more than the method name. A POST carrying a stable idempotency identity may be safe to repeat. A nominally idempotent update can still trigger duplicate notifications if the server’s implementation violates the contract.

A connector retry is reasonable when all of these hold:

• the failure is classified as transient;
• repeating the request is safe;
• the delay fits inside the caller’s time budget; and
• the connector returns one final outcome to its caller.

A timeout after sending a mutation creates a different state: the outcome is unknown. Retrying without a remote idempotency key or reconciliation query may repeat a completed side effect. INT-06 owns that identity problem in depth.

A job retries the whole unit

When a job retries, it reruns more than the HTTP call. It may reload state, reserve money, write audit history, send notifications, or perform another remote request.

The retained queue policy requires:

positive timeout
positive attempt limit or retry-until deadline
positive backoff when direct external I/O is detected

Those declarations prevent an accidental infinite loop and zero-delay hammering in the cases the rule can see. They do not prove that the job is safe to run again.

The job still needs:

• a durable identity for externally visible effects;
• a checkpoint or transaction boundary for local writes;
• classified terminal, retryable, and unknown outcomes;
• a backoff schedule appropriate to the dependency; and
• a failure path after the last attempt.

If the connector already spends three quick attempts on connection setup, the job should not spend three more attempts on the same unclassified exception by default. The job may instead own failures which outlive one process, such as a queue worker crash before local completion.

A workflow schedules another operation

A long outage should not keep one job alive for days merely to preserve intent.

The workflow can record:

operation: fetch remote statement
last outcome: provider unavailable
next eligible time: 2018-08-02T09:00:00Z
attempt generation: 4
terminal deadline: 2018-08-05T00:00:00Z

At the next eligible time, it creates or releases a new durable work unit. The decision is visible, cancelable, and repairable. Operators can see whether the business still wants the work.

That is different from a queue retry. The payload may change, credentials may rotate, the remote window may reopen, or the business deadline may pass.

A workflow attempt still contributes to the multiplication. Two scheduled operations containing three job attempts and three connector attempts permit eighteen remote calls unless the inner budgets change.

Static rules can force declarations, not judgment

The retained codebase uses static checks for several useful omissions:

• connectors without positive connection and request timeouts;
• connectors which neither throw nor retry failed responses;
• idempotent requests without multiple tries;
• mutating requests without an explicit retry-safety decision;
• non-retryable mutations which fail to set one attempt;
• queued work without timeout or termination; and
• directly visible queued external I/O without backoff.

Another rule checks selected externally visible boundaries for a uniqueness interface, cache lock, or key-shaped method call.

These rules make missing decisions noisy. They cannot establish that tries = 20 is sensible, that a method named idempotencyKey is correct, or that an indirect client call is external I/O. They also inspect each class separately, so a connector with three attempts and a job with three attempts both pass without anyone noticing the nine-call composition.

Policy lint is a prompt for review, not evidence of runtime resilience.

Test the composed timeline

Unit tests for one connector and one job can both pass while their composition remains dangerous. Add a boundary test which counts remote calls under the same persistent failure:

connector attempts: 3
job attempts:       3
expected calls:     9

Then test the failures each layer claims:

1. A brief idempotent transport failure succeeds inside the connector and does not trigger a job retry.
2. A known local transient failure retries the job within its durable bound.
3. An uncertain remote mutation stops automatic retry and enters reconciliation.
4. A prolonged outage creates a visible later workflow operation only while the business deadline remains live.
5. Exhaustion records a terminal outcome instead of silently dropping intent.

The public fixture executes the multiplication and the first three ownership decisions with synthetic failures. It does not prove that three attempts are the right budget for any real integration.

Retries do not become safer because each layer chose a small number. Write the product, assign each failure class to one owner, and stop automatically when the outcome is unknown. The remote system experiences the complete call tree, not the tidy configuration file where each branch was declared.