Retries on Brian Faust

Retry Budgets Multiply Across Layers

Wed, 01 Aug 2018 00:00:00 +0000

An HTTP connector allows three attempts. The queue job calling it also allows three attempts.

When the remote service keeps failing, the job does not make three calls. It makes nine:

job attempt 1 -> connector calls 1, 2, 3
job attempt 2 -> connector calls 4, 5, 6
job attempt 3 -> connector calls 7, 8, 9

Both settings look modest in isolation. The remote system receives their product.

Retries Can Multiply the Original Failure

Mon, 01 Aug 2016 00:00:00 +0000

Every failed job went back to the queue for another attempt 60 seconds later. There was no terminal condition and no meaningful failure classification. A malformed request received the same treatment as a short provider outage.

During long outages, roughly 50,000 jobs could sit in the system and keep trying for days. The queue backend filled, workers spent their time reproducing known failures, and the original dependency problem became sustained load on our own application.