A Rate Limit Belongs to All the Workers Sharing It

One polling job receives a 429 response saying to try again in fifteen seconds. It releases itself for fifteen seconds.

During that pause, twenty other workers poll the same provider. They have not seen the response, so each sends another request into the closed window.

The first job obeyed the limit. The application did not.

Find the scope before choosing the delay

A rate limit applies to some shared identity: an API credential, account, tenant, endpoint family, source address, or a combination of them. Every worker using that identity spends the same remote capacity.

The retained polling boundary had many independent jobs for one provider. A job was unique per tracking number, which prevented two workers from polling the same number at once. It did nothing to coordinate different numbers sharing the provider quota.

Those are different scopes:

job uniqueness:
    one worker per tracking number

remote quota:
    all tracking numbers under one provider identity

Putting the pause on the job answers only when that job may try again. The provider’s refusal answers when any caller in the quota scope may try again.

Turn the refusal into admission state

The correction introduced one shared pause deadline for that provider.

worker A sends request
    -> provider returns 429 with 15-second hint
    -> store provider paused-until deadline
    -> release worker A

worker B starts 1 second later
    -> read provider paused-until deadline
    -> release for remaining 14 seconds
    -> do not resolve a client or send a request

The preflight check matters. Deferring after client construction or request signing may still consume local work. Deferring after the remote call defeats the point entirely.

The retained test sets nine seconds of shared pause state, runs another job, and proves the provider resolver is never called. That is stronger evidence than checking only that the job was released.

Store a deadline, not a copied delay

If the first worker stores 15, a second worker reading it three seconds later cannot tell whether twelve or fifteen seconds remain.

Store an absolute deadline:

paused_until = observed_at + retry_delay
remaining = max(0, paused_until - now)

The retained store writes a Unix timestamp with cache expiry at the same instant. Readers accept an integer timestamp, calculate the remaining seconds, and never return a negative delay.

An expired or malformed cache value means no active pause. That choice keeps a broken coordination record from blocking polling forever, though it favors availability over strict quota protection.

Prefer the remote window, then fall back deliberately

The observed provider placed its retry hint inside a text field in a JSON response. The job extracts a positive number of seconds from that field and uses it for both the shared deadline and its own release.

When parsing fails, the retained policy falls back to exponential delay:

attempt 1   15 minutes
attempt 2   30 minutes
attempt 3   60 minutes
attempt 4    2 hours
...
cap         12 hours

That fallback is intentionally conservative. A malformed hint is not permission to retry immediately.

A standard Retry-After header would be preferable when the provider supplies one. A documented machine-readable field is better than prose. Text parsing is sometimes the only available contract, but it should be isolated, tested with the exact accepted shapes, and allowed to fail into a safe policy.

Clamping the parsed value to at least one second prevents a zero-second busy loop. A maximum may also be necessary if an untrusted or broken response can request an absurd delay.

Do not let one provider freeze every integration

The pause key in the retained boundary is provider-specific. Jobs for other providers continue normally.

That isolation is part of the capacity model:

pause key = provider identity + credential scope

A single global rate-limited-until key would be easy to implement and costly to operate. One constrained integration could stall unrelated work. A key which is too narrow has the opposite failure: several credentials or workers keep spending the same quota independently.

The correct key follows the remote system’s accounting boundary. If each credential has its own quota, include credential identity without storing the secret. If the provider limits an entire account across credentials, key by that account instead.

Concurrent refusals need a monotonic deadline

Shared cache alone does not complete the algorithm.

Imagine two workers receive refusals close together:

worker A observes: retry in 30 seconds
worker B observes: retry in 5 seconds

If B writes last, a simple cache assignment shortens the pause to five seconds. The safe update is:

paused_until = max(existing_paused_until, proposed_paused_until)

That comparison must be atomic across workers. A lock, compare-and-set operation, or small cache-side script can own it.

The retained implementation uses a normal cache write. It proves shared deferral in the tested sequential cases, but not monotonic behavior under concurrent 429 responses. That is a real limitation, not a reason to discard the shared gate.

A gate and a throughput limiter solve different problems

The shared deadline reacts to a refusal. It prevents peers from continuing during a known closed window.

It does not pace requests before the first 429. If the provider grants one hundred requests per minute, a proactive limiter can allocate that budget across workers and avoid the refusal in the first place.

Both mechanisms may be useful:

throughput limiter:
    may this request spend capacity now?

shared refusal gate:
    has the provider told this scope to stop until a deadline?

The first models a known budget. The second carries fresh remote feedback. Neither replaces bounded job attempts or an explicit retry owner; INT-08 takes that problem separately.

Verify that peers stay quiet

The useful tests exercise a small timeline:

1. A provider-specific 429 with a fifteen-second hint records a pause and releases the refused job for fifteen seconds.
2. A peer arriving during a nine-second pause is released without resolving the provider client.
3. An unrelated provider ignores that pause.
4. A malformed hint uses the bounded fallback rather than retrying immediately.
5. Concurrent longer and shorter hints retain the later deadline once atomic maximum updates are implemented.

The public fixture executes the first four cases and shows the extra calls made when workers keep only local delay state. It does not claim concurrent cache correctness.

Rate limiting is not an error that happened to one job. It is a capacity decision for every worker sharing the remote quota. Once the refusal becomes shared admission state, waiting stops being a private act of politeness and starts protecting the integration as a whole.