A Third-Party API Is an Unreliable Collaborator

2018.02.01

A scheduled import begins by listing remote orders. The list request returns a 502 HTML page, so there are no trustworthy item identifiers to process.

On another run the list succeeds. One order then returns a 504 while its details are being loaded. The remaining orders are still independent and can continue.

These failures share a provider, but not a recovery boundary. Treating both as “the API is down” either discards useful work or pretends to know more than the response established.

Fail at the smallest unit whose state is known

The first question after a remote failure is not “should this throw?” It is:

what work can still be proved independent of the failed call?

For the failed list request, the importer does not know which items exist. Continuing with an empty list would turn “unknown” into “nothing to do.” The run should stop without declaring completeness.

For a failed item hydration, the importer already has the list and the failed item’s identity. It can record that one gap and continue with siblings whose inputs do not depend on it.

The retained importer follows this split for intermittent server errors:

list endpoint returns 5xx:
    record the failed operation
    stop this import run

one item endpoint returns 5xx:
    record the item and failed operation
    skip this item
    continue independent items

non-5xx response:
    preserve the ordinary failure path

Tests pin 502 at enumeration and 504 during item hydration. They prove the control flow, not that every future 5xx is transient or safe to ignore.

“The next run will get it” needs evidence

Skipping one item is containment, not recovery.

Later recovery works only if all of these remain true:

another run is guaranteed to happen;
the list will still include the skipped item;
its eligibility will not change before then;
the next run can distinguish completed work from missing work;
repeating completed siblings is safe; and
someone can detect a gap which never closes.

Without those properties, “the next run” is a hope disguised as a design.

A durable recovery record can be small:

{
  "source": "provider-demo",
  "operation": "hydrate-item",
  "item_id": "item-demo-42",
  "first_failed_at": "2026-02-01T08:10:00Z",
  "last_failed_at": "2026-02-01T08:10:00Z",
  "attempts": 1,
  "state": "pending"
}

The scheduler may still rediscover the item naturally. The record exists so recovery does not depend on that assumption. It also creates an observable terminal state: completed, superseded, permanently rejected, or requiring operator attention.

The retained importer does not provide evidence of such a ledger. Its later-run recovery is therefore a design expectation, not a guarantee.

A response body can violate the advertised protocol

HTTP status is only one boundary. The response might be syntactically valid HTTP and still be unusable:

expected:
    XML booking response

received:
    <html><body>502 Bad Gateway</body></html>

A retained integration test supplies that exact shape and requires an explicit parse failure. That prevents an HTML gateway page from leaking into later code as a partially populated booking result.

The order of checks matters:

connection and TLS completed
HTTP status is classified
content type and syntax are acceptable
required response fields exist
business result is accepted

Each step can fail after the preceding one succeeded. A 200 with malformed XML is not a successful booking. A valid XML error is not malformed. A 504 with an HTML body is primarily an unavailable upstream, even though parsing it as XML would also fail.

Classification should retain the first useful boundary rather than whichever exception happened to be easiest to catch.

Unavailability has more than one shape

In first-hand operation, one accounting provider had recurring outages and expired TLS certificates. A carrier also scheduled maintenance during peak operating hours.

Those conditions reach different layers:

Condition	First failed boundary	Useful evidence
expired certificate	TLS identity and validity	certificate error, endpoint, observation time
connection refusal	network connection	target, timeout or refusal, attempt
502 or 504	HTTP upstream path	status, operation, bounded response
maintenance response	remote availability	status, retry window, provider notice
malformed payload	representation parser	expected format, bounded sample, schema version
rejected operation	remote business rules	stable error code, safe message, item identity

Collapsing them into integration_failed prevents different recovery. A certificate problem will not improve because one item is retried. A malformed request will not improve after maintenance. A declared retry window should not be replaced with an arbitrary ten-second loop.

RFC 9110 gives 502, 503, and 504 distinct HTTP meanings. It also permits a 503 response to carry Retry-After. Those semantics are inputs to policy, not proof that the remote side used them correctly.

Diagnostic context should be useful and bounded

The retained server-error path records:

HTTP status;
operation name;
remote URL;
tenant or account identity;
item identity when known; and
a whitespace-normalized, length-bounded body snippet.

That is enough to distinguish list failure from item failure and a gateway page from a structured remote error. Bounding the body avoids turning diagnostics into an accidental archive of large or sensitive responses.

The fields still need redaction. URLs can contain identifiers or query secrets. Bodies can contain personal data, credentials, and complete documents. Account and item identifiers should use the minimum form needed for correlation, with access and retention appropriate to operational logs.

Recording context is not the same as handling failure. A warning which nobody monitors and no recovery process reads is merely a better description of lost work.

Availability policy belongs to the workflow

The HTTP client should own connection, timeout, status, and representation mechanics. The workflow owns the business consequence:

remote list unavailable:
    current run incomplete
    do not infer an empty source
    schedule or await durable recovery

one independent item unavailable:
    preserve item identity
    continue siblings
    close the gap later

remote write outcome unknown:
    do not repeat until idempotency or reconciliation resolves it

permanent business rejection:
    stop automatic retry
    expose the actionable reason

This separation keeps a generic connector from deciding that a missed order is acceptable. It also stops each workflow from reimplementing TLS validation and body parsing.

The public reproduction executes these branches with synthetic items. It deliberately reports the skipped item as unrecovered until a durable recovery record is completed. It does not simulate a real provider or measure outage frequency.

Third-party availability cannot be made reliable by a broad catch. The useful promise is narrower: preserve what is known, contain failure to independent work, and leave a durable account of everything which still needs an answer.