Integrations on Brian Faust

A Third-Party API Is an Unreliable Collaborator

Thu, 01 Feb 2018 00:00:00 +0000

A scheduled import begins by listing remote orders. The list request returns a 502 HTML page, so there are no trustworthy item identifiers to process.

On another run the list succeeds. One order then returns a 504 while its details are being loaded. The remaining orders are still independent and can continue.

These failures share a provider, but not a recovery boundary. Treating both as “the API is down” either discards useful work or pretends to know more than the response established.

Silencing an Error Does Not Protect the Customer

Wed, 01 Jun 2016 00:00:00 +0000

A shipment-booking request returned without an error. The shipment appeared in the system, so the request looked successful. It had no label and no tracking number.

Persistence had failed after the remote booking completed. A broad catch swallowed the exception, emitted no report or log, and continued. The code met one operational request—do not show customers errors—by returning an object which the customer could not use.

The intent was reasonable. Customers prefer reliable bookings to error messages. But once the system can no longer produce the required result, silence does not create reliability. It converts a known failure into an ambiguous success and hands the investigation to the customer.

A Failure Timeline Can Eliminate the Best-Looking Suspect

Fri, 01 Apr 2016 00:00:00 +0000

A shipment appeared in the local database without the tracking number or label that made it useful. Payment was the first suspect because it was known to be brittle, and the visible result looked like an operation which had stopped halfway through.

But payment had completed. The remote booking had completed too. The failure happened while persisting the response, and a broad catch continued without reporting it.

The payment theory was attractive because of history, not because it explained the state in front of us. Ordering the effects made that mismatch visible before another log line was added.