Index

Debugging Real Systems

Silencing an Error Does Not Protect the Customer

A shipment-booking request returned without an error. The shipment appeared in the system, so the request looked successful. It had no label and no tracking number.

Persistence had failed after the remote booking completed. A broad catch swallowed the exception, emitted no report or log, and continued. The code met one operational request—do not show customers errors—by returning an object which the customer could not use.

The intent was reasonable. Customers prefer reliable bookings to error messages. But once the system can no longer produce the required result, silence does not create reliability. It converts a known failure into an ambiguous success and hands the investigation to the customer.

Success must name its required effects

“The request did not throw” is a control-flow observation. It is not a business result.

For this booking, success required a smaller and more concrete contract:

remote booking accepted
local shipment complete
tracking number available
label available
result returned to the customer

A remote booking reference on its own was not enough. Neither was a local row. The customer needed the tracking number and label to continue the work for which the booking existed.

Write that invariant before deciding whether an error can be ignored:

usable success =
    remote booking
    AND complete local state
    AND tracking number
    AND label

The old path returned success when only the first effect was true. The response and the stored state disagreed about what had been accomplished.

This is different from the failure-timeline problem in the preceding article. There, completed and missing effects helped locate the failed interval. Here, the failure boundary is already known. The decision is what state and response the system is allowed to leave behind.

An error can protect the next decision

A clear error is useful information:

booking could not be completed
no usable local shipment was created
retry or contact support with this reference

That message prevents the customer from printing a nonexistent label, waiting for a tracking number which will not arrive, or assuming that downstream fulfilment can continue. It also gives support a declared outcome to investigate.

The corrected path in this case did four things:

  1. rolled back the failed database work;
  2. removed the incomplete local shipment;
  3. reported the exception; and
  4. returned a clear failure with a support path.

The important property is agreement. The customer sees failure, the local system contains no usable shipment, and the exception reaches an operational surface. There is no local success-shaped object contradicting the response.

The DBG-06 fixture makes that comparison explicit. Its silent path retains an incomplete row and reports success. Its corrected path removes the row, records the failure, and returns an error. The fixture is a synthetic state machine, not a reproduction of the private application. It checks the contract among state, response, and evidence.

Catch only what you can finish handling

Broad catches are not inherently wrong. An application boundary often needs to translate many internal failures into one stable API response. Cleanup may also need to run for type errors, database exceptions, client failures, and other throwables.

The danger is confusing interception with handling.

A catch has handled an error only when it has decided:

  • which local writes remain;
  • which transaction commits or rolls back;
  • what remote effects already exist;
  • whether cleanup succeeded;
  • what the caller is told;
  • what evidence operators receive; and
  • whether retry is safe.

If it cannot answer those questions, continuing is usually the least honest branch.

The original catch did not merely simplify an error message. It changed a failed persistence operation into apparent success without repairing the missing state. That is why “never error for customers” was the wrong translation of the operational goal. The better translation was “do not expose customers to avoidable failures, and make unavoidable failures clear and actionable.”

That distinction does not blame operations for wanting a dependable product. The technical design owns how that goal is represented. Hiding a broken result is not the only way to avoid a stack trace.

A database rollback stops at the database

The remote booking had already succeeded. Rolling back a local transaction could not undo it.

The corrected result therefore looked like this:

customer response: failure
local shipment:    absent
reported error:    present
remote booking:    present

That last line matters. “Rolled back” can sound atomic when the workflow crosses systems, but the database controls only its own writes. The remote booking is an orphan unless another operation cancels, imports, or reconciles it.

In this case, the relevant carriers charged only when a label was scanned. The orphaned booking had no scanned label, so removing the incomplete local record did not create a booking charge. That commercial rule made cleanup a reasonable trade:

remote booking exists
label was not delivered or scanned
charge is not triggered
customer can retry

The article does not generalize that rule to every carrier or provider. A remote reservation may consume scarce capacity. A payment may already be captured. A booking may incur a fee at creation. A generated identifier may need audit retention even when the workflow fails.

The cleanup policy must follow the actual remote consequence.

Delete, preserve, or reconcile deliberately

Deleting partial local state is one option, not a universal rule.

Remote consequenceCredible local response
no charge or scarce resource until later useremove the incomplete local record and allow a clean retry
remote operation can be cancelled reliablycancel it, record the outcome, then remove or close the local attempt
remote operation is billable or irreversiblepreserve an explicit failed or reconciling attempt with the remote identity
remote outcome is unknownretain enough evidence to query or reconcile before retrying

A failed-attempt record is often better than a half-valid business object. It can keep the request identity, remote reference, failure stage, timestamps, cleanup attempts, and terminal decision without pretending that the shipment is usable.

That separation avoids two bad extremes:

  • delete everything and lose the only path to a billable remote effect; or
  • preserve an incomplete shipment in the normal catalogue and make every reader understand its accidental shape.

The correct state model might be:

booking attempt: failed, remote reference known, reconciliation pending
shipment:        not created

Or it might be:

booking attempt: failed, remote effect non-billable, cleanup complete
shipment:        not created

The business rules choose between them. A silent catch chooses neither.

Cleanup can fail too

The retained code contains a later example of this boundary. A local shipment row could survive when an unexpected throwable occurred after initial creation but before tracking and metadata persistence. The cleanup path was widened to cover that failure class.

The change also stopped treating deletion as infallible. If deletion failed, it emitted a critical report with enough identity to investigate the partial row.

That second failure deserves its own outcome:

original operation: failed
cleanup:            failed
local state:        potentially partial
operator signal:    required
customer response:  still failure

Cleanup failure must not overwrite the original error or turn the request into success. It increases operational urgency because the state now contradicts the response. The report should preserve both failures and the attempt identity.

finally is useful for releasing resources. It does not make compensating business actions succeed. A delete, cancellation, refund, or rollback can have its own failure modes, timeouts, and retry rules.

Report the boundary, not only the exception

An exception class and stack trace help developers. Operators also need to know which business transition failed and what remains true.

For this workflow, useful evidence includes:

attempt identifier
failure stage: response persistence
remote booking reference received: yes
local transaction outcome: rolled back
local shipment cleanup: succeeded or failed
label available: no
tracking number available: no
customer response: failure

The exact carrier payload and database exception from the historic case are no longer retained. They should not be reconstructed. The fields above describe the decision boundary which a current implementation can record.

This is also why returning a support reference matters. “Something went wrong” is honest but expensive when neither the customer nor support can connect the response to the reported failure. The reference does not need to expose internal identifiers. It needs to resolve to the same attempt in the operational record.

Test the contradiction you are removing

An error-path test should inspect more than the response code. Inject a failure after the remote effect and before required local persistence, then assert the complete outcome:

response says failure
no usable local shipment exists
exception is reported
cleanup outcome is recorded
remote reference is retained when reconciliation requires it
retry cannot duplicate a chargeable effect

The last two assertions depend on the remote contract. The DBG-06 fixture uses the supplied scan-triggered charging rule and verifies that the orphan is not charged. It also runs a bill-on-booking alternative to show why deletion alone would be insufficient there.

The synthetic control cannot prove a provider’s commercial terms. Those terms come from the supplied case and would need current contract evidence before being applied elsewhere. The control proves only that changing the charge trigger changes the safe cleanup decision.

The standard is not “customers should see more errors.” It is that every response should tell the truth about the operation the customer can perform next.

If required output is missing, return a failure the customer can act on. If a remote effect already happened, record or compensate it according to its real cost. Do not ask silence to reconcile two systems which already disagree.