Data That Must Survive on Brian Faust

Data Outlives the Code That Created It

Fri, 01 Dec 2017 00:00:00 +0000

A failed queue record kept returning. Its payload named a mail class the running application could not resolve correctly, so retrying it could only repeat the same failure.

The retained workaround was blunt: once a minute, delete failed records whose payload matched that job. It stopped an unusable delivery attempt from disturbing the queue. It did not make the payload readable.

That small incident exposes an awkward storage contract. Serializing an executable object does not merely store its arguments. It preserves facts about the code which created it: class name, namespace, property layout, and sometimes framework behavior. The payload can live much longer than those facts.

Audit History Is Different from Application Logging

Wed, 01 Nov 2017 00:00:00 +0000

When a cart appears to expire, support may ask:

Which cart changed?
What was its prior state?
What state followed?
Who or what caused the transition?
Which policy allowed it?

An engineer investigating the same report may ask:

Which request handled it?
Which worker retried?
How long did the database call take?
Which exception and stack occurred?
What else happened on that host?

Both groups need evidence. They do not need the same record.

An Index Must Serve a Real Query

Sun, 01 Oct 2017 00:00:00 +0000

A production query needed a small page of recent records requiring operational attention. The table already had an index which looked relevant. The measured scan still discarded 84,003 historical index entries and recorded 122,936 shared-buffer reads before finding 83 eligible rows.

The missing index was not “an index on the table.” It was an index for this query:

one tenant
active records with zero price
created during the last three months
newest creation time first
stable identifier as the tie-breaker
first page only

Indexing starts with the access path, not with a list of columns which sound important.

Pagination Needs a Stable Order

Fri, 01 Sep 2017 00:00:00 +0000

A financial report encoded two sort rules as one string:

lowercased customer name + "-" + customer identifier

Sharded tests with concurrently created identifiers exposed unreliable output. The correction stopped asking a compound string to express precedence. It compared names directly and consulted the stable customer identifier only when the names matched.

The retained diff does not prove that equal names alone caused the failing order. It does expose the rule every paginator depends on: the primary comparison, tie-breaker, and comparison semantics must be explicit, and each row needs exactly one reproducible position.

Soft Deletes Preserve Ambiguity Too

Tue, 01 Aug 2017 00:00:00 +0000

Adding deleted_at to a table answers one question: when was this row marked as deleted?

It does not answer whether the row was safe to mark, who should still see it, whether its credentials remain valid, whether it still occupies a unique business slot, how it can be restored, or when its data should be erased.

A retained cart cleanup could not safely reduce those questions to one nullable timestamp. Old carts might still have a shipment, a payment session, a booking attempt, or item work which had crossed a meaningful boundary. The eventual policy archives a deliberately narrow set and preserves the aggregate as evidence.

An Identifier Should Outlive Its Storage Engine

Sat, 01 Jul 2017 00:00:00 +0000

A billing export once had enough information to reconstruct a missing account, but not enough to name it.

The account still existed in an older system. Its stable identifier reached the export action. No matching row existed in the newer application’s user table, so the action built a temporary object with safe fallback details. That object had a synthetic local identifier, but its export identifier remained null. Later, a queued data object required that identifier to be a string and rejected the payload.

Time Zones Are Part of the Data Model

Thu, 01 Jun 2017 00:00:00 +0000

One metering path assigns an event to a billing month after converting its occurrence time into the account’s timezone. The same path derives its daily usage key directly from the occurrence timestamp without that conversion.

Near midnight, those two fields can disagree:

event instant: May 31, 21:30 UTC
local business time: June 1, 00:30
billing period: June
stored usage date: May 31

Each expression looks like a date calculation. They model different things. The defect appears only when the difference changes a boundary.

Money Is Not a Floating-Point Number

Mon, 01 May 2017 00:00:00 +0000

A booking-time price arrived as four values:

net amount: 407
tax rate: 25.5%
tax amount: 104
gross amount: 511

The amounts are integers in the currency’s minor unit. The rate is a decimal ratio. The contract checks that net plus tax equals gross and that applying the declared rounding rule to net multiplied by the rate produces the tax amount.

None of those details is decorative. Remove the unit, mix the rate’s two representations, or let two services round at different stages, and the same price can stop balancing.

Null Is a Domain Decision

Sat, 01 Apr 2017 00:00:00 +0000

We added four nullable fields to preserve a pricing snapshot: net amount, tax rate, tax amount, and gross amount.

Each field could now be absent. That did not mean each field was independently optional.

For older records, all four nulls meant one thing: this record predates the snapshot and does not contain captured pricing evidence. For new records, all four values had to arrive together. A row containing gross and net but no tax rate was not “partly known.” It was invalid.

A Migration Is a Production Change

Wed, 01 Mar 2017 00:00:00 +0000

We deployed an additive schema change which the database could execute without rewriting the table. It still waited indefinitely.

The statement needed an exclusive table lock. A long-running session already held a conflicting lock, so the migration queued. New application traffic then queued behind the migration. The column addition was cheap after lock acquisition; waiting to begin was the incident.

Another schema change looked cheap for the opposite reason: remove fields and tables which the application no longer read. The SQL was small. The decision would erase financial history from data sets measured in tens of gigabytes.

A Transaction Defines What Must Succeed Together

Wed, 01 Feb 2017 00:00:00 +0000

A booking flow reserved funds, completed the remote booking, and captured the funds. Only then did synchronous response processing fail.

The remote booking existed. The funds were captured. The caller received a failure.

The capture was individually valid, and its own database transaction committed cleanly. It was still too early. We had drawn the success boundary around “remote booking returned” while the operation promised something larger: a complete booking result which the caller could use.

Let the Database Reject Impossible State

Sun, 01 Jan 2017 00:00:00 +0000

Two workers asked the same reasonable question: does this owner already have an active record? Both received the same honest answer: no.

Each worker created one.

Nothing in either request was obviously careless. The application checked first. The insert happened inside a transaction. Tests which ran one request at a time stayed green. The defect lived in the interval between the check and the write, where both transactions were allowed to be correct about a state which had already become stale.