Production Bugs Can Live in the Data, Not the Code

A delivery integration returned a label, but the application could not save it. The same persistence path had passed with the fixtures used elsewhere.

The failed label was not malformed. It was simply larger than the database column could hold after Base64 encoding.

That distinction changes the investigation. Comparing deployed revisions would find nothing because the revision was not the variable. Repeating a familiar fixture would also find nothing because the fixture did not cross the same storage boundary. The investigation had treated “same code” as “same experiment” while leaving the input that decided the result uncontrolled.

The useful debugging question was not which environment had different code. It was which assumption production data had disproved.

One request crosses several size contracts

A binary document does not move through a system under one universal limit. Each boundary may describe a different representation:

carrier response
    binary label: n bytes
          |
          | Base64 encoding
          v
application value: 4 * ceil(n / 3) characters
          |
          | persistence
          v
database column: finite byte capacity

RFC 4648 defines Base64 in groups of three input octets and four output characters. Padding completes the final four-character group. The encoded value therefore grows in steps, not one byte at a time.

That growth is easy to ignore when fixtures are small. A 30 KB document becomes about 40 KB and still looks harmless. Near a storage boundary, one additional input byte can add four encoded characters and turn an accepted document into a rejected row.

The database has its own definition of capacity. MySQL documents TEXT as holding an actual byte length below $2^{16}$ , while MEDIUMTEXT uses a three-byte length prefix and permits a much larger value. Character set and the other columns matter for some string types, so “the field is text” is not a complete capacity argument. The declared type, stored representation, and byte length must agree. MySQL’s storage requirements make those distinctions explicit.

The historic column declaration and failed label size are no longer retained. I cannot reconstruct them honestly. The DBG-02 reproduction uses a MySQL TEXT capacity as a public, representative boundary instead. It demonstrates the mechanism without claiming to recreate the private schema.

For that boundary, 49,149 input bytes encode to 65,532 bytes and fit. An input of 49,150 bytes encodes to 65,536 bytes and does not:

The interesting number is not 49,150. It is the disagreement between the largest value one boundary accepts and the largest value the next boundary can represent.

A passing fixture may describe only itself

The non-production data did not represent the failing label shape. That made the environments look equivalent while leaving the decisive variable untested.

“Works in test” often compresses several claims into one:

• the same application revision executed;
• the same schema constraints existed;
• the same path ran;
• the input had the same relevant shape; and
• the same boundary observations were checked.

Only the revision claim had been established. The other claims still needed evidence. The failure needed a large encoded document, so another small valid label was not a control. It was a different experiment.

Copying the production document into every environment would have reproduced the size, but it would also have carried real operational data into places where it did not belong. A synthetic boundary fixture is safer and more durable. Generate a valid-shaped payload immediately below the declared capacity and another immediately above it. The values do not need to resemble a customer’s label; they need to exercise the same representation change and storage decision.

This is also why “test with realistic data” is too vague. Realistic averages miss boundaries. Useful fixture families include the ordinary case, the largest accepted case, and the smallest rejected case. For an encoded payload, they should be derived from the stored representation rather than guessed from the raw file size.

The schema and validator need one answer

The failure exposed three separate opinions about validity:

remote source: this is a valid label
application:   this value may proceed to persistence
database:      this value cannot be represented

The database was the first component to state a precise limit, but that does not make the database the right place for the first rejection. If the application promises to accept a document, it should know whether its chosen storage representation can preserve it before an unrelated persistence operation fails.

A useful invariant is:

maximum accepted encoded size <= maximum persisted encoded size

Better still, avoid making the encoded form the permanent storage contract unless the application needs it to be. Base64 is useful when a text protocol must carry binary data. That does not mean a relational text column is the best final home for the resulting string.

The invariant should appear in more than prose:

• validation measures the bytes that will actually be stored;
• the schema can represent every value validation accepts;
• boundary fixtures derive their sizes from the declared contract; and
• a persistence-level test proves the database agrees.

The last check matters. A unit test which compares a string length with a constant proves the comparison, not the deployed column type, character set, or strict database behavior.

Widening the column is a repair, not a policy

Changing the column to a larger type is the smallest credible repair when encoded documents belong in that table and the new bound covers the expected range. It restores the failed write without introducing another storage system.

It also leaves questions that the migration cannot answer. How large may a source document become? Does the application need to query the encoded body? How long is it retained? Will rows, backups, replicas, and transfers carry that growth comfortably? What tells an operator that upstream payload sizes are approaching the new limit?

Object storage or a binary column can be a better boundary when the database only needs document identity, metadata, and lifecycle state. That choice adds its own failure modes: a row and object can diverge, deletion needs coordination, access control moves across systems, and recovery must cover both. Moving the bytes is not automatically an architectural improvement.

I would keep the larger column when payloads are bounded, operational volume is modest, and transactional simplicity is worth more than storage separation. I would move the document when size or volume makes row storage a measured problem, or when independent document retention and delivery have become requirements. The incident alone does not decide that trade.

Compare data shape before blaming the environment

Different behavior under the same revision can still come from configuration, schema drift, caches, timing, dependencies, or environment-specific branches. Data is not the answer to every “only in production” report.

It belongs near the front of the investigation when the failure is input-sensitive. Compare dimensions, not records:

content type
raw byte length
encoded byte length
item count
nesting depth
missing or optional fields
character repertoire
relationship cardinality

Those measurements can be retained without copying a customer’s payload. They also produce better hypotheses. “Production is weird” cannot be tested. “Persistence fails when the encoded body exceeds the column’s byte capacity” can.

The decision reverses when equivalent synthetic data succeeds against the same schema. At that point, widen the comparison: SQL mode, character set, migrations, code paths, transaction handling, and infrastructure. A data-shape hypothesis earns priority only while it explains the first observed divergence.

A durable repair cannot stop at making one column bigger. The fixture that missed the boundary must stop serving as evidence about it. Same code is one controlled variable. Until the relevant data shape is controlled too, “cannot reproduce outside production” describes the experiment, not the bug.