Logging Is a Workload, Not a Side Effect

One system was writing roughly a thousand log lines per second to local disk. Debug records were part of that stream. The individual call sites were small; the combined stream was not.

It was not cheap in aggregate. Every record still had to be constructed, serialized, copied through handlers, written, rotated, and eventually removed or archived. Whether anyone queried it later did not change that work.

We made log levels configurable, kept debug output on internal nodes where it was useful, and added rotation and archival. The important shift was not from “many logs” to “few logs.” It was from treating logging as harmless commentary to treating it as a workload with a budget.

Four numbers describe the first-order cost

A rough logging budget starts with multiplication:

ingest bytes per second
    = events per second
    * serialized bytes per event
    * destinations

retained bytes
    = ingest bytes per second
    * retention seconds

Compression, batching, indexes, replication, and filesystem behavior change the eventual bill. They do not make event rate or payload size disappear.

The DBG-03 fixture makes the first-order difference concrete. It serializes two synthetic records for the same successful integration response. One embeds a 32 KiB binary document as Base64. The other keeps the event name, operation, outcome, status, trace identifier, original byte length, and a fixed digest, but omits the document body.

At one thousand events per second, the difference is large:

These are serialized byte counts from a synthetic fixture, not measurements from the old system. The historic line sizes, compression, disk throughput, and retained volume are unavailable. The event rate is the approximate rate I remember; the fixture uses it to expose the arithmetic, not to reconstruct the machine.

Removing the body cuts this example by more than two orders of magnitude. It still leaves about 28 GB of uncompressed JSON per day at the same event rate. Payload discipline and event discipline are separate decisions.

A payload can dominate an otherwise useful event

A retained private change illustrates the payload problem without belonging to the old incident. A debug record captured an XML integration response, including an embedded Base64 document. The later change replaced the document contents with a marker and preserved the surrounding response.

That is a useful shape for debugging. It keeps evidence that a document field was present and locates the response in its protocol context. It stops copying the entire binary artifact into a text channel every time the operation runs. If exact document bytes matter for audit or replay, they need a deliberate storage contract rather than accidental retention inside a debug line.

The choice is not always “body or no body.” Bounded alternatives include:

• byte length and content type;
• a digest when equality matters;
• item count and field names;
• a deliberately truncated prefix;
• a reference to separately governed evidence; or
• the full body on an explicitly enabled diagnostic path.

Each alternative answers a different question. A digest can confirm that two artifacts are equal, but it cannot explain malformed XML. A truncated prefix may expose the parser failure, but it cannot support a byte-for-byte replay. A reference preserves separation only if the referenced evidence has an owner, access policy, and retention rule.

OpenTelemetry’s log data model separates a record’s body, attributes, severity, event name, and trace context. That model makes bounded fields possible; it does not decide which payload an application should copy.

Levels control intent only when they control work

Calling everything info because it might be interesting later defeats the point of levels. So does labelling a record debug while enabling debug globally forever.

PSR-3 distinguishes detailed debugging information from informational events, warnings, errors, and more urgent conditions. The current OpenTelemetry Logs SDK specification goes further: a configured minimum severity can cause lower-severity records to be dropped before processing. That early boundary matters. Filtering after a large value has been assembled, serialized, and sent has already paid much of the cost. The specification marks that configuration section as development, so it is a useful model rather than proof that every logging library filters at the same stage.

A practical level decision asks what changed:

debug: detailed evidence for a deliberately enabled investigation
info:  a significant state transition expected during normal operation
warn:  an undesirable condition from which the operation can recover
error: an operation failed or produced an invalid outcome

Those descriptions are not an automatic mapping. A polling loop may execute normally a thousand times per second without earning a thousand informational events. An infrequent successful key rotation may deserve an informational record because the transition matters operationally.

The event rate belongs in the decision. Before adding a log to a loop, queue consumer, request middleware, or retry path, estimate its normal and failure rates. A record emitted once per deployment and one emitted once per item can share the same level while having completely different budgets.

Rotation limits files, not production

Rotation solves a real problem: one file should not grow until it fills a disk, becomes unwieldy to inspect, or cannot be removed safely. It is not a substitute for controlling emission.

A daily file with a seven-day retention window still accepts every byte written during those seven days. Archiving can move older bytes to cheaper storage, but it adds transfer, indexing, access, deletion, and recovery work. Sending the same record to local disk and a central service multiplies the destinations in the budget.

The relevant controls live at different stages:

before construction: is this level and event enabled?
during construction: which bounded attributes are necessary?
before export: should this destination receive the record?
at storage: how is it indexed, rotated, retained, and deleted?
after expiry: is it discarded or archived under another contract?

No one setting owns the whole lifecycle. Lowering retention cannot recover the application work already spent constructing and writing a record. Raising the minimum level cannot make the remaining error payloads safe or bounded.

Debug access should be deliberate and reversible

Keeping debug output on internal nodes was appropriate for the system we had: those nodes supported investigation without imposing the same stream on every machine. That topology is not a universal recommendation.

An internal node can receive different traffic, miss a timing-dependent failure, or create a debugging environment which no longer represents the affected path. Central dynamic level changes may give better coverage, but they need authorization, expiry, visibility, and a way to prevent one forgotten switch from becoming permanent.

I would prefer a time-bounded diagnostic control when the platform can support it:

scope:       named service, operation, or trace
level:       debug
payload:     bounded fields, with explicit exceptions
expires_at:  required
owner:       required
reason:      linked to the investigation

The design reverses when reproducing the issue requires full payload capture or debugging on the exact production path. Then enable the narrower exception, measure its rate, restrict access, and let it expire automatically. “We may need it someday” is not an expiry condition.

Count the work before adding another record

This article does not decide which operator question a signal should answer, how an identifier crosses queues, or whether logging caused a particular performance failure. Those are separate investigations.

Its boundary is smaller: a logger call creates work even when it is non-blocking, asynchronous, sampled later, or never queried. Before adding one, write down the expected event rate, serialized size, destinations, and retention. Then decide which level and bounded payload earn that budget.

If those four numbers are unknown, the log record is not free. Its workload is merely unpriced.