Observability Must Answer the Next Operator Question

A catalogue quote request returns 503 after its price dependency times out. One telemetry bundle produces five uncorrelated log lines, a metric labelled with request ID and raw URL, and three pages. The pages do not name an owner, affected scope, hold duration, or first query.

The other bundle produces fewer records and one page. It says which route, region, and release are affected. Its first query measures the user-visible failure. An exemplar selects a failed trace, the trace locates the dependency wait, and a correlated event names the application failure class. The runbook asks whether bypassing that dependency is safe.

Both bundles observe the same request and the same response hash. Only one reduces the operator’s next decision.

That is the useful boundary for observability. A signal earns its place by answering an operator question, not by proving that another library knows how to emit telemetry.

Start with the questions, not the signal types

Logs, metrics, and traces are storage and navigation choices. They are not an incident strategy by themselves.

The OPS-01 fixture starts with eight questions:

This order prevents a common reversal. If instrumentation begins with “collect all the logs” or “trace every function,” the operator must reconstruct the questions from whatever data happens to exist. Starting with the questions assigns each answer to the cheapest signal which can see it.

The fixture is synthetic. Its catalogue route, traffic, failure, thresholds, and runbook decision do not describe Brian’s systems. They make the reasoning inspectable without borrowing a production story.

Page on harm; diagnose the suspected cause

The noisy bundle pages separately on an application 503, a dependency timeout, and a failed retry. That is one request path represented as three interruptions. The operator still has to decide whether these pages share a cause and which one should clear after mitigation.

The decision bundle pages on the user-visible quote error ratio. Dependency latency remains available for diagnosis, but it does not create another page while users are unaffected. In the slowdown control, dependency duration rises to 180 milliseconds and the route still returns 200. A cause signal changed; the declared symptom did not.

This distinction matters during recovery as well. If a dependency gauge caused the page, a healthy-looking gauge might resolve it while users still receive errors elsewhere in the path. The fixture verifies recovery with the same user-visible query that detected the failure.

A real Prometheus server exercises the alert lifecycle. One isolated failure moves the rule to pending, then a healthy observation clears it without a page. Sustained failures move it from pending to firing. A recovered response returns it to inactive. The fixture uses 250-millisecond evaluation intervals and a one-second hold to keep the integration test fast; those are test controls, not production recommendations.

The production hold duration and threshold need evidence this fixture does not have: a service objective, traffic distribution, error budget, urgency, on-call staffing, and the cost of both missed and unnecessary pages. The Prometheus alerting guidance recommends keeping alerting rules simple and alerting on symptoms where possible. It cannot choose those service-specific values.

Aggregates need bounded dimensions

The request counter records five successful and four failed observations. The duration histogram has nine observations across declared boundaries. Route family, region, release, and outcome are useful dimensions because they bound the affected contract and deployment.

Request ID is useful too, but not as a metric label.

Adding it to the counter changes one bounded series into nine series for nine requests. It does not improve the aggregate detection query. With production traffic, the same choice would keep creating new time series for values which are rarely aggregated together.

The same rule excludes trace ID, user ID, raw URL, exception message, and customer email from metric labels. “Useful somewhere” is not the same as “useful as an aggregate dimension.” Prometheus’s instrumentation guidance warns that every unique label combination creates another time series and advises caution when cardinality can grow.

The aggregate still needs a route to one concrete failure. In the fixture, an exemplar carries the selected trace ID without turning that ID into a label on every series. The metric answers “how much harm?” and the exemplar answers “which example should I inspect?” Those are different jobs.

Correlation should shorten the search

The selected failed trace contains three spans: the quote route, the price dependency, and one bounded retry. Each changes the diagnosis. The successful control contains the route and dependency spans with successful status.

Function entry and exit spans were deliberately discarded. A trace can be perfectly connected and still waste attention if every stack frame becomes a span. The useful test is whether removing a span would hide a service boundary, wait, retry, failure, or decision needed for the next question.

The dependency span links to one structured event:

{
  "event_name": "quote.dependency_failed",
  "failure_class": "price-dependency-timeout",
  "dependency": "price-service",
  "attempt": 2,
  "release": "release-a",
  "region": "eu-north",
  "outcome": "503",
  "trace_id": "b449734daf477d0a7b0c3fabfcf71709",
  "span_id": "4282a34b54291837"
}

The event adds the bounded application classification which the dependency timing alone cannot infer. Its trace and span identifiers lead back to the selected request. The customer email used by the noisy fixture is omitted; request and response bodies, credentials, and authorization headers are not diagnostic defaults.

The complete navigation path is therefore explicit:

symptom page -> bounded aggregate -> exemplar -> failed trace
             -> dependency span -> structured event -> runbook decision

OpenTelemetry describes traces as the path of a request through an application and logs as records which can carry trace context. Correlation becomes operationally useful only when following that context actually removes a search step. Independent dashboards with compatible JSON do not prove that path exists.

A missing trace does not prove a missing request

The application and its telemetry pipeline have separate failure contracts. Conflating them makes both harder to operate.

In the exporter-failure control, the quote route continues returning its declared application response while the OpenTelemetry Collector cannot reach its downstream. Twenty span batches are submitted. The Collector accepts two, refuses eighteen under backpressure, records eighteen enqueue failures, and records one send failure from the accepted work.

“The Collector accepted the span” is therefore not the same as “the backend stored the span.” A request can also be sampled out, exceed a retention window, or occur while the export path is unavailable. In every case, the request may have happened even though the trace query returns nothing.

The Collector resiliency documentation separates sending, queueing, retry, and persistent buffering concerns. An application must still declare what happens when those mechanisms fail. It may keep serving, block, buffer within a bound, shed telemetry, or fail a request whose audit contract requires durable recording. That choice belongs to the application, not to a default exporter timeout.

Transient telemetry is also a poor substitute for durable business state. An outbox item awaiting delivery, an unsettled payment, or an unfinished import needs an owned work inventory with repair verbs. Sampling, expiry, exporter failure, and backend deletion are normal telemetry behavior; none should make business work disappear.

Every emitted record needs a disposition

The fixture gives each proposed signal a job instead of a quality score:

Retain, aggregate, sample, demote, redact, and discard are design decisions. “Collect everything for now” merely postpones them until query latency, storage cost, privacy, or alert fatigue forces a hurried answer.

There is no useful observability maturity score here. Actionability, diagnostic coverage, cardinality, retention, privacy, and cost can conflict. A weighted total would let abundant low-value telemetry conceal a missing recovery query or owner.

Nor does every service need the complete chain. A small application may be well served by structured logs and one user-visible request metric. Tracing earns its additional propagation, storage, sampling, and operational work when requests cross enough boundaries that locating one failure is otherwise slow or ambiguous.

The reproduction retains nine request scenarios, the alert states from a real Prometheus server, successful and failed traces, Collector backpressure, eight operator questions, seven signal dispositions, 20 rejected counterfactuals, 13 hashed primary sources, and a clean export of commit 5d44dbf619d4b21917a726d34608e8ab7c24eb75. It contains no production traffic, service objective, retention rule, vendor cost, legal requirement, or staffing model.

Begin the next instrumentation change with the page an operator might receive. Write down the first decision, the query which scopes it, the evidence which changes it, and the symptom which proves recovery. If an emitted record cannot move that chain forward, it has not yet justified the attention it consumes.