Partial Work Needs a Child Ledger

A queue job receives 1,000 events in one remote response. It dispatches 600 child jobs, then times out.

The parent row says processing failed. Some children have already finished. Others are waiting in the queue. The remaining 400 were never dispatched. A counter can report 573 completed events, but it cannot say which 573 they were.

Running the parent again may duplicate the first 600. Leaving it alone loses the last 400. Neither choice is defensible because the system retained a total and discarded the membership behind it.

Partial work needs a ledger of the individual units, not a more persuasive parent status.

A counter compresses away the repair information

Parent counters are useful summaries:

total:     1,000
processed:   573
failed:       12
skipped:       4

They are poor recovery state. processed = 573 does not identify the completed events. It cannot distinguish these histories:

• the first 573 events completed;
• 573 events completed in an arbitrary order;
• 574 completed, but one counter increment was lost;
• 573 counters were incremented, but one child transaction later rolled back;
• the parent was replayed and counted some events twice.

The number is a projection of the work, not the work itself.

A repairable design persists one row per child before child processing begins. The row needs a stable position or identity and an explicit state:

parent_id  sequence  state       outcome
P-42       0         processed   -
P-42       1         failed      invalid timestamp
P-42       2         pending     -
P-42       3         skipped     unknown local subject

The parent counters can then be rebuilt from those rows. If the counters disagree, the ledger wins.

Decompose before dispatch

The ordering is the important part:

1. persist every child row
2. mark the parent decomposed
3. dispatch claimers for pending rows
4. process each claimed row
5. record one terminal child outcome
6. derive parent completion from terminal children

Dispatching before persistence recreates the original gap. A worker can accept a message which the database does not yet know exists. A timeout between the last dispatch and the later insert again leaves work that cannot be enumerated.

The retained implementation uses a uniqueness constraint on parent and sequence. Repeating decomposition may attempt the same positions, but it does not create a second set of children. Invalid input is also persisted, then marked skipped with a reason. Silently dropping it would make the parent total impossible to reconcile.

The decomposed marker is a checkpoint, not a claim that every child succeeded. It says the complete set of child work is durable and future passes should resume from that set rather than parse and fan out the parent payload again.

Claim rows in bounded chunks

Persisting 1,000 children does not require one job to process all 1,000. Workers can claim pending rows in chunks:

select id
from child_work
where parent_id = ?
  and processed_at is null
  and skipped_at is null
  and failed_at is null
  and (
    processing_started_at is null
    or processing_started_at < ?
  )
order by sequence
limit 500
for update skip locked;

The exact syntax depends on the database. The useful properties do not:

• concurrent claimers do not select the same live row;
• chunk size bounds each claim transaction;
• terminal rows are excluded;
• an abandoned claim becomes eligible after a lease expires; and
• ordering makes investigation repeatable without requiring serial execution.

The lease deserves care. A short lease can release legitimate slow work while its first worker still runs. A long lease delays recovery after a crash. Choose it from observed processing time and make the child effect safe against duplicate execution. The retained implementation uses a fixed five-minute stale threshold; that proves a bounded recovery mechanism exists, not that five minutes is universally correct.

Make terminal transitions one-way

A child may finish as processed, skipped, or failed. Only one transition should win:

update child_work
set processed_at = current_timestamp
where id = ?
  and processed_at is null
  and skipped_at is null
  and failed_at is null;

If the update affects no row, another attempt already made the child terminal. This prevents two workers from recording both success and failure for the same unit.

It does not, by itself, make an external side effect exactly once. A worker can complete a remote mutation and crash before recording processed_at. The reclaimed child then has an unknown outcome. Safe recovery still needs a remote idempotency identity, an outcome query, or a manual reconciliation path. The ledger exposes the uncertainty; it does not repeal distributed-systems failures.

That boundary separates this problem from request idempotency. INT-06 addresses whether one remote operation may be repeated. The ledger here answers which operations exist, what the system knows about each one, and which remain eligible for work.

Parent failure is not child completion

Before the ledger, a parent job failure looked terminal. After decomposition, the same failure may mean only that orchestration stopped between two safe steps.

Marking that parent failed can actively prevent recovery if the scheduler selects only parents without a failure timestamp. The retained change therefore keeps an interrupted decomposed parent resumable. A later scheduler pass finds its non-terminal child rows and dispatches claimers for them.

Legacy parents require a separate rule. A failed parent without child rows cannot resume from a ledger that never existed. It must re-enter decomposition from the retained parent payload. A failed parent with ledger rows should not take that path, because reparsing it would create two competing sources of truth.

This distinction is less elegant than one universal recovery query. It is also honest about a migration in which old and new records carry different recovery information.

Reconcile summaries from durable facts

Even with guarded child transitions, updating a child and incrementing its parent counter may happen in separate database statements. A crash between them leaves the child correct and the parent stale.

That is tolerable only when the direction of authority is explicit:

total     = count(all child rows)
processed = count(children with processed_at)
failed    = count(children with failed_at)
skipped   = count(children with skipped_at)
complete  = processed + failed + skipped >= total

A reconciliation command can compare these derived values with the parent, show a dry run, then repair the projection. The retained tests deliberately construct a parent with a total of 99 and a ledger containing three terminal children. Reconciliation changes the total to three, restores the three outcome counts, and marks the parent complete. Another test clears a premature completion timestamp when its only child is still pending.

Those are executable counterexamples, not evidence about how often the inconsistency occurred in production. The history records timeouts after partial fan-out as the failure being repaired; it does not retain an incident timeline or a measured frequency.

A ledger costs more because it remembers more

The simpler alternative is still credible when a parent is small, processing is transactional, and replaying the entire input is cheap and demonstrably safe. A durable row per child adds:

• storage and indexes proportional to event count;
• insert work before useful processing starts;
• cleanup and retention decisions;
• lease and recovery policy;
• reconciliation tooling; and
• another schema whose invariants need tests.

Do not pay that cost merely because a loop exists. Pay it when interruption can leave an unenumerated middle: some child effects completed, some messages dispatched, and some work known only to the parent payload.

The public reproduction models a crash after two of four children become terminal. Its parent projection is intentionally stale. Resuming selects the two remaining child identities, and reconciliation repairs the summary without replaying the completed work. It proves the state transition model, not queue, database, or remote-provider behavior.

A parent status can say that something went wrong. A child ledger can say what is left to do. That difference turns partial failure from a replay gamble into a bounded repair operation.