The close never measures the history

This is a labelled STOP-GAP for 11.3. It removes a store-sized term from the close and makes the number that term produced durable. It does not change the shape of the generation log itself, which is what actually makes the number expensive to derive. The architectural cure is the segment log (12.0), where the committed history's weight is a property the log maintains rather than a total anybody has to walk for. Everything on this page is designed to be deleted by that change, not extended by it.

The term this removed

A brain's retention policy can cap history by BYTES — the adaptive default caps it against a budget derived from free memory, and an explicit policy may set maxBytes directly. Evaluating that cap needs one number: what the committed generation history weighs on disk.

Deriving that number is one storage read per committed generation. The running total was maintained incrementally from then on, so the cost was paid once per process — but the only thing that asked for it was the close's auto-compaction, and it was computed before that pass consulted its own time budget. So the budget did not bound it.

MEASURED (2026-09-08), a store at 110,206 committed generations:

history byte total: a COLD walk of 110,206 committed generation(s) took 10,927 ms.
=== CLOSE WALL 21,693 ms ===
  history-compact       10,932 ms      <- budget is 5,000 ms
  history-repack         5,041 ms      <- budget is 5,000 ms, honoured

history-compact spent more than double its budget and reclaimed nothing: the reclaim loop's first deadline check was already true when the walk finally returned. That contrast is the proof — the deadline bounds the loop, and nothing bounded the walk in front of it.

MEASURED, the scaling: linear at ~0.1 ms per committed generation on a warm page cache, from 7,000 to 110,206 generations. A production _generations/ tree is cold, so every read there is a device read plus a gunzip and a parse — several times that. PROJECTED at a 334k-generation production brain: tens of seconds per store, per restart.

The cure, in two halves

1. The answer is durable

_system/history-bytes.json carries the total keyed by the committed interval set it sums over:

{
  "_fmt": 1,
  "ranges": [[1, 12000], [12040, 334102]],
  "bytes": 48123904,
  "updatedAt": "2026-09-08T21:04:11.882Z"
}JSON

bytes is the sum over every generation in ranges and nothing else. The two fields are one fact and are never written apart.

It is written at three moments and no others: when a census completes, on the census's own checkpoint cadence, and at a clean close — where the running total is both known and final. A memo is never a guess: a close that never learned the total writes nothing.

2. What the memo does not cover is measured behind the doors

history-byte-census is a background job of the open, not of the close. It walks in installments that yield to foreground traffic, it checkpoints into the memo on a bounded cadence, and it measures only — it reclaims nothing, repacks nothing, and deletes nothing. The retention policy still decides what is kept; the census only supplies the number that policy is evaluated against.

Where a projection has opened a background job runtime, the census runs on it, under the same installment law, abort-at-close semantics and narration as every other background job. Where none is open — the reference engine's own indexes — the same body runs on the engine's own installment loop, yielding between installments and standing down at close. Both lanes write the same memo, and each says which one ran. Nothing measures the history on the shutdown path either way.

Exactly what invalidates the memo

Adoption is a comparison, never an act of faith. At open, the store's committed generations below the memo's own high-water mark must be byte-for-byte the intervals the memo recorded. Three outcomes:

The events that produce each, named:

• A commit extends the history above the memo's mark → partial, and the census tops up the tail. After a clean close there is no tail, so this costs nothing.

• A reclaim removes an oldest prefix of generations → the intervals below the mark no longer match → refused if the memo was not re-stamped. The reclaim path re-stamps it, so this is a crash-only case.

• A restore replaces the store wholesale → the memo is deleted outright, and would have been refused anyway.

• A repack folds cold generations into sealed segments. It is re-representation, never deletion: the intervals and the bytes are unchanged, so the memo stays valid — as it should.

A refused memo costs a walk. It can never cost a wrong number.

What a close does when the total is not yet in hand

It skips the byte cap for that close, by name, and says so on the always-visible narration channel:

[Brainy] [<store>] history byte total not measured yet — the adaptive byte cap is
         skipped for this close and the background census supplies it.

The generation and age caps are unaffected and still apply. Reclamation is deferred to the next close, by which time the census has run. Nothing is ever reclaimed on a guess, and no walk is ever smuggled back onto the close path to cover for a number that is not there.

One behaviour change to expect, stated plainly

A store may now reclaim history where it used to reclaim none. That is the retention policy finally being applied, not a new policy — but it is visible, so it is said here rather than discovered.

Before this, the byte cap was evaluated against a total the close derived with a walk. On a large-history store that walk outran the compaction pass's time budget, so the reclaim loop broke on its first iteration: the pass paid the whole walk and reclaimed nothing, every close, forever. A store big enough to need reclamation was exactly the store that never got any.

With the total in hand the same pass now evaluates the same cap against the same number the operator configured, and reclaims what that cap asks for — oldest first, never a pinned generation, still bounded by the same 5,000 ms budget.

Two guards on that, both absolute:

• Nothing is reclaimed on a guess. A close with no total skips the byte cap entirely rather than evaluating it against a number that is not the total.

• A dropped cap never becomes no cap. compact() reads "every cap undefined" as reclaim every unpinned generation, so on a policy whose only cap is maxBytes, dropping it would turn a bounded policy into a total one. The pass is skipped instead, by name.

An operator who wants the old outcome sets retention: 'all', which never consults a byte total and is never measured for one.

The counters

Brainy.historyCounters() reports the engine's own numbers, so an operator — or a pin — can read the shape of this cost without parsing a log or timing a clock. The unit is reads, deliberately: one delta read costs microseconds against a warm page cache and ~0.65 ms against a cold production tree, so a wall-clock verdict on this class either flakes on a fast box or asserts nothing on a slow one.

Brainy.historyByteCensusOwed() answers "will my next close apply its retention byte cap" on a running store, and Brainy.settleHistoryCensus(ms) waits for the measurement to finish — for the callers that legitimately need the measured state to be final. Nothing on the serving path calls either.

Pinned by

• src/native/closeDoesNotMeasureHistory.e2e.test.ts — a close performs O(1) history reads on a store with 20,000 committed generations; the memo's invalidation cases; and a genuinely interrupted census resuming from its own checkpoint rather than restarting.

• src/native/historyWalkIsBudgeted.e2e.test.ts — the walk an explicit caller asks for still respects that caller's time budget, and still caches no partial.

Both assert reads and counters rather than clocks, so neither can flake into a false green.