Flipping a brain to log-canonical
In this section
A brain that has been migrated to the segment log still runs that log as a mirror: the generation tree (the files under entities/) is the canonical, every write lands there first, and the log is written after it. The native writer (brainy serve taking writes itself) commits only into a brain whose log IS the canonical, so a mirror brain refuses every native write by name until it is flipped.
The flip is one-way. It writes a mark (_system/canonical-mode.json) recording when, under which decision and by whom; from then on the brain opens log-canonical on every engine that knows the mark, its tree's bookkeeping is frozen, and asking for the mirror back is refused by name.
npx brainy flip-log-canonical /srv/brains/acme --dry-run
npx brainy flip-log-canonical /srv/brains/acme --decision flip-acme --authorized-by opsBASHWhat it guarantees
Nothing a reader could see is lost. Before the mark, the tree and the log are compared row by row, by id, at
_revandupdatedAt. The mark is written only when every row agrees. A mark over a disagreement would make every row the log lacks vanish from every keyed read.The tree's version wins, because the tree was the canonical. Every row the tree holds and the log does not (tree-only), and every row the tree holds at a newer version (drifted, tree newer), is republished to the log first: one ordinary commit per 1,000 rows, with an empty change, through the same commit path a write takes. The log receives exactly the record a flush would have written, read back from the tree. Each commit spends one generation stamped
origin: system:reconcile, so the history says which generations were reconciliation and which were writes.A removed row's last version is kept where nothing reclaims it. The flip retires the tree's history files, and until now a removed row's last version (and the content a removed file names) lived only in them. Before the mark the flip writes every removal of the tree era into the removed-rows list (
docs/removed-rows.md), once: the row's last version, and a history reference on the content a removed file names. After the flip the answer to "what was removed" is read from that list, so listing it costs the number of removals instead of the number of generations. The pass is written in appends, each durable on its own: if the flip dies part way, the next run resumes from the list's coverage, and a run that finds nothing to write writes nothing. It changes no row and spends no generation.A flipped brain does not refuse its next open after a kill. The native writer's open rolls back a transaction a kill cut short, and for a row whose reverse record the kill tore away it can find the row's buckets only if the metadata index carries a proof that every row has a complete record; without it the open refuses the brain (
BRAINY_DEAD_TRANSACT_UNREPAIRABLE, the lease given back, nothing lost) untilrepairIndex()has run. The flip stamps that proof BEFORE it writes the mark, with one pass over the metadata index's whole value space, proves the stamp by a fresh read-only open of the closed brain, and refuses to flip a brain it could not stamp. See "The reverse-backfill proof" below.No writer can land between the proof and the mark. The flip closes the brain, takes the brain's writer lease itself, compares again and writes the mark under that lease.
A store already flipped is left alone. A second run reports the existing mark and does nothing.
What blocks it, by name
Refusal | When | What to do |
|---|---|---|
| The brain was never migrated to the segment log (no canonical pointer), or its pointer serves the tree. | Run the migration ceremony first, then flip. |
| Another process holds the brain: a lease on another host, or a live process on this one. A dry run refuses too. | Stop the process that serves the brain. Its own close drains the writes it was still holding. |
| The log holds a live row the tree does not have (log-only), or a newer version of a row than the tree has (drifted, log newer). | A ruling, per brain. See below. |
| The tree and the log still disagree after the reconciliation. No mark is written. | Report it: the reconciliation's own commits did not land. |
| The flip could not leave the metadata index's reverse-backfill proof durable (the pass stood aside, the index has no persistent surface, the native binary has no stamp door), or a fresh read-only look at the closed brain does not find it. No mark is written. | Run the flip again (the pass is resumable), on a build whose native binary carries the proof door. |
Why the log-ahead classes are refused rather than reconciled. On a mirror the log is written from the tree, so the brain's own commits never put the log ahead. Two things do. A delete whose log tombstone was still held in memory when the process died leaves the log holding the row the tree removed. A copy taken while the brain was running can capture the log after the tree. In both cases the tree-canonical answer is to publish the tree's state over the log's (a tombstone, or the older version). That choice changes what the log says happened, so the flip does not make it for you.
The reverse-backfill proof
The metadata index keeps, for every row, a reverse record: the list of buckets the row sits in, so that a row can be retracted without walking every value of every field. A kill that lands between the two halves of an update tears the batch that holds those pairs and leaves the row with no reverse record at all. The native writer's open rolls the cut-short transaction back, and for such a row it can retract only if the index carries a proof: one durable marker, written by a pass over the whole value space, saying every row has a complete record (so an absent record means "in no bucket"). Without it the rollback would walk the value space at open, which the open law forbids, so the open refuses the brain by name (BRAINY_DEAD_TRANSACT_UNREPAIRABLE, HTTP 409, the lease given back, nothing lost) until repairIndex() has run.
repairIndex() does not make the proof, before the flip or after it. MEASURED, the real brainy-serve killed at a random moment of a stream of overwrites, appends, working-state writes, removals, re-creations and attribute changes, 150 kills per way of preparing the same small brain: with no preparation 7 reopens refused, after a repairIndex() before the flip 3 refused, and after the whole-index pass before the flip none (150 of 150 reopened). The pass is what the flip now runs, before it writes the mark:
When the proof is already there (a pass an operator ran, or a metadata rebuild, which carries its own) the flip walks nothing.
When it is not, the flip waits out any metadata rebuild the open owes, then runs one resumable pass over every
(field, value)bucket, writing a reverse pair for every row each list holds, drains those writes, stamps the proof, and drains the stamp. A crash can lose the proof over durable pairs (one more pass) and never keep a proof over lost pairs.It then proves the stamp by another look: after the clean close, a fresh read-only open of the brain must find the proof. A flip that cannot leave it, or cannot find it, refuses with
ReverseProofNotStampedErrorand writes no mark.
The pass is the honest cost of the step: it walks the whole value space once. MEASURED earlier on the personal brain's post-flip copy: 1,098.7 s for 360,467 value buckets and 9,028,284 reverse pairs; seconds on a venue-sized brain. The dry run says whether the proof is there, so the cost is known where the decision is made:
Reverse-backfill proof: ABSENT - an apply walks the metadata index's whole value space once and stamps it BEFORE it writes the mark ...
Reverse-backfill proof: PRESENT - an apply stamps nothing.What the proof does not cover: the graph projection's membership roster. A kill can also leave the graph engine refusing to cold-load over a roster that does not describe its trees (BRAINY_GRAPH_COLD_LOAD_REFUSED); that stays an operator repair until its walk is ported into the Rust writer (its own piece of work).
The dry run
--dry-run takes no lease and writes nothing. It reads the lease record (and refuses if the brain is held), compares the tree and the log as they stand, takes one read-only look at the metadata index to say whether the reverse-backfill proof is there, and prints the classes, what an apply would republish, and the refusal an apply would meet.
Its comparison is unsettled. A brain whose last writer crashed with writes still held in memory shows those rows as tree-only until an open's recovery runs. The apply opens the brain (recovery included) and flushes before it compares.
The receipt
--json prints the whole receipt; without it the same facts are printed as text.
Field | What it says |
|---|---|
|
|
| The lease record as found: |
| The first comparison: per family, rows on each side and every class counted, the first 20 ids of each named, the comparison's wall and peak memory |
| The entity and relation rows republished |
| The first and last generation the reconciliation spent, and how many |
| The removed-rows list as seeded before the mark: the removals written, the generations scanned, the content references taken, the coverage it reached, and the pass's own wall ( |
| The reverse-backfill proof: what it was when the flip looked ( |
| The comparison the mark was written on |
| The mark: |
| Open and flush, compare, reconcile, seed removals, reverse proof, close, verify proof, compare again, mark, total — milliseconds, each measured |
Exit codes: 0 flipped, already marked, or a completed dry run; 1 refused (the class name and its sentence on stderr, and under refused in the JSON); 2 usage.
What it costs
One open and one clean close of the brain, two full comparisons, one commit per 1,000 reconciled rows, one pass over the brain's retained history to seed the removed-rows list, and, when the brain does not already carry the reverse-backfill proof, one pass over the metadata index's value space (see above) and one read-only look at the closed brain. A comparison walks every row of both sides once and holds one family's ids in memory at a time. The seeding pass costs in GENERATIONS, not in rows: it looks at every retained generation once (a removal is found by its tombstone where the fact log covers a generation and by the generation's own record where it does not), which is the same walk a listing of every removal makes once - after the flip it is made once for all and never again.
MEASURED on a copy of a production brain (1.37 million generations, about 95 GB allocated), in a shared build-box lane:
Step | Wall | What it found or did |
|---|---|---|
Dry run (no open) | 24–38 s | entities: tree 94,208 / log 94,190: 18 tree-only, 2 drifted (tree newer), 0 log-only · relations: tree 192,125 / log 192,113: 12 tree-only, 0 drifted, 0 log-only · peak resident 3.0 GB |
Open and flush | 1.4–1.7 s | The open's recovery replayed writes the copy's fact log held above its tree (21 VFS files, their 21 folder links, and newer versions of 6 rows) into the tree, and none of them reached the log. This is why the apply compares only after the open |
First comparison | 24.5–32.6 s | 39 tree-only and 8 drifted entities, 33 tree-only relations, nothing log-only · 192,146 ids held at peak, peak resident 5.3 GB |
Reconciliation | 0.42 s | 47 entities and 33 relations republished in 2 commits (2 generations) |
Close | 0.5 s | A clean close |
Second comparison | 22.3–30.6 s | Every row agrees |
Whole flip | 48–66 s | The 1-minute load average was 3–23 during these runs (other jobs shared the machine) |
After the flip, the same 51 reads (get, a find on a user field, related from, to and node, neighbors, a VFS folder listing and file read) on a fixed sample gave identical answers. The native write set opened the flipped copy in 1.1–1.4 s on its first claim, including a graph cold load of 84 SSTables in 95–121 ms. Later claims in the same process reused the loaded graph (0.0 ms).
What changes afterwards: blob reclaim is withheld
A flipped brain keeps no blob for deletion. A file's bytes are deleted only when no live file and no retained version names them, and the retained versions are counted from the generation record-sets the tree writes. A flipped brain writes none (its history is the log's versions), so on it a count of zero proves nothing. MEASURED on the released engine, on copies of a real brain flipped by this ceremony: a working-state file (retention: 'latest-only') that took the bytes an older version of another file named, and then moved on, deleted those bytes, and readFile(path, { asOf }) of the older version then failed with "Blob metadata not found".
So, after the flip, the engine and brainy serve delete no blob until a recount of the counts from the files and the log stamps _system/blob-history-refs.json at version 2: npx brainy recount-blob-history <ROOT>, described in the next section. Nothing is lost; disk is not freed: a working-state file that moves on leaves its superseded bytes in the store. The brain says so in three places: the open report names it, health carries a blob-history-counts row at warn ("history reference count incomplete - reclaim withheld"), and this ceremony's dry run and receipt print it under "After the flip". The flip itself stays allowed. A brain that is not flipped is unaffected.
Lifting the hold: recount the blob counts
npx brainy recount-blob-history /srv/brains/acme --dry-run
npx brainy recount-blob-history /srv/brains/acmeBASHThe recount is the second half of the flip. For every blob it counts two things: how many LIVE files name it, and how many versions the brain still keeps that name it and are no longer a file's current content (the history). It sets each stored blob's live count and history count to those numbers, and only then writes _system/blob-history-refs.json at version 2, which lifts the hold. It deletes nothing.
The live count is derived from the files, never carried over. A blob's bytes are deleted at zero live references and zero history references, so the live count a blob holds matters the moment the hold lifts, and it is not always true. Since 12.9.0 two overlapping writes to one working-state file whose bytes another file shares subtract the old bytes' count twice; the engine's own copy takes no reference for the copy; a death between a count and its commit leaves one too many. MEASURED on a capture of the personal brain: 184 session snapshots had lost their bytes outright and 570 live files sat at a live count of zero or short, protected only by a history reference. So the recount does not trust the stored live count: it walks every live file in the log, counts per blob how many name it, and sets the stored count to exactly that number, up or down. A live file it cannot read refuses the recount (LiveRowUnreadableError: a file that is not counted reads as unreferenced the moment the hold lifts), and a blob a live file names that the brain holds no bytes for is reported, never invented. The dry run says how many live counts it would raise (a live file the count did not cover: the dangerous direction) and how many it would lower (a reference nobody holds).
What it counts for the history. One reference per retained version that names a blob, from three places:
The generation tree's record-sets from before the flip: the loose
prev/files and the sealed segments the history folds into. The engine's older open-time recount listed only the loose files, so a generation folded into a segment was invisible to it and the blobs only that generation names were counted as zero.Every version the segment log keeps: all versions of a file except its newest, and all of them for a removed file. A version that a working-state write (
retention: 'latest-only') superseded takes no reference, exactly as at the commit: that class keeps no history by definition.The removed-rows ledger: history trimming keeps the last version of every row it reclaimed a removal of, and the content that version names, for the life of the store, and the ledger holds a history reference on each. For a removal trimmed before the flip it is the only place left that names the bytes, so a recount that left it out would set exactly those counts to zero.
The places can name the same superseded version (a mirror published it to the log and kept it in the tree; the log's tombstone and the ledger name the same removed file), so a blob can be counted twice. That is deliberate. On a flipped brain nothing ever releases a history reference, so a count only means "above zero or not", and an over-count only leaks bytes.
What it refuses. A brain that carries no log-canonical mark (StoreNotLogCanonicalError: its counts are the engine's own and need no recount), and, for an apply, a brain another process holds (RecountWriterLeaseLiveError). --dry-run opens the brain read-only and takes no lease, so it can run against a brain that is being served.
The order that makes it safe. It opens and flushes the brain, walks the live files and the history, sets every stored blob's live count and history count absolutely (raised where a file or a retained version was uncounted, lowered where a crash left one too many), and stamps the marker last. A run that stops before the stamp leaves the hold in place, and running it again gives the same result. Hashes a live file or a retained version names that the brain holds no blob for are reported (their bytes were already gone), never invented.
After it. Every write that supersedes or removes a version counts a history reference for that version at the commit itself, before the live write, and gives it back if the write did not happen; brainy serve does the same in its file-system doors. A later recount-blob-history is an audit: on an exact brain it raises and lowers nothing, and a count it has to raise names a write path that missed one (the receipt says so when the brain was already stamped).
exit | meaning |
|---|---|
| recounted, or a dry run that completed |
| refused: the class name and its sentence on stderr (and under |
| usage |
What it costs, MEASURED on copies of two production brains, each flipped by this ceremony, in a shared build-box lane (1-minute load 5-17):
Brain | Tree record-sets (generations, of them packed) | Log (records, keys with a history) | Dry run | Apply |
|---|---|---|---|---|
A venue brain (about 4 GB) | 63,675 (9,476) | 53,816 / 5,679 | 19 s: read-only open 4.5 s, tree 10.5 s, log 3.8 s, compare 0.2 s | 13 s: open and flush 1.4 s, tree 6.9 s, log 3.6 s, set 0.5 s, stamp 2 ms, close 0.4 s |
A large brain (104 GB allocated) | 1,505,026 (301,568) | 782,303 / 93,413 | 4.8 min: read-only open 8.2 s, tree 228 s, log 48 s, compare 1.8 s | 4.0 min: open and flush 4.4 s, tree 188 s, log 40 s, set 4.6 s, stamp 7 ms, close 0.2 s |
The tree walk is the cost, and it scales with the committed generations (one small file or one packed frame each). On the larger brain 23,451 of its 45,858 stored blobs had a count below the recounted one and none had a count above it; a second recount of the stamped copy found 45,858 unchanged, 0 raised, 0 lowered. 249,426 of its 669,489 superseded or removed versions were superseded by a working-state write and took no reference. An independent reader of the log that shares none of the recount's code found every other superseded or removed version that names a stored blob counted (0 uncounted of 50,553 that name a blob).
Rolling back: what an older engine does with a flipped brain
The mark has existed since 12.0.0, and every engine since then reads it. An older engine opening a flipped brain serves it log-canonical too. It does not reopen it as a mirror, and it refuses every door that would write the frozen tree (an enabled: false open, a state replacement, a history compaction, a rollback of the migration).
Relationships the native writer lands are the one difference. This release's related({ from }) reads relationships through the log and returns them (measured on the flipped copy: a relationship written over the wire was returned by related({ from }) and related({ to }) after a reopen). A 13.1.x engine reads related({ from }) from the tree, where the native writer writes nothing, so it does not return those relationships. related({ to }) and get return them in both.
Rows the native writer lands raise one more rollback question, answered in rolling back: a 13.2.x engine that opens a flipped brain after native writes erases the search entries of those rows (the rows stay in the log, and one metadata rebuild restores the entries), and 13.3 reads the log instead. Until 13.3 is the engine that reopens the brain, take no native write session on a brain an engine will open afterwards.
There is no command that un-flips a brain. The exact way back is to restore a snapshot taken before the flip (backup and restore). Every write since that snapshot is lost, so take one immediately before flipping.