Rolling Back a Version
In this section
A rollback is a newer store opened by an older binary. It is the reverse of the direction every release is gated on (an older store opened by the newer binary), and it has a different law, because the older binary was finished before the newer store's shape existed.
The three tiers of a store, and what each promises backwards
Tier | Written by | Backwards promise |
|---|---|---|
Canonical (entities, verbs, the generation log) | every version, one shape | Exact. Every |
Arenas (id mappers, verb endpoints, rotating logs) | base + delta + head, swapped by generation | Opens. A newer binary only sizes the files; the layout is the same. An older binary preallocates at its own sizes on its next grow, which costs apparent bytes and nothing else. |
Projections (the metadata index, columns, the vector and graph indexes) | versioned by header flags | Opens, then may read wrong. A flag an older binary documents as reserved is ignored, not refused. The table loads; lookups that depend on the new shape miss. |
The third row is the whole subject. A projection is a derived artifact and the engine treats it as one: it can always be rebuilt from the canonical. But a rebuild happens when it is asked for, and an older binary cannot know to ask for it.
The 11.2.0 → 11.1.0 case, exactly
11.2.0 writes kind-tagged posting keys in the metadata index. Every entity written, updated or converged under 11.2.0 has its postings under the tagged key. The table carries a header flag saying so; 11.1.0's format documents that byte as reserved and ignores it.
So 11.1.0 opens an 11.2.0 store without a complaint, and then:
getNoun(id)andgetVerb(id): exact, from the canonical.Field queries (
where,in, ranges) and text queries: silently miss every row touched under 11.2.0. The lookup uses the untagged key; the posting sits under the tagged one. Rows never touched under 11.2.0 still answer, which makes the miss look like data loss rather than a shape difference.
That is a wrong answer, not a crash, and a wrong answer is the worse of the two.
The procedure
There is one exact rollback and one repairing rollback. Prefer the first.
Exact: restore a snapshot. Before the swap to the newer version, take a sparse-preserving copy of each store:
rsync -aS --delete "$STORE/" "$STORE.pre-11.2.0/"
# or: cp -a --sparse=always "$STORE" "$STORE.pre-11.2.0"BASHThe arenas are sparse; a plain cp, scp or tar without its sparse flag inflates a store to its apparent size and can fill the disk (see brainy/disk-footprint once it lands with 11.2.1). A rollback is then: stop the newer binary with a clean close, restore the copy over the store, start the older binary. Zero wrong reads, no rebuild, and it costs disk for as long as you keep the copy. Rows written after the snapshot and before the rollback are in the newer store only; keep it until you have decided about them.
Repairing: roll back, then rebuild the projection. If a snapshot was not taken, start the older binary and run:
await brain.repairIndex({ rebuild: ['metadata'] })TSThe rebuild is the online generational one: the doors stay open under the serving law while a shadow index is built from the canonical in the older binary's shape, then swapped. Until it completes, field and text queries under the older binary are the wrong answers described above. Its cost is the metadata shadow build for the store (minutes on a store of tens of thousands of rows; measured per store in the rehearsal lane's r.rebuild rows).
What the engine will do about it
An older binary cannot be changed after it ships, so the law forward is:
A format change that an older reader would misread sets a must-understand flag, so the older reader refuses with a named error instead of serving a miss. 11.2.0's typed-keys flag was placed in the reserved byte; from 11.2.1 the reader treats any unknown must-understand flag as a refusal, and every later format change is declared there.
Every release note names the rollback class of the release: exact (no projection shape changed), rebuild (a projection shape changed; rollback needs the rebuild above) or snapshot (a canonical or arena shape changed, or an older binary refuses the store; rollback needs the restore above). 11.2.0 is rebuild.
The 13.1 → 13.0.x case
13.1 keeps each field's value index (the list of distinct values that exists, missing, ranges and value listings read) as a starting copy plus small change records written at each save, instead of one whole copy rewritten at every save. 13.0.x reads only the whole copy, so on a store 13.1 has saved it would load each field's values as of the last full copy and miss every value written since: the "opens, then reads wrong" case above.
So 13.1 declares the change where an older reader is made to stop. Every metadata index table it writes carries the first must-understand header flag, and 13.0.x refuses such a table by name when it opens the store, naming the flag and this page. Because 13.0.x cannot open the store at all, it cannot run the repairing rebuild either. 13.1 is rollback class snapshot: take the snapshot before the upgrade, and restore it to roll back.
A rebuild that a 13.1 close parked keeps its value counts in that same format, under the rebuild's own generation, so the same rule covers it: the snapshot taken before the upgrade holds no park, and nothing about a park needs its own rollback step.
The 13.1.1 → 13.1.0 case
13.1.1 writes each field's starting copy in a new layout that is read in place instead of being parsed whole when the field is first used. 13.1.0 does not know that layout. Every metadata index table 13.1.1 writes carries a second must-understand header flag, so 13.1.0 refuses the store by name when it opens it, as above; and the starting copy itself carries a flag 13.1.0 refuses, so even a store with no new table yet is never read as if the copy were missing. 13.1.1 is rollback class snapshot: take the snapshot before the upgrade, and restore it to roll back.
The 13.1.2 → 13.1.1 case (the column revival record)
A sorted index (one per field) keeps two records of what changed: which items were deleted, and which deleted items were written again (an update) together with the point from which their new values count. Through 13.1.1 the second record was rewritten whole at every save. 13.1.2 writes only the items updated since the field's last save, inside the change file the first record already keeps, and rewrites the whole record only when those change files are folded together.
An older binary does not know that a change file can carry these entries. So a change file that does carries a new leading tag, and 13.1.1 (and every build back to 12.13.4) refuses it as damaged: that one field is refused by name, its queries fail with a typed error instead of answering, and the brain rebuilds the field from the stored items, as it does for any damaged sorted index. A field whose change files carry no such entries reads exactly as before. This change alone is rollback class rebuild; 13.1.2 as a release keeps the snapshot class of the 13.1.1 change above it.
The relationship tier key (after 13.2.0)
After 13.2.0 a relationship's visibility tier is indexed under system.visibility instead of the bare visibility key (see the field-addressing page). An older binary reads such a store without refusing anything, but relationships it writes or updates during a rollback are indexed the old way while the store's marker still says every relationship is keyed the new way, so the newer engine's open does not warn about them — after rolling forward again, run repairIndex({ rebuild: ['metadata'] }) once to re-post them. related() reads each relationship's own record and is right in both directions.
A store the native server wrote, opened by a 13.2.x engine
This one is not a difference in file layouts. It is a read the 13.2.x engine got wrong, and it matters to any rollback to a 13.2.x engine after brainy serve has taken writes on a flipped brain.
A flipped brain's entity tree is frozen, and brainy serve's committer writes the log and no tree object. When a 13.2.x engine opens such a brain as its writer and finds commits above its metadata stamp, it re-applies the rows those commits touched. To learn what each row holds now it read the entity tree, which holds none of the server's rows. It took each of them for deleted and retracted every one of its search entries (its path, its type, every field a filter reads). Nothing was lost from the record: the rows are in the log, and a read by id (get) returns them. What stopped working is finding them by path or by field, and a folder listing that shows a file stat can no longer open.
13.3 reads that state from the log, and an end-to-end test that fails on a 13.2.x engine and passes on 13.3 holds it there. So:
Until 13.3 is the engine that reopens a brain, do not take a native write session on a brain a 13.2.x engine will open afterwards. Any write through
brainy servecounts.A brain that already met it recovers, and no row is lost. The rows are in the log. The engine's own background heal re-posts the erased search entries: on a 104 GB copy of a real brain it put 40 erased entries back after 95 seconds of one open, and one
repairIndex({ rebuild: ['metadata'] })restored 160 of 160 (1,431 seconds on a loaded machine). Search by meaning needs one more step, because the erased rows also lost their numbers: runrepairIndex({ rebuild: ['vector'] })and reopen, and the engine's own fill puts the vectors back under the rows' current numbers (200 of 200 reachable 46 seconds into the next open).
After the native server is stopped without a clean close
brainy serve keeps the log as the record and keeps the folder graph beside it, flushing the graph on its own schedule. If the server is killed (the process, or the machine's power), the log still holds every write the server acknowledged. In a test that killed it at random moments in the middle of a stream of file writes, twelve times, all 128 acknowledged files were there afterwards with exactly the bytes sent, and no file was ever half written. What can lag is the folder listing, because the graph is as of its last flush.
Reopen the brain with the native server first. Its open replays what the log holds above the graph's last flush before it answers anything. Stop it cleanly, and the listing is whole. In the same test an engine opened six of the killed brains after exactly that, and its folder listing showed every file it served.
If a 13.2.x engine opens a killed brain first, it serves every file by path (
stat,readFile,get) but lists a folder from the graph as last flushed and does not replay the log into it. In the test, all 65 files of the six brains opened this way were missing from their listings. Nothing is lost, because the rows and their links are in the log, but the listing stays short until the graph is derived again. Do not rely onrepairIndex({ rebuild: ['graph'] })for this yet: it has not been run on a brain in this state. Avoid the order instead.What is not done. Making the engine replay the graph from the log at its own open, the way it already does for the search index. It matters only while an engine stays in the path of a brain the native server writes; until then, the rule above is the whole answer.
A write the native server was making when it was killed, opened by an older engine
Every write the native server makes first puts a small note in the log naming the rows it is about to touch and, since 13.3, one more word: how it was running (at-log or at-log-shared; a transact writes no word). The native server's own next open reads that word to put a killed write back without moving any count. A 13.2.x engine reads the note by its three older names and does not look at the fourth, so it opens the brain and rolls the killed write back by its own rule, exactly as before; the new word changes what the native open does, never what an older engine does (a test runs the 13.2.3 reader over a note carrying the word). The other direction is strict: an engine that finds a word it does not know refuses to open that brain and names the word, rather than rolling the write back by a rule written for a different mode. Open such a brain with the build that wrote it.
The 13.3.0 → 13.2.x case (measured)
For a brain the in-process engine writes — a mirror brain, never written by the native server, never flipped — 13.3.0 adds no file an older engine misreads, and the class is opens, exact, measured twice on Tue 2026-11-03 on the build box:
A synthetic store (2,000 nouns, 300 relationships) written and closed by 13.3.0, opened read-write by 13.2.2: every id, every count, a page ordered by
system.createdAt, a page ordered by a user field,related()and fiveget()s answered identically; 13.3.0 then reopened it identically and kept writing. Opens: 166 ms / 132 ms / 129 ms.A copy of a real tenant store (3.4 GB allocated, 232,288 files, written by 13.2.2): 13.3.0's first open took 682 ms to serving and about 7 MB of disk (the store then grew 3.35 → 3.60 GiB during 8,100 adds and their flushes — ordinary write growth); 13.2.2 then opened the same copy in 418 ms with every read identical; 13.3.0 reopened it in 438 ms, identical, and kept writing.
What 13.3.0 writes that 13.2.x ignores is nothing on such a brain. The two cautions above about brains the native server wrote still stand; they are not this case. The pre-update sparse copy remains the proven way back for a brain that does not match this shape.