The tokens re-index ceremony
A row's token vectors are written on the way in, beside its embedding. Rows written before the projection existed have none — and on a real store that is every row you already have.
In this section
A row's token vectors are written on the way in, beside its embedding. Rows written before the projection existed have none — and on a real store that is every row you already have.
An uncovered row is not broken. It is served, it is found, and it scores zero on the late-interaction text signal, which is honest: the projection has nothing to say about it. What an uncovered store cannot do is be ordered by that signal, because a store stamped to a text source that covers a third of its rows is a store ranking by a signal most of it cannot answer.
The re-index ceremony is how a store becomes coverable.
// Start it. Returns immediately — this runs behind the doors.
await brain.index.reindexTokens()
// ...from any process, at any time, including long afterwards:
const receipt = await brain.index.reindexTokensStatus()
console.log(receipt?.summary)
// 'tokens projection COVERS this store: 11,904 of 12,000 row(s) indexed;
// 71 carried no text, 25 were excluded by rule.'TSWhat it guarantees
It never holds a door. This is a registered background job, not a call you await. It runs in installments, yields to foreground traffic between them, and stands down within one installment of a close(). It never takes priority over a query, a write or a flush — which is exactly why it is a job rather than a promise: nothing about your store's current answers depends on it finishing, so there is no reason for a caller to hold anything open across it.
It resumes where it stopped, and the resume is exact. Every installment publishes to the projection and then records the cursor it earned. That order is the whole of the guarantee. A process killed between the two re-encodes one installment on the next run — wasted work, and nothing worse. Reversed, a kill between them would leave the marker claiming rows nothing ever wrote: a hole in the projection that nothing later could report.
Its counts describe its own store. The census is merged on the same beat as the cursor, so walked === indexed + missing-text + skipped-by-rule holds at every checkpoint, including after a resume. A ceremony whose counters advanced in flight would count an installment whose publish then failed, and a resumed run's walked would exceed the number of rows it has.
A cancellation is not a failure. A close() reaches the job through its own yield. Its marker stands where the last installment put it, no retry is armed, and the runtime's failure budget does not move.
It is never silent. A run that starts, resumes, completes or fails says so, and leaves the evidence in the store rather than in a log somebody had to be watching.
What it reports, by class
Every row the walk visits lands in exactly one class. A ceremony that reported only "12,000 done" would be hiding the question you actually have.
class | what it means | what you can do about it |
|---|---|---|
| its text was encoded and its tokens published | nothing — this is the working case |
| the row carries no text to encode | nothing is lost; the row is served and scores zero on the text signal. Write a |
| a rule excluded it — see the two reasons below | depends on which |
skipped-by-rule is itself two reasons, kept apart in the census because only one of them is evidence of a problem:
reason | what it means |
|---|---|
| the engine's own |
| the row was never posted to the index, so the store's id mapper never gave it an int id to key tokens by |
The receipt carries all six numbers, and the summary line reads the totals back in a sentence. The sparse-flip guard (next) reads only skippedUnresolved.
What a row is indexed on
The same text it was embedded from, derived by the same function, with one preference in front of it: a row whose metadata carries a non-empty summary is indexed on that line.
Deriving it any other way would build a projection that is nearly right — the worst outcome, because nothing would fail. The scores would simply be answering a slightly different question than the vectors are, and no test could see it.
Why a text-less row does not stop the ceremony
The re-embed ceremony refuses outright when it meets a row it cannot convert, and it is right to: a store half in one embedding space and half in another has no coherent notion of distance at all.
This projection is different in kind. It is additive. A row with tokens and a row without are both served, both found, and both ranked by a formula that reads the row's own signals; the row without simply contributes nothing on one of the four. There is no state in which some of the store is in one space and the rest in another, so there is nothing to refuse. The text-less rows are counted, named in the receipt, and left alone.
Writes that land while it runs
They are indexed by the write path, exactly as they would be if no ceremony were running. The backfill walks canonical in a fixed order and publishes rows keyed by their own ids; a concurrent write publishes its row the same way. Two writes to one row's tokens converge on the later one, which is the row's current text.
The one case worth naming: a row created after the walk has passed its position gets its tokens from the write path and is never visited by the walk. That is correct and is why the receipt's walked is a count of what the walk saw rather than a claim about the store's size.
The sparse-flip guard
Before the flip, the ceremony checks its own census. ready.json is a claim that the projection covers the store — and a claim made over a census that says coverage is almost nothing is the exact defect this guard exists to close. Two real stores flipped ready that way the night of 2026-09-11: one indexed 1 of 40,932 rows, another 24–65 of thousands, 20,019 of 20,083 non-system rows unresolved. The cause — a multi-store id-resolution defect — is fixed, but the ceremony itself must never again write a readiness record over a census like that, whatever the cause.
The rule: if skippedUnresolved is more than 10 % of the non-system rows walked and more than 64 rows outright, the ceremony ends in refused-to-flip instead of flipping. Both bars must clear — a store with 64 or fewer unresolved rows always flips, however large a fraction of it that is, because a brand-new store whose own client has not posted most of its rows yet is not the defect this guard is watching for.
A refusal is not a failure and it is not silent:
The family is written — every row the walk could resolve was indexed. Only the flip is withheld.
No
ready.jsonis written, so the store stays exactly as safe to serve as it was:find({ profile })keeps reading the dense fallback for an uncovered row, same as any store mid-backfill.The receipt names the rule and the cure verbatim:
skipped-by-rule 20,019 of 20,083 non-system rows — the store's id mapper did not resolve them; the family was written but not flipped; make the mapper whole, then reindexTokens({ restart: true })TEXTreindexTokensStatus()reportsphase: 'refused-to-flip', andrestampFusion({ textSource: 'maxsim' })refuses by name, exactly as it does for any store whose backfill has not flipped.
The cure, when the census is wrong: fix whatever kept the id mapper from resolving those rows, then reindexTokens({ restart: true }) — a plain re-run resumes from the census that made it refuse and would refuse again with the same numbers.
The override, when the census is right: a store can be legitimately sparse — a brand-new one whose client has posted only a handful of its eventual rows, say. reindexTokens({ allowSparse: true }) flips anyway. It is explicit, never a default, and a run that resumes from a refused-to-flip marker does not re-walk the store to do it — the walk already finished; allowSparse only changes what happens at its end. The receipt records allowSparse: true on the flip it produced, alongside the same census, so a sparse flip is never mistaken for a clean one:
await brain.index.reindexTokens({ allowSparse: true })
const receipt = await brain.index.reindexTokensStatus()
receipt?.allowSparse // true — this flip overrode the guardTSThe flip, and what it does not do
The ceremony finishes with a single object write — the readiness record — naming the generation, the encoder digest and the final census. Its existence is the flip. Before it, the projection covers some prefix of the store; after it, the store is covered as of the generation it names.
What the flip does not do is change how queries are answered. It changes nothing at all about serving, because a partial projection was already safe to serve. What waits on it is a decision:
// Only once the store is covered:
await brain.restampFusion({ textSource: 'maxsim' })TSThat is the deliberate step that makes late interaction the store's text signal, and it is separate on purpose — restamping is a statement that this store's future pages differ from its past ones, and it should never be a side effect of a background job completing.
Restamping to maxsim before the projection exists refuses by name.
What it costs
Cost is dominated by the encoder: one forward pass per row with text, on the engine's own threads, never on yours. Two things bound it:
The installment. Sixty-four rows, sorted by text length before encoding. The sort is not a micro-optimisation — every sequence in one pass is padded to the longest in it, and an unsorted batch of sixteen was MEASURED at 28 % worse than encoding one at a time. Sorting is the whole of that fix, and it cannot change an answer: a row's tokens depend on that row's text alone.
The yield. The job gives the doors back between installments, so a busy store finishes later and never answers slower.
Storage is about 12 KB per row (int8, 96 components, at most 128 tokens) — PROJECTED from the format's own geometry, not measured on a fleet store.
The refusals
Never a silent skip. Each names its cure.
| what happened | cure |
|---|---|---|
| this build carries no tokens projection, or its encoder reports no identity digest | install and activate a build whose plugin registers the tokens family; until then |
| the store is already covered by this encoder | nothing — new rows get their tokens on the way in. |
| the partial backfill in this store was written by a different checkpoint |
|
| the index is shutting down and will not start new background work | nothing — the store is untouched; run the ceremony on the next open |
A ceremony that refuses this way has written nothing at all. The store is untouched and serves exactly as it did.
refused-to-flip (previous section) is a DIFFERENT thing from this table: it is not thrown, it is a terminal phase the walk reaches after doing real work — the family is written, just not the readiness record. Read it from reindexTokensStatus(), not from a caught error.
Reading the receipt
The receipt lives in the store, so it outlives the process that produced it.
const receipt = await brain.index.reindexTokensStatus()
receipt?.ready // has it flipped?
receipt?.phase // 'walking' | 'flipping' | 'done' | 'failed' | 'refused-to-flip'
receipt?.census // { walked, indexed, missingText, skippedByRule, skippedSystem, skippedUnresolved }
receipt?.generation // the family generation the last installment published
receipt?.cursor // what it has published through
receipt?.attempt // how many times it has been started, counting resumes
receipt?.resumed // did this run continue an earlier one?
receipt?.allowSparse // did a completed flip override the sparse-flip guard?
receipt?.lastError // the failure, when phase is 'failed' — or the guard's rule and cure, when phase is 'refused-to-flip'
receipt?.summary // one sentence, for an operator who was not watchingTSnull means the ceremony has never run against this store.