Content references and the recount
In this section
A file's content is stored once, however many files name it: two files with the same bytes share one stored copy, and a copy of a file shares its source's. The store keeps a count of the files that name each copy, and the count decides when the bytes may be deleted. This page is about keeping that count honest, and about the one door that recounts it from the files.
What keeps the count honest
Writes to one path take turns. A write reads the path's current content (to know which bytes it is replacing), stores the new bytes, commits, and only then lets go of the bytes it replaced. Two writes to one path that overlapped used to read the same "replaced" bytes and both let go of them, so a count fell one too far and bytes another file still named were deleted. Writes to one path now queue, in arrival order, at the file-system door (
writeFile,appendFileandunlink); writes to different paths never wait for each other. An append reads and writes in one turn, so two appends both land, and an append whose current content cannot be read refuses instead of writing the new piece over the file.A copy takes its own reference.
copy(a file or a directory) takes one count per copied file before the copy exists. A copy whose source's bytes are gone refusesEIOby name: it would only name bytes that are gone.A delete asks the files first. Before the store deletes a single byte, it asks the live files whether any of them names the content: the metadata index names the candidates and each is confirmed against its stored row. If one does, nothing is deleted, the count is repaired to the number of files naming it, and the repair is narrated. If the question cannot be answered (an index that is not serving), the delete is withheld and the bytes are kept: a leak, never a loss. A count is a counter, and counters drift; the files are the truth.
A removed file's content is held. Content a removed file names is kept for the life of the store (
docs/removed-rows.md).
repairBlobReferences - the recount
const dry = await brain.repairBlobReferences() // what is wrong; nothing written
const done = await brain.repairBlobReferences({ apply: true }) // raise the counts that fell shortTSIt walks every live entity (through the same door every listing uses, so it reads whichever copy the store treats as canonical), counts the files that name each stored content blob, and compares with the count the store keeps. It deletes nothing, in a dry run or with apply.
the report says | what it means | what |
|---|---|---|
| blobs whose count was below the files naming them ( | raises each count to the number of files naming it. A raise can only keep bytes; it never lowers a count |
| live files whose bytes the store does not hold ( | nothing: the files are named for the person who owns the store |
| blobs counted as referenced that no live file and no history names, with the size of each one's content ( | nothing: lowering such a count would make the bytes deletable, which is the owner's decision |
| blobs some file names whose count is above the files naming them - a harmless excess | nothing |
| live rows the walk could not read, so it did not count them ( | nothing: heal the rows and run the recount again before acting on |
limit (default 1,000) bounds each list; the counts and the byte totals always cover everything. The walk is a point-in-time view of a brain that may be taking writes: a file created after its page was read is not counted, so a neverReleased listing taken on a busy brain can name a blob that has since gained a file. Run it when the brain is quiet for an exact listing; the raise never lowers a count, so it is safe either way. The run is narrated: a line when it starts, a progress line every 25 pages of the walk, and one summary line with every number above.
When to run it. Once on any brain that ran concurrent writers to one working-state file (retention: 'latest-only') before this release, and whenever a file reads Blob metadata not found. A brain written only one path at a time has exact counts and the report says so.
Over the wire. POST /v1/brains/{brain}/repair-blob-references with the service key (body { apply?, limit? }); the MCP tool brainy_repair_blob_references; BrainyClient.repairBlobReferences(). The call holds the connection for the walk. brainy serve answers NotYetNativeError by name: the recount needs the writer, which a read-only attach does not have. A read-only attach can still run the dry run in process. apply needs the writer.
What the recount cannot do
Bytes that are gone cannot be recounted back. The recount names every live file whose bytes are gone, so that the person who owns the store can decide what to do. If a copy of the brain (a nightly archive, a snapshot) still holds the bytes, the next section puts them back; otherwise the owner replaces the file or accepts the loss. The recount never invents, rewrites or removes a file.
Putting bytes back: readmit
When a copy still holds the bytes a live file names, readmit is the road back into the brain, and it runs only on the owner's word. Take the store's own blob files from the copy - _cas/blob:<sha256>.gz - and put them in one folder, each renamed <sha256>.gz.
The rule a file is checked by. A stored blob file is the gzip of the JSON wrapper {"_binary": true, "data": <base64>}, and the blob's name is the SHA-256 of the decoded bytes - the bytes data decodes to - not of the gzip file and not of the wrapper. (Bytes the store compressed, zstd frames, are decompressed first when the decoded bytes alone do not match.)
What it does with each file, in order; the first failure is its refusal:
step | refused as |
|---|---|
the entry is a regular file (not a folder, not a link) |
|
the name is |
|
the store does not already hold the blob - if it does, it is left exactly as it is | ( |
a live file, or a removed file (a removed-rows-list entry), names the hash - bytes nothing names would be stored for no one |
|
the file is not larger than the largest file the store holds |
|
the decoded bytes hash to the name |
|
Readmission runs first, before the recount walks the files: which live files name a hash is a single lookup, not a walk, so the bytes are back (and counted) by the time the walk looks for them, and a recount that follows - this one, or the exact recount of a flipped store - finds them stored. A file that passes is put back (with apply) and its counts are set from the files and the list: the live count is the number of live files naming it, the history count is the removed-rows list's references. The generations that still hold a file's older versions add their own references when the history recount next runs. The folder is only read - nothing in it is changed or deleted. A dry run verifies every file and reports what it would put back, and stores nothing.
const dry = await brain.repairBlobReferences({ readmit: { directory: '/srv/recovered' } }) // verify only
const done = await brain.repairBlobReferences({ apply: true, readmit: { directory: '/srv/recovered' } })
// done.readmit: { directory, filesFound, readmitted: [{ hash, bytes, liveFiles, ledgerReferences, fileIds }], alreadyPresent, refused: [{ name, reason }] }TSOver brainy host the wire never carries a path. The operator places one folder with --readmit-dir <root> (beside --brains-dir; a folder equal to, inside or containing it refuses to start), and the request names a folder inside it, or none for the folder itself:
curl -sS -X POST "$HOST/v1/brains/self/repair-blob-references" \
-H "Authorization: Bearer $SERVICE_KEY" -H 'content-type: application/json' \
-d '{ "apply": true, "readmit": { "folder": "recovered-2026-10-29" } }'SHA host started without --readmit-dir refuses CeremonyUnavailableError (501) naming the flag; a directory, a folder that is not one name made of letters, digits, ., _ and -, or a folder that is not inside the root is a 400 BadQueryParameterError. The report shows the directory as <readmit-dir>, and any text from inside a brain that names the brains directory (the reason a row could not be read) shows <brains-dir>; the host's paths stay on the host.