Snapshot Safety

This document is a contract. Every persistent file brainy writes satisfies the property described here. Code that violates the contract does not ship.

What "snapshot" means in brainy

Brainy is the native acceleration layer underneath brainy. Brainy exposes a transactional Db API with asOf, with, and transact semantics, and persists durable state via db.persist(). Brainy's persistence model is hard-link snapshots: at persist() time, every file under the brain's root directory is hard-linked into the snapshot target.

A hard-link shares the inode with the live file. The directory entry is a separate pointer to the same on-disk data. If the live writer mutates the inode after the link is taken, those mutations are visible to any reader of the snapshot's hard-link. The filesystem provides no isolation between the two views.

That means hard-link snapshots are only point-in-time coherent for files that are immutable post-link or only modified via atomic rename. Any in-place mutation pattern — even one protected by a SeqLock or Release-Acquire pair for live concurrent reads — leaks live state into the snapshot. SeqLock makes concurrent readers retry until consistent; it does not freeze the bytes a snapshot reader holds.

The brainy promise

Brainy guarantees that any hard-link of any persistent file taken at any moment opens a coherent point-in-time view, regardless of what the live writer is doing concurrently. The view is self-contained: no coordination with brainy's pin/release protocol or transact-commit boundary is required for the snapshot reader to see consistent state.

The coordination layer (brainy's VersionedIndexProvider pin/release + the post-commit applier protocol described in the handoff Section L.2) provides cross-subsystem generation consistency — every brainy provider answers reads at the same brainy generation. The per-file snapshot safety described here is the load-bearing layer underneath it. Coordination is defense-in-depth, not the primary safety mechanism.

Why this is hard

mmap'd files plus hard-links are a foundational trap. A file that "feels safe" because individual reads are atomic (AtomicU8, AtomicU32, single-word store) is not snapshot-safe when:

• The logical state requires multiple atomic updates to publish (a bucket split touches both a payload region and a directory region; a memtable append touches both the record bytes and the header's next_offset).

• The file grows in place (a directory doubling, a bucket allocation past the current high-water mark, a memtable log growth event).

• A "frozen" region is later overwritten (a memtable log truncate-to-header pattern; a tombstone state byte replacing an occupied state byte).

Each of these has a window during which a hard-link reader sees a state that the live writer didn't intend to publish. The window can be nanoseconds; it can be milliseconds; it is non-zero. Any non-zero window violates the contract.

The pattern that ALL existing alternatives fail at:

"This file is safe to hard-link because the access pattern only writes to never-written regions" — false the moment any other caller starts overwriting. "This file is safe to hard-link because the SeqLock guard makes readers retry" — true for live readers; false for a snapshot reader holding a hard-linked inode that the writer is still mutating. "This file is safe to hard-link because brainy only snapshots at commit boundaries" — true under the current brainy-brainy coordination, false as a property of the file. Contingent safety is not safety; the moment a customer uses brainy outside brainy (or brainy changes the coordination contract) the assumption silently breaks.

The canonical pattern: LSM shadow-page

Every brainy storage primitive that holds mutable state uses the same three-layer pattern. The G4.3 verb endpoint store (crate::verb_endpoints_shadow_page::ShadowPageEndpointsStore) is the reference implementation.

{store-dir}/
  head                        # PointerFile, atomic-rename swap
  base-N.ext                  # frozen base for generation N (immutable post-write)
  delta-N+1.log               # append-only log for writes since base-N
  delta-N+1.log.K.archive     # rotation archives, retention-bounded

Three layers

1. Frozen base (base-N.ext) — the consolidated state at the start of generation N+1. Never mutated in place after it is sealed. Compaction produces base-N+1.ext as a brand new file, never edits the existing one. The old base file's inode remains reachable by any hard-link that captured it before the pointer swap; once no snapshot needs it, the directory entry is unlinked and the kernel reclaims the storage.

2. Append-only delta (delta-N+1.log) — every mutation since the base was sealed lands here as a fixed-format record (length-prefix + payload + CRC32, framed so a torn write at the tail is detected at exactly the boundary it occurred). Snapshot readers replay the delta from the head's recorded tail offset; live writers append past that offset without affecting what the snapshot reader sees. On compaction commit the delta is renamed to a .archive sibling and a fresh delta replaces it, carrying forward any record newer than the commit watermark — a record appended while the commit's durability IO was in flight stays replayable in the fresh delta, never exiled to the archive. Opens also scan retained archives for records above the watermark and recover them, so even a rotation interrupted between its two renames loses nothing.

3. Atomic head pointer (head) — a tiny text file named by PointerFile (diskann::pointer_file). Names the current base and delta files plus the current generation counter. Updated via save_atomic: write the new content to head.next, msync, rename(head.next, head). POSIX rename is atomic. A snapshot taken before the rename captures the old head's inode; a snapshot taken after captures the new. There is no window in which the snapshot sees a half-written head.

Why this is snapshot-safe

A hard-link snapshot captures a directory entry for each file present at the moment of the link. The directory entries point at inodes. After the snapshot, the live writer mutates the delta's inode (append-only growth) and creates new inodes for the next base and the new delta on compaction. The snapshot's directory entries are untouched.

A snapshot reader opening the snapshot directory resolves head to a (base, delta) pair. That base file's inode is the one the snapshot captured — frozen since it was sealed. That delta file's inode is the one the snapshot captured — the writer's subsequent appends went past the offset the snapshot recorded, so the snapshot reader's replay stops at the right boundary. Compaction events in the live store produce new base + delta files at new inodes that the snapshot's directory has no reference to.

The contract holds at every moment, not just at coordination boundaries.

Inventory: how brainy's storage primitives implement this

The "in scope" rows are the brainy 3.0 release plan items. No piece ships under the weaker "contingent safety" bar.

Forbidden patterns

1. In-place multi-word mutation of any byte range a snapshot can observe. Includes overwriting an occupied slot's payload, rewriting a directory region on growth, in-place truncate of a log file. If the operation requires multiple stores to publish the new logical state, it cannot mutate bytes a hard-link reader holds. Convert to atomic-rename of a new file.

   The one named exception: the clean-close trim. close() shrinks every preallocated arena to its watermark in place, and that is a deliberate, bounded exception to this rule rather than an oversight. It is admissible for three reasons, all of which must keep holding:

   • It removes only bytes above the file's own published watermark — past a rotating log's next_offset, past a slot array's capacity_slots, past an extendible-hash arena's allocated bucket range. Those bytes are reservation. No format reads them, no reader has ever read them, and no snapshot's logical state contains them.

   • Every instant is a legal, openable shape. For the slot arrays the trim shrinks the header's declared capacity FIRST and the file's length SECOND, because an open refuses a file whose header declares MORE slots than the file holds and accepts one that declares fewer. A snapshot taken at any instant of the trim — before it, between its two steps, or after — opens and serves exactly the same logical state. The reverse order would open a window in which a snapshot captured an unopenable file, which is the whole reason the order is fixed.

   • It refuses rather than reasons about its callers. A base a pinned view still holds, a base an off-lock maintenance build is walking, and a verb-endpoint base with no high-water mark are all NAMED refusals, not "safe because the caller happens to be closing" (forbidden pattern 2).

   Implementation: native/diskann/src/arena_trim.rs. If a future format keeps live bytes above its published watermark, this exception stops holding for that format and its trim must be removed rather than reasoned about.

2. "Safe under access pattern X" reasoning. If a primitive is only snapshot-safe because the current caller is insert-only, monotonic, single-writer, or otherwise constrained, the primitive itself is not safe. Either the primitive's contract gets tightened (its callers are checked against it) or the primitive is converted to the canonical pattern.

3. Coordination-dependent safety. If snapshot safety only holds when brainy snapshots at commit boundaries, when pin/release has been called in some specific order, or when some other external invariant is met, that's not safety. Brainy primitives must be safe in isolation. Coordination is for cross-subsystem generation consistency, not for per-file safety.

4. Hidden tombstones in frozen files. A frozen base file cannot grow new tombstone bytes after it is sealed. Tombstones are recorded in the delta log; compaction omits tombstoned entries when writing the next base.

5. Crash-recovery as a substitute for atomic publish. A recovery pass that "fixes up" half-written state on open is acceptable as a defense against crashes. It is not a substitute for atomic publish during normal operation. A snapshot taken between the half-write and the next open is not protected by crash recovery.

Verification methodology

Every storage primitive that holds mutable state ships with at least one snapshot_via_hard_link_survives_* property test:

• Set up the live store.

• Perform writes.

• Hard-link every file in the store directory to a sibling snapshot directory.

• Continue mutating the live store (further writes, compactions, rotations).

• Open the snapshot directory as an independent store instance.

• Assert that the snapshot reads return exactly the pre-snapshot state, with no live-writer drift visible.

The reference test is verb_endpoints_shadow_page::tests::snapshot_via_hard_link_survives_compaction. Every G4 conversion ships with an analogous test.

Snapshotting a brain

brain.snapshot(destDir, { mode }) copies a store root to destDir, bounded by file count, not bytes. It is the operational counterpart to the per-file contract above: that contract says a hard-link taken at any instant is coherent; this verb is how an operator actually TAKES a full, portable copy of a live store — before an engine swap, before a migration, as a point-in-time backup — without stopping the brain.

The problem, measured

Copying a house of several pooled brains before an engine swap with cp -a --sparse=always cost ~5 minutes of downtime for 1,025,426 files / 31 GB. The bytes were never the cost — 31 GB moves in seconds on NVMe. The cost was the FILE COUNT: about 3,000 file creations per second, because nearly every one of those files is a sparse arena whose live content is a small fraction of its declared size (see docs/deployment-limits.md §Disk footprint), and a naive copy pays a full open/read/write/close per file regardless.

The two modes

Both modes run behind a bounded writer hold (10 s): every write door flushes and pauses admission for the duration of the copy, and reads keep serving throughout. A write issued during the hold completes strictly after it — never refused, never lost. See native/src/snapshot.rs and src/api/snapshotCeremony.ts for the full contract, including the named refusal for every precondition (a non-empty destination, a reflink request across a filesystem boundary, an entry in the store root that is neither a file nor a directory).

After the copy, the destination's per-projection artifact census is compared against the source's, byte-exact, using the SAME native door the open-attach path already computes it with — the copy is not merely assumed correct, it is verified against the store it was taken from before the verb returns.

Measured

MEASURED on a copy of a real production-shaped store (the build box — Ryzen 9 7950X3D / 184 GB / NVMe, ext4; a 14,056-noun / 72,679-relationship fixture that opened cold — the metadata index rebuilt from canonical, 531.7 s, wholly BEFORE the snapshot below started, so none of that cost is in its wall):

ext4 does not implement FICLONE. 'reflink' mode is confirmed refused by name — both by native/src/snapshot.rs's unit tests and by an end-to-end pin against a real (small) store on this exact box — rather than measured at this fixture's full size: the refusal is a property of the filesystem, proven identically by the first file every time, so a second ~840,000-file run would only re-demonstrate what the first file already proves, at the cost of the box's shared disk and everyone else queued behind it. A 'reflink' files/second number is owed from a btrfs or XFS-with-reflink=1 host; until then, 7,434 files/second is the number an ext4 deployment should plan around, and 'reflink' should be read as a same-file-count, near-zero-byte upper bound on it — copy-on-write shares extents rather than writing them, so its wall is bounded by the same file count with none of 'copy''s per-file data-copy cost.

7,434 files/second is ~2.5× the ~3,000 files/second baseline the problem statement measured (cp -a --sparse=always) — and it is sparse-CORRECT where the baseline is sparse-preserving only by luck of the tool: the destination's 283.55 GiB apparent size against 23.45 GiB allocated (a 12.1× sparse ratio) is read back byte-for-byte identical to the source by the post-copy census, meaning every hole the source declared is a hole in the copy, never materialized zeros.

See also

• docs/performance-budget.md — the latency budget brainy commits to. Snapshot safety and performance are co-designed: the LSM shadow-page pattern's hot read path is sub-microsecond via the in-memory delta cache.

• PLATFORM-HANDOFF.md Section L.2 (brainy's post-commit applier contract) and Section R (VersionedIndexProvider interface) — the coordination layer that turns per-file snapshot safety into cross-subsystem generation consistency.

• docs/file-mutation-inventory.md — per-file analysis of the pre-G4 state and the conversions in scope.