Snapshot Safety

Name: Soulcraft
Address: US

This document is a contract. Every persistent file cor writes satisfies the property described here. Code that violates the contract does not ship.

What "snapshot" means in cor

Cor is the native acceleration layer underneath brainy. Brainy exposes a transactional Db API with asOf, with, and transact semantics, and persists durable state via db.persist(). Brainy's persistence model is hard-link snapshots: at persist() time, every file under the brain's root directory is hard-linked into the snapshot target.

A hard-link shares the inode with the live file. The directory entry is a separate pointer to the same on-disk data. If the live writer mutates the inode after the link is taken, those mutations are visible to any reader of the snapshot's hard-link. The filesystem provides no isolation between the two views.

That means hard-link snapshots are only point-in-time coherent for files that are immutable post-link or only modified via atomic rename. Any in-place mutation pattern — even one protected by a SeqLock or Release-Acquire pair for live concurrent reads — leaks live state into the snapshot. SeqLock makes concurrent readers retry until consistent; it does not freeze the bytes a snapshot reader holds.

The cor promise

Cor guarantees that any hard-link of any persistent file taken at any moment opens a coherent point-in-time view, regardless of what the live writer is doing concurrently. The view is self-contained: no coordination with brainy's pin/release protocol or transact-commit boundary is required for the snapshot reader to see consistent state.

The coordination layer (brainy's VersionedIndexProvider pin/release + the post-commit applier protocol described in the handoff Section L.2) provides cross-subsystem generation consistency — every cor provider answers reads at the same brainy generation. The per-file snapshot safety described here is the load-bearing layer underneath it. Coordination is defense-in-depth, not the primary safety mechanism.

Why this is hard

mmap'd files plus hard-links are a foundational trap. A file that "feels safe" because individual reads are atomic (AtomicU8, AtomicU32, single-word store) is not snapshot-safe when:

The logical state requires multiple atomic updates to publish (a bucket split touches both a payload region and a directory region; a memtable append touches both the record bytes and the header's next_offset).
The file grows in place (a directory doubling, a bucket allocation past the current high-water mark, a memtable log growth event).
A "frozen" region is later overwritten (a memtable log truncate-to-header pattern; a tombstone state byte replacing an occupied state byte).

Each of these has a window during which a hard-link reader sees a state that the live writer didn't intend to publish. The window can be nanoseconds; it can be milliseconds; it is non-zero. Any non-zero window violates the contract.

The pattern that ALL existing alternatives fail at:

"This file is safe to hard-link because the access pattern only writes to never-written regions" — false the moment any other caller starts overwriting. "This file is safe to hard-link because the SeqLock guard makes readers retry" — true for live readers; false for a snapshot reader holding a hard-linked inode that the writer is still mutating. "This file is safe to hard-link because brainy only snapshots at commit boundaries" — true under the current cor-brainy coordination, false as a property of the file. Contingent safety is not safety; the moment a customer uses cor outside brainy (or brainy changes the coordination contract) the assumption silently breaks.

The canonical pattern: LSM shadow-page

Every cor storage primitive that holds mutable state uses the same three-layer pattern. The G4.3 verb endpoint store (crate::verb_endpoints_shadow_page::ShadowPageEndpointsStore) is the reference implementation.

{store-dir}/
  head                        # PointerFile, atomic-rename swap
  base-N.ext                  # frozen base for generation N (immutable post-write)
  delta-N+1.log               # append-only log for writes since base-N
  delta-N+1.log.K.archive     # rotation archives, retention-bounded

Three layers

Frozen base (base-N.ext) — the consolidated state at the start of generation N+1. Never mutated in place after it is sealed. Compaction produces base-N+1.ext as a brand new file, never edits the existing one. The old base file's inode remains reachable by any hard-link that captured it before the pointer swap; once no snapshot needs it, the directory entry is unlinked and the kernel reclaims the storage.
Append-only delta (delta-N+1.log) — every mutation since the base was sealed lands here as a fixed-format record (length-prefix + payload + CRC32, framed so a torn write at the tail is detected at exactly the boundary it occurred). Snapshot readers replay the delta from the head's recorded tail offset; live writers append past that offset without affecting what the snapshot reader sees. On compaction commit the delta is renamed to a .archive sibling and a fresh delta replaces it, carrying forward any record newer than the commit watermark — a record appended while the commit's durability IO was in flight stays replayable in the fresh delta, never exiled to the archive. Opens also scan retained archives for records above the watermark and recover them, so even a rotation interrupted between its two renames loses nothing.
Atomic head pointer (head) — a tiny text file named by PointerFile (diskann::pointer_file). Names the current base and delta files plus the current generation counter. Updated via save_atomic: write the new content to head.next, msync, rename(head.next, head). POSIX rename is atomic. A snapshot taken before the rename captures the old head's inode; a snapshot taken after captures the new. There is no window in which the snapshot sees a half-written head.

Why this is snapshot-safe

A hard-link snapshot captures a directory entry for each file present at the moment of the link. The directory entries point at inodes. After the snapshot, the live writer mutates the delta's inode (append-only growth) and creates new inodes for the next base and the new delta on compaction. The snapshot's directory entries are untouched.

A snapshot reader opening the snapshot directory resolves head to a (base, delta) pair. That base file's inode is the one the snapshot captured — frozen since it was sealed. That delta file's inode is the one the snapshot captured — the writer's subsequent appends went past the offset the snapshot recorded, so the snapshot reader's replay stops at the right boundary. Compaction events in the live store produce new base + delta files at new inodes that the snapshot's directory has no reference to.

The contract holds at every moment, not just at coordination boundaries.

Inventory: how cor's storage primitives implement this

Subsystem	Primitive	Pattern	Status
Verb endpoints	`ShadowPageEndpointsStore`	LSM shadow-page (canonical)	shipped (G4.3)
Memtable log	`MemtableLog` rotation	rotation-on-commit (delta-only, no separate base)	self-contained safety in scope for G4.2 redo
Entity int → UUID	`BinaryIdMapper::I2uStore` (`int_to_uuid.bin`)	currently in-place with Release-Acquire state byte	LSM conversion in scope for G4.4 redo
UUID → entity int	`MmapKv` (`uuid_to_int.mkv`, `verb-uuid-to-int.mkv`)	currently in-place extendible hash	LSM conversion in scope for G4.1
LSM SSTables	`MmapSSTable` files	written once, atomic-rename committed, immutable thereafter	correct as shipped
DiskANN index	`.dkann` file	written once during build, mutated only at delete-marker byte	audit in scope
Column store segments	`column_store/segment-*.col`	audit in scope	audit in scope
LSM manifest	`manifest.json`	atomic-rename	correct as shipped
DiskANN build temp	`_system/scratch/build-*.adj`	excluded from brainy's persist set	correct as shipped (G4.5)

The "in scope" rows are the cor 3.0 release plan items. No piece ships under the weaker "contingent safety" bar.

Forbidden patterns

In-place multi-word mutation of any byte range a snapshot can observe. Includes overwriting an occupied slot's payload, rewriting a directory region on growth, in-place truncate of a log file. If the operation requires multiple stores to publish the new logical state, it cannot mutate bytes a hard-link reader holds. Convert to atomic-rename of a new file.
"Safe under access pattern X" reasoning. If a primitive is only snapshot-safe because the current caller is insert-only, monotonic, single-writer, or otherwise constrained, the primitive itself is not safe. Either the primitive's contract gets tightened (its callers are checked against it) or the primitive is converted to the canonical pattern.
Coordination-dependent safety. If snapshot safety only holds when brainy snapshots at commit boundaries, when pin/release has been called in some specific order, or when some other external invariant is met, that's not safety. Cor primitives must be safe in isolation. Coordination is for cross-subsystem generation consistency, not for per-file safety.
Hidden tombstones in frozen files. A frozen base file cannot grow new tombstone bytes after it is sealed. Tombstones are recorded in the delta log; compaction omits tombstoned entries when writing the next base.
Crash-recovery as a substitute for atomic publish. A recovery pass that "fixes up" half-written state on open is acceptable as a defense against crashes. It is not a substitute for atomic publish during normal operation. A snapshot taken between the half-write and the next open is not protected by crash recovery.

Verification methodology

Every storage primitive that holds mutable state ships with at least one snapshot_via_hard_link_survives_* property test:

Set up the live store.
Perform writes.
Hard-link every file in the store directory to a sibling snapshot directory.
Continue mutating the live store (further writes, compactions, rotations).
Open the snapshot directory as an independent store instance.
Assert that the snapshot reads return exactly the pre-snapshot state, with no live-writer drift visible.

The reference test is verb_endpoints_shadow_page::tests::snapshot_via_hard_link_survives_compaction. Every G4 conversion ships with an analogous test.