Upgrading to Brainy 8.0 + Cor 3.0

This guide is for anyone running Brainy 7.x + Cortex 2.x in production and upgrading to Brainy 8.0 + Cor 3.0. It covers what changes on disk, what the safe upgrade flow looks like, what zero-config defaults flip, and what the rollback story is if something goes wrong.

Scope. Cor 3.0 + Brainy 8.0 are a coordinated, lock-step release. Don't upgrade one without the other — the napi contracts and on-disk formats are tied. The peerDependency in cor 3.0 is @soulcraft/brainy@>=8.0.0.

Soulcraft posture. Soulcraft is the only customer of this stack today. Per the 2026-06-10 mandate, data formats are freely broken in 3.0. The safe upgrade pattern is backup → wipe → reinstall → restore, NOT in-place migration. Reading raw cortex 2.x on-disk files in cor 3.0 is explicitly NOT supported; the new file formats (shadow-page LSM, memtable log V2, delta log V2) reject pre-3.0 inputs at open. Brainy's persist() snapshot format is the format-independent bridge.

TL;DR — automated migration script (recommended for production)

Cor 3.0 ships scripts/migrate-cortex-2x-to-3x.mjs — a checkpointed, idempotent, online-friendly migration tool. Use this for any deployment with real data. Don't hand-roll the four-step flow below unless you're migrating a throwaway dev brain.

# Single-brain layout (Venue, Workshop, single-tenant services):
node node_modules/@soulcraft/cor/scripts/migrate-cortex-2x-to-3x.mjs \
  --brain-dir /opt/venue/data/brain \
  --backup-dir /opt/venue/backups/pre-3.0

# Multi-tenant layout (Memory, per-user/per-org brain subdirs):
node node_modules/@soulcraft/cor/scripts/migrate-cortex-2x-to-3x.mjs \
  --tenants-dir /opt/memory/data/memory/users \
  --backup-dir /opt/memory/backups/pre-3.0 \
  --concurrency 8

# Dry-run first to see what will happen:
node ... --brain-dir /opt/venue/data/brain --backup-dir /tmp/test --dry-run

# Resume after a failure:
node ... --brain-dir /opt/venue/data/brain --backup-dir /opt/venue/backups/pre-3.0 --resume

# Rollback to the filesystem backup:
node ... --brain-dir /opt/venue/data/brain --backup-dir /opt/venue/backups/pre-3.0 --rollback

The script's guarantees:

• Zero data loss. Phase 2 takes a bit-identical rsync filesystem backup BEFORE anything else. Phase 6 (wipe) refuses to run unless both the filesystem backup and the logical snapshot have completed and been verified.

• Bounded downtime. Phases 1–4 (preflight, fs-backup, logical snapshot, verify) run while your service is LIVE. Only phases 5–9 (pause-writes → restore) require writes to be quiesced. For a 100 K-entity brain, the writes-paused window is typically < 60 seconds.

• Idempotent. Every phase checkpoints its completion to <brain-root>/.cor-3.0-migration-state.json. Crashes are recoverable; re-run with --resume.

• One-command rollback. --rollback restores from the filesystem backup. Combined with npm install @soulcraft/brainy@7.31.6 @soulcraft/cortex@2.7.0, you're back on the old stack in ~5 minutes.

Run node node_modules/@soulcraft/cor/scripts/migrate-cortex-2x-to-3x.mjs --help for the full flag list.

The 4-step upgrade (manual, for reference)

If you can't run the script (e.g., your deployment environment doesn't have the cor package installed yet, or you're doing a small-scale test migration), the manual flow is:

# 1. Snapshot every brain on the old stack (Brainy 7.x + Cortex 2.x)
node -e "
  const { BrainyData } = require('@soulcraft/brainy')
  for (const path of YOUR_BRAIN_DIRECTORIES) {
    const b = new BrainyData({ storage: { type: 'filesystem', rootDirectory: path } })
    await b.init()
    await b.persist('/backups/' + path.replace(/\\//g, '_') + '.brainy-snapshot')
  }
"

# 2. Upgrade your dependencies
npm install @soulcraft/brainy@^8.0.0 @soulcraft/cor@^3.0.0
# (or `bun add ...`)

# 3. WIPE the old brain directories (their on-disk format is incompatible)
rm -rf YOUR_BRAIN_DIRECTORIES

# 4. Restore each snapshot through the new stack
node -e "
  const { BrainyData } = require('@soulcraft/brainy')
  for (const path of YOUR_BRAIN_DIRECTORIES) {
    const b = new BrainyData({ storage: { type: 'filesystem', rootDirectory: path } })
    await b.init()
    await b.restore('/backups/' + path.replace(/\\//g, '_') + '.brainy-snapshot')
  }
"

The snapshot is JSON-shaped and format-independent. Restore replays every entity through the 3.0 write path, producing 3.0-native files. No in-place data conversion runs; the data simply re-lands in the new format because every record was re-written.

Time cost: roughly one minute per 100 K entities on commodity NVMe — restore is bottlenecked by re-embedding if your Brainy config rebuilds embeddings on write. If your snapshot includes pre-computed vectors, restore is ~5-10× faster.

Breaking changes that matter for the upgrade

Cor 3.0 — on-disk format changes

Why all the rejections? Cor 3.0 ships self-contained snapshot safety (docs/snapshot-safety.md) — every persistent file is hard-link-snapshot-safe by construction. That property requires the new file structure (frozen base + atomic head + append-only delta with rotation). The 2.x in-place mutation patterns leak live state into snapshots and can't be patched. The mandate explicitly chose "break the format and migrate" over "carry compatibility code forever."

Cor 3.0 — default-flip changes (zero-config)

These defaults change ONLY for new brains created on cor 3.0. Brains restored from a 2.x snapshot retain their pre-existing IdSpace if the snapshot recorded one — the on-disk file header is authoritative on reopen (per the G4.4 wrapper fix).

Brainy 8.0 — what the SDK changed

Brainy 8.0 has its own RELEASES.md entry documenting:

• DataAPI deletion (use brain.now().persist() / restore / stats instead)

• requireSubtype: true by default — existing entities without a subtype need to be migrated via brain.fillSubtypes() (helper shipped in c446783)

• Reserved-field contract — confidence, etc. now live at the top-level entity, not in metadata (auto-remapped on update)

• Full historical query surface (db.asOf(g) / db.with(...)) backed by cor's pin lifecycle

See brainy's RELEASES.md for the exhaustive list — that's the canonical source.

The safe upgrade flow, in detail

Phase 0 — Pre-flight (15 min, on the OLD stack)

1. Run npm test / your equivalent. Confirm everything is green on 7.x + 2.x before changing anything.

2. Take a filesystem-level snapshot of every brain directory (tar, rsync, ZFS snapshot, whatever you trust). This is your atomic rollback if step 3+ goes wrong. The Brainy persist() snapshot in step 1 is a logical backup; the filesystem snapshot is a bit-identical backup.

3. Read brainy's RELEASES.md 8.0 entry (in their repo) for any SDK-shape changes affecting your call sites. Cor 3.0 doesn't change Brainy's API surface — that's brainy's responsibility — but new defaults may affect entities you create.

Phase 1 — Snapshot (1 min per 100 K entities, on the OLD stack)

Run brain.persist(path) for every brain. The snapshot is a JSON-shaped file (gzipped) containing every entity, every relation, every metadata field, every vector — format-independent.

import { BrainyData } from '@soulcraft/brainy'

const brain = new BrainyData({ storage: { type: 'filesystem', rootDirectory: BRAIN_DIR } })
await brain.init()
await brain.persist('/backups/brain.snapshot')
console.log('snapshot stats:', brain.now().stats())

For multi-tenant deployments (e.g., per-user brains via BrainyInstancePool), iterate every tenant's brain directory and snapshot each one independently. Snapshots can run in parallel across tenants — they each touch only their own files.

Phase 2 — Upgrade dependencies (2 min)

# Pin to exact RC1 versions during early adoption; relax to ^8.0.0 / ^3.0.0 after GA.
npm install @soulcraft/brainy@8.0.0-rc.1 @soulcraft/cor@3.0.0-rc.1

# Verify peer-dep satisfied:
npm ls @soulcraft/brainy @soulcraft/cor

If your environment has multiple lockfiles (monorepo, multiple services), upgrade every one in the same change so a cross-service deploy doesn't mix major versions.

Phase 3 — Wipe (1 second)

rm -rf /data/idx/_id_mapper /data/idx/_metadata /data/idx/_diskann /data/idx/_graph_adjacency
# If you have other cor-managed paths under /data/idx, see the "cor-managed paths" appendix below.

Why wipe instead of letting the new code overwrite? The shadow-page LSM stores recognize their own head pointer file. A stale 2.x head pointer + new 3.0 binary may produce undefined behavior. Wiping the cor-managed subdirectories is faster than the alternative.

If you can't tolerate the wipe (e.g., a brain you can't snapshot for some operational reason), see the "Read-only museum mode" appendix at the end.

Phase 4 — Restore (5-10× faster than the original write, on the NEW stack)

import { BrainyData } from '@soulcraft/brainy'

const brain = new BrainyData({ storage: { type: 'filesystem', rootDirectory: BRAIN_DIR } })
await brain.init()  // creates the 3.0-native empty files
await brain.restore('/backups/brain.snapshot')

// Optional: confirm the restored stats match the snapshot stats.
const stats = brain.now().stats()
console.log('restored:', stats)

Restore replays every entity through brainy's normal add() path. Vectors are NOT recomputed — they ride through the snapshot. Metadata-index posting lists, graph adjacency LSM trees, and the DiskANN index are all built from scratch on the new disk format.

For per-tenant brains, run restore in batches matching your operational concurrency cap (typically 4-8 in parallel — each restore is CPU-bound on JSON parsing).

Phase 5 — Verify (5 min)

1. Spot-check entity counts match between snapshot stats and restored stats (per tenant).

2. Run a small set of read queries with known-good answers — search, getRelations, getAllConnected, getByMetadata. Confirm the results match the pre-upgrade behaviour.

3. Run your production test suite against the new stack. If you have integration tests that exercise find() with metadata + graph + vector composition, this is the moment to verify they still pass — cor 3.0's triple-intelligence find() is sub-millisecond at 1 M (docs/verification-report.md), so anything slower than the 2.x baseline is a config problem to investigate before flipping production traffic.

4. Restart the process (no, don't skip this). Confirm the new on-disk format reopens cleanly, restore-state survives the restart, and the engine reports the expected providers via brain.diagnostics().providers (which should show vectorIndex: { source: 'plugin' } — DiskANN is engaged).

Phase 6 — Cutover (operational)

Standard blue-green or canary cutover. Cor 3.0 + Brainy 8.0 is the same JS API surface as 7.x / 2.x for the read path; the breaking changes are constructor-time and on-disk-format-time. Once the new stack is green, route traffic.

What if it goes wrong?

Rollback (any time before Phase 6)

# Restore the filesystem-level backup from Phase 0:
rsync -a /backups/brain-fs-snapshot/ /data/idx/
# Downgrade dependencies:
npm install @soulcraft/brainy@7.31.6 @soulcraft/cortex@2.7.0
# Restart the process.

The filesystem-level backup is bit-identical to the pre-upgrade state. If you skipped Phase 0's tar/rsync snapshot, the logical snapshot from Phase 1 is your fallback — but you'd be restoring an empty 7.x brain and re-importing, which is a different flow than rollback.

"I upgraded a brain in place and now nothing reads"

That's the format-mismatch case. Brainy 8.0 + Cor 3.0 reading a Cortex 2.x file directory will surface an error like:

Error: shadow_page_mmap_kv invariant: head missing or unparseable current_generation
Error: MmapKvDeltaLog: unsupported version 1
Error: MemtableLog: unsupported version 1

Fix: shut down the process, wipe the cor-managed subdirectories, restore from the Phase 1 logical snapshot. If you don't have a Phase 1 snapshot, restore from the Phase 0 filesystem snapshot and try the upgrade again with all the phases.

"Restore is slow"

A 50 K-entity brain restoring in more than 10 minutes suggests either: (a) embeddings are being recomputed on write (verify your snapshot includes pre-computed vectors), (b) network storage instead of local NVMe (cor 3.0's DiskANN auto-engagement requires local FS — cloud adapters fall back to the slower HNSW path), or (c) Brainy's BrainyInstancePool is evicting before restore finishes (raise the eviction window for the migration window).

"My idSpace is wrong"

If your old brain was u32 and you accidentally created the new brain with the cor 3.0 default u64 and restored into it — that's actually fine. The on-disk format upgrades transparently (each u32 int sign-extends into a u64). The reverse (u64 brain restored into a u32 brain) errors out at restore time with EntityIdSpaceExceeded for any int above 2^53.

If you specifically need to keep a brain on u32 (e.g., interop with a Brainy 7.x-compatible consumer), pass entityIdMapper: { idSpace: 'u32' } to the BrainyData constructor explicitly. Cor 3.0 honours the explicit choice and rolls back the production default.

Worked example — Venue upgrade (live customer data)

Venue is the canonical "real customer data, can't lose anything" deployment. Production runs ^7.30.0 brainy + ^2.7.0 cor via SDK 3.20.1 on venue-vm (GCE). The brain holds bookings, customer records, kit configurations, deployment state — none of which can be lost.

Use the automated script. Do not hand-roll.

# On venue-vm, during the morning low-traffic window:

# Phase 0: pre-flight check on the running service.
curl -fsS https://venue.example.com/api/health | jq '.brainy.ok'
# Expected: true. If false, fix BEFORE upgrading.

# Take an out-of-band filesystem backup as belt-and-suspenders.
# (The script will take its own, but this one is yours to verify
# manually before the script touches anything.)
sudo zfs snapshot tank/opt/venue@manual-pre-3.0-2026-06-11
# OR if no ZFS:
sudo rsync -a /opt/venue/data/ /opt/venue/backups/manual-pre-3.0-2026-06-11/

# Phase 1-4: snapshot + verify (CAN run while service is live;
# Venue's writes during the snapshot window will land in 7.x but
# won't make it to the snapshot — see "online-snapshot caveats" below).
sudo -u venue node /opt/venue/node_modules/@soulcraft/cor/scripts/migrate-cortex-2x-to-3x.mjs \
  --brain-dir /opt/venue/data/brain \
  --backup-dir /opt/venue/backups/cor-3.0-migration

# At this point the script will prompt you to confirm "have writes
# been paused?" before phase 5. ALSO STOP THE SERVICE HERE for safety:
sudo systemctl stop venue-service

# Phase 5-6 will then run (pause-writes confirmation, wipe).
# Phase 7 will prompt you to upgrade dependencies — do it in ANOTHER
# terminal:

# On venue-vm in a second terminal:
cd /opt/venue
sudo -u venue bun install @soulcraft/brainy@^8.0.0-rc.1 @soulcraft/cor@^3.0.0-rc.1
# Confirm 'y' back in the first terminal.

# Phase 8-9 (restore + verify) will then run automatically.
# Phase 10 writes the completion marker.

# Start the service back up.
sudo systemctl start venue-service

# Verify production traffic.
curl -fsS https://venue.example.com/api/health | jq '.brainy.ok'
# Expected: true. Confirm a few read paths work end-to-end:
#   - load a customer record
#   - list current bookings
#   - render a published page

Total operational window for Venue: typically ~10 minutes for a brain with ≤ 100 K entities. Most of that is in Phase 2 (filesystem backup) which runs while the service is live. Actual downtime — from systemctl stop venue-service to systemctl start — is the wipe + npm install + restore window, normally < 2 minutes for a Venue-scale brain.

Online-snapshot caveats (important for Venue):

Brainy 7.x's persist() captures a consistent snapshot of entities visible at call time. Concurrent writes during persist() are NOT included in the snapshot. For Venue, this means: any booking created or customer record updated between persist() start and persist() end will be LOST if the service is left running during the snapshot.

For zero-loss migration, stop the service at the start of Phase 3 (logical-snapshot) — don't wait until Phase 5 (pause-writes). The script will accept early write-pausing; just answer y to the prompt before Phase 3 too.

If your acceptable-loss tolerance is "any write in the last 30 seconds is fine to lose" (typical for blog-shape data, not for booking/financial data), pass --online-snapshot and skip the early pause. Venue should NOT pass --online-snapshot. Memory + Workshop probably can.

Brainy 8.0 + Cor 3.0's pin lifecycle makes TRUE online migration possible (Phase 3 captures a consistent snapshot WITHOUT pausing writes, because brainy holds a generation-pin during the snapshot and writes that land after the pin are filtered out at read time). But that's the FUTURE state — you can't use it to migrate TO 3.0, only between 3.0.x versions later. For 2.x → 3.0, the safe path is: stop service → snapshot → wipe → upgrade → restore → start service.

Rollback (if anything breaks):

sudo systemctl stop venue-service
sudo -u venue node /opt/venue/node_modules/@soulcraft/cor/scripts/migrate-cortex-2x-to-3x.mjs \
  --brain-dir /opt/venue/data/brain \
  --backup-dir /opt/venue/backups/cor-3.0-migration \
  --rollback
cd /opt/venue && sudo -u venue bun install @soulcraft/brainy@7.31.6 @soulcraft/cortex@2.7.0
sudo systemctl start venue-service

Total rollback time: ~5 minutes. The filesystem backup is bit-identical; downgrade puts you on the exact pre-migration state.

Worked example — Workshop upgrade

Workshop runs the same dependency stack as Venue (SDK 3.20.1 → ^7.30.0 brainy + ^2.7.0 cor). The migration flow is identical to Venue's above with these differences:

• Workshop has per-workspace brain directories under {BRAINY_DATA_PATH}/{hash8}/{workspaceId} rather than a single brain. Use --tenants-dir with the workspace root, not --brain-dir.

• Workshop's data is less write-heavy than Venue's (workshops are typically authoring sessions, not transactional). The --online-snapshot flag is safer to use here than for Venue.

• Workshop is planning to migrate to Memory's unified personal data layer per the MEM-WORKSHOP-VFS-MIGRATION thread. If that migration happens BEFORE the cor 3.0 upgrade, Workshop's per-workspace brains become Memory's responsibility and Workshop's part of the cor 3.0 upgrade trivializes (just bump the SDK peer-dep range).

Worked example — Memory team upgrade

Memory is the canonical first 3.0 consumer (production on Brainy 7.31.6 + Cortex 2.7.0 as of 2026-06-11; per-tenant brains via BrainyInstancePool with strategy: 'per-tenant', mmap-filesystem storage at /opt/memory/data/memory/users/{emailHash}).

Memory-specific gotchas:

• Storage is Brainy's standard filesystem adapter (there is no separate "mmap" storage class to configure — cor memory-maps its own index files under whatever filesystem path you give Brainy). cor 3.0's DiskANN auto-engagement WILL fire here, which is what you want.

• Per-user brain cold start was 8.3 s on Brainy 7.31.3 + Cortex 2.7.0 (per the BRAINY-COR-MMAP-VECTOR-NOT-WIRED thread). On 8.0 + 3.0 with DiskANN + the cor shadow-page LSM, cold start should drop substantially — Adaptive DiskANN auto-selects in-memory mode at the per-user scale (each user is currently ≤ 1 K memories), giving sub-millisecond reads with no warm-up. Re-measure after upgrade and update the production-impact section of that thread.

• db.asOf(g) / db.with(...) time-travel reads are now available — if Memory wants to surface "what did this user remember last week", that's a feature you can build on top of brainy 8.0's historical query surface without any cor-side work.

• No external Memory user has 1 M+ memories yet, so the migration window is operationally cheap; do it during the morning low-traffic window per current ops practice.

Cor-managed paths (appendix)

Cor 3.0 owns the following subdirectories under your storage root ({rootDirectory} in your BrainyData storage config):

{rootDirectory}/
├── _id_mapper/
│   ├── uuid_to_int/         # ShadowPageMmapKv directory (G4.1)
│   ├── int_to_uuid/         # ShadowPageI2uStore directory (G4.4)
│   ├── verb_endpoints/      # ShadowPageEndpointsStore (G4.3)
│   └── verb-uuid-to-int/    # Verb-id ShadowPageMmapKv (Piece 12)
├── _metadata/               # LSM metadata index — SSTables + memtable shards (Piece 11)
├── _diskann/                # Adaptive DiskANN index file(s)
└── _graph_adjacency/        # 4 LSM trees for typed verb tracking (Piece D)

When upgrading, wipe THESE subdirectories. Anything else under {rootDirectory} (Brainy's own state, app-level files, the VFS tree) is Brainy's or your app's — leave it alone.

Read-only museum mode (appendix)

If a brain is permanently read-only and you can't tolerate the wipe (e.g., a regulatory-archive brain that can't be re-written for compliance reasons), the supported pattern is mount the old brain on a Brainy 7.x + Cortex 2.x sidecar process, expose its read path via your service's RPC, and have the new Brainy 8.0 + Cor 3.0 service proxy reads to it for those specific brain ids. Cor 3.0 does not read 2.x on-disk files in-process.

This pattern is operationally heavier than the upgrade — only use it for genuinely-immutable archives.

What I'd watch for in the first week post-upgrade

• brain.diagnostics().providers should report vectorIndex: { source: 'plugin' } (DiskANN engaged, not the fallback HNSW). If it reports HNSW, your storage adapter isn't returning a local filesystem path from getBinaryBlobPath() — switch to { type: 'filesystem', rootDirectory: '/path' } per docs/deploy.md.

• First read after a process restart. Should be sub-millisecond at < 100 K entities. If it's > 5 ms, the engine is in cold-cache + warming mode — second read should be hot. Anything still slow is a config problem; check idSpace (should be 'u64' per getIdSpace()) and verify the DiskANN file resides on the same NVMe as the process.

• Memory pressure under multi-tenant load. AdaptiveDiskAnnModeSelector auto-demotes from in-memory to hybrid under sustained pressure — observe brain.diagnostics().memory and brain.diagnostics().diskannMode to confirm.

• find() composed latency. Cor 3.0 ships triple-intelligence find() at p50 0.230 ms / p99 0.355 ms at 1 M (docs/verification-report.md). If your production latency is meaningfully worse than that at comparable scale, the composition is broken somewhere — most often a metadata filter that the LSM index doesn't have a posting list for, falling back to a sequential scan. Run brain.diagnostics().lsm.fields to confirm every filterable field has a built posting list.