U64 Entity IDs

Cor maps every entity UUID to a compact integer id internally. Two parts of the stack key off that integer — the roaring bitmaps inside the metadata index, and the mmap-backed BinaryEntityIdMapper that persists the UUID ↔ int mapping at billion scale. Both of those layers historically used 32-bit ints (Roaring32, u32), which caps a single brain at 4 294 967 295 entities — about 4.29 B.

That ceiling is fine for almost every workload. It is not fine for the corpora cor 3.0 targets: the design point of the release is "1 B vectors on a single $1000 box", and the long tail goes well past 4 B. So cor 3.0 adds a second IdSpace — 'u64' — that lifts the ceiling to 2⁶⁴ - 1 (≈ 1.84 × 10¹⁹) while keeping every public API the same.

Picking an IdSpace

Cor 3.0's default is 'u64' — there's nothing to configure for new brains. You only need to think about this if you specifically want a brainy-7.x-compatible 4.29 B-capped brain.

import { BrainyData } from '@soulcraft/brainy'
import { register as registerCor } from '@soulcraft/cor'

// Cor 3.0 production default — no config needed for new brains.
const brain = new BrainyData({
  storage: { type: 'filesystem', rootDirectory: '/data/idx' },
})
await registerCor(brain)
await brain.init()
// brain is u64 internally — supports up to 2^64 - 1 entities.

// Explicit opt-in to u32 (brainy 7.x compatible mode):
const brainCompat = new BrainyData({
  storage: { type: 'filesystem', rootDirectory: '/data/legacy-brain' },
  entityIdMapper: { idSpace: 'u32' },
})

Use U64 (default) when:

• You're starting a new brain on cor 3.0 — it's the default for a reason.

• You expect any chance of exceeding 4.29 B entities over the brain's lifetime.

• Brain holds high-fanout graphs where the verb-int count can grow faster than the entity-int count.

Use U32 (explicit opt-in) when:

• You're restoring a brainy 7.x snapshot whose on-disk format was u32 and you want bit-identical compatibility with a brainy 7.x reader.

• You're running a cortex 2.x-compatible deployment that interops with old tooling.

• You need byte-identical wire format with cortex 2.x — existing .cidx segments and chunked metadata JSON envelopes are unchanged.

Use U64 when:

• The brain may exceed 4.29 B entities.

• You're starting fresh and want headroom — the U64 wire format has ~negligible overhead at small scale and is forward-compatible with every cor 3.x release.

• You want to use the BigInt napi sibling methods directly (e.g. when a downstream system already speaks bigint and you want to skip the safe-integer guard).

The two modes are not interchangeable on the same files. You cannot escalate a U32 brain to U64 in place — the on-disk header is authoritative and any mismatched config is rejected as a hard error at open time. Migrating an existing U32 brain to U64 is a re-export

• re-ingest into a new brain.

No sizing config — the id mapper auto-grows

You never size the id-mapper files. Both back the brain's UUID ↔ int mapping with mmap-backed sparse files that grow themselves at compaction as the brain fills:

• int_to_uuid.bin is direct-indexed (slot = header + int × 24). When the highest assigned int approaches the file's slot capacity, the next compaction rebuilds the base at a geometrically larger size (×2, with headroom). No fixed entity ceiling.

• uuid_to_int.mkv is an extendible hash. Its directory auto-doubles up to the architectural maximum depth (32), and compaction enlarges the bucket region on demand — with a retry-on-overflow loop so even highly-skewed key sets never hit a premature cap.

Growth rides the crash-safe shadow-page compaction swap (a new base is built, the head pointer is swapped atomically, the old base is unlinked), so there is no remap-while-readers-active hazard and a crash mid-grow recovers cleanly. The files are sparse: the declared size costs no real disk until the slots are actually written, and the modest defaults keep tiny / multi-tenant brains small.

Practical ceiling: a single brain scales to ~51 B entities (max_global_depth=32 at the 16-entry bucket capacity) with zero configuration. Past that — i.e. beyond the documented single-box content ceiling — cor raises a clear "shard the brain" error rather than silently degrading; horizontal scaling across brains is the architecture at that point (see why-cor).

What changes on the wire

• int_to_uuid.bin (binary mapper): v1 header for U32, v2 header for U64. The header carries the IdSpace; the file is self- describing.

• .cidx column-store segments: flags byte's FLAG_U64_IDS bit (0x01) is set; the entity-id column is u64 little-endian. The cross-language fixture in src/native/cidxU64WireFormatFixture.test.ts locks the byte layout via SHA-256.

• Metadata chunk JSON envelope: U64 chunks get a v2 envelope with "version": 2 and "idSpace": "u64" fields. U32 chunks emit the v1-compatible envelope (no extra fields) for backwards compatibility.

• PostingList (in-memory representation): the U32 brain uses croaring::Bitmap (Roaring32); the U64 brain uses croaring::Treemap (Roaring64). Both expose the same algebra (union_assign, intersect_assign, difference_assign, iter_u64, cardinality), so every query path is variant-agnostic.

What changes in the API

Nothing, for the common case. Every wrapper method that returned a number continues to return a number. Two refinements:

1. getAllIntIds() throws in U64 mode. A materialised number[] at billion scale would risk OOM, and U64 brains exist precisely because the entity count is large. Use the streaming BigInt iterator on the underlying native binding.

2. EntityIdSpaceExceeded is thrown when a U64 brain's number- typed method (e.g. getOrAssign(uuid): number) would return an int above Number.MAX_SAFE_INTEGER (2⁵³ - 1 ≈ 9.007 PB of entities). At that point you must switch to the BigInt sibling methods for the full u64 range:

   const big: bigint = mapper.getOrAssignBig(uuid)
   const uuidBack: string | undefined = mapper.getUuidBig(big)

   The BigInt siblings work in both modes. Use them in any code path that crosses the U32 → U64 escalation point so it doesn't need to branch on getIdSpace().

BigInt sibling surface (cor 3.0)

The BigInt methods are mode-independent: they work in U32 brains too, and return the same values widened to bigint. Mixing getOrAssign(uuid) and getOrAssignBig(uuid) for the same UUID returns the same underlying int.

Brainy 8.0 lockstep — EntityIdSpaceExceeded (JS fallback)

Brainy 8.0 ships the JS-fallback half of the contract. Without cor installed, brainy uses an in-RAM EntityIdMapper that also caps at u32::MAX — but the JS fallback can't widen to u64 (the bitmap layer is Roaring32 end-to-end on the JS side). So brainy 8.0's mapper throws EntityIdSpaceExceeded (in @soulcraft/brainy/internals) at the ceiling and points the caller at cor's idSpace: 'u64' mode as the migration path.

The two errors are siblings at different layers:

• brainy's EntityIdSpaceExceeded (U32_ENTITY_ID_MAX = 2³² − 1): the JS-only path hit the bitmap-width ceiling — install cor.

• cor's EntityIdSpaceExceeded (Number.MAX_SAFE_INTEGER = 2⁵³ − 1): a U64 brain's number-typed method overflowed JS's safe-integer range — switch to BigInt siblings.

Wire-format parity gates

Two hash-locked fixtures lock the format across reader implementations:

• .cidx U64 segments — src/native/cidxU64WireFormatFixture.test.ts. Three SHA-256 hashes (numeric / float / string segments) lock the byte layout. Brainy 8.0's reader targets the same fixtures.

• PostingList cross-language — round-tripped via the napi parityWriteU64NumericSegment / parityReadSegmentEntityIdsBig pair. Same write path produces same hash across 100 stress iterations (see src/native/cor30Stress.test.ts).

If a hash drifts, do not silently update the constant — root- cause the byte diff first. A non-deterministic writer or a half-applied format change is exactly the kind of bug those fixtures exist to catch.

See also

• Performance — measured roaring-bitmap ops + SIMD distance speedups.

• Scaling — multi-tenant density + the 1 B operational ceiling.

• DiskANN — the search engine that consumes these entity ids at billion scale.