Deployment — OS limits for multi-tenant pools

Running one brain per user (e.g. Workshop) puts many isolated brains in one process. Each is lean — a few open files + a few mmap regions — but they add up. Two OS limits can bottleneck a pool before RAM does:

A single big brain (e.g. Venue) doesn't need these raised. They matter at pool scale (hundreds–thousands of brains in one process).

Threads per open store

Every open store costs OS threads, not just file descriptors and mmap regions — a background job runtime (native worker pool + scheduler), and the vector/graph/column projections' own pools all spin up threads. A failed thread spawn under a tight TasksMax or memory-pressured thread-stack allocation is a refusal this engine narrates and degrades from (never a crash — see job_runtime's own docs), but sizing the ceiling correctly means the refusal never has to fire.

MEASURED (a 32-core build host, 2026-09-15, an ad hoc /proc/self/status:Threads probe against a built 12.5.x process, opening 1/2/4 filesystem-backed stores in one process and reading Threads after each open and after closing all of them):

The formula: threads(N) ≈ 7 + 68 (one-time process warm-up, paid once by the FIRST store any process opens) + 13 × N. The one-time ~68 is process-global infrastructure (pattern/embedding data load, global thread pools) that every store after the first shares for free — it is not per-store and does not repeat. The steady 13 threads per additional open store is what actually scales with tenant count: this runtime's own worker pool (MAX_NATIVE_WORKERS = 4 + 1 scheduler = 5) plus the vector/graph/column projections' own pools (~8). Closing all 4 stores released exactly 4 × 13 = 52 threads, back to the 75 the warm-up leaves behind for the rest of the process's life — confirming per-store threads are torn down cleanly on close(), not leaked.

Sizing TasksMax / ulimit -u: for a process expected to hold up to N stores open at once, budget ≈ 100 + 13 × N threads (the 100 rounds the ~75 baseline+warmup up with margin) and set TasksMax/ulimit -u well above that — the failure mode this budget exists to avoid is a thread-stack allocation refused under memory pressure at a count far below any hard ceiling (venue's own incident measured EAGAIN at 142 threads under a 12 GiB MemoryHigh, not a TasksMax anywhere near its limit — so the memory available for thread stacks matters as much as the task-count ceiling itself).

What brainy does automatically

• Raises its own soft nofile limit toward the hard cap on startup — no privilege needed, safe, zero-config (native/src/resource_limits.rs). So you only ever need to manage the hard cap.

• Warns once at pool scale if the hard cap or vm.max_map_count is below the projected need, with the exact fix command (so a missing setting is never silent). Single-brain deployments stay quiet.

• getOpenFileLimitInfo() exposes the live numbers for health checks / dashboards.

A process cannot raise its hard nofile cap or vm.max_map_count — those are the two things the deployment must set.

Recommended values

Per brain, post-lazy-shard: ~10–16 open files + ~24 mmap regions (with margin). For headroom to thousands of pooled brains:

ulimit -n        = 65536      (hard; brainy raises soft to match)
vm.max_map_count = 262144

Copy-paste config

Docker:

docker run \
  --ulimit nofile=65536:65536 \
  --sysctl vm.max_map_count=262144 \
  …your-imageBASH

Docker Compose:

services:
  workshop:
    ulimits:
      nofile: { soft: 65536, hard: 65536 }
    sysctls:
      vm.max_map_count: 262144YAML

Kubernetes (vm.max_map_count is a non-namespaced sysctl on most clusters → set it on the node, e.g. via a privileged initContainer or node tuning; nofile via the runtime):

# Node-level (initContainer or DaemonSet):
#   sysctl -w vm.max_map_count=262144
# Pod:
spec:
  containers:
    - name: workshop
      # container runtime nofile via securityContext / runtime configYAML

Bare VM / host (persistent):

# /etc/sysctl.d/99-brainy.conf
echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-brainy.conf
sudo sysctl --system
# /etc/security/limits.d/99-brainy.conf
echo '* hard nofile 65536' | sudo tee /etc/security/limits.d/99-brainy.confBASH

Verify

ulimit -Hn                      # hard open-file cap
cat /proc/sys/vm/max_map_count  # mmap region capBASH

Brainy logs both at pool scale if they're low. If you see the warning, apply the config above and restart.

Note: vm.max_map_count is Linux-only. On a macOS dev laptop it doesn't exist (and rarely matters at dev scale); ulimit -n applies on both. Production (Linux/Docker) needs both.

Disk footprint

A brain preallocates sparse arenas (the id-mapper, the verb namespace, the vector index) that show a larger apparent size than the allocated blocks they hold. du -sh <dir> reports allocated size; du --apparent-size -sh <dir> (or ls -l) reports apparent size. Some gap between the two is normal, not a leak.

The size law: which number each door reports

Every file in a brain has two sizes, and the engine keeps them apart by TYPE so a report can never quietly answer one question with the other number (src/utils/fileFootprint.ts; the twin the release and diagnostic scripts use is scripts/lib/file-footprint.mjs):

Which one each door reports, so you never have to guess:

npm run lint fails on a raw stat().size or blocks * 512 read outside those two doors, with a short allowlist that states each entry's reason — so a new size report cannot be added without deciding which number it means.

The clean-close trim

close() — settle, flush, attest — truncates every preallocated arena to its watermark, after the last write and before the attestation seals, so the attestation's artifact census records the trimmed sizes and the next open attaches against them. The trim is an ftruncate per file, so it costs a close nothing measurable; nothing is deleted and nothing moves.

For an existing store, that first clean close on 11.2.1 or later IS the migration. There is no separate step and no operator action: open the brain with the new build and close it cleanly. The close narrates it once by name, with the bytes released. Each arena then grows again from its watermark when it needs to.

One residue survives that close, and it is worth knowing about so it is not mistaken for a failure. A trim removes everything ABOVE an arena's watermark; it cannot remove reservation that sits BELOW one. The extendible-hash arenas (uuid_to_int, verb-uuid-to-int) reserve a directory region by their header's own depth — 4 × 2^28 bytes (1 GiB) at the pre-11.2.1 default, against 4 × 2^12 (16 KiB) today — and that region is inside the watermark. So a store created by an older build collapses by two or more orders of magnitude on its first clean close and then keeps roughly a gigabyte of apparent size per id-mapper arena until its next compaction, which rebuilds the base at this build's geometry and retires it. The artifact-sparseness health row names this case explicitly rather than reporting it as a trim that did not run.

The trim refuses, by name, rather than shortening a file it cannot prove is safe to shorten: an arena a time-travel view still pins keeps its reservation until the pin drops, and so does one an off-lock maintenance build is walking. A refusal is a WARN line naming the arena and the reason, never a silent skip.

The health reports carry the result. artifact-sparseness reports each artifact family's apparent bytes, allocated bytes and the ratio between them, measured once at the projection's open and once at its clean close (never inside healthReport(), which is synchronous and called on every read gate). On a store this build has cleanly closed, a ratio above 10x means the trim did not run — a defect, graded repair because the store serves every row correctly and only its declared size is wrong. On a store that has not been cleanly closed, the same ratio is the ordinary shape of a live writer's reservations and is reported as INFO.

How big that gap should be. Each arena reserves for the store's own population and doubles on demand, never for the id space: a freshly initialized empty brain is a few tens of megabytes apparent, and a populated brain's apparent size stays within a small factor of its allocated size. If you see an apparent size in the hundreds of gigabytes on a small brain, that store was created by a build before 11.2.1, which reserved 32 GiB per id-mapper arena and up to half the host's free disk for the verb-endpoint arena. Such a store keeps serving normally and is as fast as it ever was — only its apparent size is wrong — and the engine now names it: validateInvariants() / the provider health report carries an arena-reservation invariant that fails at repair grade (never rebuild, so the store stays serving). A clean close trims each file to its watermark.

Any tool that copies, backs up, or migrates a brain directory must preserve holes, or it will silently inflate the copy to its full apparent size — slow, and it can fill the target disk. Use rsync -aS (-S/--sparse; never combine with --inplace — they conflict), cp --sparse=always -a, or tar --sparse when creating an archive (extraction restores holes automatically, no flag needed). Plain scp cannot preserve holes at all — use rsync -aS instead.

For a local copy, prefer cp --sparse=always: it locates holes with a direct seek, so cost tracks allocated size. rsync -S locates holes by scanning for runs of zero bytes, so its cost tracks apparent size instead — usually fine, but on a freshly-created or lightly-populated brain (allocated size far below apparent) that scan can dominate the copy. Reach for rsync -aS when the destination is a different host — there's no local-cp equivalent across hosts.