Field Addressing

How a field name in where, orderBy, or groupBy resolves — one rule, no exceptions, identical on the pure-JS engine and the native accelerator.

The one rule

A bare field name always means user metadata. Engine scalars are reached explicitly as system.<field>.

Your metadata is yours. If you store { level: 9 }, { createdAt: 'Q3' }, or { type: 'invoice' }, then where: { level: 9 }, orderBy: 'createdAt', and groupBy: ['type'] operate on your values — an engine-internal field can never shadow a user field, no matter what it is named. This closed a real production bug class: a user field named level used to be silently dropped by the engine's internal never-index list, so filters returned [] and ordering fell back to insertion order with no error.

The ten system fields

Exactly ten engine scalars exist, and only the explicit system. prefix reaches them:

system.<anything else> does not exist and refuses (see below).

Invisible plumbing

Five names are engine plumbing — never resolvable, never indexable, never orderable, not as bare names and not through system.:

• vector

• connections

• level

• data

• _rev

Plumbing invisibility applies to the engine's fields only. A user metadata field that happens to share one of these names is ordinary user data and is reachable as the bare name, per the one rule — your { level: 9 } always works.

The index key format (frozen, cross-engine)

The law is also the physical key layout, identical on both engines and in every index surface (posting lists, column store):

• User metadata keeps today's bare flattened keys — { level: 9 } is keyed level, { a: { b: 1 } } is keyed a.b.

• The ten system scalars are keyed under the literal string system.<field> — the engine type lives at system.type (the historic internal noun key is retired), engine timestamps at system.createdAt / system.updatedAt, and so on.

• Origin — never name — decides the key. An engine scalar can only land on a system.* key; a user metadata field can only land on a bare key. The two never merge, which is what makes the one rule enforceable at the storage level rather than by resolution priority.

Existing brains migrate by a full derived-index rebuild at the epoch bump that ships with the namespace pair; the rebuild is automatic at first open.

Reserved namespace: user fields may not be named system.*

Because system. is the engine's literal key prefix, a user metadata field whose name would flatten into it is refused at write time with a typed error — a field literally named system.foo, or an object field named system (whose children would flatten to system.* keys). This is the one name-based restriction the law imposes on user metadata, and it exists so user data can never forge an engine address. A scalar user field named exactly system (no dot) is ordinary user data and indexes fine.

Refusal semantics

Anything unresolvable is a typed refusal naming both candidates — never an empty result, never a silent fallback to insertion order:

Cannot resolve field 'priority' — no user metadata field 'metadata.priority'
exists in this brain, and 'system.priority' is not one of the ten system
fields. Did you mean one of: metadata.priority (write it first) or a
system.* field?

The guarantee that matters: a query that cannot be answered throws. If you get [], the query resolved and honestly matched nothing. If you get results, they are ordered by the field you named. There is no third state.

Accepted-and-ignored options are banned under the same law: an option the engine cannot honor (cursor, includeRelations, writeOnly, …) is either implemented or refused — never swallowed.

Ordering contract

• Entities missing the orderBy field (or holding null) sort last in both directions — they are never dropped from the result.

• Ties break by id ascending, in both directions — ordering is fully deterministic.

Migration notes

Before this law, bare createdAt, updatedAt, type, and friends resolved to the engine fields. Under the law they resolve to user metadata.

• Call sites that meant the engine scalar move to system.*: orderBy: 'createdAt' → orderBy: 'system.createdAt', where: { type: 'thing' } → where: { 'system.type': 'thing' }, and so on for all ten.

• A missed site does not silently change meaning and keep running: if no user metadata field of that name exists, the bare name is unresolvable and refuses loudly, and the refusal message names the system.<field> correction. The only silent case is a brain that genuinely has a user field of that name — in which case the bare name now means exactly what the rule says it means.

Machine-readable appendix

Downstream schema layers import this by copy; it is the same vocabulary the executable spec asserts.

{
  "systemFields": [
    "id",
    "type",
    "subtype",
    "createdAt",
    "updatedAt",
    "confidence",
    "weight",
    "visibility",
    "service",
    "createdBy"
  ],
  "plumbing": ["vector", "connections", "level", "data", "_rev"],
  "rule": "bare=metadata"
}JSON

The executable spec

The law is enforced by src/native/fieldAddressingConformance.test.ts — a self-arming conformance suite that skips loudly until the namespace-capable engine pair is installed, then runs every case above (the ten-name collision bag, all five plumbing refusals on both where and orderBy, both-candidate refusal messages, nulls-last in both directions, id-ascending tie-breaks). The same suite runs in brainy's repository — it is the shared drift-proof that keeps the pure-JS engine and the native accelerator on one law.

Your field names are yours — guaranteed

Any bare name you choose is a fully ordinary field — including names the engine also uses internally. A field named data, level, confidence, or id in your metadata indexes, filters, sorts, and aggregates exactly like any other field, and never collides with the engine's own values:

await brain.add({ type: NounType.Person, metadata: { level: 9, data: 'raw', id: 'A-42' } })
await brain.find({ where: { level: 9 } })          // your field — always
await brain.find({ orderBy: 'level' })              // sorts by YOUR values
await brain.find({ orderBy: 'system.createdAt' })   // the engine's timestamp — explicitTS

This is not a documentation promise: the reopen-collider conformance case writes user metadata for every colliding name, flushes, cold-reopens the brain from disk, and asserts your values come back through find() — and both engines must pass it in every release gate, forever.

Content vs fields — what gets embedded, what gets indexed

Each entity has one embedding, computed from its content (the record's payload — the text you pass to add(), a file's body). That vector is what semantic similarity search runs against.

Metadata fields are not individually vector-indexed. They get two other kinds of indexing, matched to what fields are for: structural (exact match, ranges, sorting, aggregation) and lexical (string fields' words feed the hybrid text index, so keyword search sees them).

The rule of thumb: meaning goes in the content; exact values go in fields. If you want text found semantically, it belongs in the content; if you want a value filtered or sorted, it is a metadata field — and string fields get keyword findability for free.

On disk, each entity is two files: the record (engine scalars and the content slot at the top level, your metadata bag nested verbatim beside them) and the vector derived from that content. Indexes are projections of these canonical files and can always be rebuilt from them.

Fields you store but do not query — the unindexed declaration

Every metadata value is indexed, one posting per array element. That is why an array longer than 256 elements is refused at the write door by name (MetadataArrayTooLargeError): an unbounded array is an unbounded write.

But real data has lists nobody filters element by element — a seen set, a quote history, an alias list, a tag trail. They are read back whole and never queried for membership, and they are exactly the shape the bound refuses. Until this door existed the only cures were to truncate real data or move it into data, where it stops being a field.

A field declared unindexed keeps every value it is given, of any length, in the record — and writes no postings at all.

const brain = new Brainy({
  storage: { path: '/data/brain' },
  metadata: { unindexedFields: ['seen', 'quotes', 'aliases'] }
})
await brain.init()

// A 5,000-element array is accepted, stored whole, and comes back whole.
await brain.add({
  type: NounType.Person,
  data: 'Ada Lovelace',
  metadata: { name: 'ada', seen: fiveThousandIds }
})

const row = await brain.get(id)
row.metadata.seen.length        // 5000 — get(), find() hydration, export, the
                                //        change feed all serve it unchangedTS

What it costs: the field is not searchable, and says so

Because there are no postings, there is nothing for a filter, a sort, an aggregation or a value enumeration to answer from — so every one of them refuses by name:

await brain.find({ where: { name: 'ada' } })   // served — an ordinary field
await brain.find({ where: { seen: 'x' } })     // throws UnindexedFieldError
await brain.find({ where: { name: 'ada' }, orderBy: 'seen' })  // throws
brain.defineAggregate({ groupBy: ['seen'], ... })              // throwsTS

This is the same law as the rest of this page: a query that cannot be answered throws. Returning [] would be indistinguishable from "nothing matched", which is precisely the silent-drop defect the 256-element bound replaced. The error carries field and door, so a handler can route on it without parsing a message, and it names the cures: read the value off rows a query on an indexed field returns, keep the queried values in a separate indexed field, or make the field indexed again (below).

explain() says the same thing before you run anything — a declared field reports path: 'none' with a note that names the declaration, rather than the "no index entries" note that would send you hunting for an unflushed writer.

The declaration belongs to the store

It persists at _system/field-index-policy.json, beside the other open-time handshake artifacts, so every writer and reader of one brain agrees about which fields carry postings — including a process that never passed the option. metadata.unindexedFields at open is therefore a declaration, not a session setting: names it adds are written to the record and survive close and reopen.

It is additive. Dropping a name from the config does not re-index the field, because re-indexing needs a rebuild:

await brain.declareUnindexed(['quotes'])                     // durable, idempotent
brain.unindexedFields()                                      // ['aliases', 'quotes', 'seen']
await brain.declareIndexed(['quotes'])                       // throws — no postings exist
await brain.declareIndexed(['quotes'], { rebuild: true })    // rebuilds, then servesTS

The loudness is deliberate. Flipping the flag alone would leave the field claiming to be searchable while every query on it answered an empty page.

Declaring a field that already has postings

Existing postings are not erased synchronously — that would be an O(store) walk hidden inside a one-line call. Queries refuse from the first moment, and the postings retire on the two paths the engine already uses for convergence:

• per entity, as each row takes its next update;

• wholesale, on repairIndex({ rebuild: ['metadata'] }).

Until then the health report names the field and counts the remainder down — unindexed (N legacy postings remain) — graded repair, never rebuild, so the store keeps serving everything else while it converges. getStats() carries the same two facts as unindexedFields and unindexedLegacyPostings.

What may not be declared

The ten system.* scalars and the derived text field __words__ are the engine's own reading of a row — type counts, visibility tiers, ordering and search() all resolve through them. Declaring one refuses with FieldIndexPolicyError at the declaration door, before anything is written.

A declaration covers the field and its dotted descendants: declaring history also covers history.quotes. The boundary is the dot — historyOfArt is a different field and keeps its postings.

MAX_INDEXED_ARRAY_LENGTH keeps applying to indexed fields only. Nothing about the bound changed; a declared field simply has no postings for it to bound. Its refusal now names the declaration as its first cure, so a caller who hits the bound on a list they never query learns about this door at the moment it would have helped.

Moving a brain that uses it

export() carries entities and relationships, not the store's _system/ artifacts — so a brain imported into a fresh store carries no declaration, and the first row with a long list is refused by the array bound (by name, with the cure). Declare the fields on the target before importing:

const target = new Brainy({
  storage: { path: '/data/copy' },
  metadata: { unindexedFields: source.unindexedFields() }
})
await target.init()
await target.import(dump)TS

A whole-directory copy (db.persist(), a filesystem snapshot, a rehearsal fixture) carries _system/field-index-policy.json with everything else and needs nothing.