Long list fields

A metadata field can hold a list — tags, aliases, memberIds, years. Each value in the list is indexed on its own, so a filter on any one of them finds the record:

await brain.add({ data: 'Release notes', type: 'document', metadata: { tags: ['release', 'engine', 'q4'] } })
await brain.find({ where: { tags: 'engine' } })   // finds itTYPESCRIPT

What indexes

• 256 is the write door's own bound. A write whose list on an indexed field holds more than 256 values is refused with MetadataArrayTooLargeError, naming the field. A field the store declares unindexed (brain.declareUnindexed(['seen'])) carries a list of any length and is never filterable — see Field Addressing.

• A list of more than ten decimals is the shape of a vector, and a vector in the metadata index is hundreds of near-unique values, one entry each, for every record that carries one. Those lists are stored whole and returned by get(), but a filter will not find a record by them. The first time a process sees such a list on a field it logs one line naming the field. To search by meaning, pass the embedding as the record's vector; to filter on the values, store them as whole numbers or words. In 13.1.3 there is no way to declare a decimal-list field indexable; that declaration arrives in 13.2.

• Whole numbers are decided by value. 2.0 and 2 are the same whole number; the list [1.0, 2.0, …] indexes.

• Ten values or fewer index exactly as before, decimals included, so no existing entry in any store changes.

What a long list costs

Every value in an indexed list is one index entry, so a record whose long lists change pays for the values it changed — not for the size of the brain. Measured on one build at production shape (12,000 records with 90 fields each, 200 records updated and saved per cycle, on a memory-backed disk):

Loading the 12,000 records took 38 s without the lists and 92 s with them. A real disk adds the cost of each durable write on top of both rows.

Records written before 13.1.3

Before 13.1.3, a list with more than ten values was skipped by the index entirely, whatever it held, and nothing said so. A record whose tags held eleven words was stored, and a filter on any of those words did not find it.

13.1.3 indexes those lists on every new write. A record written before 13.1.3 is indexed under the new rule when either of these happens:

1. The record is updated. Any update() of the record re-indexes its metadata, and the long list is indexed with it. Nothing else is needed for records your application rewrites anyway.

2. The metadata index is rebuilt. For records that are never updated, a rebuild of the metadata index walks every record and indexes each one under the current rule:

   await brain.repairIndex({ rebuild: ['metadata'], only: ['provider'] })TYPESCRIPT

   only: ['provider'] runs the rebuild alone, without the other repair walks.

Do you need the rebuild? Only if records written before 13.1.3 carry a list of 11 to 256 words, ids, booleans or whole numbers in a field you filter on, and those records are not going to be updated. A store whose lists never held more than ten values, or held only decimals, gains nothing from it.

What the rebuild costs. The rebuild walks every record in the brain, so its time grows with the brain: seconds on a small one, much longer on a large one. It builds the new index beside the live one and swaps it in when it is done, so filters keep being answered from the current index while it runs — but it reads every record and takes disk and CPU for the whole walk, and a busy brain shares them with it. Measure it on a copy of the brain before running it on the live one, and run it when the brain is quiet. A close of the brain stops the rebuild cleanly (RepairStoppedError); the unfinished index is cleared at the next open and the rebuild can be run again.