Brainy Brainy
Docs Brainy

Pictures described in the engine

In this section

A picture is invisible to search. Its bytes say nothing a text query can match, and the only text a file write embeds for it is its name and size. So brainy describes pictures the way it embeds text: through a describe provider that sits beside the embed provider, reached through one door on a GPU box with one token. Every brain gets it with no code change.

When a picture lands in a brain, the engine sends it to the describe service once, in the background, and stores on the picture's record:

  • a description of up to 2,400 characters (short paragraphs joined by a blank line),

  • a short tag list — the things in it, the roles they play, the places it shows, and any words visible in it,

  • what skin the picture shows, when the describe service says so,

  • how the picture reads — its mood, colours, light, setting, style, how many people are in it, how good the photograph is, and a short alt text — and a second, closer look at how revealing it is, when the service says so,

  • the description's embedding, as the record's own vector.

After that the picture turns up in an ordinary find({ query }), its tags can be filtered like any other field, and the change feed announces them. Nothing blocks the save: the write returns before the describe service has seen the picture.

What counts as a picture

A VFS file (brain.vfs.writeFile(path, bytes)) whose MIME type is image/* and whose bytes live in the blob store. The MIME type comes from the file name's extension. A working-state file (retention: 'latest-only') is never described: it never embeds at all, by design.

What the record carries

metadata.vision = {
  description: 'A photograph of a lighthouse standing alone under a grey sky.', // up to 2,400 characters; paragraphs joined by a blank line
  tags: {
    things: ['lighthouse'],   // lower-case, at most 12 per list
    roles: ['landmark'],
    places: ['coast'],
    text: [],                 // words visible in the picture
  },
  exposed: [{ part: 'breast', score: 0.97 }],  // only when the service answered it
  exposedParts: ['breast'],                    // the same parts, as a plain list
  reading: {                                   // only when the service answered it; every member optional
    mood: { words: ['calm', 'warm'], confidence: 0.8 },
    palette: { colours: [{ hex: '#c8a165', name: 'sand' }], confidence: 0.7 },
    light: { value: 'golden-hour', confidence: 0.9 },
    setting: { text: 'a harbour wall at dusk', confidence: 0.6 },
    style: { value: 'candid', confidence: 0.5 },
    people: { count: 2, confidence: 0.95 },
    quality: { sharpness: 0.8, noise: 0.1, word: 'sharp', confidence: 0.85 },
    alt: { text: 'Two people on a harbour wall at golden hour.', confidence: 0.7 },
  },
  readingTerms: ['calm', 'warm', 'sand', 'golden-hour', 'a harbour wall at dusk', 'candid'],  // the engine's own plain list
  secondLook: { verdict: 'artistic-nude', confidence: 0.64, crops: 9, at: '2026-10-14T16:20:00.000Z' },
  model: 'the-model-name',
  version: '1.0',
  describedAt: '2026-10-14T16:20:00.000Z',
  sourceHash: '<the blob hash of the bytes this describes>',
  confidence: 0.92,
  served: ['description', 'tags', 'text', 'exposed', 'reading', 'secondLook'],
}TS

exposed and exposedParts say what skin the picture shows, so a product can keep nude pictures off public pages, prints and social posts. An empty list means the service looked and found none; when the two fields are absent, the service did not say, and the picture needs a look. served lists what the engine asked the service for when it made this description. See Exposed skin and The richer reading and the second look.

It is written through the engine's ordinary update path, by the engine's own housekeeping actor:

  • The change feed carries it as an update event whose metadata holds the tags — a subscriber learns a picture's tags without reading the record.

  • The metadata index serves every part of it as a field: find({ where: { 'vision.tags.things': 'lighthouse' } }) returns the pictures tagged so (a tag list indexes one entry per tag).

  • The vector. The record's embedded text becomes the description followed by every tag, joined, and is embedded through the same background embedding path every file write uses. find({ query: 'lighthouse on a cliff' }) finds the picture. A description may run to 2,400 characters and is stored whole on the record, but the embedder reads only the first 256 tokens of the text (about 1,000 characters of English), so a long description embeds up to that limit; the rest, and the tags after it, are searchable by their fields but do not enter the vector.

sourceHash is the blob hash of the bytes the description was made from. When a file is written again with new bytes, its record still carries the old vision until the job describes the new bytes; a reader can tell by comparing vision.sourceHash with metadata.storage.hash. The new description replaces the old one whole.

A file written again with the same bytes is not sent again. The write puts the record's embedded text back to the file's name, so the job puts the description back as the embedded text, without a call to the service.

The door

The engine speaks to the describe service through one fixed contract.

POST /v1/describe — Authorization: Bearer <token>

{
  "requestId": "…",
  "items": [ { "id": "…", "mime": "image/png", "bytes": "<base64>", "locale": "en" } ],
  "want": ["description", "tags", "text", "exposed", "reading", "secondLook"]
}JSON

want always holds description, tags and text; exposed, reading and secondLook are each added only when the service's health lists it in serves, so a service is never asked for something it has not declared. At most 8 items per call and 20 MB in total. The body also carries maxDescriptionChars (2400), the longest description this client accepts; the service treats an absent field as 600, so an older client never receives a long answer and the service can restart with long descriptions without a fleet flag day. An item may carry url instead of bytes; the engine always sends bytes.

It answers 200:

{
  "model": { "name": "…", "version": "…" },
  "results": [
    { "id": "…", "description": "≤ 2,400 characters; short paragraphs joined by a blank line",
      "tags": { "things": [], "roles": [], "places": [], "text": [] },
      "confidence": 0.0, "ms": 0,
      "exposed": [ { "part": "…", "score": 0.0 } ],
      "reading": { "mood": { "words": [], "confidence": 0.0 }, "…": "…" },
      "secondLook": { "verdict": "…", "confidence": 0.0, "crops": 9, "at": "…" } }
  ],
  "refused": [ { "id": "…", "code": "…", "message": "…" } ]
}JSON

Tags are lower-case, at most 12 per list; text is the words visible in the picture. exposed is optional — a service that does not answer it is still in contract; when it is there it is a list of at most 32 entries, each a lower-case part and a score from 0 to 1. reading and secondLook are optional the same way and are bounded in The richer reading and the second look. One bad item never fails the batch — it comes back in refused.

Refusals: 400 unsupported-mime | too-large | bad-image (per item, in refused); 413 batch too large; 429 busy (with Retry-After, and an expectedWaitMs in the body from services that know it); 503 model unavailable; 401 bad token.

GET /v1/health answers { model, version, device, queue, msPerImageP50, expectedWaitMs, concurrency, batch, serves, supersedes }. queue counts pictures and msPerImageP50 is one picture's model wall under load. batch is optional: the most pictures the service wants in one call, a whole number of at least 1; absent is the contract's 8, and anything else is refused by name as a protocol error. concurrency is optional: how many pictures the model works at once, a number of at least 1; absent is read as 1, never 0. expectedWaitMs is optional: the service's own estimate of how long a call sent now waits before it is worked, already divided by its concurrency; when present the engine uses it in place of queue × msPerImageP50 / concurrency, and an absent one is never read as zero. serves is optional: the wants the service answers, e.g. ["description", "tags", "text", "exposed", "reading", "secondLook"]. A service that leaves it out is taken to answer the first three. supersedes is optional: a list of model names (name, any version) or name@version entries that the service's current model replaces; absent means none is declared, and anything that is not a list of non-empty strings is refused by name as a protocol error.

The job

The work runs on the engine's background job runtime as the job media-describe — a maintenance job, so it never holds a door, it stands down at close, and the operator's maintenance hold holds it.

  1. On landing. After a picture's write returns, the engine drops a small queue entry in the store (_brainy_describe/queue/<id>.json) and asks for a run. The entry is durable: a process stopped before the run finds it again at its next open. A picture written twice before the run has one entry.

  2. The drain. A run reads up to 32 queued pictures, plans calls of at most 8 pictures and 20 MB, and makes at most 4 calls before handing the runtime back and asking to run again — so a large backlog drains in short runs other jobs can interleave with. A queue entry is removed only after its record is written.

  3. One lane per service, per process, as wide as the service. Every brain in a process speaks to the service through one lane with as many slots as the service's health declares it works at once (concurrency; absent is 1, and it is read again from every health, so a service that changes it changes the lane on the next run). At most that many calls are in flight to it at a time, however many brains are open — the service is kept full and never sent one more — and a brain whose turn is behind waits for a slot, first come first served, never on a timeout. The pace belongs to the lane: after a call carrying n pictures, the next call — from any brain — is not sent until n × msPerImageP50 / concurrency milliseconds after it, so the process never sends faster than the service says it describes — and a service that works four pictures at once is fed four times as fast as one that works one. A call carries at most the batch the service's health declares (read again from every health; a plan cut before the read is split, never sent larger). The health's median says nothing about the pictures in hand, so when a call is abandoned at its deadline the lane remembers it for that service, in memory, for this process: the next call carries half as many pictures and its deadline is sized at the per-picture wall the abandoned call showed, until a call is answered. The service keeps reading the abandoned call (the client cannot cancel it), so the smaller call queues behind it.

  4. The queue, read before every call. Holding the lane, the job reads GET /v1/health. The service's backlog is its own expectedWaitMs when the health carries one, otherwise queue × msPerImageP50 / concurrency. The call's deadline is sized to it: (backlog + n × msPerImageP50) × 1.25 + 30 s (the call's own n pictures at the per-picture wall, undivided), at least 120 s and at most 10 minutes (the quarter more is there because the median runs short of the wall summed over a long queue). When even 10 minutes would not cover it, nothing is sent: the run ends without failing, prints one line ("the describe door … has 63 queued × 7683 ms ≈ 484 s of backlog … this pass waits 484 s"), once per process however many brains meet it, and the job runs again after the backlog — at least 8 s, at most 10 minutes. A picture landing meanwhile runs the job at once, which reads the health again and sends as soon as the backlog has cleared.

  5. A busy service. A 429 is waited out for the wait it names (the longer of Retry-After and an expectedWaitMs in its body) and the call tried again, at most 3 tries — when that wait is 30 s or less. Past the tries, or at once when the service names a longer wait, the run ends without failing, prints one line, keeps every brain in the process off the service until the wait has passed, and the job runs again after it (at least 8 s, at most 10 minutes). Nothing is counted against the job, so a busy service never latches it.

  6. The receipt. Every call prints one line: how many pictures were described and refused, the wall it took, the service's median per picture, and the model.

The daily pass

The job is a cadence with a 24-hour interval: once a day while the brain is open it walks every picture in the brain and describes the ones that were never described, or whose bytes changed since (vision.sourceHash differs from the current blob hash). An open on a brain that holds pictures and whose last complete walk is more than a day old runs one, so a brain that was closed for a week catches up when it opens. Pictures that landed while no service was configured are found this way. The walk also records which wants the engine that made it knew (description, tags, text, exposed, reading, secondLook today) and which rules it applied. An open whose last walk was made by an engine that knew other wants or applied other rules runs one at once, however recent that walk was — so pictures described before the engine learned reading are reached the first time the brain opens on the new engine, not a day later.

What a daily walk costs when the service never answers a want it lists. Every picture missing that answer is sent once per walk: one call per 8 pictures (fewer for large ones) and one health read per call, each day. A brain of 1,661 pictures whose service lists reading and never answers it costs 208 calls a day; at the MEASURED hosted median of about 2.2 s per picture in a batch, about an hour of the service's time a day. The pass prints one line with the count each time, so the cost is visible until the service is fixed or stops listing the want.

The walk is durable: _brainy_describe/sweep.json records the last picture a walk finished, and a process stopped mid-walk resumes after it.

A better reader

When a better picture-reading model arrives, the pictures it replaces are read again by it. The service says which models its current one replaces in the supersedes list of its health. A picture whose description was made by a model that list names is owed one new description from the next daily walk (rule 6):

  • An entry name matches a record whose vision.model is name, at any version. An entry name@version matches that version only.

  • When the new description lands, the record's vision.model and vision.version become the service's current model, so it is never owed again for that reason — unless a later model supersedes that one too.

  • Only the daily walk applies the rule, never the queue of newly landed pictures. Within a pass the re-reads go oldest description first, inside the walk's usual limits (512 pictures examined and four calls per pass); no new limit is added.

  • A change of model the service does not declare re-reads nothing. The walk remembers the last model it saw, and when it sees a different one that the new model's supersedes does not cover, it prints one line, once per change: the service's model changed from the old to the new and declares no supersedes, nothing is re-read, and the old model must be declared in supersedes to re-read it.

  • Every brain's first walk after upgrading to an engine with this rule runs at once, however recent the last walk was, so existing pictures are examined under it.

Confining the walk to the operator's hours. A re-read of a whole library is heavy GPU work. BRAINY_DESCRIBE_WALK_WINDOW=HH:MM-HH:MM, in the host process's local time (its TZ), allows the daily walk to run only inside that window; the start is inclusive, the end exclusive. A window that crosses midnight, such as 22:00-06:00, is valid. Outside the window the walk is simply not due: it prints one line a day (the daily walk waits for its window), and the job is armed to run when the window opens. Pictures that land are never windowed — they are described promptly. A malformed value is refused by name when the describe job is built. Unset means any time.

Exposed skin

The describe service can also say what skin a picture shows. The engine asks for it ("exposed" in want) on every call to a service whose health lists exposed in serves, and writes what comes back onto the record as vision.exposed (each part with its score, in the service's order) and vision.exposedParts (the part names, each once). The plain list is what a filter reads:

const exposed = await brain.find({ where: { 'vision.exposedParts': 'breast' } })TS
  • The parts the engine knows are breast, genitals, buttocks and anus (exported as DESCRIBE_EXPOSED_PARTS). The service may name others; a part the engine does not know is kept as the service said it, never dropped.

  • Absent means the service did not say. A service that does not answer exposed writes neither field. A product should treat such a picture as one that needs a look, not as a clean one.

  • One re-pass when the service learns it. A picture described before its service answered exposed is described again once, by the daily pass, after the service's health starts listing exposed in serves. The new record's served names exposed, so it is never sent again for that reason. A service that does not list it is sent nothing extra, and is never asked for exposed.

  • A service that was asked and did not say is named, and asked again the next day. When a service that lists exposed answers a picture without it, the picture keeps its description and carries no exposed (so it reads as needing a look). Each pass that saw this prints one line with the count — the service must answer exposed on every result, an empty list when it finds none. The picture stays owed its answer: every daily walk that starts after its description was made sends it once more, while the service still lists exposed, until the service answers. Only the daily walk asks again — never the moment a picture lands — and a walk never sends the same picture twice.

  • Never in search. The words for exposed parts never enter the picture's embedded text: its vector is still made from the description and its tags only.

  • Out of contract is a refusal. An exposed that is not a list, holds more than 32 entries, or has a part that is not lower-case or a score outside 0..1 makes that picture's answer answer-out-of-contract, retried by the next daily pass.

The richer reading and the second look

Two more things the describe service can say about a picture. The engine asks for each ("reading", "secondLook" in want) only on a call to a service whose health lists it in serves, and writes what comes back onto the record. They follow the same rules as exposed skin: absent means the service did not say, one re-pass when a service starts declaring them, a reading the service was asked for and left off is named once per pass and asked for again by each daily walk until the service answers it, and neither ever enters the picture's embedded text. The second look is the one difference: the service takes it only when its first read of the picture is uncertain, so a result without secondLook is the ordinary case. The picture's record is stored in full without it, nothing is printed, and it is not sent again for it.

vision.reading is how the picture reads, in plain words. Every member is optional — a service may answer only some — and every member carries its own confidence, a number from 0 to 1:

member

what it holds

bound

mood

words: plain lower-case words

at most 3, each at most 24 characters

palette

colours: { hex, name }

at most 5; hex is #rrggbb in lower-case, name at most 24 characters

light

value

one of golden-hour, daylight, studio, night, mixed

setting

text: one short phrase

at most 60 characters

style

value

one of candid, editorial, glamour, product, landscape, portrait, other

people

count: how many people are in it

a whole number, 0 or more

quality

sharpness, noise, and a word

the two numbers 0 to 1; word one of sharp, soft, noisy, clean

alt

text: a short alt text

at most 120 characters

A member the contract does not name is ignored, not stored.

vision.secondLook is the service's second, closer opinion on how revealing the picture is: { verdict, confidence, crops, at }. verdict is one of safe, suggestive, artistic-nude, exposed, explicit; confidence is 0 to 1; crops is the whole number of crops of the picture the service looked at; at is the ISO 8601 instant it looked.

vision.readingTerms is the engine's own plain list, made from the reading and present exactly when reading is: the mood words, the palette colour names, the light, the setting and the style, lower-cased, each once, at most 32. It is what a filter reads for a colour name:

const warm = await brain.find({ where: { 'vision.readingTerms': 'golden-hour' } })
const sandy = await brain.find({ where: { 'vision.readingTerms': 'sand' } })TS

The reading's own members are also fields the metadata index serves: vision.reading.light.value, vision.reading.style.value, vision.reading.people.count and vision.reading.mood.words (a list of words, one entry each). A list of objects' sub-field is not in the index, so vision.reading.palette.colours.name finds nothing; use vision.readingTerms for colour names.

  • Out of contract is a refusal of the whole picture's answer. A present member outside its bound — six palette colours, an alt text of 121 characters, a confidence of 1.2, a light that is not on the list, a secondLook without a whole number of crops — makes that picture's answer answer-out-of-contract, naming the fault. Nothing of that answer is stored partially, and nothing is trimmed into shape. A picture already described well keeps its description, tags and vector, with the refusal beside them; the next daily pass tries again.

  • Absent means the service did not say. A service that does not answer reading writes neither reading nor readingTerms; one that does not answer secondLook writes no secondLook. A service that answers only some of the reading's members writes only those.

  • New bytes replace all of it. A file written again with new bytes is described again and its reading, readingTerms and secondLook are replaced with the new answer, or dropped if the service no longer answers them.

Refusals are rows, never silence

A picture the service refuses gets a refused row instead of a description:

metadata.vision = {
  sourceHash: '<the blob hash of the refused bytes>',
  refused: { code: 'bad-image', message: 'the bytes are not an image', at: '2026-10-14T…' },
}TS
  • unsupported-mime, too-large, bad-image cannot change while the bytes do not, so the same bytes are not sent again. New bytes are tried again.

  • A picture too large to fit in one call even alone is not sent at all; it gets too-large from the engine with that reason in its message.

  • answer-out-of-contract — the service answered this picture outside its own contract (a description over 2,400 characters, a tag list over 12 or not lower-case, a confidence outside 0..1, an exposed list, a reading or a secondLook out of shape). The answer is recorded as a refusal, never trimmed into shape. Retried by the next daily pass.

  • unanswered — the service answered neither a result nor a refusal for it. Retried by the next daily pass.

  • detector-error — the service's skin detector failed on this one picture. Retried by the next daily pass.

A picture already described well and sent again only to learn its exposed skin, its reading or its second look keeps its description, tags and vector when the service refuses it: the refusal is recorded beside them in vision.refused, never in their place.

When the whole call is refused, nothing is written and the pictures stay queued. Except for a busy service (and a backlog, which is never sent into), the job's run fails with DescribeProviderUnavailableError, whose reason says which:

reason

what happens

unreachable

the runtime backs off and retries — the connection was refused, the name did not resolve, or no answer came on a call whose health was not read ("could not be reached")

queued

the runtime backs off and retries — the service answered its health just before, and the call's deadline, sized to that health's queue, passed ("timed out waiting on a queued door (queue N, P ms per picture, deadline D s)")

busy

not a failure: waited out and retried; past 3 tries, or at once for a wait over 30 s, the run ends and the job runs again after the service's wait

model-unavailable

a failure, not a full service: the runtime backs off and retries — including the service's 503 detector-unavailable, after which its health stops listing exposed (or reading, or secondLook) and the engine stops asking for it

bad-token

stops; set BRAINY_DESCRIBE_TOKEN — the next picture to land, or the next open, runs again

batch-too-large

stops; the engine's batching and the service disagree

protocol

stops; the service answered outside its contract

not-configured / opted-out

nothing is sent — see below

Configure it

Zero-config is one name: run the describe service where brainy-describe resolves (a hosts line or a DNS record) on port 8201, and set BRAINY_DESCRIBE_TOKEN to its token. This mirrors the embedding service (brainy-embed, port 8200, BRAINY_EMBED_TOKEN).

variable

meaning

BRAINY_DESCRIBE_ENDPOINT

host:port (or http://host:port) — wins over the name. The value none turns describing off on purpose, and nothing is resolved

BRAINY_DESCRIBE_TOKEN

the bearer token the service requires

BRAINY_DESCRIBE_WALK_WINDOW

optional HH:MM-HH:MM in the host's local time: the hours the daily walk may run (see "A better reader"); landing pictures are never windowed

Discovery runs once per process and is shared by every brain in it. A name that did not resolve is asked again after ten minutes, so a service that comes up later is found without a restart.

With no service configured, nothing is described and no record is touched — the engine never writes a description a model did not make. The first picture to land prints one line saying it was not described and how to configure the service; later pictures print nothing more.

Cost

  • A picture landing: one small queue object written after the save has returned. A write that is not a picture pays one check of its metadata and nothing else.

  • A brain with no pictures pays nothing: no job runtime, no queue, no cadence. When a describe service is configured, the open lists the brain's media files once, from the metadata index, to decide whether to start the job.

  • Describing: one service call per 8 pictures (fewer for large ones), one health read before each, and one record update per picture — the same update any caller's update() costs, plus the background embedding of the description.

  • Many brains, one service: at most concurrency calls in flight per service per process, so the service sees the process's pace, not each brain's. PROJECTED from the pacing law (not measured), taking each call to be worked in one of the service's slots at its declared 7.7 s per picture: 16 brains in one process and a service working 4 at once (concurrency: 4) keep 4 calls of 8 in flight, sent about 15 s apart, each answered about 61 s after it was sent — about 31 pictures a minute (the service's own ceiling at those figures, 4 × 60 s ÷ 7.7 s), a 395-picture walk in about 13 minutes — where one call at a time reached about 8 a minute and each brain sending alone would put 16 calls on the service at once. A service reporting 63 queued at that rate has 63 × 7.7 s ÷ 4 ≈ 2 minutes of backlog, which one call's deadline covers (about 258 s), so the call is sent and its first picture is read about 2 minutes in. The same queue on a service that works one picture at a time (≈ 8 minutes of backlog) is sent nothing until it clears: one health read and one line, the brains re-armed about 8 minutes out, instead of sixteen calls that each time out and are sent again.

  • The daily pass: one metadata-index read listing the brain's media files, then one record read per picture, in runs of at most 512. Only owed pictures reach the service.

  • Wall time is the service's, not the engine's. MEASURED on the hosted describe service (2026-10-14): 1.5–4.6 s per picture sent alone, about 2.2 s per picture in a batch, and a declared msPerImageP50 of 3,538 ms. Because the lane paces to that median, a batch of 8 owns the service for about 28 s. PROJECTED from that median: a first daily pass over 1,000 pictures takes about an hour. Each pass that moves the walk prints where it stands ("daily pass at N of M pictures"), and a stopped process resumes after the last picture it finished.

  • The index: each description is a near-unique string, indexed like any other field. A store that never filters on it can declare vision.description unindexed and keep the stored value.

In code

import { DescribeProviderUnavailableError, resolveDescribeProvider } from '@soulcraft/brainy'

const { provider } = await resolveDescribeProvider()
if (provider.configured) {
  const health = await provider.health()
  console.log(`${health.model} ${health.version}, ${health.msPerImageP50} ms per picture`)
}

await brain.vfs.writeFile('/photos/harbour.jpg', bytes) // returns at once
// … later, once the job has run:
const hits = await brain.find({ where: { 'vision.tags.places': 'harbour' } })TS