Pictures described in the engine
In this section
A picture is invisible to search. Its bytes say nothing a text query can match, and the only text a file write embeds for it is its name and size. So brainy describes pictures the way it embeds text: through a describe provider that sits beside the embed provider, reached through one door on a GPU box with one token. Every brain gets it with no code change.
When a picture lands in a brain, the engine sends it to the describe service once, in the background, and stores on the picture's record:
a description of up to 2,400 characters (short paragraphs joined by a blank line),
a short tag list — the things in it, the roles they play, the places it shows, and any words visible in it,
what skin the picture shows, when the describe service says so,
how the picture reads — its mood, colours, light, setting, style, how many people are in it, how good the photograph is, and a short alt text — and a second, closer look at how revealing it is, when the service says so,
the description's embedding, as the record's own vector.
After that the picture turns up in an ordinary find({ query }), its tags can be filtered like any other field, and the change feed announces them. Nothing blocks the save: the write returns before the describe service has seen the picture.
What counts as a picture
A VFS file (brain.vfs.writeFile(path, bytes)) whose MIME type is image/* and whose bytes live in the blob store. The MIME type comes from the file name's extension. A working-state file (retention: 'latest-only') is never described: it never embeds at all, by design.
What the record carries
metadata.vision = {
description: 'A photograph of a lighthouse standing alone under a grey sky.', // up to 2,400 characters; paragraphs joined by a blank line
tags: {
things: ['lighthouse'], // lower-case, at most 12 per list
roles: ['landmark'],
places: ['coast'],
text: [], // words visible in the picture
},
exposed: [{ part: 'breast', score: 0.97 }], // only when the service answered it
exposedParts: ['breast'], // the same parts, as a plain list
reading: { // only when the service answered it; every member optional
mood: { words: ['calm', 'warm'], confidence: 0.8 },
palette: { colours: [{ hex: '#c8a165', name: 'sand' }], confidence: 0.7 },
light: { value: 'golden-hour', confidence: 0.9 },
setting: { text: 'a harbour wall at dusk', confidence: 0.6 },
style: { value: 'candid', confidence: 0.5 },
people: { count: 2, confidence: 0.95 },
quality: { sharpness: 0.8, noise: 0.1, word: 'sharp', confidence: 0.85 },
alt: { text: 'Two people on a harbour wall at golden hour.', confidence: 0.7 },
},
readingTerms: ['calm', 'warm', 'sand', 'golden-hour', 'a harbour wall at dusk', 'candid'], // the engine's own plain list
secondLook: { verdict: 'artistic-nude', confidence: 0.64, crops: 9, at: '2026-10-14T16:20:00.000Z' },
model: 'the-model-name',
version: '1.0',
describedAt: '2026-10-14T16:20:00.000Z',
sourceHash: '<the blob hash of the bytes this describes>',
confidence: 0.92,
served: ['description', 'tags', 'text', 'exposed', 'reading', 'secondLook'],
}TSexposed and exposedParts say what skin the picture shows, so a product can keep nude pictures off public pages, prints and social posts. An empty list means the service looked and found none; when the two fields are absent, the service did not say, and the picture needs a look. served lists what the engine asked the service for when it made this description. See Exposed skin and The richer reading and the second look.
It is written through the engine's ordinary update path, by the engine's own housekeeping actor:
The change feed carries it as an
updateevent whose metadata holds the tags — a subscriber learns a picture's tags without reading the record.The metadata index serves every part of it as a field:
find({ where: { 'vision.tags.things': 'lighthouse' } })returns the pictures tagged so (a tag list indexes one entry per tag).The vector. The record's embedded text becomes the description followed by every tag, joined, and is embedded through the same background embedding path every file write uses.
find({ query: 'lighthouse on a cliff' })finds the picture. A description may run to 2,400 characters and is stored whole on the record, but the embedder reads only the first 256 tokens of the text (about 1,000 characters of English), so a long description embeds up to that limit; the rest, and the tags after it, are searchable by their fields but do not enter the vector.
sourceHash is the blob hash of the bytes the description was made from. When a file is written again with new bytes, its record still carries the old vision until the job describes the new bytes; a reader can tell by comparing vision.sourceHash with metadata.storage.hash. The new description replaces the old one whole.
A file written again with the same bytes is not sent again. The write puts the record's embedded text back to the file's name, so the job puts the description back as the embedded text, without a call to the service.
The door
The engine speaks to the describe service through one fixed contract.
POST /v1/describe — Authorization: Bearer <token>
{
"requestId": "…",
"items": [ { "id": "…", "mime": "image/png", "bytes": "<base64>", "locale": "en" } ],
"want": ["description", "tags", "text", "exposed", "reading", "secondLook"]
}JSONwant always holds description, tags and text; exposed, reading and secondLook are each added only when the service's health lists it in serves, so a service is never asked for something it has not declared. At most 8 items per call and 20 MB in total. The body also carries maxDescriptionChars (2400), the longest description this client accepts; the service treats an absent field as 600, so an older client never receives a long answer and the service can restart with long descriptions without a fleet flag day. An item may carry url instead of bytes; the engine always sends bytes.
It answers 200:
{
"model": { "name": "…", "version": "…" },
"results": [
{ "id": "…", "description": "≤ 2,400 characters; short paragraphs joined by a blank line",
"tags": { "things": [], "roles": [], "places": [], "text": [] },
"confidence": 0.0, "ms": 0,
"exposed": [ { "part": "…", "score": 0.0 } ],
"reading": { "mood": { "words": [], "confidence": 0.0 }, "…": "…" },
"secondLook": { "verdict": "…", "confidence": 0.0, "crops": 9, "at": "…" } }
],
"refused": [ { "id": "…", "code": "…", "message": "…" } ]
}JSONTags are lower-case, at most 12 per list; text is the words visible in the picture. exposed is optional — a service that does not answer it is still in contract; when it is there it is a list of at most 32 entries, each a lower-case part and a score from 0 to 1. reading and secondLook are optional the same way and are bounded in The richer reading and the second look. One bad item never fails the batch — it comes back in refused.
Refusals: 400 unsupported-mime | too-large | bad-image (per item, in refused); 413 batch too large; 429 busy (with Retry-After, and an expectedWaitMs in the body from services that know it); 503 model unavailable; 401 bad token.
GET /v1/health answers { model, version, device, queue, msPerImageP50, expectedWaitMs, concurrency, batch, serves, supersedes }. queue counts pictures and msPerImageP50 is one picture's model wall under load. batch is optional: the most pictures the service wants in one call, a whole number of at least 1; absent is the contract's 8, and anything else is refused by name as a protocol error. concurrency is optional: how many pictures the model works at once, a number of at least 1; absent is read as 1, never 0. expectedWaitMs is optional: the service's own estimate of how long a call sent now waits before it is worked, already divided by its concurrency; when present the engine uses it in place of queue × msPerImageP50 / concurrency, and an absent one is never read as zero. serves is optional: the wants the service answers, e.g. ["description", "tags", "text", "exposed", "reading", "secondLook"]. A service that leaves it out is taken to answer the first three. supersedes is optional: a list of model names (name, any version) or name@version entries that the service's current model replaces; absent means none is declared, and anything that is not a list of non-empty strings is refused by name as a protocol error.
The job
The work runs on the engine's background job runtime as the job media-describe — a maintenance job, so it never holds a door, it stands down at close, and the operator's maintenance hold holds it.
On landing. After a picture's write returns, the engine drops a small queue entry in the store (
_brainy_describe/queue/<id>.json) and asks for a run. The entry is durable: a process stopped before the run finds it again at its next open. A picture written twice before the run has one entry.The drain. A run reads up to 32 queued pictures, plans calls of at most 8 pictures and 20 MB, and makes at most 4 calls before handing the runtime back and asking to run again — so a large backlog drains in short runs other jobs can interleave with. A queue entry is removed only after its record is written.
One lane per service, per process, as wide as the service. Every brain in a process speaks to the service through one lane with as many slots as the service's health declares it works at once (
concurrency; absent is 1, and it is read again from every health, so a service that changes it changes the lane on the next run). At most that many calls are in flight to it at a time, however many brains are open — the service is kept full and never sent one more — and a brain whose turn is behind waits for a slot, first come first served, never on a timeout. The pace belongs to the lane: after a call carryingnpictures, the next call — from any brain — is not sent untiln × msPerImageP50 / concurrencymilliseconds after it, so the process never sends faster than the service says it describes — and a service that works four pictures at once is fed four times as fast as one that works one. A call carries at most thebatchthe service's health declares (read again from every health; a plan cut before the read is split, never sent larger). The health's median says nothing about the pictures in hand, so when a call is abandoned at its deadline the lane remembers it for that service, in memory, for this process: the next call carries half as many pictures and its deadline is sized at the per-picture wall the abandoned call showed, until a call is answered. The service keeps reading the abandoned call (the client cannot cancel it), so the smaller call queues behind it.The queue, read before every call. Holding the lane, the job reads
GET /v1/health. The service's backlog is its ownexpectedWaitMswhen the health carries one, otherwisequeue × msPerImageP50 / concurrency. The call's deadline is sized to it:(backlog + n × msPerImageP50) × 1.25 + 30 s(the call's ownnpictures at the per-picture wall, undivided), at least 120 s and at most 10 minutes (the quarter more is there because the median runs short of the wall summed over a long queue). When even 10 minutes would not cover it, nothing is sent: the run ends without failing, prints one line ("the describe door … has 63 queued × 7683 ms ≈ 484 s of backlog … this pass waits 484 s"), once per process however many brains meet it, and the job runs again after the backlog — at least 8 s, at most 10 minutes. A picture landing meanwhile runs the job at once, which reads the health again and sends as soon as the backlog has cleared.A busy service. A
429is waited out for the wait it names (the longer ofRetry-Afterand anexpectedWaitMsin its body) and the call tried again, at most 3 tries — when that wait is 30 s or less. Past the tries, or at once when the service names a longer wait, the run ends without failing, prints one line, keeps every brain in the process off the service until the wait has passed, and the job runs again after it (at least 8 s, at most 10 minutes). Nothing is counted against the job, so a busy service never latches it.The receipt. Every call prints one line: how many pictures were described and refused, the wall it took, the service's median per picture, and the model.
The daily pass
The job is a cadence with a 24-hour interval: once a day while the brain is open it walks every picture in the brain and describes the ones that were never described, or whose bytes changed since (vision.sourceHash differs from the current blob hash). An open on a brain that holds pictures and whose last complete walk is more than a day old runs one, so a brain that was closed for a week catches up when it opens. Pictures that landed while no service was configured are found this way. The walk also records which wants the engine that made it knew (description, tags, text, exposed, reading, secondLook today) and which rules it applied. An open whose last walk was made by an engine that knew other wants or applied other rules runs one at once, however recent that walk was — so pictures described before the engine learned reading are reached the first time the brain opens on the new engine, not a day later.
What a daily walk costs when the service never answers a want it lists. Every picture missing that answer is sent once per walk: one call per 8 pictures (fewer for large ones) and one health read per call, each day. A brain of 1,661 pictures whose service lists reading and never answers it costs 208 calls a day; at the MEASURED hosted median of about 2.2 s per picture in a batch, about an hour of the service's time a day. The pass prints one line with the count each time, so the cost is visible until the service is fixed or stops listing the want.
The walk is durable: _brainy_describe/sweep.json records the last picture a walk finished, and a process stopped mid-walk resumes after it.
A better reader
When a better picture-reading model arrives, the pictures it replaces are read again by it. The service says which models its current one replaces in the supersedes list of its health. A picture whose description was made by a model that list names is owed one new description from the next daily walk (rule 6):
An entry
namematches a record whosevision.modelisname, at any version. An entryname@versionmatches that version only.When the new description lands, the record's
vision.modelandvision.versionbecome the service's current model, so it is never owed again for that reason — unless a later model supersedes that one too.Only the daily walk applies the rule, never the queue of newly landed pictures. Within a pass the re-reads go oldest description first, inside the walk's usual limits (512 pictures examined and four calls per pass); no new limit is added.
A change of model the service does not declare re-reads nothing. The walk remembers the last model it saw, and when it sees a different one that the new model's
supersedesdoes not cover, it prints one line, once per change: the service's model changed from the old to the new and declares nosupersedes, nothing is re-read, and the old model must be declared insupersedesto re-read it.Every brain's first walk after upgrading to an engine with this rule runs at once, however recent the last walk was, so existing pictures are examined under it.
Confining the walk to the operator's hours. A re-read of a whole library is heavy GPU work. BRAINY_DESCRIBE_WALK_WINDOW=HH:MM-HH:MM, in the host process's local time (its TZ), allows the daily walk to run only inside that window; the start is inclusive, the end exclusive. A window that crosses midnight, such as 22:00-06:00, is valid. Outside the window the walk is simply not due: it prints one line a day (the daily walk waits for its window), and the job is armed to run when the window opens. Pictures that land are never windowed — they are described promptly. A malformed value is refused by name when the describe job is built. Unset means any time.
Exposed skin
The describe service can also say what skin a picture shows. The engine asks for it ("exposed" in want) on every call to a service whose health lists exposed in serves, and writes what comes back onto the record as vision.exposed (each part with its score, in the service's order) and vision.exposedParts (the part names, each once). The plain list is what a filter reads:
const exposed = await brain.find({ where: { 'vision.exposedParts': 'breast' } })TSThe parts the engine knows are
breast,genitals,buttocksandanus(exported asDESCRIBE_EXPOSED_PARTS). The service may name others; a part the engine does not know is kept as the service said it, never dropped.Absent means the service did not say. A service that does not answer
exposedwrites neither field. A product should treat such a picture as one that needs a look, not as a clean one.One re-pass when the service learns it. A picture described before its service answered
exposedis described again once, by the daily pass, after the service's health starts listingexposedinserves. The new record'sservednamesexposed, so it is never sent again for that reason. A service that does not list it is sent nothing extra, and is never asked forexposed.A service that was asked and did not say is named, and asked again the next day. When a service that lists
exposedanswers a picture without it, the picture keeps its description and carries noexposed(so it reads as needing a look). Each pass that saw this prints one line with the count — the service must answerexposedon every result, an empty list when it finds none. The picture stays owed its answer: every daily walk that starts after its description was made sends it once more, while the service still listsexposed, until the service answers. Only the daily walk asks again — never the moment a picture lands — and a walk never sends the same picture twice.Never in search. The words for exposed parts never enter the picture's embedded text: its vector is still made from the description and its tags only.
Out of contract is a refusal. An
exposedthat is not a list, holds more than 32 entries, or has a part that is not lower-case or a score outside 0..1 makes that picture's answeranswer-out-of-contract, retried by the next daily pass.
The richer reading and the second look
Two more things the describe service can say about a picture. The engine asks for each ("reading", "secondLook" in want) only on a call to a service whose health lists it in serves, and writes what comes back onto the record. They follow the same rules as exposed skin: absent means the service did not say, one re-pass when a service starts declaring them, a reading the service was asked for and left off is named once per pass and asked for again by each daily walk until the service answers it, and neither ever enters the picture's embedded text. The second look is the one difference: the service takes it only when its first read of the picture is uncertain, so a result without secondLook is the ordinary case. The picture's record is stored in full without it, nothing is printed, and it is not sent again for it.
vision.reading is how the picture reads, in plain words. Every member is optional — a service may answer only some — and every member carries its own confidence, a number from 0 to 1:
member | what it holds | bound |
|---|---|---|
|
| at most 3, each at most 24 characters |
|
| at most 5; |
|
| one of |
|
| at most 60 characters |
|
| one of |
|
| a whole number, 0 or more |
|
| the two numbers 0 to 1; |
|
| at most 120 characters |
A member the contract does not name is ignored, not stored.
vision.secondLook is the service's second, closer opinion on how revealing the picture is: { verdict, confidence, crops, at }. verdict is one of safe, suggestive, artistic-nude, exposed, explicit; confidence is 0 to 1; crops is the whole number of crops of the picture the service looked at; at is the ISO 8601 instant it looked.
vision.readingTerms is the engine's own plain list, made from the reading and present exactly when reading is: the mood words, the palette colour names, the light, the setting and the style, lower-cased, each once, at most 32. It is what a filter reads for a colour name:
const warm = await brain.find({ where: { 'vision.readingTerms': 'golden-hour' } })
const sandy = await brain.find({ where: { 'vision.readingTerms': 'sand' } })TSThe reading's own members are also fields the metadata index serves: vision.reading.light.value, vision.reading.style.value, vision.reading.people.count and vision.reading.mood.words (a list of words, one entry each). A list of objects' sub-field is not in the index, so vision.reading.palette.colours.name finds nothing; use vision.readingTerms for colour names.
Out of contract is a refusal of the whole picture's answer. A present member outside its bound — six palette colours, an alt text of 121 characters, a confidence of 1.2, a light that is not on the list, a
secondLookwithout a whole number of crops — makes that picture's answeranswer-out-of-contract, naming the fault. Nothing of that answer is stored partially, and nothing is trimmed into shape. A picture already described well keeps its description, tags and vector, with the refusal beside them; the next daily pass tries again.Absent means the service did not say. A service that does not answer
readingwrites neitherreadingnorreadingTerms; one that does not answersecondLookwrites nosecondLook. A service that answers only some of the reading's members writes only those.New bytes replace all of it. A file written again with new bytes is described again and its
reading,readingTermsandsecondLookare replaced with the new answer, or dropped if the service no longer answers them.
Refusals are rows, never silence
A picture the service refuses gets a refused row instead of a description:
metadata.vision = {
sourceHash: '<the blob hash of the refused bytes>',
refused: { code: 'bad-image', message: 'the bytes are not an image', at: '2026-10-14T…' },
}TSunsupported-mime,too-large,bad-imagecannot change while the bytes do not, so the same bytes are not sent again. New bytes are tried again.A picture too large to fit in one call even alone is not sent at all; it gets
too-largefrom the engine with that reason in its message.answer-out-of-contract— the service answered this picture outside its own contract (a description over 2,400 characters, a tag list over 12 or not lower-case, a confidence outside 0..1, anexposedlist, areadingor asecondLookout of shape). The answer is recorded as a refusal, never trimmed into shape. Retried by the next daily pass.unanswered— the service answered neither a result nor a refusal for it. Retried by the next daily pass.detector-error— the service's skin detector failed on this one picture. Retried by the next daily pass.
A picture already described well and sent again only to learn its exposed skin, its reading or its second look keeps its description, tags and vector when the service refuses it: the refusal is recorded beside them in vision.refused, never in their place.
When the whole call is refused, nothing is written and the pictures stay queued. Except for a busy service (and a backlog, which is never sent into), the job's run fails with DescribeProviderUnavailableError, whose reason says which:
reason | what happens |
|---|---|
| the runtime backs off and retries — the connection was refused, the name did not resolve, or no answer came on a call whose health was not read ("could not be reached") |
| the runtime backs off and retries — the service answered its health just before, and the call's deadline, sized to that health's queue, passed ("timed out waiting on a queued door (queue N, P ms per picture, deadline D s)") |
| not a failure: waited out and retried; past 3 tries, or at once for a wait over 30 s, the run ends and the job runs again after the service's wait |
| a failure, not a full service: the runtime backs off and retries — including the service's |
| stops; set |
| stops; the engine's batching and the service disagree |
| stops; the service answered outside its contract |
| nothing is sent — see below |
Configure it
Zero-config is one name: run the describe service where brainy-describe resolves (a hosts line or a DNS record) on port 8201, and set BRAINY_DESCRIBE_TOKEN to its token. This mirrors the embedding service (brainy-embed, port 8200, BRAINY_EMBED_TOKEN).
variable | meaning |
|---|---|
|
|
| the bearer token the service requires |
| optional |
Discovery runs once per process and is shared by every brain in it. A name that did not resolve is asked again after ten minutes, so a service that comes up later is found without a restart.
With no service configured, nothing is described and no record is touched — the engine never writes a description a model did not make. The first picture to land prints one line saying it was not described and how to configure the service; later pictures print nothing more.
Cost
A picture landing: one small queue object written after the save has returned. A write that is not a picture pays one check of its metadata and nothing else.
A brain with no pictures pays nothing: no job runtime, no queue, no cadence. When a describe service is configured, the open lists the brain's media files once, from the metadata index, to decide whether to start the job.
Describing: one service call per 8 pictures (fewer for large ones), one health read before each, and one record update per picture — the same update any caller's
update()costs, plus the background embedding of the description.Many brains, one service: at most
concurrencycalls in flight per service per process, so the service sees the process's pace, not each brain's. PROJECTED from the pacing law (not measured), taking each call to be worked in one of the service's slots at its declared 7.7 s per picture: 16 brains in one process and a service working 4 at once (concurrency: 4) keep 4 calls of 8 in flight, sent about 15 s apart, each answered about 61 s after it was sent — about 31 pictures a minute (the service's own ceiling at those figures, 4 × 60 s ÷ 7.7 s), a 395-picture walk in about 13 minutes — where one call at a time reached about 8 a minute and each brain sending alone would put 16 calls on the service at once. A service reporting 63 queued at that rate has 63 × 7.7 s ÷ 4 ≈ 2 minutes of backlog, which one call's deadline covers (about 258 s), so the call is sent and its first picture is read about 2 minutes in. The same queue on a service that works one picture at a time (≈ 8 minutes of backlog) is sent nothing until it clears: one health read and one line, the brains re-armed about 8 minutes out, instead of sixteen calls that each time out and are sent again.The daily pass: one metadata-index read listing the brain's media files, then one record read per picture, in runs of at most 512. Only owed pictures reach the service.
Wall time is the service's, not the engine's. MEASURED on the hosted describe service (2026-10-14): 1.5–4.6 s per picture sent alone, about 2.2 s per picture in a batch, and a declared
msPerImageP50of 3,538 ms. Because the lane paces to that median, a batch of 8 owns the service for about 28 s. PROJECTED from that median: a first daily pass over 1,000 pictures takes about an hour. Each pass that moves the walk prints where it stands ("daily pass at N of M pictures"), and a stopped process resumes after the last picture it finished.The index: each description is a near-unique string, indexed like any other field. A store that never filters on it can declare
vision.descriptionunindexed and keep the stored value.
In code
import { DescribeProviderUnavailableError, resolveDescribeProvider } from '@soulcraft/brainy'
const { provider } = await resolveDescribeProvider()
if (provider.configured) {
const health = await provider.health()
console.log(`${health.model} ${health.version}, ${health.msPerImageP50} ms per picture`)
}
await brain.vfs.writeFile('/photos/harbour.jpg', bytes) // returns at once
// … later, once the job has run:
const hits = await brain.find({ where: { 'vision.tags.places': 'harbour' } })TS