The speech doors

The model service that serves /v1/embed and /v1/rerank also serves speech: one binary, one port, one bearer token, one licence check. Two of the three doors are WebSockets, because neither transcription nor synthesis is a request/response and modelling them as one throws away the number they are judged on.

Every door needs Authorization: Bearer <token>, /v1/speech/health included — the same rule the embedding doors follow, and the same token.

The bar

First transcript token and first audio frame under 300 ms. That is the whole acceptance test for a voice reflex, and it is why both doors stream: a transcription that answered when the audio ended could not report a first token, and a synthesis that returned a finished clip could not report a first frame. Both numbers are on the wire — firstTokenMs on the first transcript delta, firstFrameMs on the first audio frame of each utterance — so a caller measures the thing the bar names rather than something correlated with it.

The bar is a GPU number. /v1/speech/health reports latencyBarMs beside the device that has to meet it. The measured CPU numbers are at the bottom of this page; they are the reason the CPU path is functional, not a fallback.

The models are mounted, not shipped

No speech weight is in this package. Not in the npm tarball, not in the optional platform packages, not in the binary. The voice an assistant speaks in and the model that hears it are one versioned artifact with its own release cadence, and tying them to the engine's would mean an engine release every time a voice changed — and a gigabyte in every install that never speaks.

So the service loads them from a MOUNT, named once:

embed-service --bind 10.0.0.5:8200 --speech-assets /opt/anima/voice-models/current
#   or
BRAINY_SPEECH_ASSETS=/opt/anima/voice-models/current embed-service --bind 10.0.0.5:8200BASH

The flag wins over the variable. There is no default and no search — no probe beside the binary, no probe beside the working directory, no bundled copy to fall back on. A service that found a voice somewhere nobody named would produce output carrying an identity nobody can check.

With neither set, the service starts and serves. /v1/embed and /v1/rerank are completely unaffected; the three speech doors answer 503 speech_assets_not_mounted, naming the flag, the variable and the mount point. That is a supported deployment, not a broken one: most deployments of this service do not speak.

What the mount must contain

<mount>/
  MODELS.sha256
  models/whisper-small/model.safetensors
  models/whisper-small/tokenizer.json
  models/whisper-small/config.json
  models/whisper-small/melfilters.bytes
  models/kokoro-82m/model.onnx
  models/kokoro-82m/voices/af_heart.binTEXT

Those six files are required, at exactly those paths relative to the mount root. Further voices are models/kokoro-82m/voices/<name>.bin; each one needs a manifest line too, and every <name>.bin in that directory becomes a selectable voice.

MODELS.sha256 — the artifact's own identity

A GNU sha256sum file in the mount root, paths relative to the mount, plus exactly one # version: comment line:

# brainy speech assets
# version: anima-voice-models-2026-09-10
1d7734884874f1a1513ed9aa760a4f8e97aaa02fd6d93a3a85d27b2ae9ca596b  models/whisper-small/model.safetensors
27fc476bfe7f17299480be2273fc0608e4d5a99aba2ab5dec5374b4482d1a566  models/whisper-small/tokenizer.json
e6a2b489da1b5aed65a8eb8d1e7466fa867ad5643a8bc138ba708bd56b2875c4  models/whisper-small/config.json
85818f156f7e189453901a515e4726d270d307f976e161cf9403e8caab405498  models/whisper-small/melfilters.bytes
8fbea51ea711f2af382e88c833d9e288c6dc82ce5e98421ea61c058ce21a34cb  models/kokoro-82m/model.onnx
d583ccff3cdca2f7fae535cb998ac07e9fcb90f09737b9a41fa2734ec44a8f0b  models/kokoro-82m/voices/af_heart.binTEXT

Rules, each of them enforced at start:

sha256sum -c MODELS.sha256, run in the mount, is the same check by hand.

The digests are the artifact's, not the build's. No speech digest is compiled into this binary any more: a new voice release publishes new digests and a new version string, and no engine release is needed to accept it. The engine's own embedding model is the opposite case — it is vendored into the package and proved against the digests the build was stamped with.

A mounted artifact that fails any rule above refuses the start, by name, with the failing file and the cure. A corrupt or half-synced mount is a deployment error and is never served around. Only "nothing mounted at all" is a serving configuration.

The version string

Opaque to this service: whatever the artifact's publisher put on the # version: line, reported verbatim. /v1/speech/health reports it as assets.version and /health as speech.assetsVersion, so comparing two boxes is comparing one string. Both builds — CPU and CUDA — read it the same way; the compute cap changes which execution provider serves, never where the models come from.

/v1/transcribe

Open with a standard WebSocket handshake (Sec-WebSocket-Version: 13) plus the bearer token. The service answers 101, then immediately sends ready.

Client → server

Server → client — text frames, always JSON:

{"type":"ready","modelId":"whisper-small","language":"en","sampleRate":16000,
 "encoding":"pcm16le","channels":1,"partialCadenceMs":1000,"segmentMaxMs":30000,
 "device":"cuda","executionProvider":"cuda"}

{"type":"delta","segment":0,"kind":"partial","text":"and so my fellow",
 "t0Ms":0,"t1Ms":1000,"decodeMs":141.0,"firstTokenMs":126.5}

{"type":"delta","segment":0,"kind":"final",
 "text":"And so my fellow Americans, ask not what your country can do for you…",
 "t0Ms":0,"t1Ms":10000,"decodeMs":388.0}

{"type":"error","name":"queue_full","message":"…"}JSON

• A partial is a view of the open segment, re-decoded from its first sample every partialCadenceMs of newly arrived audio. It will be superseded.

• A final ends a segment and is the text to keep. A segment ends when it reaches segmentMaxMs (the encoder's own 30-second window), when the client sends end, or when the client closes — a client that closes mid-utterance still gets the words it already paid for.

• t0Ms and t1Ms are the span of audio the delta covers, counted from the stream's first sample. They come from the sample count, not from the model: the decode runs with timestamp tokens suppressed, so the door reports what it fed in rather than the model's guess about it.

• firstTokenMs is on the first delta of the stream and nowhere else. It is measured from the arrival of the first audio byte.

• decodeMs is on every delta: the wall from the stream's first sample to that delta's last token. firstTokenMs is the number the bar names; decodeMs is the one that says whether the partial cadence is keeping up with the audio arriving. Both figures above are illustrative — read them from the door.

Transcription is English (<|en|> fixed). Language detection costs a forward pass before the first token and would be spent on every partial.

/v1/synthesize

Client → server — text frames. Each frame is one utterance, spoken in the order it arrived. A binary frame is refused: audio only ever travels outward on this door. There is no end control, because on this door a text frame is content; the client closes when it has nothing more to say, and the service finishes the utterance in flight first.

Server → client

{"type":"ready","modelId":"kokoro-82m","voice":"af_heart","sampleRate":24000,
 "encoding":"pcm16le","channels":1,"device":"cuda","executionProvider":"cuda"}

{"type":"audio","utterance":0,"seq":0,"samples":24000,"durationMs":1000.0,
 "text":"Hello there.","firstFrameMs":166.2}
<binary frame: 48000 bytes of PCM16 little-endian>

{"type":"audio","utterance":0,"seq":1,"samples":12000,"durationMs":500.0,
 "text":"And this follows."}
<binary frame: 24000 bytes>

{"type":"utteranceComplete","utterance":0,"chunks":2,"durationMs":1500.0}JSON

Every audio frame is a pair: a JSON header, then the binary PCM it describes, immediately after it on the same socket. Audio is not base64 in a JSON envelope — that is a third more bytes on every frame of a latency-budgeted path — and the metadata is not a packed binary header, which no one can read in a console and nobody can version. A client reads the two as one unit and should refuse a header whose samples does not describe the frame that follows.

An utterance is chunked by sentence, and each sentence's audio is sent as soon as it exists. That is what makes a paragraph usable: its first frame arrives at the same latency as a single clause's, and the rest streams behind it. firstFrameMs is per utterance, measured from the arrival of its text.

/v1/speech/health

{
  "warm": true,
  "device": "cpu",
  "executionProvider": "cpu",
  "latencyBarMs": 300,
  "assets": {
    "mount": "/opt/anima/voice-models/current",
    "version": "anima-voice-models-2026-09-10",
    "manifest": "MODELS.sha256",
    "filesProved": 6
  },
  "transcription": {
    "modelId": "whisper-small",
    "source": "huggingface.co/openai/whisper-small @ 973afd24965f",
    "license": "Apache-2.0",
    "runtime": "candle",
    "weightsDigest": "…", "tokenizerDigest": "…", "configDigest": "…",
    "melFiltersDigest": "…",
    "sampleRate": 16000, "encoding": "pcm16le",
    "partialCadenceMs": 1000, "segmentMaxMs": 30000,
    "loadMs": 0.0, "warmupMs": 0.0, "residentDeltaBytes": 0,
    "queueDepth": 0
  },
  "synthesis": {
    "modelId": "kokoro-82m",
    "source": "huggingface.co/onnx-community/Kokoro-82M-v1.0-ONNX @ 1939ad2a8e41",
    "license": "Apache-2.0",
    "runtime": "onnxruntime",
    "modelDigest": "…", "voiceDigest": "…",
    "voice": "af_heart", "sampleRate": 24000, "encoding": "pcm16le",
    "loadMs": 0.0, "warmupMs": 0.0, "residentDeltaBytes": 0,
    "queueDepth": 0
  },
  "sessions": { "open": 0, "cap": 128 },
  "processResidentBytes": 0,
  "vramBudgetBytes": 4294967296, "vramInUseBytes": 0, "overVramBudget": false
}JSON

The shape is exact; the figures above are placeholders. loadMs, warmupMs, residentDeltaBytes and processResidentBytes are what THIS process measured of itself at startup, so they differ per deployment — read them from the door rather than from this page. residentDeltaBytes is a delta of VmRSS across that model's load: the host-side figure a CPU deployment is sized by, and not a measure of device memory, which vramInUseBytes reports per process.

warm gates both doors. Both models load at start and a warm-up pass runs before the socket is bound, so a service that is listening is a service that has already spent the first pass through each model — which is several times slower than every one after it, and is exactly the number nobody should ever measure. A session opened before that finishes is refused 503 speech_not_warm.

The digests are the ones this process measured at startup, hashing every file in the mount and comparing it to the mounted artifact's own MODELS.sha256. assets.version is that artifact's version string, verbatim, and assets.filesProved counts the manifest lines that were verified — 6 with one voice, more with more. The models are never fetched at run time; a mismatch refuses the start.

/health's speech section carries the same facts in short: built (was this binary compiled with the models' runtimes), mounted (was an artifact given), assetsVersion (which one, or null) and warm.

Refusals, by name

Refused before the upgrade, as an HTTP status a caller can pace against:

Refused inside a session, as an {"type":"error","name":…} frame followed by a close:

There is no fallback. A voice reflex has no slower-but-correct answer to degrade to: on the same hardware, the CPU misses the bar for every model the GPU clears it on. So a door that cannot serve refuses, by name, and the caller decides.

Backpressure

The service is a bounded guest on a box whose first duty is elsewhere. The number of concurrent speech sessions is capped, and the cap is enforced at the handshake — 429 with Retry-After, before the upgrade, so a refused caller gets a status and an interval rather than a socket that opens and then fails. Inside a session, the work queue is a bounded channel: nothing is dropped, and a client that sends audio faster than the model decodes it is slowed by TCP rather than served stale.

One forward pass runs at a time per model. That is not a tuning choice — it is the "one stream" term of the resource budget this service enforces on itself, and it is a property of the code rather than a promise: one thread owns the transcription model, and one permit gates the synthesis model.

The CPU numbers, measured

MEASURED 2026-09-08 on a 32-core AMD host, CPU only, one warm-up run then five measured runs, p50, with the model resident before the clock starts. The full matrix, including the smaller Whisper checkpoints, is in the engineering record that produced these doors.

What the two models cost this service

MEASURED 2026-09-09 by the service's own /v1/speech/health door, on a CPU build, and reported by the gate that starts it (sd-carry). These are size and start-up figures, not latency figures, so they do not need an idle box — the latency numbers above do, and were taken on one.

Resident after both are loaded and warm: 1,720,262,656 bytes (1.60 GiB).

That is a smaller number than the 3.0 GB the bench above reports, and the two are measuring different things rather than disagreeing. The bench figure is VmHWM — the PEAK any of the two separate processes reached while transcribing and synthesising. This one is VmRSS after loading and one warm pass in ONE process whose weights are memory-mapped, so pages the forward pass has not touched are not resident yet. Size a box on the peak; size a container's steady state on this.

The warm-up is where the first-pass cost goes, and it is why it happens before the socket is bound: 3.2 seconds for Whisper's first decode against 0.26 s for Kokoro's. A caller who was served that pass would have measured a transcription latency an order of magnitude off.

The CPU path is functional, not fast, and adding cores does not fix it: at every model size and every clip length, 32 threads finished slower than 8. The encoder saturates early and the decode loop is sequential, so the usual CPU lever is already spent. The accurate checkpoint is the one that costs two seconds: whisper-tiny reaches 350 ms to first token and drops the last word of a ten-second sentence.

Run the doors on a CPU for development, for a smoke test, and for anything where a late transcript is still a transcript. Do not run a voice reflex on one.

Notes for a client

• Send audio in small frames. 20 ms of 16 kHz PCM16 is 640 bytes. The door accepts any size up to 4 MiB, but a partial cannot be produced from audio that has not arrived yet.

• Keep one session per stream. Opening a session per utterance pays the handshake and the session-cap check every time.

• Answer pings. The service answers a client's ping with the identical payload; a client should do the same.

• Mask every frame you send. RFC 6455 requires it of a client, and the service refuses an unmasked frame by name rather than guessing.

Measured on the production GPU (Wed 2026-09-09)

The carry smoke against the CUDA build, every door called once under node: firstTokenMs 117.1 ms (transcription, the 10-second fixture clip) and firstFrameMs 208.9 ms (synthesis, the fixed sentence) — both under the 300 ms bar the doors are held to. CPU numbers above are the functional floor, never the bar.