Skip to main content

The worker protocol and the capability document

The contract, written before the code.

Rhapsode is a multi-engine speech server: one HTTP contract, many TTS models behind it, each in its own process. MaroonedSoftware/rhapsode, published on Docker Hub as maroonedsoftware/rhapsode, with the engine SDK as the Python package rhapsode-worker. A rhapsode was a performer who recited written verse aloud, which is the job description.

Two audiences read this file and they need different things. A client author needs the public API and the capability document, and should never learn that workers exist. An engine author needs the worker protocol and the Python SDK, and should never learn that the core is TypeScript. The seam between them is the point of the design, so each half is written as if the other were somebody else's project.

Every rule below that looks arbitrary is one somebody already paid for, and says so.

1. The one shape​

A worker speaks the engine-scoped subset of the public API. There is no second protocol.

Public APIWorker
GET /engines(the core aggregates)
GET /engines/{id}/capabilitiesGET /capabilities
GET /engines/{id}/voicesGET /voices
POST /speak (with engine)POST /speak
POST /engines/{id}/dialogue (§ 6)POST /dialogue
GET /healthGET /health
(core policy)POST /load, POST /unload, POST /terminate
POST /engines/{id}/pull (§ 10)POST /fetch

The core also answers routes no worker has, for installing and removing engines. They are about the box rather than about speech, and § 10 describes them.

This is the single decision everything else falls out of, and it buys four things:

  • An engine author develops and tests against the worker alone, with curl. The core is not a dependency of writing an adapter.
  • A remote worker is a URL. Local and remote are the same code path, not two, so "engines on another box" needs no new protocol and no new bugs.
  • The core is a router, a registry and a supervisor. It holds no engine knowledge, which is what keeps the "core never imports torch" rule structural rather than aspirational.
  • One shape to learn, document and version.

Transport: HTTP over a Unix domain socket by default, TCP when configured. Same protocol either way. The socket avoids port allocation in the common case and dies with its directory; TCP is what a remote worker uses. Chunked transfer encoding is what makes streaming audio work without inventing a framing layer, which is the reason this is HTTP and not a pipe with length-prefixed messages.

2. Worker lifecycle​

Spawn​

The core starts a worker with three environment variables and nothing else:

RHAPSODE_WORKER_LISTEN=unix:/run/rhapsode/workers/chatterbox.sock   # or tcp:127.0.0.1:0
RHAPSODE_WORKER_ENGINE=chatterbox
RHAPSODE_WORKER_CONTRACT=1

RHAPSODE_WORKER_CONTRACT is the highest contract the core speaks, not a demand. The worker answers at the highest version it supports that is not above it, and says which in the handshake. §9 has the rest.

Handshake​

The worker prints exactly one JSON line to stdout when it is bound and ready, and nothing else to stdout ever. Logs go to stderr as JSON lines, and the core forwards them into its own logger with the engine id attached.

{"ready":true,"contract":1,"engine":"chatterbox","listen":"unix:/run/rhapsode/workers/chatterbox.sock"}

A line rather than a poll loop because polling is how you end up with TTS-WebUI's 120-second socket wait: a worker that died on an import error is indistinguishable from one that is slow to start, and the operator waits two minutes to find out. A worker that exits before printing the line has failed to start, and its stderr is the error message.

Stdout is reserved for this one line so that a print statement in somebody's adapter cannot corrupt the handshake. Say so loudly in the SDK docs, and have the SDK capture sys.stdout and redirect it to the log stream after the handshake, so the rule cannot be broken by accident.

Stop​

SIGTERM means drain: refuse new requests as overloaded, finish what is in flight, exit 0. The error table in § 6 is what fixes the status, and it says 429 and retryable, which is the answer this case wants: a request refused by a draining worker is one the core should send somewhere else or send again, not one it should write off. (This paragraph said 503 and the table said 429. The table wins, because a caller reads the code and the retryable flag rather than the prose.) The core sends SIGKILL after a grace period it owns. POST /terminate (§3) is the same sequence asked for over HTTP rather than signalled, and is what reclaims a card from a worker the core cannot signal.

Which one the core reaches for, and in what order. An eviction sends /terminate first and signals only if the verb does not land, because that is the one reclaim a remote worker can also be given: a core that only signals degrades to unload over TCP and leaves behind what an unload strands (§ 3). A shutdown signals without asking, which is the same sequence by the paragraph above and one round trip shorter on a path where every worker is going anyway. Either way the grace period is the core's and ends in SIGKILL: a worker that took the verb and then hung is not a worker to wait on forever. A remote worker gets the verb and keeps its connection, because the process restarting is somebody else's supervisor's business and the core will want to talk to its replacement.

An exit the core did not ask for is a crash, and the core restarts it with backoff, holding the restart count against it. An exit that follows a SIGTERM or a /terminate is not, however it is timed: a worker that finishes draining a second after the core stopped waiting has done its job.

A core stops what it spawned however it is itself stopped. Closing the server is a shutdown whether or not it ever listened: an embedder or a test that builds the core, answers through it in process and closes it has started workers exactly as a listening core does, and they are stopped by the same sequence. The core's tests built and closed it without listening, the workers were never stopped, and by 2026-10-07 one Mac held 1,010 tone workers dating back three weeks, about 9.5 GB resident.

When the core is gone​

A worker whose core has died drains as if it had been sent SIGTERM, and leaves on its own if the drain has not finished within a grace period. A core can die without stopping anything: a crash, the OOM killer, SIGKILL from an operator, a test runner torn down mid-run. Its local workers are reparented to init, each holding its model, and no signal is ever coming, so the worker has to notice for itself. The grace period is the core's default drain grace, ten seconds, because the SIGKILL that ends a hung drain is the core's and there is no core left to send it. The exit after that grace is non-zero, since it is not the clean drain this section otherwise promises.

A worker detects this by its parent process changing, which is portable and needs nothing from the core: polling getppid() once a second costs nothing. Linux's PR_SET_PDEATHSIG would be immediate, but a Mac has no equivalent. Reading stdin to end of file would also work for a core that held a pipe open, and would end every worker started by hand from a terminal or with stdin on /dev/null, so stdin stays unused and § 2's three environment variables stay the whole of the spawn contract.

Two consequences follow. The worker's parent must be the core, so an engine command that wraps the interpreter has to exec it: a shell or uv run that stays alive becomes the parent, and the worker watches it instead. And a parent of PID 1 is not read as orphaned, because a core run as PID 1 without an init in front of it is a real parent; the image runs tini, and an operator's own may not. A core that dies before its worker has started watching is missed, which is a window of the adapter's imports. A remote worker is unaffected: it was never the core's child, and the section above already says its process is somebody else's supervisor's business.

3. Process residency and model residency are different things​

This distinction is load-bearing and is the thing most servers in this space get wrong.

  • A worker process costs tens of megabytes. Starting one is cheap.
  • A loaded model costs gigabytes of VRAM. Loading one is expensive and exclusive.

So the state machine is per worker:

down → starting → up(unloaded) → loading → up(loaded)
                       ↑                        |
                       +------- unloading ------+

GET /health reports both:

{"process":"up","model":"loaded","variant":"turbo","device":"cuda:0","vramBytes":4831838208,"modelBytes":3355443200}

vramBytes is what the card holds and modelBytes is what this model took of it, measured across the load by the SDK and reported while it stays loaded. They are different questions and only the second one helps a core decide what to evict: a budget counted in models assumes every model is the same size, and Kokoro is 82M parameters where Dia is 1.6B.

modelBytes is optional, because no measurement covers every device, and it is approximate where it is given: it is a card-wide delta, so it counts the runtime's own context and anything else that allocated during the load, which is the right answer for "can another model fit" and the wrong one for "how big are these weights". An adapter that knows the real figure returns it from Engine.memory_bytes and is believed. What a worker must never report is 0 or a negative: a core adding up a card reads those as a model that is free, so a measurement that failed is left out instead.

A card-wide delta also counts what other processes freed during the load, and a card shared with anything else frees memory often. Chatterbox turbo measured 2.6 GiB on its earlier loads on an RTX 4070 Ti SUPER, then loaded onto a full card in October 2026 while a failed Breeze load and an Ollama model were giving memory back, and its delta came out at zero or less, so it reported nothing. Where the device has an allocator that belongs to the worker's process alone (torch's, on CUDA and ROCm), the SDK reads that across the load too. A card-wide delta smaller than what this process's own allocator took proves somebody else freed memory, so the allocator's figure is reported instead. It is a floor, missing the context and anything outside torch, but it is the model's own and a floor beats nothing to a core deciding what fits. The driver's per-process list is no help here: inside a container it names processes by host PID, which a worker cannot match to its own.

Three rules, each paid for​

POST /speak loads on demand. If the model is not resident, /speak loads it and then speaks. It does not fail with "no model loaded" and it does not require the client to call /load first. The alternative was measured on a running station: the client ended up calling a model-info endpoint before every single synthesis to find out whether the previous request's unload had emptied the server, because nothing else would notice. That is a round trip per utterance to ask a question the server already knows the answer to.

A failed load unloads before retrying, exactly once. A CUDA OOM strands its own partial allocations. 3.5 GiB was measured stranded on a 16 GiB card, so an immediate retry throws itself at a GPU it just filled. Unload-then-load-once covers the common case of a card that was briefly full and has since freed up. This belongs in the SDK so every adapter gets it without knowing about it.

Unloading does not reclaim everything; terminating does. An unload gives back the model and keeps the process: its CUDA context, about 300 MiB, which only the process exiting returns, and anything held outside the worker's Python, such as a vLLM engine in a child process, which no unload in the adapter reaches. So the core needs both verbs and must know they differ: unload for "I may want this again shortly", terminate for "I need the card back". A residency manager with only unload keeps a context on the card for every engine it has ever loaded.

This was written as "an unload reclaims roughly 70%" until October 2026, and the cause given, the graphics runtime keeping the rest, was wrong. Measured on an RTX 4070 Ti SUPER, Chatterbox turbo's own unload freed none of its 2.65 GiB: the model sat in reference cycles until Python's collector next ran, so how much came back depended on when that happened to be. The SDK now collects after every unload (§ 8), and three unloads in a row each left the card at 311 MiB, the context alone.

Which is why terminate is a worker verb and not a core one. The obvious implementation is for the core to signal the process it spawned, and that works for exactly as long as every worker is local. A remote worker (§1) is a URL on somebody else's box, so a core that terminates by signalling silently degrades to unload over TCP, and what that cannot reclaim goes unreported. POST /terminate means drain and exit 0: the worker ends its own process, and whatever supervises it locally restarts it. The core still sends the signal to a local worker that has stopped answering, but the verb is what it reaches for first, and it is the only thing that works in both places.

Core policy, not worker policy​

The worker obeys; the core decides. Default policy, all of it settable, and without a restart (§ 10, "Settings"):

  • maxResidentModels (default 1, which is the right answer for one GPU)
  • LRU eviction when a /speak needs a model and the budget is full
  • keepAliveSeconds (default 300): how long a model with nothing left to do stays on the card. -1 keeps it until something else needs the room, 0 frees it as the last request lets go. Settable per engine as well as server-wide, because the cold start it trades against is per engine: a model that takes forty seconds to load has earned a longer deadline than one that takes two.

An expiry unloads rather than terminates. Rule 3 keeps the two verbs apart, and what separates them was measured on an RTX 4070 Ti SUPER in October 2026: once the SDK collects after an unload (§ 8), an unload leaves the worker's CUDA context and nothing else, 250 to 310 MiB for Breeze, Fish, Chatterbox and Orpheus's full build, whose vLLM engine exits with it. The model comes back in seconds where a terminated worker would have to start again first: Breeze reloaded in 1.4 s, Chatterbox in 4 s, Orpheus in 13 s. So a deadline that runs every few minutes costs a context per idle engine and saves a process start each time. An eviction still terminates, because it needs the card back now, and so does POST /engines/{engine}/unload by default. This replaces the idleUnloadSeconds and idleTerminateSeconds pair, which was two deadlines for two verbs and is one deadline now that there is one verb.

Until then an expiry terminated, on the strength of "an unload leaves roughly 30% behind", which rule 3 now corrects.

The default is on, which is a reversal. Off was defensible while an expiry was something an operator opted into: it trades a cold start for memory nobody asked for. It stops being defensible as the default on a box with one card, because the memory is being asked for, by whatever wants to run next and finds the card held by a model idle since this morning. A cold start is a cost the request that pays it can see; a card held by nothing is a cost that lands on somebody else.

Putting this in the core rather than in each worker is what stops every adapter author reinventing an idle timer, and it is the only component that can see the whole card.

GET /residency, for the operator asking where the memory went​

{
  "resident": 1, "max": 1, "waiting": 0,
  "models": [{
    "engine": "dia", "variant": "full", "leases": 0,
    "lastUsedAt": "2026-09-20T11:04:02.118Z", "expiresAt": "2026-09-20T11:09:02.118Z",
    "keepAliveSeconds": 300, "sizeBytes": 3355443200
  }]
}

expiresAt is absent while a model is speaking, because the deadline starts when the last request lets go, and absent when its keep-alive says never. keepAliveSeconds is the one actually in force after the precedence below, not the server's default. sizeBytes is what the worker measured and is absent where it could not.

It reads the core's own state: it spawns no worker, loads nothing and never blocks on one, which is the promise /health and /engines already make. Asking each worker instead would make listing what is loaded a reason to start processes that are not, which is the opposite of what somebody looking at a full card wants.

It is separate from /health because they answer different questions to different readers: a monitor polls /health and gets every engine's licence with it. And it is not a route to consult before speaking. /speak loads on demand, and a client that reads this first to decide whether it needs to has rebuilt the round trip per utterance that rule 1 exists to remove.

POST /engines/{engine}/unload, for giving the memory back now​

?mode=terminate by default, or ?mode=unload to keep the process for a faster next load. It is a management route (§ 10), behind the same guard as install: somebody who can empty a card can make every synthesis on the box pay a cold start.

It is idempotent, like the worker verb it reaches for, and answers 200 with the engine's summary. terminate ends the process even when no model is loaded, which is how an operator reclaims what a variant switch left stranded (rule 3: an unload keeps the process's context until it exits).

It is refused with 409 while the engine is speaking, for the reason § 7 gives about uninstall: cutting off a stream in progress hands that caller a truncated file for something they could not have predicted. On a busy box this can refuse for as long as the box is busy, which is the honest answer. Draining instead, unloading once the current request finishes, was considered and left out: the next request may set a new keep-alive, so a promise to unload afterwards is one the core cannot keep, and a 202 that silently becomes nothing is worse than a 409 that says what is true now.

A request may say how long it wants its model kept​

POST /speak and POST /engines/{engine}/dialogue take keepAliveSeconds, meaning the same thing it means in configuration, for the model that request loads. It is a hint about what happens next, which is the one thing the server cannot know and the caller often does: a batch about to send another two hundred lines says -1, and a page that just wants one sentence read aloud says 0 and gives the card back.

Precedence is request, then engine, then server. Three rules, because each answers a different question: the server knows what the box is for, the engine knows what its own cold start costs, and only the request knows whether there is more coming.

The last request to take a lease wins. Two requests speaking the same model cannot both be right about how long it stays, and the newer one is the one whose caller is still waiting. A request that says nothing puts the engine's own answer back rather than leaving the previous request's in place, so a -1 from a batch that has finished does not outlive it.

A keep-alive is not a reservation. -1 means "do not expire this", not "do not evict this": a /speak for another engine with the budget full still takes it by LRU (rule 1 above would otherwise fail for everybody else the moment one caller pinned a card). That is also why the field is safe on a public route: the worst a stranger can do with -1 is what this server did by default before there was a deadline at all.

The worker is never told. keepAliveSeconds is on the public request shapes and not on the ones the core forwards, because an adapter that can see a keep-alive is an adapter that will eventually act on one, and then there are two idle timers disagreeing about the same card.

4. The capability document​

This is the product. Everything else is plumbing that three other projects already have.

GET /capabilities
{
  "contract": 1,
  "engine": {
    "id": "chatterbox",
    "displayName": "Chatterbox",
    "adapterVersion": "0.3.1",
    "upstreamVersion": "0.1.4"
  },
  "license": {
    "code": "MIT",
    "weights": "MIT",
    "weightsCommercialUse": true,
    "notes": "https://github.com/resemble-ai/chatterbox"
  },
  "device": { "type": "cuda", "name": "NVIDIA GeForce RTX 4090", "vramBytes": 25757220864 },
  "current": {
    "variant": "turbo",
    "cues": ["laugh", "chuckle", "sigh", "gasp", "cough", "clear throat", "sniff", "groan"],
    "deliveries": [],
    "dials": {},
    "languages": ["en"],
    "maxCharacters": 4096,
    "segmentation": { "supported": true, "segmentCharacters": 300 },
    "cloning": { "supported": true, "referenceSeconds": [5, 10], "formats": ["wav", "mp3", "flac", "ogg"] },
    "blending": { "supported": false },
    "streaming": { "supported": true, "granularity": "chunk" },
    "nativeFormat": { "encoding": "pcm_s16le", "sampleRate": 24000, "channels": 1 }
  },
  "variants": {
    "turbo":        { "cues": ["laugh", "..."], "deliveries": [], "dials": {},
                      "cloning": { "supported": true, "referenceSeconds": [5, 10] } },
    "original":     { "cues": [], "deliveries": ["hushed", "frantic"],
                      "dials": { "exaggeration": {"min":0,"max":2,"default":0.5},
                                 "cfgWeight":    {"min":0,"max":1,"default":0.5} } },
    "multilingual": { "cues": [], "deliveries": ["hushed", "frantic"], "dials": { "...": {} },
                      "languages": ["en","fr","de","es","..."] }
  },
  "formats": ["wav", "mp3", "opus", "flac", "pcm"]
}

Why current and variants are separate, and why this is the hardest part​

An engine's capabilities can depend on which build is loaded. Chatterbox is the proof: the turbo build performs the paralinguistic tags and silently discards the expressiveness dials (upstream logs that it is ignoring them), while original and multilingual are the other way round. So on that engine you get cues or dials and never both, and which one is a fact about the weights that are resident right now.

A single flat capability list cannot express that, and a client that assumes one will send dials that vanish without a word. The two-level document is what lets a client ask "what can I do with what is loaded" and "what would I get if I asked for a different variant" as separate questions.

Engines with one build report a single variant and current mirrors it. The shape costs them nothing.

variants lists what this worker can load, not everything the engine has. Orpheus's full build runs on vLLM, which needs a CUDA card; on a Mac, or on a box without vLLM installed, the worker does not list it. Listing it there would be a variant that every load fails, which a client cannot tell from a variant that is only slow to load, and the first sign would be an error on a real request.

current is absent when no model is resident, because a worker in up(unloaded) has nothing to describe and an invented answer is worse than no answer. variants is always present, so the question "what could this engine do" is always answerable and only "what can it do right now" goes away. A client that reads current unconditionally will find this on the first request after a restart, which is the cheapest possible time to find it.

So nothing in the core may depend on current. It resolves the effective variant first, from the request's variant if it named one, from the resident variant if one is loaded, and from the engine's default otherwise, then reads variants[effective]. That rule is total, and it gives the same answer as current in every case where current means anything. It also covers the case the two-level document exists for: a request that names a variant which is not the one loaded, where current describes weights that are about to be evicted.

Each variant says whether it clones, as cloning, the same shape current carries. Cloning needs no model (§ 7), so whether it is offered cannot wait on one being resident: a client reading only current could not tell an engine that cannot clone from one that is idle, and offering a clone form that every submission fails is the first sign it would get. It is per variant rather than per engine because nothing makes it engine-wide: a build that conditions on reference audio and a build that only knows its trained speakers can be the same engine, as Orpheus's base model and its finetune are. cloning is optional, since contract 1 shipped without it; a reader that finds it absent does not know, and falls back to current, then to offering it and letting the worker refuse.

blending sits beside it on every variant and in current, for the same reason: a blend reads the voices it mixes and never the model (§ 7). Absent means no, because an engine blends only if its adapter says so, and a worker that predates the field cannot.

dialogue is declared per variant, and absent where it is not performed. Dia's 1.6b says "dialogue": { "maxSpeakers": 2 }: the variant answers POST /dialogue (§ 6), and one request may have up to two speakers. It is on the variant rather than the engine for the reason everything else is: nothing in the core may depend on current, and a second build of the same engine can differ. A client that does not know the field ignores it (§ 9), which is the whole of what an engine without dialogue asks of anybody.

license carries code and weights separately​

The weights licence is the one that package metadata never reveals, and it is the one that decides whether a commercial user may ship the thing. Apache-2.0 inference code over research-only weights is a real and common combination; Breeze TTS 2 is exactly that. A licence scanner reads the package and reports Apache-2.0, and it is wrong in the way that matters.

code is the licence of what the worker process runs, not of the adapter package. A scanner makes the same mistake one layer down: kokoro-onnx is MIT, and turning text into the phonemes its model takes goes through phonemizer and eSpeak NG, which are GPL-3.0-or-later. An adapter reports the most restrictive licence among what it loads, and says in notes which dependency is responsible, because that is the one a commercial user will ask about.

So the adapter declares both, by hand, and GET /engines surfaces them. Nobody else in this space does this, and for anyone using the server commercially it is the most useful field in the document. The catalog entry for an engine carries the same two fields, so the licence is visible before install, not after, and an install of weights that may not be used commercially has to name the licence it accepts (§ 10).

A licence belongs to an engine, so builds under different licences are different engines. Gradium's Phonon 31M is Apache-2.0 and its 40M builds are CC BY-NC 4.0. As variants of one engine there were two answers and both were wrong: report the most restrictive and a commercial user is told the Apache build is non-commercial, or report the other and an install of the 40M builds accepts nothing. So they are phonon and phonon-40m, two catalog records over one adapter package. A licence per variant would have had to change what an install accepts, since an install names no variant unless it pulls one, for the sake of the one upstream that needs it.

5. The standard vocabulary​

The whole reason a client can stay engine-agnostic.

Cues​

Things a speaker does that are not words. They ride inside the text, written [laugh], because a laugh happens at a place in a sentence and a separate field would have to invent a way to say where.

laugh  chuckle  sigh  gasp  cough  clear throat  sniff  groan

A closed set. The engine translates: (laughs) for Dia, [laugh] for Chatterbox turbo, <laugh> for Orpheus, nothing at all for Piper.

An engine claims the cues it performs and not the set: Orpheus has a tag for seven of these and none for clear throat, so it claims seven, and a [clear throat] sent to it is stripped like any other cue it does not claim.

The vocabulary is the only way in. An adapter removes its engine's own tag syntax from the text before it translates, so <yawn> sent to Orpheus is dropped, not performed. The alternative is a client that learns an engine's private tags and is then tied to that engine, which is the thing this section exists to prevent. A tag worth having on two engines is a word for this list.

The core strips every cue the loaded variant does not claim, before dispatch. This rule is worth more than it looks: it means an engine that implements no cues never sees one, and the failure where an engine reads the word "laugh" out loud cannot happen. Claiming a cue you cannot perform is the only way to break it, which makes the honesty requirement on adapters exactly one line long.

A removal closes the space it leaves behind, so word [laugh] word does not render with a double space that every consumer would have to know to tidy.

Text a speaker cannot read​

The core removes emoji and markdown emphasis before dispatch, on every request. Text written by a language model arrives decorated, and an engine voices the decoration. Chatterbox turbo read I really like Jon, and this is great. in 2.65 seconds averaged over three seeds. The same sentence with *really*, **Jon** and __great__ took 4.33, and with three emoji 4.19: over half as long again, spent saying something that was never words. No engine gives either a meaning, so nothing is lost by removing them, and every client would otherwise have to know to do it.

  • Emphasis is removed only where it is emphasis. *, **, _ and __ go when they open against a word and close against one, the way markdown reads them. The text inside stays. So snake_case, 2 * 3 and a lone asterisk survive, because none of them is emphasis.
  • Brackets are never touched. Square, angle and round brackets are where every engine's tag syntax lives ([laugh] here, <laugh> for Orpheus, (laughs) for Dia), and § 5's cue rule is the only thing that decides what happens to them.
  • It is silent, as stripping an unclaimed cue is. It makes the request performable rather than discarding something the client asked for, so it is not reported.
  • The ceiling counts the text as it arrived. maxCharacters is applied before either removal, as it is for cues, so whether a request fits does not depend on how decorated it was.

Deliveries​

How a whole line is read. Two, and there is deliberately no word for "ordinary":

hushed  frantic

A request with no delivery is the voice's own ordinary reading, which is what nearly every line should be. A third word meaning "ordinary" would be a second way to ask for nothing, and would key a second cache entry for identical audio.

A word and not a number. exaggeration: 0.9 means something to one family of models and nothing to the next, so a client that sent it would be tied to the engine it was written against, which is the thing this whole document exists to prevent. The numbers live in dials, per engine, and the adapter is what turns a word into them.

Translate relative to the voice, not to a fixed point. A voice that is intense at rest should still be more intense than its neighbours when hushed. An adapter that hard-codes exaggeration = 0.2 for hushed has thrown away the voice's own character.

Widen this list only when two engines can both perform the new word. One engine's feature is a dial, not a delivery.

6. Speaking​

POST /speak
Content-Type: application/json
{
  "text": "Right, that was The Verve Pipe. [laugh] Nobody warned me about that intro.",
  "voice": "narrator_02",
  "variant": "turbo",
  "format": "opus",
  "delivery": "hushed",
  "params": { "exaggeration": 0.7 },
  "seed": 20260917,
  "stream": true
}
FieldNotes
textRequired. Cues already stripped by the core to what this variant claims.
voiceEngine-scoped id from GET /voices. Absent means the engine default.
variantAbsent means whatever is loaded, or the engine's default if nothing is.
formatFrom capabilities.formats. The response's Content-Type is authoritative.
languageFrom the effective variant's languages. Absent means the first one it lists.
deliveryOnly ever one the effective variant claimed; the core drops the rest.
paramsAgainst the effective variant's dials (§4). Unknown keys are refused, not ignored.
seedOptional. Reproducibility for engines that can.
keepAliveSecondsHow long to keep this model once the request is done. §3. Not sent to the worker.
streamtrue streams chunked; false buffers and sets Content-Length.

Response: 200, Content-Type: audio/opus, chunked.

X-Rhapsode-Duration-Ms is an ordinary header when stream: false, because the duration is known before the headers go out, and a trailer when stream: true, because it is not. Treat the trailer as best effort and never require it: fetch exposes no trailer API in Node or in a browser, so only a client built on a raw HTTP library can read one. A client that needs the duration on every response should ask for stream: false.

One note on the opus label, because it will look like a bug to somebody: what goes on the wire is Ogg-encapsulated Opus, for which audio/ogg; codecs=opus is the precise answer. audio/opus is what this contract says, it is what every client in this space already sends and accepts, and the response's own Content-Type is authoritative in any case.

language selects among what the variant declares​

variants[...].languages has been in the capability document from the start and there was no way to ask for one, which made a multilingual build indistinguishable from an English one. A variant that lists a single language ignores this field; one that lists several reads it, and a language it does not list is unsupported.

It is a separate field rather than something inferred from the text because detection is a guess, and a guess that silently picks the wrong language produces a whole take in the wrong accent with nothing in the response to say so.

Unknown keys in params are refused​

The tempting alternative is to ignore them, and it is wrong for the same reason a silently discarded dial is wrong: the client believes it asked for something. A 400 naming the key is a bug report delivered to the right person in under a second.

Failing after the headers have gone​

This is the one that bites, and it needs saying in the spec rather than being discovered per client.

Once a 200 and a Content-Type are on the wire, the status cannot be taken back. A worker that fails mid-stream must abort the connection rather than close it cleanly, so the core sees a truncated body rather than a short successful one.

The core additionally applies a floor: a body under 256 bytes is not audio, whatever the status said. This exists because a server answering 200 with a JSON complaint about an unknown voice produces a segment that airs as a click, and the only place to notice is at the end of the stream. The check belongs at the end, not on the first chunk, because a short error body can arrive in several pieces and "was any of this plausibly audio" is only answerable once it stops.

The floor behaves differently in the two modes, and that is deliberate. With stream: true the headers are long gone by the time the count is known, so a body under the floor is an abort and the client cannot be told why. With stream: false the core has the whole body before it writes anything, so the same failure is an ordinary error envelope with a code and a retryable flag.

Which is the general rule and worth stating once: stream: false reports failures strictly better than stream: true does, because every failure is still a pre-headers failure. Streaming buys first-byte latency and pays for it in diagnosis. A caller rendering a file rather than feeding a player should ask for stream: false and will get a better error the day something breaks.

Errors​

{ "error": { "code": "unknown_voice", "message": "no voice \"narrator_99\"", "retryable": false } }
CodeHTTPRetryableMeaning
bad_request400noMalformed, or a params key the variant does not have
unknown_engine404noNot in GET /engines. Core only; a worker is one engine
unknown_voice404noNot in GET /voices
unsupported422noA format, language or feature this variant does not do
model_unavailable503yesLoading, evicted, a load that ran out of time, or a worker stopped from outside before it started
oom503yesOut of device memory
overloaded429yesDraining, or at the concurrency limit
internal500noThe adapter threw, or its worker refused to start
forbidden403noA management route (§ 10) called from somewhere it does not answer
conflict409noA management route (§ 10) asked for something the current state rules out

retryable is a field and not something the client infers from the status. The distinction that matters is between "this request was wrong" and "this request was fine and the server was not", and a caller that conflates them either retries a permanent failure forever or discards work that would have succeeded on the next pass. A cold start that ran out of its budget is model_unavailable, and a client that understands that keeps the job rather than writing it off.

A worker that exits with a status before its handshake has refused to start, and that is internal. It has not loaded a model yet, since the handshake comes first, so whatever stopped it stops the next attempt too: Breeze TTS 2 on a Mac exits 1 saying it needs an NVIDIA card, and was reported as a retryable model_unavailable, which a client trusting the flag would retry forever. An interpreter that does not exist, a wrong venv path, is internal for the same reason. A worker killed by a signal before its handshake was stopped from outside, by the OOM killer or an operator, and stays model_unavailable, as does one that did not announce itself in time.

Dialogue​

POST /engines/{engine}/dialogue          (worker: POST /dialogue)
Content-Type: application/json
{
  "turns": [
    { "speaker": "a", "text": "Did you hear that? [gasp]" },
    { "speaker": "b", "text": "[laugh] It's only the cat." },
    { "speaker": "a", "text": "It is never only the cat." }
  ],
  "voices": { "a": "narrator_02" },
  "variant": "1.6b",
  "format": "wav",
  "seed": 7,
  "stream": false
}

Some engines make a conversation in one pass, with the timing, the overlaps and the reactions of two people in one room, which stitching separate takes together cannot make. Dia is the first. That does not fit /speak, whose one voice is one reader, and it is a second route rather than a second shape of text so that no engine without it has a request shape it must refuse half of. A variant that performs it says so in dialogue (§ 4); one that does not answers the route unsupported.

A turn names a speaker, not a voice. Speakers are labels the request makes up, and voices maps the ones that should sound like a particular voice to its id. A speaker with no voice is read in one the model picks, which a seed holds fixed, exactly as /speak with no voice is. A voice on each turn could not say "the same person as turn one, whoever that is", which is the ordinary case for an engine with no voices of its own. A voice the engine does not have is unknown_voice, as in § 7.

FieldNotes
turnsRequired, at least one. Each is { speaker, text }, and text has cues stripped per turn to what the variant claims.
voicesOptional. Speaker label to voice id. A label no turn uses is bad_request.
variant, format, language, params, seed, streamAs for /speak.
  • More distinct speakers than maxSpeakers is unsupported, since the variant said it cannot.
  • The ceiling is maxCharacters over the sum of every turn's text. A dialogue is one take, so it is one budget. Splitting it into turns does not buy a longer one.
  • There is no delivery. A delivery reads a whole line one way, and a dialogue has more than one reader. It is left out rather than half-defined; a delivery per turn waits for an engine that can perform one.
  • Everything about the response is /speak's: the formats, the 256-byte floor, aborting rather than closing when a stream fails after its headers, the duration header or trailer, and the errors.

Speaking as the text arrives​

GET /speak/stream
Upgrade: websocket

A language model writes its answer a few characters at a time, and a caller that waits for the whole answer before /speak makes its listener wait for both. This route takes the text as it is written and speaks each sentence once it is complete, so the first audio follows the first full stop rather than the last. Measured with Chatterbox turbo on an M-series Mac, on a 218-character paragraph written at 30 tokens a second: the first audio 1.2 s after the first token, before the model had finished, against 8.9 s through /speak once it had.

// client → server, text frames
{ "type": "start", "engine": "kokoro", "voice": "af_bella", "seed": 7 }
{ "type": "text", "text": "The quick brown fox jum" }
{ "type": "text", "text": "ps over the lazy dog. And then" }
{ "type": "flush" }
{ "type": "end" }

// server → client
{ "type": "ready", "engine": "kokoro", "variant": "fp16" }
{ "type": "format", "contentType": "audio/L16; rate=24000; channels=1" }
<binary frames: PCM>
{ "type": "spoken", "index": 0, "characters": 44, "durationMs": 2810 }
<binary frames: PCM>
{ "type": "spoken", "index": 1, "characters": 8, "durationMs": 640 }
{ "type": "done" }

It is /speak, one piece at a time, and not a second implementation of it. Each piece goes to the worker's ordinary POST /speak, so the worker protocol does not change and an engine needs nothing to take part. Each piece passes through everything /speak applies: the ceiling, the cue and decoration stripping of § 5, the dial check and the floor. So nothing that is true of /speak can stop being true here.

  • start is /speak's fields without text and stream: engine, voice, variant, format, language, delivery, params, seed and keepAliveSeconds. A field it does not know is bad_request, as it is at /speak. Nothing else may come first.
  • format is pcm or absent. PCM pieces join into one stream by concatenation. A WAV, an Ogg or a FLAC per piece would be a header in the middle of the audio, and making one file of them is audio work the core does not do. The rate arrives once, in a format frame before the first audio, carrying the worker's Content-Type verbatim. It is not in ready, because the core holds no format table and knows the rate only once a worker has answered.
  • The core cuts at sentence ends, by the SDK's own rule (§ 8): ., !, ? or … followed by whitespace, or a CJK stop. A full stop at the very end of what has arrived is not yet a sentence end, because 3. may be about to become 3.5; the next character, a flush or the end decides.
  • The first piece is the first sentence, and later ones are everything complete since. Pieces are spoken one after another, and while one is being spoken the text keeps arriving. Whatever sentences are complete when the worker is free go together, up to the variant's segmentCharacters (or maxCharacters where it declares no split), so the latency is one sentence and the rest is spoken in pieces the engine reads as prose rather than as a list of lines.
  • Text with no sentence end is not held forever. Past the same limit with no sentence end, the core cuts at the last clause or word boundary within it, as the SDK would.
  • flush speaks what has arrived, finished or not. It is how a caller says the model has stopped writing for now. end is a flush that also closes the session once everything is spoken.
  • seed is the first piece's, and piece n gets seed + n, the rule the SDK applies to the pieces it cuts (§ 8). A seeded session reproduces itself when its text is cut the same way, which is only when it arrives in the same frames.
  • spoken follows each piece's audio, with its index, how many characters it was, and its duration from the worker's trailer, so a caller can follow along without counting samples.
  • The residency lease is held for the whole session, from start to the socket closing, so the model is not evicted between two sentences of one answer. A session that sends nothing for 60 seconds while nothing is being spoken is closed as bad_request, because a client that vanished without closing the socket would otherwise hold the lease for good.
  • Every error ends the session. It arrives as { "type": "error", "error": { code, message, retryable } }, the envelope of § 6, and the server closes the socket after it. Audio already sent stays sent; a client that needs all or nothing wants /speak. An unknown voice, for instance, is found by the first piece, after ready.
  • Closing the socket stops the piece being spoken, as closing /speak's connection does.

It answers the callers /speak answers, except a web page from elsewhere. A browser does not ask a server whether a page may open a WebSocket to it, as it asks before a cross-site POST of JSON, so without a check any page the operator visits could use this route to drive the engines on their machine and hear the result. So a request carrying an Origin is refused as forbidden unless the origin is this machine's own or is listed in management.origins, the rule /mcp applies (§ 12).

It is not in GET /openapi.json (§ 9): OpenAPI describes requests and responses, and this is a conversation. The OpenAI shim and the MCP server do not offer it, because neither has a request shape for text that has not been written yet.

7. Voices​

GET /voices
[
  {
    "id": "narrator_02",
    "label": "Narrator 02",
    "description": "Cloned from narrator_02.wav",
    "spec": "narrator_02@turbo",
    "tags": ["cloned", "en"],
    "previewUrl": "/voices/narrator_02/preview"
  }
]

spec is an opaque token that changes whenever the rendering would. Clients key cached previews on it. The reason it exists rather than clients keying on id: the id is exactly the part that does not change when somebody edits what is under it, so a remapped voice served its old preview forever, and the only fix that works is a token the engine mints. Treat it as opaque; never parse it.

previewUrl is worker-scoped, and the core rewrites it. The worker knows nothing about engine ids in paths, so it answers /voices/{id}/preview, which is correct on its own socket and a 404 to anybody who followed it from the public API. The core rewrites each one to /engines/{engine}/voices/{id}/preview on the way out. This is the only field the core edits while proxying, and the only reason it does is that the worker cannot know its own prefix. A client should follow the URL it was given and never build one.

A voice id is a name: letters, digits, - and _, starting with a letter or digit, at most 64 characters. Anything else is bad_request, checked before an adapter sees it. An adapter makes a file of an id, and before the rule an id of ../../x wrote a clone outside the voice directory and an id of * matched whichever voice sorted first.

A voice the engine does not have is unknown_voice, never a substitute. An engine that falls back to its default voice for an id it does not know hands the caller audio in a voice they did not ask for, with a 200, which is the silent discard this document exists to prevent.

Creating a voice, where the engine supports it:

POST /engines/{engine}/voices          (worker: POST /voices)
Content-Type: multipart/form-data
  id=narrator_03  label="Narrator 03"  reference=@clip.wav  transcript="What the clip says."
  id=host         label="Host"         blend="af_bella(2)+af_sky(1)"
DELETE /engines/{engine}/voices/{id}   (worker: DELETE /voices/{id})

transcript is optional: the words spoken in the reference. An engine that clones by continuing from the clip, as Dia does, has to be told what was said in it as well as how it sounded, and one that is given the wrong words clones worse without any error, so there is nothing to infer it from safely. Such an engine refuses a create without one as bad_request, naming the field; the capability document has no way to say in advance that a transcript is needed, and a refusal naming the field is the whole of the discovery. An engine that does not read it ignores it, which is the one exception to unknown input being refused: every client can send it to every engine, so a client does not have to know which engine reads it.

The answer to a create is the new voice, as GET /voices would list it. Creating an id that exists replaces it, which is how a voice is re-recorded; its spec changes, so a cached preview does too.

A create carries exactly one of reference or blend, and both or neither is bad_request.

reference is whatever this engine makes a voice from, and current.cloning.formats says which file types that is. For Chatterbox it is a clip of somebody speaking. For Kokoro it is a style vector, the thing a Kokoro voice actually is: a .npy array, or a .pt voicepack as Kokoro-FastAPI ships them, of shape 510 x 1 x 256. That is how a voicepack outside the engine's own set arrives, Kokoro-FastAPI's v0 voices included. A reference whose filename names a type the engine does not list is unsupported, as any format the engine does not do is (§ 6), and one of the right type and the wrong contents is bad_request. A voicepack is read as data and never unpickled: a .pt file is a pickle, and unpickling an upload runs whatever the uploader wrote.

blend is a mix of voices the engine already has, where current.blending.supported says so. The recipe is name(weight)+name(weight), each weight optional and 1 when absent, positive, and normalised so they sum to 1. The syntax is Kokoro-FastAPI's, because that is what an operator coming from it already has typed into a config. Every name is a voice id (above) and must be one the engine lists, built in or created, else unknown_voice.

A blend is resolved when it is created, into a stored voice like any other. So deleting or re-recording a component later does not change a blend made from it, and its spec is minted from the result rather than from the recipe. The alternative, a recipe resolved at every speak, is a voice that changes under a caller who never touched it, which is the failure spec exists to catch and would here be invisible to it.

Why a created voice and not a recipe written into voice at speak time: a recipe is not a name, and the id rule above is what stops a voice id from being a path or a pattern. A named blend also gets what every other voice has, a place in GET /voices, a spec, and a preview.

An engine that makes no voices answers a create with unsupported, and one that makes voices one way and not the other answers the other with unsupported too.

The worker owns its voice store. The core keeps no shadow registry, because two registries is one more than the number that can be right. It streams the upload to the worker as it arrives and keeps no copy, so a clip costs the core no memory and no disk. It refuses a body over 25 MB as bad_request: a usable reference is 5 to 20 seconds, which is under 1 MB of 24 kHz mono WAV and under 6 MB of 48 kHz stereo at 24 bits, so the cap is generous for any clip and small enough that a single request cannot fill a disk.

Creating and deleting a voice are management routes (§ 10): they write files on the box, so the same guard applies and a page from another machine is refused. Listing voices and hearing a preview stay open, like speaking.

Creating a voice loads no model and takes no residency slot. An adapter's create_voice stores the reference and returns. Work that needs the model, such as computing a speaker embedding, happens at the first speak in that voice, under the lease speak already holds. The alternative is a clone that evicts whatever another caller is using, from a route nobody would expect to touch the GPU, and Chatterbox, which clones from the reference on every call anyway, has nothing to do early. A Kokoro blend holds to the same rule: it reads the voices file, 28 MB, and never the graph, which is up to 325 MB.

8. The Python worker SDK​

An adapter author installs one package, writes one class, and never learns what the core is written in. If that is not true, the extension model is decoration and you will write every engine yourself.

from rhapsode_worker import Engine, Capabilities, Variant, Voice, SpeakRequest, NativeFormat, serve

class ChatterboxEngine(Engine):
    id = "chatterbox"
    display_name = "Chatterbox"
    license = dict(code="MIT", weights="MIT", weights_commercial_use=True)

    # The SDK encodes. Yield PCM and stop thinking about mp3.
    native_format = NativeFormat(encoding="pcm_s16le", sample_rate=24_000, channels=1)

    def variants(self) -> dict[str, Variant]:
        return {
            "turbo": Variant(cues=ALL_CUES, deliveries=[], dials={}),
            "original": Variant(
                cues=[],
                deliveries=["hushed", "frantic"],
                dials={"exaggeration": (0.0, 2.0, 0.5), "cfgWeight": (0.0, 1.0, 0.5)},
            ),
        }

    def load(self, variant: str) -> None:
        # Each build is its own class upstream rather than an argument, which is a fact about that
        # engine and not about this protocol. A variant name is whatever the adapter says it is.
        self.model = BUILDS[variant].from_pretrained(device=self.device.torch())

    def unload(self) -> None:
        del self.model

    def voices(self) -> list[Voice]:
        return [Voice(id=p.stem, label=p.stem, spec=f"{p.stem}@{self.variant}")
                for p in self.voice_dir.glob("*.wav")]

    def speak(self, req: SpeakRequest) -> Iterator[bytes]:
        # An engine that streams yields as it goes. One that does not, and chatterbox does not,
        # synthesises whole and chunks the result: the SDK's contract is an iterator of PCM, not a
        # promise that the model is incremental.
        yield from self.pcm_of(self.model.generate(req.text,
                                                   audio_prompt_path=self.path_for(req.voice),
                                                   **self.dials_for(req)))

serve(ChatterboxEngine())

What the SDK does so no adapter has to​

  • Binds the socket, prints the handshake line, captures stray stdout, frames logs as JSON on stderr.
  • Serves /health, /capabilities, /voices, /load, /unload, /terminate, /fetch, /speak and /dialogue. capabilities() is assembled from variants(), native_format, license and the detected device, and current is omitted entirely while nothing is loaded.
  • Encodes. The engine yields its native PCM; the SDK produces wav, mp3, opus, flac or raw through ffmpeg. Without this, every adapter reimplements format conversion and they all do it differently. This is the single largest reduction in adapter burden in the design.
  • Applies the unload-then-load-once retry around load(), so the OOM lesson is free.
  • Collects after every unload, then empties torch's cache if an adapter imported torch. A model held in a reference cycle is freed by the collector, not when the adapter drops it, so an adapter's own empty_cache finds nothing to return: Fish Audio S2 Pro left 3.93 GiB of the 11.97 GiB a load took on the card after its unload, and every reload ran out of memory, until a collection ran.
  • Measures what a load cost, as a device-wide delta across load(), and reports it as modelBytes (§ 3), falling back to the process's own allocator delta when another process freed memory during the load. An adapter that knows better overrides memory_bytes(); one that does not gets a figure for free, and one on a device nothing can measure reports nothing rather than zero.
  • Maps exceptions onto the error taxonomy, with retryable set correctly, and aborts the connection rather than closing it cleanly when speak() raises mid-stream.
  • Enforces maxCharacters and validates params against the declared dials before speak() is called, so an adapter never receives a request it did not declare support for.
  • Splits long text where a reader would pause, so maxCharacters stops being the client's problem. The subsection below is why this is the SDK's job and not each adapter's.
  • Serialises requests by default. One model, one utterance at a time is the correct default for a GPU; an engine that can genuinely batch sets concurrency > 1 and takes responsibility.
  • Waits for an abandoned synthesis to end before it loads, unloads or starts another. A model's generate() is one blocking call, so a client that hangs up stops the stream only when that call returns, and until then the model is still on the device. Measured with Dia on Metal: a load that started under an abandoned generation killed the process with "failed assertion _status < MTLCommandBufferStatusCommitted".
  • Offers a cache for analysed references, ReferenceCache, which a cloning adapter fills with whatever its model makes of a clip and clears when it loads or unloads. It is offered rather than imposed, because only the adapter knows what the analysis is and which request fields it depends on. Upstream code tends to analyse the clip inside generate on every request: Chatterbox turbo on MPS spent 4.9 s on a cloned line against 2.2 s for the stock voice until it kept the analysis. An entry is keyed on the clip's path, modification time and size, so a re-recorded voice is analysed again without the adapter having to notice.

Long text, and who splits it​

maxCharacters is the most text a request may carry. It is not the most text one generation gets. Those were the same number until three adapters had each written the code that makes them different, and the fourth had not.

  • Kokoro splits on sentences and packs them back up. A 4096-character request is about four and a half minutes of audio, and at the 2.5x realtime measured on int8 that is 110 seconds before the first byte, against the core's 120-second wait for headers. Speaking a group at a time makes the wait the length of the first group instead.
  • Dia and Orpheus split on sentences, then clauses, then words, and pack the pieces back greedily so that a run of short sentences is one generation rather than several. Dia also carries an audio prompt from one piece into the next, because its voice drifts across generations that have nothing in common.
  • Chatterbox did none of this, and paid for it. Every build stops at 1000 speech tokens at 25 a second, so one generate cannot say more than 40 seconds whatever it is handed. A 1,060-character news bulletin sent to turbo whole came back as 26.7 seconds of skipped and crammed speech, where a 488-character one read cleanly in 30.8.

_fit and _pack in rhapsode_engine_dia.prompt and rhapsode_engine_orpheus.prompt are the same twenty lines, character for character, and their sentence and clause patterns are the same two regular expressions. Kokoro has a third copy, weaker in two ways nobody chose: no clause fallback, so a sentence longer than the limit reaches the model whole, and a sentence pattern that misses the ellipsis the other two match. Three copies of one idea, diverging quietly, is the shape of something that belongs one level down.

So the SDK splits, and speak() receives one segment at a time. Breaking text where a reader would pause is text handling, not engine knowledge: it is the same argument that put encoding in the SDK, and it is the single largest thing left that every adapter still pays for itself.

  • The algorithm is fixed and shared. Sentences, then clauses, then words, packed back greedily. A single word longer than the limit stays whole, because splitting it has the model read two halves of a word. A cue is one character to the splitter, so [clear throat] is never torn.
  • Chinese and Japanese break without spaces. 。！？ end a sentence and ，、；： a clause with nothing after them, a closing 」 stays with the stop before it, and an ellipsis breaks before a CJK character. Two CJK pieces packed together are joined with nothing, which is what was between them. Before this the splitter broke only at whitespace, and 420 characters of 今天天气很好。 reached a 300-character engine as one piece of 420. A run of Han or kana with no punctuation at all is cut at the limit: there a word can end at any character, and the alternative is a piece of any length. Korean spaces its words and is split like Latin script.
  • The size is the adapter's. 180 characters for Orpheus, 250 for Dia, 200 for Kokoro, 300 for Chatterbox. The number is a fact about the model's attention and its generation budget; the splitting is not. An adapter that declares no size receives the whole text.
  • The joint is the adapter's too. Kokoro puts 250 ms of silence between groups. Dia puts none, because its next piece continues from the audio of the last. An adapter that needs to carry state across a boundary keeps its own loop and calls the shared splitter for the text alone.

The capability document declares the split, because a client cannot see it and is affected by it. segmentation.segmentCharacters is what one generation gets, beside maxCharacters for what the request may carry. supported: false means text over maxCharacters is refused rather than split: where every engine stood before this rule, and where an engine whose take is indivisible stays. Each size was arrived at on the engine that declares it. Chatterbox's was the last, held back until the 40-second ceiling above had cost a bulletin, because guessing it would have been the kind of unpaid-for rule this document does not keep. 300 is about 19 seconds at the 16 characters a second that bulletin's voice read at: under half the ceiling, so a slower voice still fits.

A client needs this for two reasons and § 4 is the place that owes it to them. A seed reproduces a generation, so a request split four ways is four seeded generations rather than one. And prosody does not carry across a joint unless the engine carries it, which is a thing engines differ on and therefore a thing this document says out loud.

fetch, the one optional verb​

fetch(variant) downloads a variant's weights to wherever the engine keeps them, without putting them on the device. The SDK serves it as POST /fetch with { "variant": "turbo" }, and an adapter that does not override it answers unsupported. It exists because the alternative is the first /speak doing the download: Chatterbox's turbo is 3.8 GB and took about 75 seconds on a first load, and a caller waiting on one utterance cannot tell that from a hang. An engine whose weights ship inside its package, or that has none, leaves it alone.

fetch may bring upstream's code as well as its weights, where that code cannot ship in a package. Fish Audio S2 Pro's inference code is fish-speech, which cannot be installed as a package: its pyproject depends on pyaudio, which needs PortAudio's headers and a compiler that the server image does not have, though nothing that runs inference imports it. And its code is under the same research licence as its weights, so a copy in a rhapsode package would be rhapsode distributing it. So the adapter fetches upstream's archive at one pinned commit, keeps only the package and its licence, and checks the unpacked files against a pinned digest of them rather than the archive's bytes, which GitHub does not promise to keep the same. The operator's box downloads it from upstream, after an install that accepted that licence by name (§ 10). The adapter's own dependencies are the ones inference imports, found by importing it.

compiles, a variant whose first load builds what later loads reuse​

A variant declares "compiles": true when its first load on a box compiles kernels into a cache on disk that every later load reads instead. Absent means it does not, which is every engine but one. Orpheus's full is that one: vLLM compiles the graph with Inductor on a cold cache, and on an RTX 4070 Ti SUPER that first load took 35 seconds against 12 for every load after it, and held 8.05 GB until it was unloaded against 7.26 GB. It is declared rather than left to the first /speak because the core can pay it at install (§ 10), when nobody is waiting on an utterance, and only the adapter knows its runtime does this. The SDK spells it Variant(compiles=True).

dialogue, which is declared rather than discovered​

dialogue(request) speaks a conversation (§ 6) and yields PCM as speak does. An adapter that overrides it has every variant declare dialogue with its max_speakers, and one that does not answers /dialogue with unsupported and declares nothing. Whether a variant has it is derived from the override, as whether an engine clones is derived from create_voice, so an adapter cannot claim dialogue it did not write. The SDK checks the speakers, the voices, the ceiling and everything /speak checks before dialogue() is called.

Unlike fetch, a client needs to know before it asks, because the answer decides what it offers: the web page shows a conversation editor only where a variant declares one.

An ONNX engine is an adapter like any other​

Piper, Kokoro and Supertonic ship ONNX weights, and onnxruntime-node could run them inside the core with no Python at all. They run as workers instead, through onnxruntime in their own virtualenv, and the core has one kind of engine. This was open until v1 (§ 14) and was decided on what running them in-process would actually have cost:

  • The model is the easy half. These models take phonemes, not text, and every maintained route from one to the other goes through eSpeak NG, which is GPL-3.0-or-later, in Python and in Node alike. In a worker it runs in a process and a virtualenv of its own, behind a catalog entry that shows the licence before install (§ 4). In the core it would be loaded into an MIT server that holds no engine knowledge, which is the rule § 1 rests on.
  • onnxruntime-node is 301 MB unpacked, and would be a dependency of every core, including the ones that never run an ONNX engine.
  • The benefit was a first run with no Python, and the installer (§ 10) already builds each engine its own virtualenv with uv. The gain that remains is a first engine that needs no GPU and a download measured in megabytes, and a CPU worker delivers that.

Phonon is the same case with a different runtime. ptts is a Rust library behind Python bindings, needs neither torch nor onnxruntime, and is a worker for the same reasons, on the same CPU footing.

What this asks of the SDK is only what it already promises: it must never depend on torch, or an ONNX adapter could not install it. device is cpu for such an engine and means it; nothing in this document treats CPU as a degraded case.

What an adapter must do honestly, and it is only one thing​

Claim only what the loaded variant can actually perform. Every guarantee in this document rests on that and nothing else checks it.

9. Versioning​

contract is an integer in the handshake, in /capabilities, and in the public API.

  • Changes are additive only within a major. New optional fields, new enum members at the end.
  • The core offers its own maximum in RHAPSODE_WORKER_CONTRACT at spawn.
  • The worker answers at the highest version it supports that is not greater than the offer, and reports that number as contract in its handshake line and in /capabilities.
  • If the offer is below everything the worker supports, the worker exits non-zero before printing a handshake, naming both numbers on stderr.
  • The core refuses a worker whose reported contract exceeds its own, with a message naming both.
  • Clients ignore fields they do not know, and must not fail on an unrecognised cue or dial name.

The negotiation is one-sided on purpose: the core states a ceiling and the worker picks under it. The alternative, both sides declaring a range and meeting in the middle, needs a range on the wire, and there is nowhere to put one that an old core would understand. A single integer in each direction is enough because the only question that has ever needed answering is "can you speak the version I speak", and additive-only evolution means the answer is yes for every version at or below the offer.

The core and its workers ship separately, so they will disagree in the field. Designing for that on day one costs about forty lines. Retrofitting it costs a client that sniffs your OpenAPI document to work out what you accept, which is a real thing a real client had to do against a real server, and is the specific future this section exists to prevent.

Package versions are not the contract​

Every released package carries one version, and they release together. The server ships as a Docker image tagged with it, and @maroonedsoftware/rhapsode-sdk is published to npm at it. The rest carry it without being published: the workspace packages (@rhapsode/contract, @rhapsode/core and rhapsode), because the core reads its own to pin engines, below, and the Python packages (rhapsode-worker, rhapsode-conform and every engine in the catalog), because every way to run the core today installs engines from the sources it ships with, so nothing reads the package index yet. Versioning them anyway means publishing one later is a release like any other, not a renumbering. The first release is 0.1.0.

That version and contract answer different questions. contract says what a core and a worker can say to each other, and moves only by the rules above. The package version says which code this is. A release that changes no shape leaves contract where it was; a new contract major always comes with a new package version, and the reverse is almost never true. A client reads contract to decide what it may send and never parses a package version to find out.

The core reports its own package version as version on GET /health, beside contract, so that a page can show a person which release this box runs without fetching and parsing /openapi.json. It is for display. Every question that would need two versions compared is answered by the core: whether an engine is behind it (outdated, below) and whether a newer release exists (GET /update, below). A client that compared version strings itself would need its own ordering of them, and two clients would disagree about a prerelease.

One version rather than one per package because § 14.5 keeps the protocol and both of its implementations in one repository so that they move in one commit, and that holds for a user only if they also move in one release. Separate versions would need a hand-kept table of which core goes with which engine, and the catalog would have to consult it on every install.

An engine installed from the package index is pinned to the core's own version. A core at 0.3.0 installs rhapsode-engine-kokoro==0.3.0, and every engine pins rhapsode-worker to its own version exactly. Unpinned, pip resolves the newest engine on the index, which may have been built for a core this box does not have yet. Negotiation would refuse it, but only at the first spawn, after the virtualenv, the install and possibly gigabytes of weights were already paid for. The cost of the pin is that an engine fix reaches a server only with a core upgrade, which under one version is how every other fix reaches it too.

None of this narrows what the rest of this section allows. A remote worker, or an engine the operator configured by hand, is whatever version it is, and negotiation is still what decides whether the core will speak to it.

The pin is made at install and broken by an upgrade​

An install pins, and nothing re-pins. The virtualenvs outlive the core that made them, deliberately: they are gigabytes, they sit on the volume an upgrade is careful to keep, and rebuilding them on every release would make upgrading cost what installing cost. So a core that has been upgraded is talking to workers pinned to the version it replaced, and that is the ordinary state of an upgraded box, not an error.

Negotiation does not catch it and is not supposed to. It refuses a worker whose contract is too new, because that one cannot be spoken to at all; a worker that is too old speaks a contract this core still supports, which is exactly what additive-only evolution promises. What the operator loses is quieter: every field added inside the contract major since that worker was built is a field it never sends, and an optional field nobody sends is indistinguishable from one nobody could answer. A core two releases ahead of its worker reported no modelBytes (§ 3) and read as a card that could not be measured.

So the core reports, per engine, the version of rhapsode-worker installed in that engine's virtualenv, as workerVersion on the engine summary of GET /engines and GET /health.

  • It is a diagnostic and never an input to negotiation. Nothing in the core or in a client may branch on it, and the paragraph above stands unchanged: a client reads contract to decide what it may send and still never parses a package version to find out. This field exists so that a human, or a doctor acting for one, can see a pin that an upgrade has broken.
  • It is read from the virtualenv, not asked of the worker. A worker would have to be running to answer, and the normal state of an engine is down — a model loads when something speaks and leaves when nothing does (§ 3). A version visible only while an engine happens to be loaded would be absent exactly when the question is asked. Reading the installed distribution's metadata needs no process, so it answers for a down engine, which is the case that matters.
  • It is absent rather than guessed. A remote engine has no virtualenv here, an operator-configured command may point anywhere, and a virtualenv whose metadata cannot be read is one the core should not invent a number for. Absent means "not something this core can say", the same way a missing modelBytes means "nothing measured it".

A workerVersion that differs from the core's own version is the signal, and the core says so itself: the same summaries, and each entry of GET /catalog, carry outdated: true for an engine whose workerVersion is not this core's version, false for one that matches, and nothing where workerVersion is absent. It is "differs", not "is older": a core rolled back to an earlier release breaks the pin the same way, and the remedy is the same. Like workerVersion, it is computed when the engine is declared and not per request, so it adds nothing to the path a healthcheck polls.

The remedy is to reinstall that engine (§ 10), which builds a new virtualenv beside the old one and does not take the engine away while it does. It is worth a warning and never a refusal, and it is never done unless the operator asked for it: the worker is contract-legal, it works, a reinstall costs a cold load and the network, and an operator who has reasons to run an engine at another version is inside what this section allows. Asking once is enough: with update.reinstallOutdated on, the core reinstalls every engine behind it each time it starts (§ 10, "Reinstalling"). It is off by default, and the Docker image turns it on in the config it writes on first boot.

The core asks whether it is current​

GET /update answers whether a newer release of rhapsode exists. It is open to every caller, like /health and /catalog: the version it reports is already info.version on /openapi.json, the latest release is public, and the only other thing it says is whether this box checks at all.

{
  "version": "0.1.9",
  "check": "ok",
  "latest": "0.2.0",
  "updateAvailable": true,
  "releaseUrl": "https://github.com/maroonedsoftware/rhapsode/releases/tag/v0.2.0",
  "checkedAt": "2026-09-21T09:14:02.000Z",
  "distribution": "docker"
}

  • check is off when the operator turned it off, pending before the first answer has come back, ok once one has, and failed when the last attempt did not get one. latest, updateAvailable, releaseUrl and checkedAt are present only with ok.

  • The core asks GitHub's API for the repository's latest release, which by GitHub's own definition is neither a draft nor a prerelease, so a prerelease is never offered. The request carries a User-Agent naming rhapsode and its version, and nothing else about the box: no token, no engine list, no identifier.

  • It asks at most once a day, and only when asked. A request finding no answer, or one older than a day, starts one check in the background and answers at once with what it has, so a first call after boot reads pending. A failed check is retried after six hours. There is no timer: a box nobody asks never phones out, and there is nothing to stop at shutdown. The route never waits on the network, and nothing about the check reaches /health, whose answer a container's healthcheck turns into a restart.

  • updateAvailable is the core's comparison, of x.y.z release versions. A core newer than the latest release (a checkout, or an image built locally) is not offered a downgrade, and a latest it cannot parse offers nothing.

  • distribution is docker in the published image and source everywhere else. It says which instructions to show, because the core cannot follow them itself.

  • upgrade is those instructions for this box exactly, present with updateAvailable in the image when compose told the container which image it named (RHAPSODE_IMAGE, which compose.yaml sets from its own image:). image is that reference, version is what to set RHAPSODE_VERSION to in .env first, absent when the tag already takes the release, and commands are what to run after it, in order, beside compose.yaml:

    "upgrade": {
      "image": "maroonedsoftware/rhapsode:0.1",
      "version": "0.2",
      "commands": ["docker compose pull", "docker compose up -d --wait", "curl -fsS -X POST http://127.0.0.1:8081/api/installs/outdated"]
    }

    The tag decides version. A release (0.1.22) becomes the latest release, a minor line (0.1) stays where it is when the latest release is on it and becomes the latest's minor line when it is not, and latest stays. Any other tag (local, a digest) gives no upgrade, because the box was not built from a published image and nothing here knows how it is rebuilt. The curl is left out when update.reinstallOutdated is on, because the start does it (§ 10). It is the core's to work out rather than the page's because it orders versions, which no client does, and because a generic "change the pin if you pinned it" is what an operator read on a box whose pin and registry were both in the way, and the upgrade did nothing. A compose.yaml from before RHAPSODE_IMAGE sends nothing, and gets no upgrade: the instructions for it are the generic ones.

It is on by default, because the failure it exists to prevent, a box that quietly stays several releases behind, is invisible from inside the box, and a check that has to be turned on is one only the operators who already watch releases will turn on. update.check: false in the config or the settings (§ 10), or RHAPSODE_UPDATE_CHECK=0 in the server's environment, turns it off, and the route then answers check: off without ever having asked. The environment can only turn it off: a setting of true does not override RHAPSODE_UPDATE_CHECK=0, because whoever set the environment decided the box would not phone out, and a page is not the place to overrule them.

POST /update/check asks now. A day is the right wait for a box nobody is looking at and the wrong one for a person who has just read that a release is out: on the box this was written for, 0.1.10 was published at 11:50 and the answer the core held from before it said 0.1.9 until the next morning. So a caller can ask for a check, which answers with the same document as GET /update once GitHub has, waiting at most the check's own ten-second timeout. It takes no body.

  • It is open to every caller, like GET /update, because the person asking is often at a page opened from another machine, and the answer is public.
  • It cannot make the box phone out more than once in five minutes. An answer less than five minutes old is returned as it is, and a request arriving while a check is in flight waits for that one. That is what lets an open route start outbound traffic: pressed as fast as anyone can, it asks GitHub twelve times an hour, a fifth of the sixty GitHub allows an address without a token, so a box behind a shared address cannot spend its neighbours' allowance.
  • It does not override off. With the check turned off it answers check: off without asking, for the reason the environment cannot turn it back on: somebody decided this box does not phone out, and a button is not the place to overrule them.
  • A failure is an answer, not an error. GitHub unreachable is check: failed with a 200, exactly as GET /update reports it, and the next background check follows the six-hour retry as before. An answer from here stands for a day like any other.

The core does not update itself. In the image it could only do so with the Docker socket, which is control of the host, and the one mechanism every operator already trusts is docker compose pull. So the core says that a release exists, and once the new image is running, outdated says which engines to reinstall.

The core describes its own API​

GET /openapi.json answers with the public API as an OpenAPI 3.1 document, generated from contracts/ like every other shape. It is open to every caller, like /health and /catalog: it says nothing a caller could not learn by trying, and withholding it only sends people to a copy on the web that describes some other version.

  • Public routes only: § 6 and § 7, the management routes of § 10, and the OpenAI shim of § 11. Never a worker route. A client author should not learn that workers exist, and one document for both cannot even be written, because /health and /speak are different operations on the same paths. The first combined document had the worker's in both places and described a /speak that takes no engine.
  • info.version is the running core's package version, set when the document is served, so it describes this box rather than whichever build published the docs. It is still not the contract. A client reads contract from /health to decide what it may send, and reads this document to learn the shapes, whether it is a person or a generator building a client. Nothing negotiates from it.
  • The worker protocol is described by docs/openapi.worker.yaml, committed and served by nobody. An engine author reads it next to rhapsode.worker.ck, and a worker is not required to describe itself: rhapsode-conform is what says whether it speaks the protocol.

10. Managing engines​

Installing an engine is part of the API rather than a set of instructions, so that a terminal client and a web page can both drive it and neither holds logic the other lacks. By hand it was five steps: a virtualenv, a pip install, an edit to the config, a restart, and a first /speak that silently downloaded 3.8 GB. The core still ships no interface of its own (§ 13). These routes are what one is built on.

RouteDoes
GET /catalogEvery engine that exists, installed or not, with both licences
POST /engines/{id}/installStarts an install job; 202 with the job. ?pull=turbo fetches that variant too, and ?accept= names the weights licence where it must be accepted
POST /engines/{id}/reinstallStarts a job that rebuilds an installed engine beside itself and swaps it in; 202 with the job. ?accept= as for install
POST /installs/outdatedStarts a reinstall for every outdated engine this API installed; 202 with the jobs and what it skipped
DELETE /engines/{id}Stops and removes an engine this API installed
POST /engines/{id}/pullStarts a job that downloads a variant's weights; body { "variant": "turbo" }
POST /engines/{id}/unloadFrees the model now (§ 3); ?mode=unload keeps the process
GET /installsEvery job this process knows about, newest first
GET /installs/{job}One job
GET /installs/{job}/eventsThe job's progress as server-sent events
GET /settingsEvery setting: the value in use, where it came from, and whether a change to it applies now
PATCH /settingsChanges some of them; those that can apply now do, the rest wait for the next start

Who may call them​

Management routes answer loopback callers, and nobody else unless a token is configured. With management.token set, a caller presenting Authorization: Bearer <token> is admitted from anywhere. Everything else gets forbidden. GET /catalog is the exception and answers everybody, because it only reads and the licences in it are the thing § 4 promises before install.

The reason is that the server has no other authentication and binds every interface by default, and an install runs pip. A management route open to the LAN is remote code execution for anyone on it. The speech routes stay open because the worst a stranger can do with them is make it talk.

A web page counts as where it came from, not where it is running. Any page the operator visits can make their browser send a request to localhost, and the browser is a loopback caller: a cross-site POST /engines/tone/install from Origin: https://evil.example was measured answering 202 and installing. So a request carrying an Origin is refused unless that origin is this machine's own (localhost, 127.0.0.1, [::1], any port) or is listed in management.origins, and that holds with a valid token too. A browser attaches Origin to every cross-site request and every POST, and a page cannot forge it, which is also what defeats DNS rebinding: a rebound page's origin is still the attacker's hostname. Clients that are not browsers send no Origin and are judged on the rest.

A proxy on this machine does not make its clients local. The web app's dev server and nginx both connect from loopback, and without this rule a web app served to the LAN would hand the LAN the install routes. A request from a loopback peer carrying X-Forwarded-For is local only if every address in that header is. The header is believed only from a loopback peer, and it can only make a caller less trusted: leaving it out gains a local caller nothing it did not already have.

The guard runs before the request body is read and before any stream opens, so a refused caller is refused cheaply and cannot hold a connection.

Settings​

Everything in the config file can also be changed through the API, so that a person can configure the box from a page or a terminal without editing a file and restarting. Configuring used to mean finding the file (inside a container, on a volume), editing JSON by hand with no check that a value was allowed until the next boot refused it, and a restart that dropped every model on the card whether or not the change needed one.

Where a value comes from. Three layers, each over the one before: the defaults, the operator's config file, and the settings in the state database (below). The database wins. The file is what a box starts with, the database is what it has been told since, and PATCH /settings writes only the database. A setting cleared from the database falls back to the file's value, and then to the default. The file is still never written by the core, for the reasons under "The state database", and it still has to exist first: it is what says where the database is.

The reverse order, the file over the database, was the rule for engines and was considered here. It makes every value the file names read-only on the page, and the file is exactly where a first boot puts values (the Docker image seeds four), so the settings a person would most want to change are the ones a page could not.

GET /settings answers the whole document:

{
  "values": {
    "server": { "port": 8080, "host": "::", "shutdownGraceMs": 20000 },
    "log": { "level": "info" },
    "residency": { "maxResidentModels": 1, "evictionWaitSeconds": 30, "keepAliveSeconds": 300 },
    "workers": { "socketDir": "/tmp/rhapsode", "voiceDir": "/config/voices", "startupTimeoutSeconds": 60, "drainGraceMs": 10000, "maxRestarts": 5, "restartDecaySeconds": 300 },
    "install": { "venvDir": "/data/.rhapsode/venvs", "sourceDir": "/app/python", "python": "3.12" },
    "management": { "tokenSet": true, "origins": [] },
    "update": { "check": true, "reinstallOutdated": false },
    "engines": { "kokoro": { "keepAliveSeconds": 900 } }
  },
  "fields": [
    { "key": "residency.keepAliveSeconds", "source": "default", "applies": "live" },
    { "key": "server.port", "source": "config", "applies": "restart", "saved": 9090 },
    { "key": "engines.kokoro.keepAliveSeconds", "source": "database", "applies": "live" }
  ]
}
  • values is what the running process is using, not what is saved. A setting that waits for a restart shows its old value here until the restart.
  • fields has one entry per setting, by dotted key. source is the layer the value came from: default, config or database. applies is live or restart. saved is present only when the database holds a value the process is not using yet, which is a change waiting for a restart, and a client says so.
  • The token is never in it. management.tokenSet says whether one is configured, and a new token waiting for a restart is saved: true on its field rather than the token. The token is a root password for the box (above), and a document read to draw a form is not a place to carry one.
  • engines has an entry for each engine in the registry, holding the settings that are per engine. The engine's entry itself (its virtualenv, its module, its env) is not a setting: it is what the install recorded or the operator wrote, and it is changed by reinstalling or by editing the file.

PATCH /settings takes any part of values, and answers the whole document as it stands after the write. null clears a setting from the database. management.token is written as a string, management.tokenSet is never written, and a key the document does not have is bad_request, so a misspelt setting is refused rather than saved and ignored. An array replaces the one below it rather than adding to it: management.origins set in the database is the whole list. The write is one transaction, so a patch that is refused changes nothing, even where some of its keys were allowed.

What applies now, and what waits. Six settings apply as soon as they are written: residency.maxResidentModels, residency.evictionWaitSeconds, residency.keepAliveSeconds, each engine's keepAliveSeconds, update.check and update.reinstallOutdated. They are the ones the core reads when it uses them rather than once at boot (update.reinstallOutdated is used at a start, so a change to it is read by the next one without waiting to be applied), and they are the ones a person tunes while watching a box run: a keep-alive is set by seeing how long a model sits idle on the card, and a restart to try a value would drop the model being measured. A change to either keep-alive moves the deadline of every model idle now, rather than from its next request, because a deadline nobody sees move reads as a setting that did nothing. A lower maxResidentModels evicts nothing on its own: the next load that needs the room evicts, by § 3's rules, exactly as it would have.

Everything else waits for the next start, for a reason each:

  • server.port and server.host are the socket the request that changed them arrived on.
  • workers.* and install.* are read by a worker or an install as it starts, and several are directories that engines and voices already live in. Changing one moves nothing: an engine installed under the old install.venvDir runs from there until it is reinstalled, and voices under the old workers.voiceDir are not found under the new one. Move them first.
  • management.token and management.origins are what the Docker image's proxy is built from when the container starts (it presents the token for the page). Applied live, the core would refuse the page its own proxy serves until the next start, which is the same restart later with a broken page in between.
  • log.level and server.shutdownGraceMs are read once, when the server is built.

Refusals. A value out of range is bad_request, naming the setting: a port outside 1 to 65535, a keep-alive below -1, a maxResidentModels below 1, a negative wait or grace, a log.level outside error, warn, info, debug, trace, an origin that is not an absolute http or https origin with nothing after the host and port. A per-engine setting for an engine that is not in the registry is unknown_engine. The contract's numbers are coerced, as its booleans are (§ 10, "Installing"), so "8080" is read as 8080; null is not a number and always means "clear".

A server started without a state database (a core embedded as a library can be) answers unsupported, because it has nowhere to keep a setting, as it has nowhere to record an install.

A change that would lock its caller out is conflict. Two, and both are checked before anything is written:

  • A caller that is not local may not clear the token or set it empty. It could only have been admitted by the token, and without one the management routes answer loopback callers only, so its next request, and every one after, would be forbidden.
  • A caller whose Origin is not this machine's own may not leave that origin out of management.origins. Its page would be refused from the next start on. A page on this machine is admitted by its origin whatever the list says, so it is never refused this.

Neither rule stops an operator: at the machine, or from a page they have listed, both changes are allowed, and the file and the database are theirs to edit by hand.

Both routes are management routes, behind the guard above, reading included. GET /settings names directories on the box, the origins it trusts and whether it has a token, which is a map for somebody deciding where to push next, and a page that cannot change settings has no use for it either. The web page and pnpm wizard settings are the two clients, and the page has nothing the wizard does not (§ 13).

The catalog​

[
  {
    "id": "chatterbox",
    "displayName": "Chatterbox",
    "license": { "code": "MIT", "weights": "MIT", "weightsCommercialUse": true },
    "package": "rhapsode-engine-chatterbox",
    "defaultVariant": "turbo",
    "installed": "yes",
    "managed": true
  }
]

installed is no, installing or yes. managed says whether this API installed it and can therefore remove it; an engine the operator configured by hand is installed: yes, managed: false. GET /engines is unchanged and still lists only what is configured: the catalog is what exists, and /engines is what this box has. An installed engine's entry also carries workerVersion and outdated as its engine summary does (§ 9), so a page listing the catalog can say which engines an upgrade left behind without a second request. An engine this API installed carries warmed as well, the variants recorded as warmed (below), absent when none has been.

Installing​

An install is four steps, and a job reports which one it is on:

  1. venv: create <install.venvDir>/<id>, with uv venv when uv is on the path and the interpreter's own venv module otherwise. A directory already there that no registered engine points at is the remains of an install that did not finish, and is removed first, and so is <id>.alt, the second slot a reinstall uses (below). An engine whose dependencies install on a narrower range of Python than the SDK's says so in its catalog record. uv is handed the range and finds or fetches an interpreter inside it; without uv, the configured interpreter is asked first and the job fails naming both, because pip given an interpreter outside the range does not fail. It resolves the newest release that still claims to support it: kokoro-onnx 0.6.1 declares <3.14, and on 3.14 pip quietly installed 0.4.7.
  2. packages: pip install the adapter. If <install.sourceDir>/<package> exists it is installed from there, whatever version the directory holds; otherwise it is installed by name from the package index, pinned to the core's own version (§ 9). The first is how a checkout and the server image work, and the second is how an installed core works once adapters are published. An engine whose upstream has no packaging names companions in its catalog record: distributions found the same way and installed in the same resolve as the SDK and the adapter. Breeze TTS 2's inference code is a repository with no pyproject, which pip cannot install, so rhapsode-vendor-breeze is a copy of it at one commit with a pyproject added, and the Breeze adapter depends on it by name like any other package.
  3. verify: import the engine's module with the new interpreter. That is the command the core will spawn, minus serving, and it is the check that a virtualenv pip abandoned halfway fails, although its bin/python runs perfectly well.
  4. register: record the engine in the state database, with the weights licence the install accepted, and add it to the running registry. No restart: the next /speak for it spawns a worker.
  5. weights, only when asked: fetch a variant, exactly as a pull does (below).
  6. warm, only after step 5, and only for a variant that declares compiles (§ 8): load it and unload it, so the compiling is done before anybody asks to be spoken to.

An install can fetch its weights. POST /engines/chatterbox/install?pull=turbo adds step 5 for the variant named; a client that wants the default reads it from the catalog's defaultVariant. Without it an install stops at register, as it always has. A pull that is empty or repeated is bad_request. It is one job rather than a client queueing a pull behind its install, because a client that goes away between the two (a closed browser tab, a terminal that lost its connection) leaves an engine whose first /speak sits through the whole download: 3.8 GB for Chatterbox's turbo.

It is a query parameter rather than a body so that a bare POST, which is every install so far, stays one. ServerKit refuses a request without a body on a route that declares one, with a 411, so a body here could not have been optional. And it names a variant rather than being a flag, because the contract's booleans are coerced and ?pull=false would have read as true.

The engine is registered before step 5 starts, so a failed download leaves it installed: the job fails at weights, and the way on is a pull, not another install. An engine whose worker does not implement fetch has nothing to do in step 5; the job says so in its output and succeeds, and the weights arrive on first load as they would have.

An install warms what compiles. Where the variant step 5 fetched declares compiles, step 6 loads it and unloads it again, through residency exactly as a /speak would load it: it takes a slot, it may evict an idle model to get one, and it waits for one behind models that are speaking, for as long as a /speak would. Unlike a pull it therefore costs the card for as long as a load takes, which is why it happens only for a variant that says it needs it. The model is unloaded rather than left resident, because a load that compiled holds what compiling took until it unloads (0.8 GB for Orpheus full), and the next load, from the cache, holds only what the model needs. A load that fails here fails the job at warm and leaves the engine installed with its weights: it is the load every /speak would have tried, and finding out now is the point. The variants an install warmed are recorded with the engine as warmed, for a reinstall to warm again (below).

RHAPSODE_PIP_TRUSTED_HOSTS is honoured from the server's own environment and cannot be set through the API. Weakening certificate verification stays a decision made on the box.

Installing an engine that is already installed or already installing is conflict, and so is installing one the operator's config names, even disabled: that entry would win over whatever the install recorded. An id that is not in the catalog is unknown_engine. A server started without a state database (a core embedded as a library can be) answers unsupported, because it has nowhere to record the result.

Accepting a weights licence​

An install of weights that may not be used commercially names the licence it accepts. Where the catalog says weightsCommercialUse: false, the install carries ?accept= with the catalog's weights string exactly, ?accept=CC-BY-NC-4.0, and without it is refused as bad_request before a job exists, in a message naming the licence and the query that accepts it. An accept that is empty, repeated, or names anything else is refused the same way. An engine whose weights are commercial needs none, and an accept sent to one must still name its licence, so a client can send it on every install without first working out which kind of engine it is.

It is enforced here rather than left to the clients because § 4 put the licence in the catalog to be read before install, and a promise kept only by the clients that happen to show it is not kept for curl, the SDK, or the next client somebody writes. The web page and the wizard showed both licences and asked; a script calling this route installed whatever it named and never saw either.

It names the licence rather than being a flag, for the reason pull names a variant: the contract's booleans are coerced, and ?accept=false would have read as true. Naming it also makes the acceptance of the licence the client displayed. A catalog that changed between the read and the install, because the server was upgraded and the weights relicensed, refuses the install rather than accepting terms nobody was shown.

It is asked only where the weights are not for commercial use. A condition on every install is one a client learns to satisfy without reading, and the refusal is worth something only while it is rare. A pull asks nothing: the engine it fetches for was installed, and its licence accepted, or it was configured by the operator, whose own file is their decision. A client still does not accept for a person. The web page and the wizard send accept only once somebody has seen both licences and said yes, and the wizard with nobody at the terminal accepts only when given --yes.

The state database​

The core keeps what it has been told in rhapsode.db, a SQLite database beside the config file: the engines it installed, and the settings written through PATCH /settings. The core never writes the operator's file. That file may carry comments that a rewrite would lose, may sit on a read-only path, and is somebody's hand-kept record of the box. One writer per file is the rule that keeps both honest.

It is a database rather than the JSON file it replaces, rhapsode.engines.json, because there are now two writers, installs and settings, and a file rewritten whole by each needs both to hold one lock to avoid losing the other's write. SQLite gives each a transaction, and it ships with Node, so it costs no dependency and no build step. It is not a database server: rhapsode is one process on one box, and a second service to install, upgrade and back up would buy nothing that one file does not.

  • It keeps its journal only while writing. A backup of the config directory taken while the core runs is then the database as of its last write. Write-ahead logging would leave the latest writes in a second file beside it, which a copy of the one file misses.
  • It is opened, and written to once, at boot, so a database the core cannot write to fails the start naming its path, rather than the first install hours later.
  • A core that finds rhapsode.engines.json beside the config imports it into the database in one transaction and renames it rhapsode.engines.json.imported. It is left there rather than deleted so that a downgrade has something to go back to.
  • It records the version of the core that last opened it, as coreVersion, and a core that finds another version there, or none, copies the config directory's state first, into rhapsode.backups/<time>-before-<version>/ beside the config: the database, the config file, a rhapsode.engines.json that has not been imported yet, and the voices directory when there is one beside the config, as the Docker image keeps it. That is everything an upgrade's migrations could change and the one thing (a cloned voice) that cannot be downloaded again. It happens before the database is opened for writing, so before any import or migration, and the three newest copies are kept. Operators were told to docker cp the config directory out before every upgrade, about 6 MB on a box with cloned voices, and a step that guards against a rare failure is the first one skipped. The database is copied with VACUUM INTO, which is consistent even while another process holds it. A copy that fails is logged and does not stop the start: refusing to boot is a worse outcome than the one the copy guards against.

Engines keep the rule the file had: where the operator's config and the database name the same engine, the operator's config wins. It is the opposite of the rule for settings, on purpose. An engine entry is what makes an engine exist, and an operator who configured one by hand is the only party who can remove it (an uninstall refuses theirs, below). Were the database's entry to win, an install over an operator's engine would replace it until somebody opened the database, and removing the engine from the file would no longer remove it. A setting only tunes something that exists.

An engine's entry is an engine entry like one in the config, plus one field the core keeps for itself: accepted, the weights licence the last install or reinstall accepted, as the accept query named it, and absent when none was needed. And warmed, the variants step 6 of the install or a warm since (below) warmed, absent when nothing has been. Boot ignores both. A reinstall reads both (below).

Uninstalling​

DELETE /engines/{id} terminates the worker, removes the engine from the registry and from the state database, along with its per-engine settings, and deletes its virtualenv, and the other slot's if one is there, but only a virtualenv inside install.venvDir. An engine the operator configured is conflict: it is theirs to remove, by editing their file. So is an engine that is speaking, because an uninstall that cut a stream off would hand that caller a truncated file for a reason it could not have predicted; the removal happens under the same lock a load takes, so nothing can start one in between. An engine that is not installed is unknown_engine.

Weights are left alone. They live in the engine's own cache (for Chatterbox, the Hugging Face cache in the user's home), which other tools on the box share, and 9.7 GB is not something to delete as a side effect.

Reinstalling​

POST /engines/{id}/reinstall rebuilds an engine this API installed, from what this core ships, and swaps the new virtualenv in once it works. It is how an engine left behind by an upgrade (§ 9) catches up, and how a virtualenv somebody broke is mended. It answers 202 with a job whose kind is reinstall and whose steps are an install's first four, then warm for what the install warmed; there is no weights step, because the weights are in a cache the old virtualenv never owned and are still there.

It is its own route rather than an install over the top because the two refuse opposite things: an install refuses an engine that is installed, a reinstall one that is not. Before this route the remedy was an uninstall followed by an install, two calls with a stretch between them where the engine did not exist and a failed install left nothing at all.

The new virtualenv is built beside the old one, and the engine keeps working until it is ready. An engine has two slots, <install.venvDir>/<id> and <install.venvDir>/<id>.alt, and the venv in its state database entry says which one it runs from. A reinstall removes whatever is in the other slot (the remains of one that did not finish), builds there, and verifies there, so the interpreter it imports the engine with is the one the core will spawn. A virtualenv is never moved: it is not relocatable, and every script in its bin names its own path in its first line, so a renamed one runs bin/python and breaks bin/pip. Two fixed slots rather than a directory per version, because a reinstall at the version already installed (every retry, and every checkout) would otherwise build over the virtualenv the engine is running from.

A failure in any step before register fails the job and changes nothing: the engine, its virtualenv and its workerVersion are as they were.

register waits for the engine to stop speaking, rather than refusing. It takes the lock a load takes, as an uninstall does, and while the engine holds a lease it waits for the lease to end, for up to ten minutes; past that the job fails with conflict and nothing has changed. An uninstall refuses instead, because its caller is present to be told; a reinstall's caller has usually gone, and on a busy box a reinstall that refused whenever the engine was speaking would never finish. Under the lock it records the new slot in the state database, stops the old worker, and declares the engine again, which reads the new workerVersion and outdated. The database is committed first, so a core that dies at any point after it boots on the new virtualenv, and one that dies before it boots on the old. The old slot is deleted last.

Then it warms again what the install warmed. A reinstall is how an upgrade reaches an engine, and an upgrade that moves the compiler or the runtime moves the key the compile cache is filed under, so the first load after it would compile from cold. Each variant in the engine's warmed that the new worker still declares compiles for is loaded and unloaded as step 6 of an install does it, after register. Only those: warming every variant that declares compiles would download weights the box never pulled, which is the side effect the missing weights step exists to avoid. A failure here fails the job at warm with the new virtualenv already in place, because the swap is done and undoing it would put back the engine the upgrade was meant to replace.

What it costs, and what a client should say before asking for it:

  • A cold load. The old worker is stopped, so the model it held leaves the card, and the next /speak for the engine loads it again.
  • Anything installed into the virtualenv by hand. A reinstall builds from the catalog record, so a package an operator added to the old virtualenv with its pip is not in the new one.

The refusals are an install's, reversed, and all of them come before a job exists: an engine that is not installed, or is not in the catalog, is unknown_engine; one the operator's config names is conflict, as for uninstall; and one with a job queued or running is conflict. A server without a state database has installed nothing, so every engine it has is the operator's.

A reinstall asks for the licence only when what was accepted no longer covers it. Where the catalog says weightsCommercialUse: false, a reinstall needs no accept if the engine's recorded accepted is the catalog's weights exactly, because somebody accepted those terms and nothing since has changed them. It needs one, and is refused as an install is without it, when the entry has no accepted (an engine installed by a core that did not record one) or names another licence (the catalog relicensed the weights, which is the case the rule under "Accepting a weights licence" exists for: an upgrade is how a catalog changes). An accept that is sent is checked as an install's is, and recorded.

POST /installs/outdated reinstalls everything behind. It queues one reinstall for each engine this API installed whose outdated is true, in id order, and answers 202 with them:

{
  "jobs": [{ "id": "01J8Z6Q4B7", "engine": "kokoro", "kind": "reinstall", "state": "queued", "createdAt": "2026-09-21T09:20:00.000Z" }],
  "skipped": [{ "engine": "research", "reason": "licence" }]
}

It never refuses as a whole. An engine it cannot reinstall without asking somebody is skipped with a reason: licence where a single reinstall would need accept, busy where a job for it is already queued or running, uncatalogued where the catalog no longer has it. Nothing behind is jobs: [], which is a success, so the call can sit at the end of an unattended upgrade and be run every night. It is a route rather than a loop in each client because the question of which engines it covers is the core's to answer, and the one client that can always reach an upgraded container is curl.

With update.reinstallOutdated on, the core does the same when it starts. Once the server is ready, it queues what POST /installs/outdated would, and logs the jobs and every engine skipped, with the reason, at info. An upgrade in the image is then docker compose pull and docker compose up -d and nothing else. The curl at the end of the upgrade was the step that got forgotten, and forgetting it failed silently, which is the case outdated exists to make visible: on a box upgraded from 0.1.3 to 0.1.6, Chatterbox went on running a 0.1.2 worker until somebody noticed /residency reporting no sizeBytes.

  • It is off by default, for the reasons in § 9 that a reinstall is the operator's to ask for. A setting is asking. The Docker image writes true into the config it seeds on first boot, since an upgrade there always replaces the core under engines it did not build, and a box that already has a config keeps what it says.
  • Nothing it skips is retried until the next start, and a skip is not a failure. An engine whose licence needs accepting again waits for a person, as it does for the route.
  • Every start with an engine behind queues a reinstall, not only the first after an upgrade. A reinstall that failed leaves the engine behind, so the next start tries again, which is what a person would do; one that keeps failing costs one failed job per start, in GET /installs.

Pulling weights​

POST /engines/{id}/pull asks the engine's worker to fetch a variant (§ 8): download the weights without loading them. The worker process is started if it is not running, which costs tens of megabytes, and no model is loaded, so it takes no residency slot and evicts nothing. An engine that does not implement fetch answers unsupported, and its weights arrive on first load as before. A pull never warms, even a variant that declares compiles: evicting nothing is the promise that lets a client pull on a busy box, and warming is asked for by name (below).

Warming​

POST /engines/{id}/warm with { "variant": "full" } loads a variant that declares compiles (§ 8) and unloads it again, exactly as an install's step 6 does, as a job whose kind is warm and whose one step is warm. An absent variant is the engine's default, as for a pull. It is the way to warm an engine an install did not: one installed without ?pull, one installed by a core that had no step 6, and one the operator configured. An Orpheus installed before step 6 existed has no warmed, so the reinstall an upgrade asks for would rebuild it and leave its first request to compile from cold, 35 seconds against 12.

It is a route of its own rather than a pull that also warms, because the two make opposite promises about the card: a pull evicts nothing, and a warm takes a slot as a /speak would, may evict an idle model for it, and waits behind one that is speaking. A client that asks for a warm has asked for that by name.

  • A variant that does not compile fails the job with unsupported, and so does one the worker does not declare. Which variants compile is the worker's to say, so it is a job's answer and not a refusal before one: the core has not asked the worker yet.
  • What it warms is recorded, added to the engine's warmed when this API installed it, so a reinstall warms it again from then on. An engine the operator configured is warmed and nothing is recorded, because the state database does not hold an entry for an engine it does not own, and a reinstall refuses that engine anyway.
  • The refusals before a job are a pull's: an engine that is not installed is unknown_engine, and one with a job queued or running is conflict.

What is left to warm is the variants that compile, less what was warmed. A client reads the first from the engine's capability document, the variants that declare compiles, and the second from the engine's catalog entry, warmed. The core does not answer it for them in the catalog, because only the worker knows which variants compile and the catalog is read on every page load: a listing that started a worker for each installed engine to ask would cost tens of megabytes apiece for an answer that changes only at an install. Where the two disagree only because nothing was recorded, as for an engine the operator configured, the client is told nothing it can act on, and should not keep offering a warm it cannot see the end of.

Jobs​

{
  "id": "01J8Z6Q4B7",
  "engine": "chatterbox",
  "kind": "install",
  "state": "running",
  "step": "packages",
  "createdAt": "2026-09-18T14:02:11.000Z",
  "startedAt": "2026-09-18T14:02:11.004Z"
}

kind is install, pull, reinstall or warm. variant is the variant a pull fetches, or an install fetches in step 5; an install without it stops at register. step is one of venv, packages, verify, register, weights and warm. state is queued, running, succeeded or failed; a failed job carries error, an ordinary error envelope body. Its message names the command and the line of its output that says why it failed, not the line it printed last: pip ends a failed build with a footer naming the package, and an Orpheus install whose error was ╰─> llama-cpp-python had its reason, a missing C compiler, only in the event feed. One job runs at a time and the rest queue, because two pip installs racing for one disk and one network connection finish later than the same two in a line, and a failure in one is easier to read without the other interleaved.

An id GET /installs/{job} does not know is bad_request with a 404, as an unknown route is: the request named something that is not there, and a finished job is only kept until fifty newer ones have finished after it.

Jobs live in memory. A restart forgets them, and a job running at shutdown is stopped and marked failed. That loses nothing that matters: a half-built virtualenv is caught by step 1 of the next install.

GET /installs/{job}/events streams the job's events: step changes as progress, each line pip and the interpreter print as log, and the outcome as status or error. A client that reconnects with Last-Event-ID resumes where it left off, and one whose resume point has already fallen out of the replay buffer is sent a resync event and should re-read the job. The stream does not end when the job does: a job's last event is a progress whose status is done or failed, and a client closes the stream when it sees one. Ending it from the server would make a browser's EventSource reconnect to a finished job over and over. The filter is pinned to the job in the path, so a query parameter cannot widen it to another job's output. The frame format is ServerKit's server feed, so the same client code reads it wherever ServerKit is used.

11. The OpenAI shim​

The native API is the contract, and adoption runs through other people's clients. Most of those speak OpenAI's POST /v1/audio/speech, and pointing one at this server should need a base URL and nothing else.

POST /v1/audio/speech
Content-Type: application/json
{ "model": "chatterbox:turbo", "input": "Right, that was The Verve Pipe. [laugh]", "voice": "narrator_02", "response_format": "opus" }

The shim is a translation into /speak, not a second implementation of it. It builds a native request and hands it to the same code, so the ceiling, the cue stripping, the dial check, the residency lease and the rules for failing after the headers (§ 6) are the ones /speak applies, and cannot drift from them. The /v1 in the path is OpenAI's and says nothing about this contract's version, which is § 9's business.

FieldBecomes
modelRequired. An engine id, or engine:variant. An exact engine id is tried first, so an id with a colon in it is still reachable.
inputtext. Cues in it are handled as /speak handles them.
voicevoice. Absent means the engine's default, although OpenAI's own API requires it.
response_formatformat. Absent means mp3, because that is OpenAI's default. aac is unsupported.
speedA dial named speed on the effective variant. Absent or 1 sends nothing.
instructionsunsupported unless absent or empty.
stream_formataudio or absent. sse is unsupported.

The response always streams, as OpenAI's does. A caller that wants the duration header or the better error that stream: false gives (§ 6) wants the native API.

model names an engine, and tts-1 is not one​

A model that is not an installed engine is unknown_engine, and the message lists the ones that are. There is no alias that maps tts-1 to a default engine, for the reason § 7 gives for voices: a substitute is audio nobody asked for, delivered with a 200. A client hardcoded to tts-1 is misconfigured, and the refusal is what tells its operator so.

alloy is not a voice either, and is unknown_voice like any other id the engine lacks. An operator whose client cannot be told another name can clone a voice with the id alloy: voice ids are theirs to choose.

What has no equivalent is refused, not dropped​

The shim could drop instructions and a speed it cannot honour and answer 200. That is the silent discard § 6 refuses for dials, and for the same reason: the client believes it asked for something. So a non-empty instructions is unsupported, and so is a speed other than 1 on a variant with no speed dial. A speed outside the dial's range is bad_request, as any dial is.

A field the shim does not know is bad_request naming it. OpenAI's own API refuses an unrecognised field with a 400, so strictness holds a client to nothing it was not already held to. delivery, params, seed, language and keepAliveSeconds are not accepted: the native API takes them, and adding them here would make a second native API with a worse name. A shim request gets the engine's keep-alive, or the server's, as anything that does not ask for one does. A multilingual variant speaks the first language it lists, as /speak does when language is absent.

A cue in input is different. Stripping one the variant does not claim is § 5 making the request performable, not the shim discarding a field, and it happens silently here exactly as it does in /speak.

The shim has no dialogue. OpenAI's speech request has one voice, so it translates into /speak and only ever reaches one reader, on an engine that performs dialogue as on any other.

mp3 by default needs ffmpeg​

Because OpenAI's default is mp3, a client that says nothing gets mp3, which needs an ffmpeg with libmp3lame on the worker's box. Without one the answer is unsupported saying so. The shim does not fall back to wav: a client that saves the body as speech.mp3 never reads the Content-Type, and a WAV with the wrong extension is a bug report that arrives weeks later from somebody else.

pcm is the engine's rate, not OpenAI's​

OpenAI's pcm is 24 kHz, 16-bit, mono. Here it is the engine's native format, as it is in /speak, and the Content-Type says which: audio/L16; rate=24000; channels=1. The core does not resample, because it holds no audio knowledge. Chatterbox is 24 kHz, so it matches; an engine at 22.05 kHz plays about 9% fast in a client that assumes. A client that cannot read a Content-Type should ask for wav, whose header carries the rate.

Errors​

The envelope is OpenAI's, because that is what the clients parse. The status is § 6's, and the taxonomy survives in code:

{
  "error": {
    "message": "no voice \"alloy\"",
    "type": "invalid_request_error",
    "param": "voice",
    "code": "unknown_voice",
    "retryable": false
  }
}

type is invalid_request_error when the request was wrong (not retryable, and under 500), and server_error otherwise. param names the request field the failure is about, when there is one.

Every error carries x-should-retry, set from retryable. OpenAI's SDKs retry 408, 409, 429 and every 5xx twice by default, deciding by status, so without the header an internal 500, which § 6 says is not retryable, is sent three times. Both official SDKs read x-should-retry before the status, so with it they retry exactly what the taxonomy says to.

Who may call it​

Anybody who may call /speak, which is anybody. OpenAI's clients will not start without an API key and send it as Authorization: Bearer; the shim ignores it. The header is stripped before any handler runs (§ 10), so a key typed into a client is never logged.

Only this route is served. GET /v1/models and the rest of OpenAI's API wait for a client that needs them.

12. The MCP server​

An agent that can call tools should be able to find an engine, pick a voice and speak a line without anybody writing glue for it. The Model Context Protocol is how those agents are told what a server can do, so the core serves it.

POST /mcp
Content-Type: application/json
Accept: application/json, text/event-stream

MCP is a translation into the public routes, as the OpenAI shim is (§ 11). Every tool builds a request to a route of § 6 or § 7 and runs it through the core's own router, in process, and reads the answer. The ceiling, the cue stripping, the dial check, the residency lease, the floor and the error taxonomy are that route's, so an agent is held to exactly what curl is held to, and a rule added to a route reaches its tool without anybody remembering that the tool exists.

ToolRunsTakes
list_enginesGET /enginesnothing
engine_capabilitiesGET /engines/{engine}/capabilitiesengine
list_voicesGET /engines/{engine}/voicesengine
speakPOST /speak with stream: false/speak's body without stream
speak_dialoguePOST /engines/{engine}/dialogue with stream: falsethe dialogue body with engine and without stream

A tool's input schema is the request contract's, from contracts/, so the two cannot describe different requests. The read-only tools say so in their annotations, and answer with structured content whose outputSchema is the route's response contract, so a client checks a result against what the route promises rather than against prose. MCP requires structured content to be an object, so a list goes out as { "items": [...] }. The same JSON goes out as text too, for a client that predates structured results.

The tools are the public API and nothing more. Nothing in § 10 is a tool. The management routes run pip and change settings, and they answer loopback callers and a token (§ 10); an agent that can reach /mcp is not thereby somebody who may install an engine. A client that wants those routes calls them.

Stateless, over Streamable HTTP​

Each POST carries one JSON-RPC message and is answered with one JSON response, or with 202 and no body for a notification. The server keeps no session and sends no Mcp-Session-Id, so any request may be the first one, a restart loses nothing, and a load balancer needs no affinity. Every tool is a request and its answer; nothing here has progress to report or anything to push, so a session would be state kept for no reader. GET /mcp and DELETE /mcp answer 405, which is how the transport says a server offers no stream.

stdio is not served. A client that only speaks stdio runs a bridge to the URL; an agent on another machine, which is the case this was built for, cannot spawn the core as a child anyway.

Audio arrives as audio content​

speak answers with two content items: the audio as MCP audio content, base64 with the Content-Type /speak gave it as mimeType, and a line of text naming the engine and voice asked for, the type, the size and the duration when the worker reported one. speak_dialogue answers the same way. The route runs buffered because a tool result is one message, which is the case § 14.2 already allows for: one response, in memory, bounded by the variant's maxCharacters.

format defaults to wav, not to what /speak would choose. mp3 and opus need ffmpeg on the worker's box and wav needs nothing, so a box without ffmpeg can still answer the call an agent makes when it says nothing. pcm keeps its audio/L16; rate=…; channels=1 type verbatim, for the reason § 11 gives: the rate is the engine's.

Errors are tool results​

A route's refusal becomes a tool result with isError: true whose text carries the envelope's code, message and retryable. That is how MCP reports a tool that ran and failed, as opposed to a call that was malformed, and it is the form an agent reads: unknown_voice naming the voices there are is something it can act on, where a JSON-RPC error is something its client reports and stops at. A call to a tool that does not exist, or a body that is not JSON-RPC, is a protocol error as the transport defines one.

Who may call it​

Anybody who may call /speak, which is anybody, with one refusal. The transport requires a server to check Origin, because a page the operator visits can make their browser POST to a server on the LAN as easily as to localhost. /mcp applies the rule the management routes do (§ 10): a request carrying an Origin is refused as forbidden unless the origin is this machine's own or is listed in management.origins. An agent is not a browser and sends no Origin, and is admitted. A bearer token is ignored and, as everywhere, stripped before any handler runs.

/mcp is not in GET /openapi.json (§ 9). It is one JSON-RPC endpoint whose operations are the tools, and tools/list is where they are described, to the only readers that can use them.

13. Deliberately not in v1​

Named so that nobody has to guess whether they were forgotten.

  • STT. A different problem wearing a similar hat. Say no once, in the README.
  • A delivery in a dialogue. § 6 says why it is left out, and it waits for an engine that can perform one per turn.
  • Word timestamps. Wanted, cheap enough as an optional sidecar response, and not worth blocking v1. Leave room: a X-Rhapsode-Timings-Url header or a timings field in a multipart response.
  • Batching. Adapters declare concurrency and that is the whole of it for now.
  • Any format but pcm from /speak/stream. § 6 says why: one file from many pieces is audio work. An encoder that streams across pieces is the way in, and it waits for a client that needs one.
  • Engines inside the core. Every engine is a worker, ONNX ones included. § 8 says why.
  • A UI in the core. The gap this project fills is that everything else has one, and has put its API behind it. A web page for installing engines is a client of § 10 like any other, and gets no route the terminal client does not. Its API reference renders GET /openapi.json (§ 9), the document every other client can read, rather than a copy of its own.

14. Open questions​

  1. Dialogue. Decided: a second endpoint, declared per variant (§ 4, § 6). A structured text would have put a shape into every engine's request that most of them must refuse half of; a second route splits the API only for the engines that have something to put there. Turns name speakers rather than voices, because an unvoiced speaker is the ordinary case for Dia.
  2. Does the core ever hold audio? Decided: only when stream: false was asked for, and only the one response, in memory. § 6 already depends on it: the floor can only become an error envelope if the core has the whole body before it writes a status, and Content-Length and the duration header can only be headers if the length is known. A stream: true response goes through at constant memory. The buffer is bounded by the variant's maxCharacters, not by anything the client sends. Retry-on-truncation and content-addressed caching were the arguments for buffering everything. Neither is in v1, and whichever arrives first reopens this.
  3. Voice namespacing across engines. Decided: not this server's job. A voice id is scoped to its engine (§ 7) and names one engine's rendering of one reference. The same clip cloned on two engines gives two voices that sound related, not one voice, and an alias spanning them would promise a sameness the server cannot deliver and add a second id space for the § 7 rules to police. A client that means "the same person" on three engines keeps its own table of (engine, voice) pairs, which is also the only place that knows which of those it thinks are close enough.
  4. In-process ONNX engines. Decided: no. ONNX engines are Python workers like every other engine, for the reasons in § 8, and "engine" in the registry keeps meaning one thing.
  5. Where the core lives. Decided: MaroonedSoftware/rhapsode, one repository holding the protocol, the core and the Python worker SDK. MaroonedSoftware already publishes ServerKit and ContractKit, which the core builds on. A change to the protocol has to land in both implementations at once or one of them is wrong, and one repository with one pnpm test is what makes that a single commit rather than a coordinated pair of releases. The cost is that the SDK cannot version on its own cadence. Revisit when it has outside users who need it to.