One endpoint, five models

The first version of this API had five job endpoints. Vocal separation, enhancement, speaker separation, super-resolution, and SI-SDR scoring were each callable on their own, and you chained them by passing the previous job's id instead of a file. It was a clean design, it worked, and we deleted it.

Today there is one job-creation endpoint, POST /v1/jobs/separation. The old paths answer 404 with a pointer to it. This post is about why the composable version was the wrong shape, and what it takes to run five models in one job without the whole thing becoming a single unobservable black box.

What five endpoints actually cost the caller

Composability sounds like a gift to the caller, and sometimes it is. Here it wasn't, because there was only one path anyone ever took. Every real integration submitted the same five stages in the same order, which meant the flexibility we shipped was being spent on:

• Five round trips and five polling loops instead of one, with the client responsible for noticing that stage three failed and not submitting stage four.
• Five billing events. Each stage was priced separately, so one result cost five charges for what the user experienced as a single operation.
• Five chances to get the order wrong, and no way for us to tell a caller who ran the stages in the wrong order that they had just made their output worse.
• An ordering contract encoded in client code, which meant we could not change the pipeline without breaking every integration: no reordering a stage, inserting one, or retuning a model's placement.

That last one is the real argument. The order of these five models is a research result, not a user preference. Freezing it into every caller's code turned an internal decision into a public API contract we hadn't meant to make.

They were five separately-billed jobs for one result. Now the sequencing lives in the worker, and the user pays once instead of five times.

One job also means one privacy story. Under the old design the intermediates had to survive between calls, because the next call referenced them. Now a stage's working directory is deleted as soon as the next stage has produced its own output, and only the final pair is ever uploaded off the GPU box. The four intermediates are the user's audio too, and nothing downloads them.

Why the five models cannot share a process

Having decided on one job, the obvious implementation is one Python process that imports five models and calls them in order. That implementation is not available to us, and the reason is dependency arithmetic rather than architecture:

Every model stage runs as a subprocess. That is not a style choice: the stage venvs carry mutually incompatible torch builds, so importing two of them into one process is impossible.

apps/worker/README.md

The separation and music-separation models want a current CUDA-13 torch build. The bandwidth-extension model needs an older one. There is no single environment that satisfies both, and pinning everything to the older build would mean giving up the newer kernels on the two stages that use them most.

So each model stage runs as a subprocess under its own interpreter. Note that this is four interpreters, not five. The mapping is less tidy than the stage list suggests, and it is worth showing honestly:

venvs/mss     → music_separation
              → separation
              → super_resolution  (outer driver only)
venvs/frcrn   → enhancement
venvs/apbwe   → super_resolution  (the inner inference script)
in-process    → si_sdr

Subprocess isolation is usually described as a cost, and it is one: process spawn, model load, and audio marshalled through the filesystem between stages. But it buys two things worth having. A stage that segfaults or exhausts GPU memory takes down a subprocess, not the worker. And upgrading one model's torch build is a change to one venv, not a coordinated migration across five.

Five clocks, one progress bar

The thing you lose by collapsing five jobs into one is observability. Under the old design the caller knew exactly where they were, because they were the one submitting each stage. Under the new one, a job could plausibly report running for several minutes with nothing else to say.

So job.stage names the phase in progress (queued, vocal_separation, separation, enhancement, super_resolution, si_sdr, done), and progress runs 0 to 1 across the whole pipeline. Each stage owns a share of that range: 0.25, 0.30, 0.20, 0.15, and 0.10 respectively. The five are asserted to total exactly 1.0 at import, because a rounding error there is a job that never reaches 100%. Switch a stage off and its share is dropped, with the rest renormalized over whatever will actually run, so a shortened pipeline still spans the whole bar instead of stopping short of the end.

Within a slice, the fraction complete is elapsed wall clock measured against what that stage's real-time factor predicts, capped at 0.95:

progress = sum(weights before this step) + this step's weight * fraction

# fraction = elapsed / (input_seconds * stage RTF), capped at 0.95.
# The cap matters: a step that under-runs its estimate must not sit at
# the end of its slice while it is still working. That reads as a hang.
# 1.0 for a slice is written once, when the outputs are on disk.

The construction is monotonic by design. Progress cannot go backwards, even when a stage overruns its prediction, because the cap holds it inside its own slice until real output appears.

The ETA is computed from a different set of numbers, and the distinction is easy to miss. Weights say how the bar should look; real-time factors say how long the work takes. Those are not the same thing: speaker separation costs roughly 0.134 seconds per second of audio, the most of any model stage, while super-resolution costs about 0.004 — more than an order of magnitude less — yet still owns 15% of the bar. Sizing each bar slice by cost would leave the cheap stages crawling and the dear one lurching. So the remaining time is summed from RTFs, and the bar is drawn from weights.

Canceling something that isn't yours

Single-job execution makes cancellation harder than it looks. The API can cancel a queued job synchronously, so it comes off the Redis queue and its credit hold is released. A running job is different: the API sets a flag, the worker polls it roughly twice a second on its wait loop, and the response still reads running until the worker acknowledges. Cooperative, not immediate.

The subtle part is what gets killed. The worker's direct child is often a driver script, and the actual GPU work is a grandchild. That is exactly the case for super-resolution, where the outer driver shells out to a second interpreter. Killing the child alone would leave the grandchild holding the GPU. So cancellation kills the whole process group, SIGTERM first and SIGKILL after a five-second grace period.

Whatever the outcome, the scratch tree is removed in a finally block, and the worker sweeps for orphaned scratch directories at startup to cover the paths a finally cannot: a SIGKILL, an out-of-memory kill, a power loss.

One charge, quoted before the work starts

Billing follows the same collapse. Every job response carries estimated_cost_seconds at submission, computed from the duration of the recording you uploaded, so a client can show a price before any work happens. That amount is held at submission and charged only if the job succeeds. A canceled job is never charged, and neither is a failed one.

One detail we chose deliberately: the charge is the figure we quoted, not what the worker later reports having processed. The worker's number is advisory, logged when it drifts and never billed. A price you were shown before you agreed to it should not change afterwards because a model padded a short input.

What we gave up

This is not a free trade, and the honest accounting is short. You can no longer run enhancement on its own, or score a pair of tracks you separated somewhere else. Individual stages can be switched off, and doing so does shorten the job, but the price is quoted for the pipeline either way: a recording that only needs denoising costs what a full separation costs. And the worker runs one job at a time with no in-process concurrency, so the queue is genuinely serial.

We think that is the right position for now, because the one path that mattered got dramatically simpler and the pipeline stopped being frozen into other people's code. If a single-stage need turns out to be real, it comes back as its own endpoint with its own contract, not as five endpoints and an ordering convention we hoped callers would follow.

The stage-by-stage view of what runs inside that one job is in Separating the Inseparable. The endpoint itself is in the docs.