[Performance]: Per-request Python preprocess loop dominates decode step time at high concurrency (8.5 ms of a 10.9 ms step at 32 requests) — also a frame-cadence blocker for realtime/duplex

Proposal to improve performance

The per-request Python preprocess loop in OmniGPUModelRunner dominates decode step time at high concurrency. For models with has_preprocess, the runner calls self.model.preprocess(...) once per request per step inside a serial Python loop, with per-request dict lookups, tensor slicing, and _merge_additional_information_update (0.20.0: vllm_omni/worker/gpu_model_runner.py:1307-1352; the same per-request loop is still the default path on main @ 29339673, gpu_model_runner.py:1654+).

Measurement (1B Moshi/Hibiki-style AR talker — same plugin/model family as #4355 — H100, CUDA graphs on, steady-state decode, per-step engine trace): at 32 concurrent decode requests, the preprocess loop accounts for ~8.5 ms of a ~10.9 ms step — i.e. ~78% of the step is host-side Python bookkeeping, not model compute. The cost scales linearly with batch size (~0.27 ms/request/step), so it sets the throughput ceiling precisely in the regime where continuous batching should shine.

Proposal:

1. Promote the batched-decode preprocess path introduced in #3662 (preprocess_decode_batch, currently implemented only by qwen3_tts_talker.py) to a documented, generic model interface, so any AR talker model can opt in.
2. Vectorize the runner-side bookkeeping that wraps the loop (intermediate-buffer lookups, span arithmetic, additional_information merge — cf. RFC #1351) so models without the batch hook also stop paying O(N) Python per step.
3. Consider extending the batched path to prefill rows / mixed steps (today it covers only span_len == 1 and not is_prefill rows of has_talker_mtp models).

Report of performance regression

Not a regression — a standing ceiling. Two ways it binds:

• Throughput: with the loop at ~0.27 ms/request/step, a 32-request decode step on a 1B model is ~4× slower than its GPU compute alone; larger models amortize better but the host cost is constant per request, so it returns at every batch-size increase.
• Realtime/duplex latency (cf. RFC #3745, PR #3907): for frame-cadence speech serving (12.5 Hz, 80 ms/frame, Moshi-style), the loop runs every frame and scales linearly with concurrent sessions. We measured the full stock-0.20 streaming round trip (update → park/wake → 1-token re-entry decode → DELTA out) at ~10 ms p99 with 8 live sessions on the 1B — the decode step, dominated by this loop, is the majority of the cycle. At N≈32 sessions the loop alone (~8.5 ms) approaches a third of the 80 ms frame budget before any model compute, codec work, or jitter reserve. This is the single biggest engine-side obstacle we see to duplex serving of small talker models.

Misc discussion on performance

The loop's contents per request: model_intermediate_buffer.get, optional per-model attach, query_start_loc slicing, the Python preprocess call, conditional talker_mtp staging copies, _merge_additional_information_update (0.20.0 gpu_model_runner.py:1342). None of it is per-request-dependent in shape during steady decode (all spans are 1), which is what makes a batched fast path natural — as #3662 demonstrated for one model.

We carry no workaround for this (it needs runner changes); we simply operate at the resulting ceiling.

Your current environment (if you think it is necessary)

Same environment/venv as #4355: vllm-omni 0.20.0, vLLM 0.20.2+cu129, PyTorch 2.11.0+cu126, Python 3.12.12, Ubuntu 22.04, NVIDIA H100 80GB. Line references re-verified against installed 0.20.0 and main @ 29339673 (2026-06-12).

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.