[RFC] Full-Duplex Session Architecture for vLLM-OMNI

Motivation

A new class of speech models — MiniCPM-o 4.5 (Omni-Flow / TDM), Nemotron VoiceChat (nemotron_duplex_h), SoulX-Duplug, Moshi— are full-duplex by design: they perceive and respond concurrently. The user can speak while the model is speaking, interrupt mid-sentence, and the model keeps a single conversation context alive for minutes. This is the "Doubao / Gemini Live / GPT-4o voice" experience: no push-to-talk, no turn button, sub-300ms barge-in.

vLLM-OMNI cannot serve this today. Three model PRs (#3642, #3512, #1967) have independently hit the same wall: the runtime is request-oriented (prompt → output → done, KV freed at finish), the models are stream-oriented (continuous audio in while generating out, conversation-lifetime KV cache). Each PR is at risk of growing its own ad-hoc streaming path that we later have to reconcile. This RFC defines the duplex session primitive once, and enumerates how each model conforms.

This is also explicitly shared infrastructure with World Models (#1987): "client pushes observations, server returns predictions, repeat" is the same persistent bidirectional loop as "client pushes audio, server returns audio."

Use Cases

Use Case	Input cadence	Barge-in	Session length	Latency target
Live voice assistant (Doubao-style)	80–160 ms audio	Required	minutes	< 400 ms v2v
Real-time interpreter / dubbing	continuous	Soft	minutes	< 1 s
Embodied agent (#1987 world model)	obs stream	Required	unbounded	< 500 ms
Call-center / IVR	continuous	Required	minutes	< 700 ms
Accessibility narration	continuous	Optional	unbounded	< 1 s

---

The Impedance Mismatch (grounded in current code)

flowchart LR
    subgraph REQ["vLLM-OMNI today — request-oriented"]
        A1[add_request] --> A2[prefill prompt]
        A2 --> A3[decode loop]
        A3 --> A4[finish]
        A4 --> A5["_free_request → _free_blocks<br/>KV returned to manager"]
    end
    subgraph DUP["Duplex models — stream-oriented"]
        B1[open session] --> B2[continuous chunks in]
        B2 --> B3[generate out concurrently]
        B3 --> B2
        B2 --> B4[barge-in / turn change]
        B4 --> B3
        B3 --> B5[close session<br/>minutes later]
    end

Concrete blockers found in the codebase:

#	Where	Current behavior	Why it blocks duplex
1	OmniARScheduler._free_request / _free_blocks (omni_ar_scheduler.py:650/732)	KV blocks returned to the manager on request finish	No conversation-lifetime KV — every turn re-prefills
2	OmniSchedulerMixin._replace_session_with_streaming_update (omni_scheduler_mixin.py:18)	On a streaming update sets num_computed_tokens = 0, clears token buffers	Existing "streaming" path re-prefills each segment — KV is not preserved across turns
3	OmniSchedulingCoordinator.process_pending_full_payload_inputs (omni_scheduling_coordinator.py:117)	Requests without a ready next chunk are removed from the waiting queue before super().schedule(), re-inserted after	Streaming requests drop out of the GPU batch whenever a chunk is momentarily late → under-batching (vklimkov-nvidia: ~½ throughput, 2× ITL)
4	Orchestrator (orchestrator.py)	Assumes requests have a begin/end; _route_output finalizes on finish	No concept of a persistent session that owns long-lived per-stage requests
5	Per-token path core → client → core over ZMQ (stage_engine_core_client.py)	Next audio chunk round-trips through the orchestrator client	vklimkov-nvidia: +3–5 ms/token, grows under concurrency
6	StageExecutionType enum (stage_config.py:132)	Only LLM_AR, LLM_GENERATION, DIFFUSION	No stage type owns turn-taking / VAD

Prior art (#1481 WS /v1/audio/conversation, #1480 input.cancel/audio.cancelled) solved the wire protocol well (it tracks the OpenAI Realtime API) but is half-duplex turn-taking and API-layer only — it assumes an explicit input.audio.done and says nothing about persistent KV or the request→stream mismatch. It is carried forward as the protocol layer on top of this primitive, not revived standalone.

---

Target Experience

sequenceDiagram
    participant U as User (mic)
    participant WS as WS /v1/audio/conversation
    participant S as DuplexSession (orchestrator)
    participant DV as duplex_vad stage
    participant TH as Thinker (LLM_AR, session-pinned KV)
    participant TK as Talker / Token2Wav

    U->>WS: session.config {model, voice}
    WS->>S: open_session(sid)
    S->>DV: open_session(sid) — KV lease acquired
    loop continuous, never "done"
        U-->>WS: PCM chunk (80–160 ms)
        WS->>S: push_chunk(sid)
        S->>DV: chunk → turn-state stream
        DV-->>S: user_nonidle / user_complete
        alt user_complete
            S->>TH: gate open — append chunk tokens (KV kept)
            TH-->>WS: response.text.delta
            TH->>TK: hidden states (async chunk)
            TK-->>WS: response.audio (binary)
        else user speaks while model speaking
            U-->>WS: PCM (barge-in)
            DV-->>S: user_nonidle (epoch++)
            S->>TK: flush stale epoch, abort talker req
            S->>TH: keep KV, resume listening
            WS-->>U: audio.cancelled
        end
    end
    U->>WS: session.close
    S->>DV: close_session(sid) — KV lease released

The session is the unit, not the request. The thinker's KV cache is leased to the session and survives every turn; barge-in flushes downstream stages and bumps an epoch but never frees the conversation KV.

---

Architecture

flowchart TB
    subgraph API["Protocol layer (#1481 / #1480, OpenAI-Realtime-aligned)"]
        WS["WS /v1/audio/conversation<br/>serving_duplex.py"]
    end
    subgraph ENG["Engine layer"]
        DS["DuplexSession registry<br/>engine/duplex_session.py<br/>TTL-GC, epoch, ring buffer"]
        SDC["StageDuplexClient<br/>engine/stage_client.py<br/>open/push_chunk/signal_turn/barge_in/close"]
    end
    subgraph SCHED["Scheduler layer"]
        DSCH["OmniDuplexScheduler<br/>session KV lease + coalescing window"]
    end
    subgraph STAGE["Stage layer"]
        DV["StageExecutionType.DUPLEX_VAD<br/>owns turn-taking"]
        AR["LLM_AR (duplex-append mode)"]
    end
    WS --> DS --> SDC --> DSCH --> DV
    DSCH --> AR
    SDC -. "direct SHM chunk ring<br/>(bypass ZMQ round-trip)" .-> STAGE

1. DuplexSession — the new unit of work

vllm_omni/engine/duplex_session.py, owned by the Orchestrator. Keyed by session_id. Holds:

• the set of long-lived per-stage request IDs (one resumable Request per stage),
• the KV lease handle (blocks pinned for the conversation),
• an input ring buffer of incoming chunks (audio / observation, opaque typed payload — modality-agnostic for #1987),
• the turn state machine (driven by the duplex_vad stage),
• a monotonically increasing barge-in epoch,
• TTL-based GC (idle session reclaimed; lease released).

The Orchestrator is taught that a session-bound request is never finalized by _route_output until close_session.

2. StageDuplexClient — model-agnostic stage interface

Added in vllm_omni/engine/stage_client.py alongside StagePoolLLMClient. Extends StagePoolClient with:

class StageDuplexClient(StagePoolClient, Protocol):
    def open_session(self, sid: str, params: DuplexSessionParams) -> None: ...
    def push_chunk(self, sid: str, chunk: DuplexChunk) -> None: ...      # SHM ring, not ZMQ
    def signal_turn(self, sid: str, event: TurnEvent) -> None: ...
    def barge_in(self, sid: str, epoch: int, scope: Literal["current","all"]) -> None: ...
    def close_session(self, sid: str) -> None: ...

A model conforms by implementing a thin DuplexModelAdapter (chunk-in hook, turn-state hook, KV-append hook) — see Conformance.

3. duplex_vad stage type — owns turn-taking

New StageExecutionType.DUPLEX_VAD in vllm_omni/config/stage_config.py:132; _resolve_scheduler() (:140) maps it to a new OmniDuplexScheduler (vllm_omni/core/sched/omni_duplex_scheduler.py). This stage runs the streaming VAD / dialogue-state model (the SoulX-Duplug pattern: 160 ms block-causal chunks → states user_idle / user_nonidle / user_backchannel / user_complete / user_incomplete), or the perceive half of an end-to-end duplex model. It gates the downstream thinker/talker and is the single owner of turn boundaries and barge-in detection.

4. Conversation-lifetime KV cache

Two scheduler changes:

• Session KV lease. OmniDuplexScheduler overrides _free_request/_free_blocks (cf. omni_ar_scheduler.py:650/732): a session-bound request's blocks are not returned on segment finish; they are released only on close_session / TTL-GC.
• Duplex-append mode. Replace the num_computed_tokens = 0 reset in _replace_session_with_streaming_update (omni_scheduler_mixin.py:18) with an append path for session requests: keep num_computed_tokens, extend the block table with the new chunk's tokens, only invalidate from a checkpoint on rollback (ASR correction — required by #1967). This is the deepest change and gets its own sub-design + tests.

5. Scheduler cadence fix (vklimkov-nvidia bottleneck #1)

flowchart LR
    subgraph NOW["Today"]
        N1[chunk late] --> N2["remove from waiting queue"] --> N3[step batches subset] --> N4[under-batched]
    end
    subgraph NEW["OmniDuplexScheduler"]
        M1[chunk in-flight] --> M2{within<br/>coalescing window?}
        M2 -->|yes, low concurrency| M3[wait ≤ window_ms]
        M2 -->|high concurrency| M4[step now, others fill batch]
        M3 --> M5[batch streaming reqs together]
        M4 --> M5
    end

Replace the unconditional remove-before/restore-after in OmniSchedulingCoordinator.process_pending_full_payload_inputs (omni_scheduling_coordinator.py:117) with a bounded adaptive coalescing window: a session declares its chunk period (e.g. 160 ms / 80 ms); the step may wait up to duplex_batch_window_ms for in-flight chunks to land so they batch together. Adaptive — short/zero wait under high concurrency (throughput), small wait under low concurrency (latency). This is exactly Sy0307's stated requirement: fast streaming under low concurrency, batching under high concurrency.

6. IPC round-trip elimination (vklimkov-nvidia bottleneck #2)

The +3–5 ms/token comes from per-chunk core → client → core over ZMQ. Design: a session-keyed direct shared-memory ingress ring, reusing the existing SharedMemoryConnector infra (shm_connector.py, already does codec_streaming / codec_chunk_frames). The client writes audio chunks straight into the stage-0 process's ring; only control events (turn signals, barge-in) traverse ZMQ. Co-locate the perceive stage and its consumer where possible — same logic as the #3642 t2w fold (commit b59fa3f) that removed a no-op inter-stage hop.

7. Barge-in / cancel semantics (from #1480)

Session-scoped epoch. Every output chunk carries (session_id, turn_id, epoch). A barge-in (VAD-detected user speech during model speech, or client input.cancel) bumps the epoch; all stages drop in-flight work tagged with a stale epoch (cross-stage flush via the connector + scheduler abort of the talker request) — but never the session KV lease. Then resume listening. cancel_scope: current | all carried verbatim from #1480; audio.cancelled emitted to the client.

8. Protocol layer (from #1481 / #1480)

entrypoints/openai/serving_duplex.py is a thin translator: OpenAI-Realtime-aligned WS events ↔ DuplexSession operations. No persistent-KV / impedance logic in the API layer — that was the structural defect of the standalone RFCs. Reused verbatim: session.config, response.start/done, audio.start/done, input.cancel, audio.cancelled.

---

Phased Implementation

gantt
    title Full-Duplex Session Architecture Roadmap
    dateFormat YYYY-MM-DD
    axisFormat %b

    section Phase 0
    #3642 lands req/resp + t2w fold (b59fa3f) :done, p0, 2026-05-12, 7d
    section Phase 1
    DuplexSession + StageDuplexClient + KV lease :p1, after p0, 14d
    section Phase 2
    DUPLEX_VAD stage + OmniDuplexScheduler + cadence :p2, after p1, 14d
    section Phase 3
    Barge-in epoch + WS protocol (#1481/#1480) :p3, after p2, 10d
    section Phase 4
    SHM direct-ingress ring + HC batching :p4, after p3, 10d
    section Phase 5
    Generalize to World Models #1987 :p5, after p4, 14d

Phase 0 — Request/response bring-up (done / in flight)

#3642 lands as request/response so the community can use MiniCPM-o 4.5 in turn-taking mode now; no-op t2w stage folded into the talker (b59fa3f). #3512 and #1967 likewise land as bring-up. Duplex does not gate these PRs.

Phase 1 — Session primitive + conversation-lifetime KV

flowchart LR
    OS[open_session] --> RL[Request becomes session-bound resumable]
    RL --> AP["append mode:<br/>keep num_computed_tokens"]
    AP --> KL["KV lease:<br/>no _free_blocks on segment finish"]
    KL --> CS[close_session releases lease]

DuplexSession + StageDuplexClient + session KV lease + duplex-append mode. Validation: MiniCPM-o 4.5 (#3642) running multi-turn with persistent context — no re-prefill between turns, KV held across the conversation. Measurable: TTFT on turn N ≫ improves vs. re-prefill baseline.

Phase 2 — duplex_vad stage + scheduler

StageExecutionType.DUPLEX_VAD + OmniDuplexScheduler + adaptive coalescing window (bottleneck #1). Validation: SoulX-Duplug (#1967) as a pure duplex_vad stage gating a downstream STT→LLM→TTS cascade — proves the model-agnostic split (VAD owns turn-taking; cascade conforms unchanged). Measurable: batch occupancy under N concurrent streams vs. today's drop-out behavior.

Phase 3 — Barge-in + protocol

sequenceDiagram
    participant C as Client
    participant S as DuplexSession
    participant TK as Talker
    C->>S: input.cancel (or VAD barge-in)
    S->>S: epoch++
    S->>TK: drop stale-epoch chunks, abort talker req
    S-->>C: audio.cancelled {scope}
    Note over S: KV lease intact — resume listening

Epoch-based cross-stage flush + WS /v1/audio/conversation (#1481) + input.cancel/audio.cancelled (#1480). Validation: end-to-end full-duplex with #3642 (Omni-Flow, end-to-end duplex) and #3512 (nemotron_duplex_h thinker → eartts talker) — listen-while-speak + sub-400 ms barge-in.

Phase 4 — IPC + high-concurrency batching

Session-keyed SHM direct-ingress ring (bottleneck #2) + high-concurrency batching policy in OmniDuplexScheduler. Validation: per-token overhead and ITL under concurrency vs. the ZMQ round-trip baseline; reproduce/close vklimkov-nvidia's profile.

Phase 5 — Generalize to World Models (#1987)

DuplexChunk is an opaque typed payload; only stage adapters are modality-specific. Validation: an #1987 observation→prediction loop reuses DuplexSession unchanged — confirming one session-management system, not two.

cc @linyueqian @yinpeiqi @Sy0307 @hsliuustc0106 @tc-mb @vklimkov-nvidia — feedback welcome here or in the WeChat group.