vllm-project/vllm-omni

[RFC]: Encoder Embedding Data Sharing for Disaggregated Omni-Modality Inference

Open

#3,427 opened on May 7, 2026

View on GitHub
 (5 comments) (4 reactions) (1 assignee)Python (1,067 forks)github user discovery
help wantedhigh priority

Repository metrics

Stars
 (4,990 stars)
PR merge metrics
 (PR metrics pending)

Description

Motivation

Omni-modality models such as Qwen-Omni, Bagel, and Ming-Omni pass multimodal inputs through dedicated encoder stages before the resulting embeddings reach the LLM backbone. A single video frame through a ViT-L encoder produces roughly 8 MB of embedding data and takes 20–80 ms of GPU compute; a 60-second audio clip produces approximately 24 MB across ~3,000 audio tokens. In a disaggregated deployment, these embeddings must also cross node boundaries to reach the prefill stage, adding transfer latency on top of encoding latency.

The current architecture has no mechanism to detect or reuse identical encoder outputs. If the same image appears in ten concurrent requests, the encoder runs ten times independently and the result is transferred ten times. In a replicated prefill pool with four replicas, this is multiplied further: each replica either re-encodes or receives a separate TE transfer of the same bytes. At meaningful QPS, encoder GPU utilization is dominated by redundant computation on previously seen inputs, product images, shared system prompts with images, reference audio clips for voice applications.

Three concrete gaps drive this RFC. First, there is no content-addressed lookup to detect that two requests contain the same encoder input and can share a result. Second, the MooncakeTransferEngineConnector's 1-sender→1-receiver constraint means 1→N embedding distribution requires N separate transfers or N encoder runs; there is no broadcast primitive. Third, when encoder and prefill co-locate on the same node, the embedding still routes through the connector stack rather than shared memory.

This RFC proposes an EmbeddingCache component, a two-layer cache-backed sharing mechanism that resolves all three gaps: content-addressed hashing detects reusable inputs before the encoder runs; a local LRU+TTL layer serves hits within a process at sub-millisecond cost; and pull-based broadcast via MooncakeStoreConnector enables N prefill replicas to independently fetch the same embedding that was written once, without modifying the TE connector's core semantics. Cache reuse and sharing are inseparable: the cache is the mechanism by which sharing avoids both redundant encoding and redundant transfer.

1. Overview

In vLLM-Omni's disaggregated inference architecture, multimodal encoder stages (vision, audio, video) produce large embedding tensors that must be transferred to the LLM prefill stage across process and node boundaries. Currently, there is no mechanism to detect or reuse identical encoder outputs across requests: the same image or audio clip is re-encoded for every request that contains it, and in a replicated prefill pool each replica independently receives or re-computes the same embedding. This wastes significant encoder GPU compute and adds avoidable transfer overhead.

This document proposes a cache-backed encoder embedding sharing mechanism an EmbeddingCache component that detects reusable encoder inputs via content-addressed hashing, serves repeated inputs from memory without re-running the encoder forward pass, and enables pull-based broadcast to N prefill replicas via the existing MooncakeStoreConnector. Cache reuse is the primary means by which we avoid redundant transfers; the caching and sharing concerns are inseparable and are addressed together here.

Related:

2. Scope & Objectives

Goals

  1. EmbeddingCacheKey: A content-addressed hash of the encoder input and all preprocessing parameters, computable before the encoder runs, enabling a lookup that can avoid the forward pass entirely.
  2. EmbeddingCache local layer: An in-process LRU + TTL cache that serves repeated inputs from memory within a single stage process.
  3. EmbeddingCache remote layer: Publication of new embeddings to the MooncakeStoreConnector so that remote prefill replicas can pull the same embedding, enabling 1→N sharing without modifying the MooncakeTransferEngineConnector's 1:1 semantics.
  4. Single-flight semantics: Concurrent misses for the same key are coalesced so that only one encoder forward runs; all concurrent waiters receive the same result.
  5. OmniBase integration: Opt-in embedding cache initialization via stage config, with a per-model encoder forward hook that checks the cache before running the encoder.

Cache Value Types

The EmbeddingCache value type must accommodate two distinct forms:

  1. Dense encoder embedding tensors e.g. image, video, and audio encoder outputs (ViT-L, audio transformer, etc.). These are the primary artifact type described in the original design.
  2. Token-sequence / hidden-state prefix artifacts e.g. TTS voice-reference codec tokens (Qwen3-TTS, Fish Speech, VoxCPM2) that are reused by the LLM prompt path. These behave more like a prompt prefix than a single encoder embedding tensor, and the same tokens may also appear in the LLM prompt.

This distinction has implications for serialization format, the store publish/pull path, key-namespace alignment (see RFC #1184 note below), and how the cache entry interacts with the LLM decode pipeline.

RFC #1184 Key-Namespace Alignment

When the cached value is a codec-token prefix (case 2 above), the EmbeddingCacheKey namespace must align with RFC #1184 (Prefix Caching with Hidden-State I/O). Without this alignment, the same voice reference can be independently hashed and cached at two layers once as an EmbeddingCache artifact and again as a prefix-cache artifact missing a large part of the intended reuse.

Voice-reference cache entries must therefore use a shared or composable key with the prefix-cache layer when the reusable artifact is a codec-token prefix. This is a hard design requirement for avoiding duplicate cache namespaces and preserving the intended reuse, though lack of alignment degrades performance and reuse efficiency rather than changing model outputs.

Correctness Criteria

  • Two requests with the same EmbeddingCacheKey receive bit-identical embeddings whether the result came from a cache hit or a live encoder forward.
  • With N concurrent misses for the same key, the encoder forward() is called exactly once within a single process. This exact-once guarantee relies on the in-process threading.Event coalescing in get_or_compute(). Cross-process duplicate computes are not prevented by this mechanism; remote sharing is best-effort deduplication after the first successful publish. A distributed reserve/commit path or an atomic store primitive would be required to extend this guarantee across process boundaries.
  • enable_embedding_cache: false (the default) produces behavior and performance identical to the current system.

Performance Targets

  • Cache hit path (local): sub-millisecond retrieval, no serialization.
  • Cache hit path (remote store pull): < 20 ms for a 100 MB embedding over RDMA (see Section 3 : this target is gated on the optimized EC connector path; the current MooncakeStoreConnector object path should not be assumed to provide this without the planned tensor/EC fast path).
  • Encoder skip on cache hit: verified by asserting forward() call count = 0 for repeated inputs in the E2E test.

3. Design

Architecture

Three deployment topologies are supported with a uniform EmbeddingCache API:

SCENARIO A: Same-node (intra-process)
══════════════════════════════════════════════════════════════════
  Encoder Stage
      │  get_or_compute(key, fn)
      ▼
  ┌─────────────────┐
  │  EmbeddingCache │  Local LRU hit → return tensor (no copy, no connector)
  │  (local layer)  │
  └─────────────────┘
      │
      ▼
  Prefill Stage (same process)


SCENARIO B: Cross-node, single prefill replica
══════════════════════════════════════════════════════════════════
  Encoder Node                          Prefill Node
  ┌──────────────────────┐              ┌────────────────────────┐
  │ EmbeddingCache.put() │─── async ───▶│ MooncakeStore          │
  │  → local store       │  publish     │                        │
  │  → store publish     │              │ EmbeddingCache.get()   │
  └──────────────────────┘              │  → store pull (RDMA)   │
                                        │  → local cache insert  │
                                        └────────────────────────┘


SCENARIO C: Cross-node, N prefill replicas (pull-based broadcast)
══════════════════════════════════════════════════════════════════
  Encoder Node                          Prefill Replica 1..N
  ┌──────────────────────┐              ┌────────────────────────┐
  │ EmbeddingCache.put() │              │ EmbeddingCache         │
  │  encoder runs once   │─── write ───▶│  .get_or_compute()     │
  │  (local single-      │  once to     │                        │
  │   flight per process)│  store       │  local hit: return     │
  └──────────────────────┘              │  store miss: RDMA pull │
                                        │  (each replica pulls   │
                                        │   independently)       │
                                        └────────────────────────┘

graph TD
    A[Incoming Request] --> B{EmbeddingCache.get_or_compute}
    B -->|local hit| C[Return cached tensor]
    B -->|remote hit| D[MooncakeStore RDMA pull]
    B -->|miss, elected| E[Encoder forward]
    B -->|miss, waiter| F[Block on Event]
    D --> G[Insert local cache]
    E --> H[Insert local + async publish to MooncakeStore]
    F -->|elected completes| C
    G --> C
    H --> C

API & Interface Changes

Component File Change
EmbeddingCacheKey embedding_cache/key.py (new) Content-addressed key; from_raw_bytes() and from_decoded_tensor() constructors
EncoderPreprocessingConfig embedding_cache/key.py (new) Frozen dataclass of all preprocessing parameters that affect encoder output
EmbeddingCache embedding_cache/cache.py (new) Two-layer LRU+TTL + MooncakeStore; get_or_compute() as primary API; value type supports both dense tensors and codec-token prefix artifacts
_BoundedPublisher embedding_cache/publisher.py (new) Bounded ThreadPoolExecutor for async store publish with backpressure
OmniBase vllm_omni/entrypoints/omni.py ~30-line init block; optional _embedding_cache field; background eviction loop; cleanup_request() releases all cache references acquired by the request
OmniModelState (per-model) vllm_omni/v1/worker/omni_model_state.py _encoder_forward_with_cache() hook; opt-in per model
Stage config schema (documented) New optional enable_embedding_cache and related fields

Immutability contract: EmbeddingCache.get() and get_or_compute() return tensor.detach(). This detaches autograd state but does not make the returned tensor read-only mutations to the returned tensor write through to the same storage as the cached entry, silently corrupting future hits. Consumers must treat cache hits as immutable and call .clone() before mutation. Debug builds may optionally return guarded wrappers or cloned tensors to catch accidental mutation.

Key Technical Decisions

Decision: EncoderPreprocessingConfig as an explicit required parameter for key construction.

  • What we chose: A frozen dataclass covering model ID, encoder config hash, processor/tokenizer version, resize/crop mode, image size, normalization mean/std, audio sample rate, audio codec identity, video frame stride, and input dtype.
  • Alternatives: Hash only the raw input content; infer preprocessing from the model config at key-construction time.
  • Rationale: Any preprocessing parameter that affects encoder output, if omitted from the key, can produce a false cache hit, incorrect results served silently with no error. Explicit required parameters force the caller to reason about what affects the output. The dataclass being frozen ensures keys are not accidentally constructed with mutable state.
@dataclass(frozen=True)
class EncoderPreprocessingConfig:
    cache_schema_version: int

    model_id: str
    model_revision: str
    encoder_config_hash: str
    processor_version: str

    resize_mode: str
    image_size: tuple[int, int]
    crop_mode: str
    normalize_mean: tuple[float, ...]
    normalize_std: tuple[float, ...]

    audio_sample_rate: int
    audio_codec_version: str        # e.g. "Encodec", "DAC", "ChatTTS-DVAE-v2"
    audio_codec_config_hash: str    # hash of codebook count, bandwidth, hop size,
                                    # quantizer settings, model revision

    video_frame_stride: int
    input_dtype: str

audio_codec_version captures the codec family and version (e.g. Encodec, DAC, or ChatTTS DVAE version). audio_codec_config_hash captures configuration that affects output tokens or embeddings codebook count, bandwidth, hop size, quantizer settings, or model revision. A codec implementation bump can silently change the produced tokens/embeddings while preserving the same raw audio input; omitting these fields would allow false cache hits that are particularly difficult to detect because the model receives plausible-looking but wrong codec tokens.

For non-audio modalities, audio_codec_version should be set to "" and audio_codec_config_hash to "".

Decision: Two explicit hash modes: from_raw_bytes (preferred) and from_decoded_tensor (fallback). No automatic mode selection.

  • What we chose: Caller selects mode explicitly at the integration point based on what data is available.
  • Alternatives: Auto-detect which mode to use; always use tensor hashing.
  • Rationale: Automatic fallback could produce different keys for the same logical input depending on what data happened to be available at call time, causing spurious cache misses or hits. Explicit mode selection makes the hash contract visible in the calling code. from_raw_bytes is faster (< 0.5 ms for 200 KB JPEG); from_decoded_tensor is the correct fallback when raw bytes are unavailable or the codec is non-deterministic (incurs a D2H copy for GPU tensors).

Note: from_raw_bytes and from_decoded_tensor are each stable within their own hash mode. They are not required to produce the same key, because raw-byte identity and decoded-tensor identity are different namespaces.

Decision: Single-flight via threading.Event coalescing in get_or_compute().

  • What we chose: An _in_flight: dict[str, threading.Event] tracks keys with in-progress encoder forwards. Concurrent misses for the same key receive an Event to wait on rather than launching a second forward.
  • Alternatives: Per-key mutex (simpler but serializes all access including reads); process-level distributed lock (complex, adds a coordination service dependency).
  • Rationale: In-process Event-based coalescing handles the common case (multiple concurrent requests for the same image in a batch) with no external dependency and sub-millisecond wait overhead when the forward completes quickly. This exact-once guarantee is scoped to a single process. For the cross-process case (multiple prefill replicas all missing the same key), there is no distributed reserve/commit primitive; two replicas can independently elect themselves to compute the same embedding. The MooncakeStore provides best-effort deduplication after the fact: whichever replica publishes first wins, and others can pull from the store on the next miss. A stronger cross-process guarantee would require a distributed lock or atomic store primitive and is left as a future extension.
# CacheValue is a dense tensor, a token-id sequence, or a hidden-state prefix artifact.
CacheValue = Union[torch.Tensor, list[int], "HiddenStatePrefix"]

def get_or_compute(
    self,
    key: EmbeddingCacheKey,
    compute_fn: Callable[[], CacheValue],
) -> CacheValue:
    # Fast path: local hit
    with self._lock:
        if (entry := self._local.get(key.hex)) and not self._is_expired(entry):
            entry.ref_count += 1          # acquire reference on hit
            entry.last_accessed = time.monotonic()
            return _detach_if_tensor(entry.value)
        # Check if another thread is already computing
        if key.hex in self._in_flight:
            event = self._in_flight[key.hex]
        else:
            event = threading.Event()
            self._in_flight[key.hex] = event
            event = None           # this thread is the elected compute thread

    if event is not None:          # waiter path
        event.wait(timeout=self._compute_timeout_s)
        with self._lock:
            if entry := self._local.get(key.hex):
                entry.ref_count += 1      # acquire reference on hit
                return _detach_if_tensor(entry.value)
        raise RuntimeError(f"EmbeddingCache: compute for {key.hex} failed")

    try:                           # elected compute path
        value = compute_fn()
        # put() inserts with ref_count=1 so the computing request holds a reference
        # from the moment of insertion. cleanup_request() releases it.
        self.put(key, value, initial_ref_count=1)
        return _detach_if_tensor(value)
    finally:
        with self._lock:
            ev = self._in_flight.pop(key.hex, None)
        if ev:
            ev.set()

Decision: ref_count acquired on cache hit and on elected compute path, released via OmniBase.cleanup_request().

  • What we chose: get() and get_or_compute() increment ref_count before returning a live cache entry. The elected compute path calls put(key, value, initial_ref_count=1), so the entry is inserted already referenced by the computing request rather than momentarily unreferenced between insert and the first acquire. OmniBase.cleanup_request() releases all cache references acquired by the request, including entries inserted by it. TTL eviction skips entries with ref_count > 0. If an entry is expired but still referenced, it becomes eviction-eligible only after the final release.
  • Alternatives: Release on first access completion; fixed TTL with no ref-counting; insert with ref_count=0 and acquire separately (creates a window where a newly inserted entry is immediately eviction-eligible).
  • Rationale: Long streaming TTS or video generation can outlive the nominal TTL. TTL eviction alone is not a safe lifetime boundary for streaming requests. Inserting with ref_count=1 closes the lifecycle gap on the compute path: there is no window between put() and the first acquire() during which TTL eviction could remove the entry. This makes cleanup_request() the definitive release hook and prevents a long synthesis from losing a voice-reference entry midway through generation. This applies to TTS voice cloning, streaming video generation, and any code path that reuses codec-token prefixes.

Required lifecycle semantics:

  • put() inserts with ref_count=1 on the elected compute path, so the entry is never unreferenced between insertion and the computing request's first acquire.
  • get() / get_or_compute() acquires a reference before returning a live cache entry on cache hit (local or remote pull).
  • OmniBase.cleanup_request() releases all cache references acquired or inserted by the request, decrementing ref_count for each.
  • TTL eviction skips entries with ref_count > 0.
  • If an entry is expired but still referenced, it becomes eviction-eligible only after the final release.

Decision: Bounded async publisher (_BoundedPublisher) for store writes, not threading.Thread per put().

  • What we chose: A ThreadPoolExecutor(max_workers=2) with a Semaphore(max_queue) for backpressure. Queue-full publishes are logged and dropped (not blocked, not silently lost).
  • Alternatives: Raw threading.Thread per put() call.
  • Rationale: Unbounded threads under high QPS exhaust file descriptors and memory. Exceptions from raw threads are silently lost. The bounded executor provides backpressure, error counting, and stats() visibility into dropped publishes.

Decision: Two store unavailability modes "degrade" (default) and "strict".

  • What we chose: "degrade" operates local-only without error; "strict" fails at init with a clear message.
  • Alternatives: Always degrade silently; always fail hard.
  • Rationale: "degrade" is the safe default for environments where the store may not always be present. "strict" is correct for production deployments where silent degradation would be operationally misleading. The mode is caller-controlled.

Decision: Two-track remote implementation plan (initial + optimized).

The remote layer tracks two possible implementations:

The performance target for remote pulls (< 20 ms for 100 MB) is gated on the optimized path. The current MooncakeStoreConnector object path should not be assumed to provide zero-copy tensor transfer or this latency target without the planned tensor/EC fast path.

Dependencies & Risks

Item Impact Mitigation
Phase 2 Opt B (tensor-dict fast path) Remote store pulls require the dict fast path for zero-copy transfer Schedule Phase 2 Opt B to land before Phase 4 PR 5 (remote layer)
raw_bytes availability in Qwen3-Omni pipeline Preferred hash mode requires raw bytes before decode at stage boundary Confirm with @wtomin / @linyueqian; fallback to from_decoded_tensor if unavailable (see Open Questions)
False cache hit from incomplete EncoderPreprocessingConfig Incorrect embeddings served silently; for TTS paths, codec token errors are plausible-looking and hard to detect EncoderPreprocessingConfig is a required explicit parameter including audio_codec_version and audio_codec_config_hash; no optional fields; unit tests verify that any changed field produces a different key
compute_timeout_s expiry under slow encoders Waiters in get_or_compute() receive a RuntimeError Default timeout is generous (30s); configurable; failed compute removes the key from _in_flight in the finally block
Persistent memory overhead of local cache max_local_entries embeddings held in GPU/CPU memory Operator-configured; stats() exposes total cache size bytes for monitoring
Streaming TTS / video generation outliving TTL Voice-reference or video embedding evicted mid-generation ref_count acquired on hit; TTL eviction skips ref_count > 0 entries; cleanup_request() is the definitive release hook
RFC #1184 key-namespace misalignment for codec-token prefixes Same voice reference hashed independently at two cache layers, missing intended reuse Voice-reference entries use shared or composable key with prefix-cache layer when value type is a codec-token prefix
Upstream EC path not yet landed Remote pull latency target unachievable on current connector Target gated on optimized EC path; initial implementation documented as best-effort via MooncakeStoreConnector

Performance

Expected improvements over baseline (no cache):

Scenario Before After (cache hit) Improvement
Repeated image, same process ~30–80 ms encoder forward < 1 ms local LRU hit ~100×
Repeated image, cross-node single replica ~30–80 ms encoder + ~10 ms TE transfer ~5–20 ms store RDMA pull (optimized path) 3–10×
Same image, N=4 prefill replicas 4× encoder forward 1× forward + N store pulls ~4× encoder GPU reduction
Repeated TTS voice reference re-encode per request local LRU hit; RFC #1184 prefix reuse encoder + prefix compute eliminated

Performance-relevant design choices:

  • from_raw_bytes hashing: < 0.5 ms for 200 KB JPEG; < 2 ms for 2 MB BF16 tensor. One-time cost per unique input.
  • get_or_compute() lock hold time is minimized: the lock is released before compute_fn() runs. The waiting path uses a lock-free Event.wait().
  • Store publish is fully async and off the encoder forward critical path. Queue-full drops are visible in stats() but do not affect latency.

Trade-offs:

  • Local cache uses GPU/CPU memory proportional to max_local_entries × embedding_size. For 256 entries at 8 MB each (ViT-L image): ~2 GB. Operators must size deliberately.
  • from_decoded_tensor fallback incurs a D2H copy for GPU tensors (~5 ms for 100 MB). Confirming raw_bytes availability in Qwen3-Omni's pipeline eliminates this cost.
  • The < 20 ms remote pull target requires the optimized EC connector path to land first.

4. Correctness & Testing Plans

Core invariant: get_or_compute(key, fn) returns a tensor that is bit-identical to the output of fn() for the given key, whether the result came from a local cache hit, a remote store pull, or a live compute, across all supported modalities.

L1 : Unit tests

  • test_key.py:

    • Same input + same config → same key (all modalities).
    • Any change to any EncoderPreprocessingConfig field → different key; this includes audio_codec_version and audio_codec_config_hash.
    • from_raw_bytes and from_decoded_tensor are each stable within their own hash mode. They are not required to produce the same key, because raw-byte identity and decoded-tensor identity are different namespaces.

  • test_cache.py:

    • LRU eviction triggers when max_local_entries is exceeded.
    • TTL eviction removes expired entries; entries with ref_count > 0 are not evicted even after TTL expiry, they become eviction-eligible only after the final cleanup_request() release.
    • get() and get_or_compute() increment ref_count on a cache hit; put() inserts with ref_count=1 on the elected compute path so the entry is never transiently unreferenced between insertion and the first acquire.
    • OmniBase.cleanup_request() decrements ref_count for all entries acquired or inserted by the request; an entry at ref_count == 0 after release becomes normally evictable.
    • Debug-mode immutability guard catches accidental mutation, or clone-on-return mode prevents mutation from affecting the cached entry. tensor.detach() does not enforce immutability; enforcement is the responsibility of either the debug guard or the consumer calling .clone().

  • test_single_flight.py:

    • N=10 concurrent threads within one process all miss the same key; compute_fn is called exactly once; all 10 threads receive the correct result. (This guarantee is scoped to a single process; cross-process deduplication is best-effort via the store.)
    • Compute failure (compute_fn raises): all waiters receive RuntimeError; key is removed from _in_flight; subsequent calls can retry.

  • test_publisher.py:

    • Queue-full path logs a warning and drops the publish without blocking.
    • Publish errors are counted in stats().
    • store_unavailable_mode="strict" raises at init if store is unreachable; "degrade" logs a warning and continues local-only.

L2 : Integration tests

  • Two-process test (encoder + prefill): encoder computes and publishes an embedding; prefill pulls from store; assert bit-identical result; assert encoder forward() call count = 1 for same-process concurrent misses; for N=2 prefill replicas, assert the second replica can pull from the store after the first successful publish (cross-process deduplication is best-effort, not guaranteed exact-once).
  • E2E Qwen3-Omni with enable_embedding_cache: true: send same image twice in consecutive requests; assert encoder forward() call count = 1; assert model output matches single-node reference within BF16 tolerance.
  • store_unavailable_mode="degrade" fallback: with store unreachable, encoder runs normally per-request; no error raised; output matches reference.
  • TTS voice-reference streaming test: send a streaming TTS request with a shared voice reference; assert the cache entry's ref_count > 0 throughout generation; assert the entry is not evicted mid-stream even if TTL elapses; assert ref_count returns to 0 after cleanup_request().

Smoke check

Single-node Qwen3-Omni request with enable_embedding_cache: true completes without error. Cache stats() returns local_entries=1 after the first request.

5. Open Questions & Discussions

  1. raw_bytes availability in Qwen3-Omni pipeline: Does Qwen3-Omni's input processing pipeline preserve raw JPEG/PCM bytes at the encoder stage boundary, or does it decode before the stage is entered? If decode-first, from_decoded_tensor must be used, incurring a D2H copy for GPU tensors. Requesting confirmation from @wtomin /@linyueqian.

  2. EncoderPreprocessingConfig ownership and versioning: Should this dataclass live in embedding_cache/key.py (self-contained) or be derived from each model's existing processor/config class? The latter is more convenient for callers but couples the cache key format to model configuration evolution any field rename or addition (including future codec fields) must bump cache_schema_version to prevent stale hits across upgrades.

  3. Store TTL coordination across replicas: If a local entry expires while the store entry is still valid, the next local miss correctly pulls from the store. But if the store entry expires first, a warm local hit returns an embedding that new replicas cannot access from the store. Should store TTL be set as a multiple of local TTL (e.g., store_ttl = local_ttl × expected_N_replicas)? Or should a successful local hit refresh the store TTL ("touch on local hit")?

  4. ref_count release point for streaming generation: get() / get_or_compute() should acquire a reference, and OmniBase.cleanup_request() should release all cache entries acquired by the request. This needs review against TTS voice cloning, streaming video generation, and any code path that reuses codec-token prefixes.

  5. get_or_compute vs. reserve / commit / abandon API: The compute_fn closure API is clean but may be awkward for callers with complex forward signatures. An alternative: reserve(key) → bool (True if caller should compute), commit(key, embedding), abandon(key) giving the caller explicit control while still providing single-flight protection. Should we offer both?

6. References

Contributor guide

Research direction
Study the existing vLLM Omni embedding transfer path, understand the MooncakeTransferEngineConnector's 1:1 constraint, and prototype a content addressed cache with local LRU+TTL and remote store publish/pull using MooncakeStoreConnector. Focus on the EncoderPreprocessingConfig key construction and single flight coalescing.
Tech stack
pythonpytorch
Domain
backendinfrastructureai infra agents
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonDistributed SystemsPyTorch
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20