[RFC]: Mooncake and vLLM-Omni Collaboration Roadmap

Motivation.

1. Overview

This RFC outlines the proposed collaboration roadmap between the Mooncake project and vllm-omni. The goal of this collaboration is to enhance the disaggregated inference architecture for omni-modality models, optimize transfer engine components, introduce high-performance communication backends, and implement efficient data sharing for multimodal embeddings.

2. Motivation

As vllm-omni scales to support more complex omni-modality models, efficient resource utilization and low-latency communication across disaggregated components become critical. Based on our recent meeting, we have identified four key strategic areas where Mooncake and vllm-omni can collaborate to significantly improve system performance, particularly focusing on the Qwen-omni architecture, TE (Transfer Engine) connector efficiency, underlying network transport, and the handling of massive multimodal embedding data.

3. Proposed Roadmap

The collaboration is structured into four main phases:

Phase 1: Qwen-Omni Disaggregation Architecture Support

Objective: Enable and validate the disaggregated inference architecture specifically for Qwen-omni models.

• Details:
  • Integrate Mooncake's disaggregation capabilities with the vllm-omni framework to support Qwen's omni-modality characteristics (e.g., handling multimodal KV-cache or intermediate state transfers).
  • Define the APIs and boundaries between the prefill (context/vision encoding) and decode nodes for Qwen-omni.
  • Ensure correctness and stability of the model outputs when running in a separated prefill-decode architecture.

Phase 2: Mooncake TE Connector Optimization via Performance Profiling

Objective: Identify bottlenecks and optimize the Mooncake Transfer Engine (TE) connector.

• Details:
  • Profiling: Conduct deep performance profiling on the current Mooncake TE connector implementation within the vllm-omni pipeline. Focus on serialization/deserialization overhead, memory copies, and data transfer latency.
  • Optimization: Refactor and optimize the TE connector based on profiling results. This may include implementing zero-copy mechanisms, better memory pool management, and reducing CPU overhead during cross-node data transfers.
  • Metrics: Establish baseline metrics and target metrics (e.g., Time to First Token (TTFT) and inter-node transfer latency) to measure optimization success.

Phase 3: High-Performance ZMQ Backend (RDMA Integration)

Objective: Replace or upgrade the standard ZMQ communication layer with a high-performance backend.

• Details:
  • Develop a high-performance version of the ZeroMQ (ZMQ) communication layer.
  • Introduce RDMA (Remote Direct Memory Access) support to bypass the OS networking stack, drastically reducing communication latency and freeing up CPU cycles.
  • Ensure backward compatibility or provide a seamless fallback mechanism to standard TCP ZMQ for environments where RDMA/RoCEv2 is not available.

Phase 4: Encoder Embedding Data Sharing

Objective: Efficiently manage and share encoder embedding data across disaggregated nodes to minimize communication overhead.

• Details:
  • Multimodal Context Handling: Omni-modality models generate heavy embedding data from visual, audio, or other sensory encoders. Implement a mechanism to efficiently share these continuous embeddings between the standalone encoder components and the LLM backbone (prefill/decode stages).
  • Zero-Copy & Caching: Leverage the optimized TE connector (from Phase 2) and RDMA backend (from Phase 3) to enable zero-copy reads or distributed caching (via Mooncake) of encoder embeddings.
  • Latency Reduction: Prevent redundant data transfers of heavy multimodal embeddings, ensuring that prefill and decode instances can access the required context with maximum throughput and minimum latency.

4. Success Criteria

• Functionality: Qwen-omni runs successfully on a disaggregated architecture using Mooncake + vllm-omni without degradation in model quality.
• Performance: Demonstrable reduction in transfer latency for the TE connector post-optimization, and highly efficient routing of encoder embeddings.
• Scalability: The RDMA-backed ZMQ implementation shows significant throughput improvements and latency reductions in multi-node deployments compared to the baseline TCP implementation.

5. Next Steps & Action Items

1. Assign Owners: Determine assignees from both the Mooncake and vllm-omni teams for each of the four phases.
2. Design Docs: Draft a detailed technical design document specifically for the Phase 1 Qwen-omni disaggregation and Phase 4 Embedding Data Sharing.
3. Environment Setup: Set up the profiling environment to capture the current baseline performance of the TE connector.

Proposed Change.

Please provide the detailed design document of the RFC using the template.

Feedback Period.

No response

CC List.

@hsliuustc0106

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.