[RFC]: Ray Connector

Motivation.

Disclosure: this RFC is authored with help from AI.

vLLM-Omni already uses on Ray for distributed inference. But users still need to set up external KV connector services (Mooncake or Yuanrong) in addition to the Ray cluster. We may, instead, implement a connector directly using Ray object store as the backing storage, easing the deployment process.

Proposed Change.

1 Overview

Add a new omni connector, RayConnector, that uses the Ray object store for storage, providing multi-node KV cache transfer with zero additional infrastructure beyond a running Ray cluster.

1.1 Motivation

vLLM-Omni already relies on Ray for distributed execution. However, existing multi-node connectors require setting up additional external services in addition to a Ray cluster:

Connector	Extra Setup Required
MooncakeStoreConnector	Mooncake Master
MooncakeTransferEngineConnector	ZMQ (not really something the user has to setup, but do require firewall configuration etc.)
YuanrongConnector	Yuanrong Datasystem workers and etcd

This creates an operational barrier for teams that already deploy on Ray. A Ray-native connector would eliminate the need for any external service by using the Ray object store. It handles cross-node object transfer transparently via Ray's distributed object protocol, while providing performance comparable with MooncakeTransferEngine or other RDMA based connectors with the help of Ray Direct Transport (RDT): https://docs.ray.io/en/latest/ray-core/direct-transport.html#direct-transport

1.2 Target

Feature

• New connector class: RayConnector(OmniConnectorBase)

Accuracy

• Compatibility: Must work with existing stage configurations without breaking changes. Existing pipelines that do not configure RayConnector are unaffected. And should work with chunked/async transfer manager etc.

Performance

• Single-node: Expect performance close or equal to SharedMemoryConnector on single node clusters.
• Multi-node: Expect performance close or equal to MooncakeTransferEngineConnector on multi node RDMA equipped clusters.

2 Design

2.1 Overview of Design

Two Ray primitives:

1. Ray object store — data plane for storing blobs.
2. RayRefStore named actor — distributed dict mapping keys to ObjectRefs (which in turn maps to objects).

  Stage 0 (Sender)                          Stage 1 (Receiver)
  ┌─────────────────┐                      ┌─────────────────┐
  │ serialize(data) │                      │ actor.get(key)  │
  │ ray.put(blob)   │                      │ ray.get(ref)    │
  │ actor.put(key,  │                      │ deserialize()   │
  │         obj_ref)│                      │ return data     │
  └─────────────────┘                      └─────────────────┘
            │                                       ▲
            ▼                                       │
  ┌──────────────────────────────────────────────────┐
  │           RayRefStore (Named Actor)              │
  │  _store: dict[str, ObjectRef]                    │
  │  put / get / delete / delete_by_prefix / keys    │
  └───────────────────┬──────────────────────────────┘
                      │
  ┌───────────────────▼──────────────────────────────┐
  │            Ray Object Store                      │
  │  Node A ──auto cross-node transfer──► Node B     │
  └──────────────────────────────────────────────────┘

Key design decisions:

Decision	Rationale
Actor based distributed hash map instead of metadata-passing	get() is self-sufficient — no dependency on control-plane metadata. Also necessary for compatibility with async transfer manager, see discussion in https://github.com/vllm-project/vllm-omni/pull/1643

2.2 Use Cases

1. Multi-Node Inference on Ray Cluster

Deploy Qwen3-Omni (Thinker on Node A → Talker on Node B) with RayConnector between stages. No Mooncake/etcd/RDMA setup needed — the RayRefStore actor and Ray object store run inside the existing Ray cluster.

2. Development / CI Testing

Test multi-stage pipelines with connector-based transfer on a single-node Ray cluster (ray start --head). Exercises the full code path (serialize → ray.put() → actor registration → actor lookup → ray.get() → deserialize) without special hardware.

2.3 API Design

Current Component Changes

• factory.py: Add _create_ray_connector() + register_connector("RayConnector", ...). Additive; no existing behavior modified.
• disaggregated_inference.md: Add RayConnector row to Connector Choices table.

New APIs

RayRefStore (Ray Actor) — ray_connector.py

Method	Signature	Description
put	(key: str, ref: ObjectRef) -> None	Register ref under key
get	(key: str) -> ObjectRef | None	Look up ref by key
delete	(keys: list[str]) -> list[ObjectRef]	Remove keys, return freed refs
delete_by_prefix	(prefix: str) -> list[ObjectRef]	Remove all keys matching prefix
keys	() -> list[str]	List all keys (debug/health)

RayConnector(OmniConnectorBase) — ray_connector.py

Config parameters:

Parameter	Default	Description
ray_address	"auto"	Ray cluster address
ray_namespace	None	Ray namespace for isolation
actor_name	"vllm_omni_ray_ref_store"	Named actor identity; override for multi-pipeline isolation

YAML example:

runtime:
  connectors:
    connector_of_ray:
      name: RayConnector
      extra:
        ray_address: "auto"

stage_args:
  - stage_id: 0
    output_connectors:
      to_stage_1: connector_of_ray
  - stage_id: 1
    input_connectors:
      from_stage_0: connector_of_ray

Implementation:

TBD

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.