Support OpenAI Responses API affinity across multiple providers/deployments

Why do you need it?

OpenAI Responses API contains protocol-level state that is tied to the provider or deployment that produced the previous response.

The important fields are:

• previous_response_id
• reasoning item ids
• encrypted_content

When AI Route load balances a follow-up request to a different provider or deployment, the upstream provider cannot resolve state created by another backend. This breaks multi-turn Responses API workloads under multi-provider or multi-deployment routing.

The problem is visible with Azure OpenAI multi-deployment routing: a first request can succeed, while a follow-up request with previous_response_id fails when routed to another deployment.

How could it be?

Add optional Responses API affinity support in ai-proxy.

Recommended configuration:

responsesAffinity:
  enabled: true
  encodeResponseId: true
  encodeItemId: true
  encodeEncryptedContent: true
  providerUnavailablePolicy: failFast

Request processing:

1. Detect Responses API requests.
2. Decode provider or deployment identity from previous_response_id.
3. If previous_response_id is absent, scan input items for encoded reasoning item ids.
4. If encoded item ids are absent, scan encrypted_content for affinity metadata.
5. If an affinity target is found, route to that provider or deployment.
6. Restore the original upstream ids and content before sending the request to the provider.
7. If no affinity marker is found, use normal AI Route provider selection.

Response processing:

1. Encode provider or deployment identity into response.id.
2. Encode reasoning item ids that can be referenced by later requests.
3. Encode or wrap encrypted_content affinity metadata.
4. Preserve the client-facing OpenAI Responses API shape.

Streaming processing:

1. Apply the same encoding rules to streaming response events.
2. Preserve SSE event type and ordering.
3. Preserve compatibility with OpenAI SDK clients.

Provider unavailable policy:

• failFast is required for stateful Responses API references.
• Routing to another provider is not a valid fallback when previous_response_id, reasoning item ids, or encrypted_content are bound to one provider.

LiteLLM reference design

LiteLLM provides Responses API continuity through router pre-call checks:

• responses_api_deployment_check: routes requests with previous_response_id back to the originating deployment.
• encrypted_content_affinity: routes encrypted-content follow-up requests back to the originating deployment.
• session_affinity: supports explicit session-id based stickiness.
• deployment_affinity: supports broader deployment stickiness.
• deployment_affinity_ttl_seconds: controls deployment affinity TTL.

LiteLLM also supports model-group scoped affinity configuration, so affinity can be enabled for stateful model groups without forcing it on all models.

References:

• https://docs.litellm.ai/docs/response_api
• https://docs.litellm.ai/docs/proxy/load_balancing

The same pattern maps to Higress:

Responses request
  -> decode previous_response_id / item id / encrypted_content
  -> force provider/deployment selection when affinity metadata exists
  -> restore original upstream protocol fields
  -> call selected provider
  -> encode response ids and encrypted_content for later calls

Acceptance criteria

• Follow-up Responses API requests with previous_response_id route to the originating provider or deployment.
• Requests containing encoded reasoning item ids route to the originating provider or deployment.
• Requests containing encoded encrypted_content route to the originating provider or deployment.
• Original upstream ids and content are restored before forwarding to the provider.
• Client-facing response ids and item ids carry enough metadata for later routing.
• Streaming and non-streaming Responses API both work.
• Existing AI Route behavior is unchanged when Responses API affinity is disabled.

Other related information

This feature is different from generic sticky session support.

Generic sticky session uses an explicit session key. Responses API affinity is protocol-level routing based on IDs and encrypted content returned by the provider.