Support provider-level session affinity for AI Route / ai-proxy

Why do you need it?

When an AI Route is configured with multiple AI providers or multiple deployments of the same provider, some workloads need provider-level session affinity.

Today, AI Route can distribute traffic across multiple upstream AI providers, but there is no generic way to pin a logical session to the same provider instance.

This is important for LLM gateway scenarios where the selected provider/deployment may hold provider-side state, cache, or session context. Without provider-level affinity, consecutive requests from the same logical session may be routed to different providers, causing inconsistent behavior or request failures.

The session key should not be limited to HTTP headers only. In LLM protocols, the session key may come from different places, for example:

• Request header
• Cookie
• JSON body field
• Dynamic metadata set by an earlier plugin

A concrete example is a request body field such as:

{
  "model": "gpt-4o",
  "callOptions": {
    "stickySessionId": "session-123"
  }
}

In this case, all requests with the same logical session key should be pinned to the same named provider instance.

How could it be?

AI Route / ai-proxy could support provider-level session affinity with configurable session key extraction and provider selection.

Example configuration idea:

  sessionAffinity:
    enabled: true
    key:
      source: body
      jsonPath: "$.callOptions.stickySessionId"
    scope:
      - consumer
      - model
    mode: persistent
    ttlSeconds: 3600
    onProviderUnavailable: fallback

  Possible key sources:

  key:
    source: header
    name: x-session-id

  key:
    source: cookie
    name: llm_session

  key:
    source: body
    jsonPath: "$.callOptions.stickySessionId"

  key:
    source: metadata
    name: selected_session_key

Expected behavior:

1. For the same session key, AI Route / ai-proxy should consistently select the same named provider instance.
2. The affinity mapping should be scoped by configurable dimensions such as consumer, model, route, or custom metadata.
3. The selected provider instance should be observable in logs, metrics, or filter state.
4. Provider-unavailable behavior should be explicit and configurable:
   • fail fast
   • fallback to another provider and update affinity
   • fallback without updating affinity
5. The feature should support both persistent affinity with TTL and hash-based affinity.

This would allow AI Route to support more advanced LLM gateway scenarios while keeping the provider selection logic inside Higress instead of forcing users to generate one route per provider instance.

Other related information

This feature is related to, but different from, protocol-specific affinity.

Provider-level session affinity is the generic routing foundation. Some protocols, such as OpenAI Responses API, also need protocol-aware affinity based on response IDs or encrypted reasoning content. That protocol-specific requirement is better tracked separately.