grafana/mimir

[helm] Ingress routes to non-existent mimir-alertmanager service when zoneAwareReplication is enabled

Open

#14,334 opened on Feb 12, 2026

View on GitHub
 (3 comments) (5 reactions) (0 assignees)Go (808 forks)auto 404
bughelmhelp wanted

Repository metrics

Stars
 (5,181 stars)
PR merge metrics
 (PR metrics pending)

Description

What is the bug?

After upgrading from mimir-distributed-5.8.0 to mimir-distributed-6.0.5, the Ingress for alertmanager paths breaks with:

<error: services "mimir-alertmanager" not found>

This affects all alertmanager ingress paths:

  • /alertmanager
  • /multitenant_alertmanager/status
  • /multitenant_alertmanager/configs
  • /api/v1/alerts

Root Cause

PR #13064 introduced this breaking change in 6.0.0:

[CHANGE] Ingress: Change default routes to point to non-headless services. #13064

The ingress template was changed to use mimir-alertmanager (ClusterIP) instead of mimir-alertmanager-headless. However, when alertmanager.zoneAwareReplication.enabled=true, the chart never creates a mimir-alertmanager ClusterIP service — only zone-specific services and the headless service are created.

Services that exist with zoneAwareReplication enabled:

mimir-alertmanager-headless   ClusterIP   None    8080/TCP 
mimir-alertmanager-zone-a     ClusterIP   <IP>   8080/TCP 
mimir-alertmanager-zone-b     ClusterIP   <IP>    8080/TCP 
mimir-alertmanager-zone-c     ClusterIP   <IP>   8080/TCP 

Steps to Reproduce

  1. Install mimir-distributed chart with alertmanager.zoneAwareReplication.enabled: true
  2. Enable ingress.enabled: true
  3. Upgrade from any 5.x chart to 6.0.x
  4. Run kubectl describe ingress mimir -n <namespace>
  5. Observe <error: services "mimir-alertmanager" not found> on alertmanager paths

Expected Behavior

Ingress should route alertmanager paths to mimir-alertmanager-headless when zoneAwareReplication.enabled=true, since the non-headless service is not created in that configuration.

Actual Behavior

Ingress tries to route to mimir-alertmanager which does not exist, causing all alertmanager paths to return errors.

Workaround

Override alertmanager ingress paths in values.yaml to explicitly use the headless service:

ingress:
  paths:
    alertmanager:
    alertmanager-headless:
      - path: /alertmanager
      - path: /multitenant_alertmanager/status
      - path: /multitenant_alertmanager/configs
      - path: /api/v1/alerts

Suggested Fix

The ingress template should conditionally use the headless service for alertmanager when zoneAwareReplication.enabled=true:

{{- if .Values.alertmanager.zoneAwareReplication.enabled }}
name: mimir-alertmanager-headless
{{- else }}
name: mimir-alertmanager
{{- end }}

Environment

Chart version mimir-distributed-6.0.5
Mimir version 3.0.1
Kubernetes AKS (Azure)
Ingress controller nginx
zoneAwareReplication enabled (3 zones)

Helm values (relevant section)

alertmanager:
  zoneAwareReplication:
    enabled: true
    zones:
      - name: zone-a
        ...
      - name: zone-b
        ...
      - name: zone-c
        ...

ingress:
  enabled: true
  hosts:
    - mimir.example.com

Relevant Links

Contributor guide