grafana/mimir

Add alert for ring members in non-active state

Open

#2,342 opened on Jul 8, 2022

View on GitHub
 (0 comments) (0 reactions) (1 assignee)Go (808 forks)auto 404
enhancementgood first issuemonitoring/alerts

Repository metrics

Stars
 (5,181 stars)
PR merge metrics
 (PR metrics pending)

Description

As an extension to #1960, we should consider adding an alert when ring members are in a non-active state for an unexpected amount of time.

I had suggested something like min by(cluster, namespace) (cortex_ring_members{name="distributor",state!="ACTIVE"}) > 0 for longer than 30min, but feedback from @pracucci highlighted:

  • For some components not supporting the auto-forget feature (e.g. distributors till very recent times) it's expected to have non-ACTIVE ring members for a long time and everything works as expected anyway
  • I think the query you're suggesting would suffer the false positives issue during rollouts

Contributor guide

Research direction
Tech stack
Domain
Issue type
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Prerequisites
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.