grafana/mimir

helm: Stateful components emptyDir inMemory option

Open

#8,169 opened on May 22, 2024

View on GitHub
 (6 comments) (0 reactions) (0 assignees)Go (808 forks)auto 404
enhancementhelmhelp wanted

Repository metrics

Stars
 (5,181 stars)
PR merge metrics
 (PR metrics pending)

Description

Describe the bug

"emptyDir: {}" is hardcoded in statefulset templates. In our case this causes impossibility to set "medium: Memory" for compactor's persistent volumes when one can't use PVC. Therefore that volumes always use k8s ephemeral storage, the default behavior of which has changed greatly in k8s v1.27. As a result, the compactor pods go into the "Completed" state a few seconds after start (time depends of load). If you perform "kubectl describe" compactor pod at this moment, you can see error "Pod ephemeral local storage usage exceeds the total limit of containers <some_size>" This is default for k8s 1.27+ and if you aren't cluster admin you can do nothing to fix this behavior.

Another stateful components (Ingester etc) have similar problem.

It is necessary to add the ability to set emptyDir location (memory/disk) in chart values. The Grafana Tempo chart provides such an opportunity for stateful components (e.g. ingester) and we have no similar problems with Tempo after upgrade k8s to 1.27.

Contributor guide