grafana/mimir

[mixin] filterNodeDiskContainer: make sda filter configurable

Open

#6,925 opened on Dec 13, 2023

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Go (808 forks)auto 404
good first issuemixin

Repository metrics

Stars
 (5,181 stars)
PR merge metrics
 (PR metrics pending)

Description

Is your feature request related to a problem? Please describe.

Running Mimir on Kubernetes on a relatively unknown cloud provider, the "Disk reads" and "Disk writes" panels show no data because of the filter used to ignore the root device.

Digging in mimir-mixin , I've found the use of filterNodeDiskContainer, which assumes the root device is sda.

On this cloud provider, the root device is shown as /dev/vda on the host, and each subsequent volume mount shows as sda, sdb, sdc... (note that sda is not skipped: it's the first volume mount). As a result, the expression actually filters out the device on which /data is mounted and returns 0 rows.

(At least, that's my interpretation, which could be flawed: I have two ingesters using /dev/sda and one using /dev/sdb according to container_fs_writes_bytes_total, yet I get no result at all)

Describe the solution you'd like

Assuming my interpretation of the problem is correct, I would like the ability to configure the device filter as a variable in the mixin, and perhaps even as a helm value to not have to build the mixin myself is this value needs to be customized.

Describe alternatives you've considered

Perhaps a completely different approach to the expression that doesn't rely on this filtering could be used for the same results. I'm relatively inexperienced in promql-fu though, and lack the knowledge to explore this possibility further.

Additional context

I've yet to write or even compile mixins myself and have no knowledge so far of jsonnet, hence why this is an issue instead of a PR. However, I'd be happy to contribute with some light guidance!

Cheers,

Contributor guide