Bug: Mimir creates thousands of TCP connections and DNS requests to/for S3 storage and does not use existing connections.

What is the bug?

I am running am Mimir environment on prem. It contains 4 monolithic instances in a cluster sending metrics to our local s3 (Nutanix / MinIO). I configured mimir to only use 25 concurrent connections per host, 25 idle connections per host and 1h idle timeout. I set "dualstack" to false to only use IPv4.

The problem is that mimir opens in a ~40s tcpdump 5000 tcp connections to the s3 server. this results in 5000 DNS requests per 90s. In addition it send AAAA requests (IPv4) which I disabled via "dualstack = false".

How to reproduce it?

Connect Grafana Mimir v 3.0.0 to an s3 storage. check the dns communication and the requests made to s3 storage. Run some queries which need s3.

What did you think would happen?

Open up to configured max value of concurrent connections per host (25) and keep these connections open for up to 1h (idle) and send all requests through these existing connections. Only on DNS request per each connection.

In addition no DNS IPv4 requests (AAAA) if dualstack is disabled.

What was your environment?

Grafana Mimir 3.0.0 binary on RHEL8 in monolithic mode.

Any additional context to share?

N/A