grafana/loki

Loki uses HTTP for S3 instead of HTTPS

Open

#7,290 opened on Sep 29, 2022

View on GitHub
 (8 comments) (0 reactions) (1 assignee)Go (3,997 forks)batch import
good first issuehelp wantedtype/docs

Repository metrics

Stars
 (28,187 stars)
PR merge metrics
 (Avg merge 5d 20h) (522 merged PRs in 30d)

Description

Describe the bug When using the s3 storage backend with an s3-compatible API, Loki (allegedly) attempts to connect to the specified endpoint with an http scheme rather than https, even though s3_storage_config.insecure is false.

This is an issue for me as the object store I have to use only accepts the https scheme and not http (unrelated to the port).

To Reproduce Using the official Loki Docker image version 2.6.1, and pushing some logs to it.

Relevant config:

schema_config:
  configs:
    - from: 2022-01-01

      store: boltdb-shipper
      object_store: s3

      schema: v11
      index:
        prefix: loki_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: s3

  aws:
    s3: s3://access_key:secret_key@endpoint:port/bucket
    s3forcepathstyle: true

    # HTTPS
    insecure: false
    sse_encryption: false

Expected behavior Loki runs and is able to sync index and chunks on the provided object store through https.

Actual behaviour Loki tries to use http. Which in my situation correctly results in empty replies.

Environment:

  • Infrastructure:
  • Loki image : Official Docker in an Ubuntu virtual machine
  • Object store : Swift (OpenStack) ; (I need to use i through S3, and anyhow this seems like an issue that could affect other users than me who only have S3)
  • Deployment tool: docker compose

Screenshots, Promtail config, or terminal output

level=info ts=2022-09-29T11:31:16.471455302Z caller=loki.go:374 msg="Loki started"
level=info ts=2022-09-29T11:31:19.413882007Z caller=scheduler.go:682 msg="this scheduler is in the ReplicationSet, will now accept requests."
level=info ts=2022-09-29T11:31:19.414200346Z caller=worker.go:209 msg="adding connection" addr=172.18.0.2:9095
level=info ts=2022-09-29T11:31:21.472486286Z caller=compactor.go:386 msg="this instance has been chosen to run the compactor, starting compactor"
level=info ts=2022-09-29T11:31:21.472818993Z caller=compactor.go:413 msg="waiting 10m0s for ring to stay stable and previous compactions to finish before starting compactor"
level=info ts=2022-09-29T11:31:26.415269849Z caller=frontend_scheduler_worker.go:101 msg="adding connection to scheduler" addr=172.18.0.2:9095
level=info ts=2022-09-29T11:32:15.310369764Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-29T11:32:15.314246588Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-29T11:33:04.406700343Z caller=table_manager.go:180 msg="downloading all files for table loki_19264"
level=error ts=2022-09-29T11:33:04.468122657Z caller=index_set.go:265 table-name=loki_19264 user-id=fake msg="sync failed, retrying it" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2Ffake%2F\": EOF"
level=error ts=2022-09-29T11:33:04.477475008Z caller=index_set.go:265 table-name=loki_19264 user-id=fake msg="sync failed, retrying it" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2Ffake%2F\": EOF"
level=error ts=2022-09-29T11:33:04.477573053Z caller=index_set.go:104 table-name=loki_19264 user-id=fake msg="failed to initialize table loki_19264, cleaning it up" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2Ffake%2F\": EOF"
level=error ts=2022-09-29T11:33:04.477611957Z caller=table.go:294 table-name=loki_19264 msg="failed to init user index set fake" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2Ffake%2F\": EOF"
level=error ts=2022-09-29T11:33:04.477647243Z caller=table.go:312 table-name=loki_19264 org_id=fake msg="index set fake has some problem, cleaning it up" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2Ffake%2F\": EOF"
ts=2022-09-29T11:33:04.477742042Z caller=spanlogger.go:80 table-name=loki_19264 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=70.030233ms
level=error ts=2022-09-29T11:33:04.488331735Z caller=index_set.go:265 table-name=loki_19264 msg="sync failed, retrying it" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2F\": EOF"
level=error ts=2022-09-29T11:33:04.498580977Z caller=index_set.go:265 table-name=loki_19264 msg="sync failed, retrying it" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2F\": EOF"
level=error ts=2022-09-29T11:33:04.498710982Z caller=index_set.go:104 table-name=loki_19264 msg="failed to initialize table loki_19264, cleaning it up" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2F\": EOF"
level=error ts=2022-09-29T11:33:04.498764433Z caller=table.go:294 table-name=loki_19264 msg="failed to init user index set " err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2F\": EOF"
level=error ts=2022-09-29T11:33:04.498799709Z caller=table.go:312 table-name=loki_19264 org_id=fake msg="index set  has some problem, cleaning it up" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2F\": EOF"
ts=2022-09-29T11:33:04.498889649Z caller=spanlogger.go:80 table-name=loki_19264 org_id=fake level=info msg="downloaded index set at query time" duration=21.080771ms
ts=2022-09-29T11:33:04.499242534Z caller=spanlogger.go:80 user=fake method=query.Label level=info org_id=fake latency=fast query_type=labels length=1h0m0s duration=94.005268ms status=200 label= throughput=0B total_bytes=0B total_entries=2
level=info ts=2022-09-29T11:33:04.501500869Z caller=metrics.go:170 component=frontend org_id=fake latency=fast query_type=labels length=1h0m0s duration=97.358476ms status=200 label= throughput=0B total_bytes=0B total_entries=2

Contributor guide

Research direction
Examine how the S3 endpoint URL is constructed in the Loki configuration, specifically where the scheme (http vs https) is determined based on the 'insecure' flag. Look in the storage/aws package for the S3 client initialization and ensure the scheme respects the setting.
Tech stack
go
Domain
backend
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
Go
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60