grafana/loki

grafana-loki lacks basic feature of extracting nested json labels

Open

#6,994 opened on Aug 29, 2022

View on GitHub
 (11 comments) (9 reactions) (0 assignees)Go (3,997 forks)batch import
component/fluent-bit-plugingood first issuetype/feature

Repository metrics

Stars
 (28,187 stars)
PR merge metrics
 (Avg merge 5d 20h) (522 merged PRs in 30d)

Description

Is your feature request related to a problem? Please describe. I am running a java spring-boot application on AWS ECS and want to ship logs to loki/grafana. In order to have the java stack trace as a single log line in grafana I log as json to console using

<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>

The resulting json from AWS ECS looks like this:

{
  "container_id": "ea7b47de37024af8a71629fc4c435e09-285964202",
  "container_name": "backend",
  "ecs_cluster": "dev-fargate",
  "ecs_task_arn": "arn:aws:ecs:eu-central-1:xxx:task/dev-fargate/ea7b47de37024af8a71629fc4c435e09",
  "ecs_task_definition": "dev-backend:72",
  "log": {
    "@timestamp": "2022-08-29T08:13:22.893Z",
    "@version": "1",
    "message": "Running with Spring Boot v2.7.3, Spring v5.3.22",
    "logger_name": "com.example.Application",
    "thread_name": "main",
    "level": "DEBUG",
    "level_value": 10000
  }
}

Describe the solution you'd like I want to extract also lables from the log element, i.e. log level, logger name, ... and only keep the log.message as the log text.

So therefore I need configuration like this (cloudformation yaml config), which is currently not working/supported:

...
LogConfiguration:
  LogDriver: awsfirelens
  Options:
    Name: grafana-loki
    Url: https://loki:3000/loki/api/v1/push
    Labels: "{source=\"console\"}"
    LabelKeys: container_id,ecs_task_arn,ecs_task_definition,ecs_cluster,container_name,log.level,log.logger_name,log.thread_name
    RemoveKeys: source,log.level_value,log.@version
    LineFormat: key_value
    insecure_skip_verify: true
  SecretOptions:
    - Name: TenantID
      ValueFrom: !Sub arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/config/loki/tenant-id
- Name: log_router
  Image: grafana/fluent-bit-plugin-loki:2.6.1-amd64
  Essential: false
  Memory: 512
  Cpu: 256
  FirelensConfiguration:
    Type: fluentbit
    Options:
      enable-ecs-log-metadata: true
  LogConfiguration:
    LogDriver: awslogs
    Options:
      awslogs-stream-prefix: firelens
      awslogs-group: !Ref LogGroup
      awslogs-region: !Ref AWS::Region
...

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. I have also tried using LabelMapPath, but this didn't work either. Also tried LineFormat=json but has same result.

Additional context This is really a basic feature I bet a lot of people need!

And maybe another option would be great to specify the final message target, i.e. "log.message" in my case... and other properties should be ignored. Would extremly ease the config avoiding the need of specifying tons of "RemoveKeys"...

Contributor guide