knative/eventing

Inform user of dropped events

Open

#8,572 opened on Apr 24, 2025

View on GitHub
 (18 comments) (1 reaction) (0 assignees)Go (632 forks)auto 404
good first issuekind/feature-requesttriage/accepted

Repository metrics

Stars
 (1,551 stars)
PR merge metrics
 (PR metrics pending)

Description

Problem Whenever events are being dropped by event mesh, we should inform the end-user about that. Just logging is not enough for such a case. IMHO, the best to create a dedicated warning level Kubernetes event, which is designed for such a case: https://www.cncf.io/blog/2023/03/13/how-to-use-kubernetes-events-for-effective-alerting-and-monitoring/

Background With the recent addition of the EventTransform API, the chance of misconfiguration by the end-user raises greatly, as they could easily create infinite loops by transforming the source event in a way the original trigger matches again. The TTL mechanism should eventually break that loop, but in that case, we should inform the end-user about it.

Persona: System Operator, Developer

Exit Criteria The end-user could easily identify the Event Mesh configuration is invalid, and some messages are getting dropped. The best would be to allow use of well known K8s monitoring tools - using K8s events should be adequate.

Time Estimate (optional): 5d (events are dropped in number of places across the codebase)

Additional context (optional) My proposal to solve this is to reconcile a Kubernetes event whenever such a situation occurs. Such an event may look like:

apiVersion: events.k8s.io/v1
kind: Event
eventTime: 2025-04-23T09:09:54Z
metadata: 
  namespace: user-evening-bits-namespace
  name: knative-eventing-mt-broker.1838e7822e31b835
  labels:
    eventing.knative.dev/event-type: my-event-type
    eventing.knative.dev/event-source: my-event-source
regarding: 
  apiVersion: eventing.knative.dev/v1
  kind: Broker
  name: default
  namespace: user-evening-bits-namespace
type: Warning
action: event-dropped
reason: EventLoop
note: Event of type "my-event-type" and source "my-event-source" has reached internal TTL, which most likely signals an event loop. The event was dropped.
series: 
  count: 351
  lastObservedTime: 2025-04-23T09:09:54Z

Notice the series.count. It should be bumped whenever "same event" occurs again. In this case, the reconciler should match the K8s events using metadata.labels of eventing.knative.dev/event-type and eventing.knative.dev/event-source, and bump the series.count when the next message is being dropped.

Contributor guide