perf: tooling and guidance for profiling etcd revision overhead of Tekton objects

Problem

Tekton operators and platform builders need to understand how much etcd storage their PipelineRun workloads consume, but there is no tooling or documentation to help analyze this.

etcd stores every mutation of a Kubernetes object as a full copy (a "revision"). A single TaskRun may accumulate 10-20 revisions during its lifecycle as multiple controllers update it (Pipeline Controller, Chains, Results, and platform-specific controllers). Between compaction cycles, these revisions represent the actual etcd storage cost — which can be 10-20x the "live snapshot" size of the object.

Understanding this revision profile is critical for:

• Identifying which controllers contribute the most revisions (and therefore etcd pressure)
• Distinguishing between revisions that reflect meaningful state changes vs. no-op or cosmetic updates
• Making data-driven decisions about which optimizations (fewer objects, smaller objects, fewer writes) will have the most impact
• Capacity planning for concurrent PipelineRun limits

What's needed

1. Documentation on how to analyze etcd revision overhead

A guide explaining how operators can profile their Tekton workloads' etcd impact. This would cover:

• How to use etcdctl to inspect revision history for a specific key (TaskRun, PipelineRun)
• How to calculate per-object revision count and total storage cost
• How to identify which controller/actor created each revision (using managedFields or audit logs)
• How to estimate total etcd cost of a pipeline execution (sum across PipelineRun + TaskRuns + Pods + Events)

2. OpenTelemetry span enrichment for write vs. no-op reconciliations

Tekton already has OpenTelemetry tracing spans around ReconcileKind and key operations. Adding a span attribute or metric that captures whether a reconciliation resulted in an actual etcd write or was short-circuited by the DeepEqual check (in the knative/pkg reconciler) would give operators the data to answer:

• How many reconciliations are triggered vs. how many result in writes?
• What's the actual write frequency per object type?
• Which event sources trigger the most writes?

This builds on existing tracing work (#9701, #9796, #9798 and others).

3. Example analysis

As a reference, here's what production data from a Konflux deployment shows for the docker-build-oci-ta pipeline:

• 1 PipelineRun creates ~22 TaskRuns + 22 Pods + ~150 Events
• Each TaskRun is ~68 KB and accumulates ~10-20 revisions
• Total etcd cost per PipelineRun: ~22-25 MB
• 300 concurrent PipelineRuns consume ~7 GB of the 8 GB etcd limit
• The "live snapshot" (1 revision per object) would be ~1-2 MB per PipelineRun — the 10-20x multiplier from revisions is the dominant factor

Without profiling tooling, operators cannot determine whether their specific bottleneck is revision count (too many controller writes), object size (too much data per write), or object count (too many TaskRuns per pipeline).

Proposed deliverables

1. Documentation page in the Tekton docs: "Understanding and profiling etcd usage" — covering the concepts, tools, and methodology
2. OpenTelemetry enhancement: add reconcile.write_outcome (or similar) attribute to reconciliation spans indicating write, no-op, or status-only
3. (Optional) Script or tool: a helper that, given a namespace and PipelineRun name, produces a revision profile summary showing per-object revision counts and sizes

Related

• #9573 — strip managedFields from informer caches
• #9701 — improve distributed tracing instrumentation coverage
• #9686 — scalability and reconcile behavior for completed PipelineRuns