test: revisit E2E framework and testcase matrix across Docker and Kubernetes

Summary

Revisit the E2E framework, profiles, and testcase matrix so the same meaningful behavior contracts can run consistently across Docker/local and Kubernetes environments, while removing duplicate or low-value coverage and adding targeted high-value tests.

Parent roadmap: #2287 Related: #1519, #2375, #2376, #2334, #2331, #2332, #2355 Related code and assets: e2e/cmd/e2e, e2e/pkg/framework, e2e/pkg/testcases, e2e/pkg/testmatrix, e2e/pkg/docker, e2e/pkg/cluster, e2e/pkg/helm, e2e/profiles, e2e/testcases, e2e/testing, tools/agent/e2e-profile-map.yaml, tools/agent/task-matrix.yaml, tools/make/agent.mk, .github/workflows, docs/agent/testing-strategy.md

Motivation

The repo already has a broad E2E surface: reusable testcases, profile registration, Kind/Helm deployment, Docker image build/load, profile-to-path mapping, workflow integration suites, and manual-only profiles. That breadth is valuable, but H2 work adds more cross-environment behavior: multiple recipes, Docker/Kubernetes endpoint parity, dashboard config flows, inference-aware backends, Router Memory/Learning, streaming, security, and edge deployments.

We need the E2E system to be easier to reason about:

• a behavior contract should say which environments it can run in;
• Docker/local and Kubernetes profiles should share testcase semantics where possible;
• testcases should have meaningful pass/fail thresholds, not only smoke/report-only checks;
• duplicate testcases should be consolidated or clearly scoped;
• benchmarks and stress probes should be named as such and not substitute for acceptance coverage;
• new H2 features should land with valuable E2E coverage rather than one-off scripts.

This is broader than #1519. #1519 focuses on expanding module-level E2E coverage, especially dashboard workflows. This issue focuses on the E2E framework and testcase matrix itself.

Scope

Audit and improve:

• Framework and environment contract

  • separate testcase behavior contracts from environment lifecycle concerns;
  • make it clear which testcases can run against Docker/local, Kind/Kubernetes, workflow integration suites, or manual-only profiles;
  • review whether TestCaseOptions, ServiceConfig, Kubernetes client usage, and profile registration need a cleaner environment abstraction;
  • keep local Docker image flow and K8s/Helm flow consistent with repo harness commands.

• Profile and suite taxonomy

  • review tools/agent/e2e-profile-map.yaml, e2e/pkg/testmatrix, and profile GetTestCases() lists;
  • classify profiles as baseline contract, focused integration, workflow integration, manual-only, benchmark/reporting, or debug-only;
  • ensure manual-only profiles have explicit rationale and local command guidance;
  • remove accidental duplicate ownership of the same contract across profiles.

• Testcase value and deduplication

  • inventory all e2e/testcases/*.go and e2e/testing/** tests;
  • mark each as acceptance, smoke, regression, benchmark, stress, observability, or debug helper;
  • remove, merge, or demote tests that only duplicate another profile's contract;
  • ensure routing, classification, plugin, fallback, API, dashboard, security, and integration tests have explicit failure conditions.

• Additional high-value tests

  • add or split tests for H2 behavior gaps discovered by the audit;
  • likely candidates include multi-recipe/entrypoint behavior, endpoint/backend config parity, dashboard config preservation, recipe presets, Router Memory/Learning guarded behavior, inference-aware backend diagnostics, streaming/fallback, and security-sensitive paths;
  • prefer targeted reusable testcases over slow full-stack expansion.

• Documentation and CI wiring

  • update E2E README, profile map, and testing strategy to explain how to run affected tests locally and in CI;
  • keep make agent-e2e-affected CHANGED_FILES=... and profile selection rules aligned;
  • make Docker/local and Kubernetes execution paths discoverable for contributors.

Non-goals

• Do not make every testcase run in every environment.
• Do not require live GPU, gated models, or external provider credentials for default CI.
• Do not turn benchmark or stress tests into release blockers without explicit thresholds and stability analysis.
• Do not replace unit or integration tests with broad E2E coverage when a narrower test is sufficient.
• Do not hide flaky tests by weakening assertions; classify and fix or quarantine them explicitly.

Acceptance criteria

• An E2E framework audit documents profiles, testcases, ownership, environment support, duplication, and coverage gaps.
• Docker/local and Kubernetes execution paths share a clear testcase contract where possible, with environment-specific setup isolated in profiles or workflow suites.
• Every testcase has a declared role: acceptance, smoke, regression, benchmark, stress, observability, or debug helper.
• Acceptance testcases encode meaningful pass/fail thresholds and avoid report-only success criteria.
• Duplicate or overlapping testcases are merged, removed, or assigned distinct ownership in e2e/pkg/testmatrix and tools/agent/e2e-profile-map.yaml.
• Manual-only profiles remain documented with rationale, required environment, and local command guidance.
• New high-value H2 testcase gaps are either implemented or split into focused child issues with affected surfaces and validation commands.
• Docs and harness profile selection stay aligned with the actual runnable profiles.
• The work cross-links #1519 for dashboard/module coverage expansion instead of duplicating its owned scope.

Validation

Use the repo harness to scope follow-up PRs:

make agent-report ENV=cpu CHANGED_FILES="e2e tools/agent/e2e-profile-map.yaml tools/agent/task-matrix.yaml tools/make/agent.mk .github/workflows docs/agent/testing-strategy.md"
make agent-lint CHANGED_FILES="e2e tools/agent/e2e-profile-map.yaml tools/agent/task-matrix.yaml tools/make/agent.mk .github/workflows docs/agent/testing-strategy.md"
make agent-ci-gate CHANGED_FILES="e2e tools/agent/e2e-profile-map.yaml tools/agent/task-matrix.yaml tools/make/agent.mk .github/workflows docs/agent/testing-strategy.md"

When implementing concrete changes, also run the smallest affected E2E path, for example:

make agent-e2e-affected CHANGED_FILES="e2e/testcases e2e/profiles tools/agent/e2e-profile-map.yaml"