GoogleCloudPlatform/kubectl-ai

Add github action to run lightweight evals as presubmit job

Open

#119 opened on Apr 28, 2025

View on GitHub
 (5 comments) (0 reactions) (1 assignee)Go (708 forks)auto 404
good first issue

Repository metrics

Stars
 (7,532 stars)
PR merge metrics
 (PR metrics pending)

Description

We want to run a few eval tasks (may be difficulty level easy) as pre-submit tasks to ensure that new PRs are getting some basic e2e tests.

Details:

  • The pre-submit job should bring up a kind cluster
  • The pre-submit job should also bring up ollama or llamacpp server
  • Run evals with the difficulty level easy or to begin with --task-pattern=scale
  • Add contributing section in the README.md to use this job as e2e test

Contributor guide

Research direction
Set up a GitHub Actions workflow that creates a kind cluster, installs Ollama server, and runs evaluation tasks with difficulty level 'easy' or pattern 'scale'. Document in README.
Tech stack
godockerkubernetesgithub actions
Domain
devopsbackendtesting
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GitDockerKubernetes basicsGitHub Actions
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40