confident-ai/deepeval

Add JudgeLM as a way to evaluate and compare historical test runs

Open

#262 opened on Oct 30, 2023

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Python (1,677 forks)auto 404
enhancementhelp wanted

Repository metrics

Stars
 (16,939 stars)
PR merge metrics
 (PR metrics pending)

Description

Currently, metrics are computed based on test cases that run during evaluation. However, there's currently no way to compare historical test runs' performances except for comparing metric scores for each test run.

I would like to introduce another way to choose the best configurations / hyperparameters to help devs speed up development using DeepEval, and that would be to implement a JudgeLM (eg. https://github.com/baaivision/JudgeLM) to choose the best test run out of a set of test runs.

from deepeval import judge

best_performing_test_run = None
for test_run in test_runs:
      best_performing_test_run = judge(test_run, best_performing_test_run)

This assumes the test runs were ran on the same test cases / evaluation dataset. There should be error handling for edge cases that don't fit into this criteria.

Contributor guide

Research direction
Study the current test run comparison system, understand the data structure of test runs, and implement a judge function that compares two test runs using JudgeLM. Ensure proper error handling for mismatched test cases.
Tech stack
python
Domain
aitesting
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonLLMs
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65