EleutherAI/lm-evaluation-harness

CoQA's implementation only predicts the last answer of each text

Open

#1,231 opened on Jan 1, 2024

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Python (3,306 forks)auto 404
buggood first issue

Repository metrics

Stars
 (12,755 stars)
PR merge metrics
 (Avg merge 15d 7h) (11 merged PRs in 30d)

Description

For CoQA, in coqa/utils.py, only the last answer of each text (i.e. the answer for the last turn_id, with all the previous questions and answers in the context window) is predicted. On the website of the authors of CoQA, they seem to consider all turn_id (see their sample prediction file where they have answers for every turn_id, and the official evaluation script where they do an average of the result for every turn_id).

Here is an excerpt from the paper (https://arxiv.org/pdf/1808.07042.pdf) :

image

I haven't found how it's implemented with other popular LLM evaluation frameworks, but I'm pretty sure that predicting only the answer to the last question is not what is intended by the authors of CoQA.

Contributor guide

Research direction
Inspect coqa/utils.py to understand how predictions are generated. The issue states only the last answer is predicted; modify the logic to predict answers for all turn ids per dialogue as in the original CoQA evaluation. Refer to the sample prediction file and evaluation script linked in the issue.
Tech stack
python
Domain
backendtesting
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GitPython
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60