Repository Issues

EleutherAI/lm-evaluation-harness

A framework for few-shot evaluation of language models.

View on GitHub
Stars
 (12,755 stars)
Forks
 (3,306 forks)
Indexed issues
 (24 indexed issues)
open beginner issues
 (0 open beginner issues)
Latest indexed
Jul 17, 2026
Last GitHub push
May 11, 2026
License
MIT License
Contributing guide
Contributing guide
Code of conduct
No code of conduct
Dominant language
Python
PR merge metrics
 (Avg merge 15d 7h) (11 merged PRs in 30d)
Beginner labels
No beginner labels indexed

Issues

23 open indexed issues

RSSAtom
Open
reproduce llama 3 evals
good first issuevalidation

EleutherAI/lm-evaluation-harness #2,557 opened Dec 10, 2024 · Python · 12,755 stars

Why recommendedNo assignee yet · Has a beginner-friendly label
No assignee yetHas a beginner-friendly labelContributing guide available
8 comments2 reactions0 assignees
Open
Add Regression Testing
feature requestgood first issuehelp wanted

EleutherAI/lm-evaluation-harness #1,883 opened May 24, 2024 · Python · 12,755 stars

Why recommendedNo assignee yet · Has a beginner-friendly label
No assignee yetHas a beginner-friendly labelContributing guide available
6 comments0 reactions0 assignees
Open
Add better test coverage for models
feature requestgood first issuehelp wanted

EleutherAI/lm-evaluation-harness #1,613 opened Mar 20, 2024 · Python · 12,755 stars

Why recommendedNo assignee yet · Has a beginner-friendly label
No assignee yetHas a beginner-friendly labelContributing guide available
3 comments0 reactions0 assignees
Open
Add New Lambada Translations
good first issue

EleutherAI/lm-evaluation-harness #1,501 opened Feb 29, 2024 · Python · 12,755 stars

Why recommendedHas a beginner-friendly label · Contributing guide available
Has a beginner-friendly labelContributing guide available
2 comments0 reactions1 assignee
Open
Add ZeroScrolls Benchmark
feature requestgood first issuehelp wanted

EleutherAI/lm-evaluation-harness #1,083 opened Dec 8, 2023 · Python · 12,755 stars

Why recommendedNo assignee yet · Has a beginner-friendly label
No assignee yetHas a beginner-friendly labelContributing guide available
2 comments0 reactions0 assignees
Open
[New Task] COLLIE
feature requestgood first issuehelp wanted

EleutherAI/lm-evaluation-harness #1,013 opened Nov 21, 2023 · Python · 12,755 stars

Why recommendedNo assignee yet · Has a beginner-friendly label
No assignee yetHas a beginner-friendly labelContributing guide available
1 comment0 reactions0 assignees
Open
Quac Dataset
feature requestgood first issue

EleutherAI/lm-evaluation-harness #827 opened Sep 4, 2023 · Python · 12,755 stars

Why recommendedNo assignee yet · Has a beginner-friendly label
No assignee yetHas a beginner-friendly labelContributing guide available
8 comments1 reaction0 assignees