EleutherAI/lm-evaluation-harness
View on GitHubSpeed up + streamline prompt template rendering runtime
Open
#1,286 opened on Jan 15, 2024
feature requesthelp wanted
Repository metrics
- Stars
- (12,755 stars)
- PR merge metrics
- (Avg merge 15d 7h) (11 merged PRs in 30d)
Description
In some situations (for example, N-shot with very large N) we currently spend an annoying amount of time redundantly rendering our Jinja2 prompt template strings.
There are two ways we can significantly cut down on this runtime, without impacting code complexity significantly (e.g., no multiprocessing when generating requests / Instances, etc.)
- At the start, render
doc_to_target,doc_to_text,doc_to_choiceonce each for every doc, and cache that result so that we don't need to repeatedly render our prompts when we want to call these methods multiple times for every test instance. - cache these attributes, or the actual Instance objects, created when a task is run once, so that we can skip this whole step's processing in future. The easiest way to do so would be to leverage HF datasets'
dataset.map()function which can cache results based on pickling a function. However, we'd want to be extremely cautious of whether this might create weird bugs when someone is trying to change + rerun a task but the cached intermediate input objects get silently used, preventing them from seeing the expected changes in behavior when they change code.
Change 1 should suffice for some pretty nice QoL improvements. We should however also make sure this doesn't negatively affect runtime when we'd have to perform it on a large train set that we're drawing few-shot examples from.