EleutherAI/lm-evaluation-harness
View on GitHublog_samples File name too long. Need truncation or override
Open
#1,454 opened on Feb 21, 2024
feature requestgood first issuehelp wanted
Repository metrics
- Stars
- (12,755 stars)
- PR merge metrics
- (Avg merge 15d 7h) (11 merged PRs in 30d)
Description
When setting --log_samples, the default logic for lm_eval.main is the following:
if args.log_samples:
for task_name, config in results["configs"].items():
output_name = "{}_{}".format(
re.sub("/|=", "__", args.model_args), task_name
)
filename = path.joinpath(f"{output_name}.jsonl")
samples_dumped = json.dumps(
samples[task_name],
indent=2,
default=_handle_non_serializable,
ensure_ascii=False,
)
filename.write_text(samples_dumped, encoding="utf-8")
If a user passes in a lengthy --model_args such as pretrained=${model_path},tokenizer=${tokenizer},tensor_parallel_size=1,dtype=float16,gpu_memory_utilization=0.8,data_parallel_size=1
The filename can become extremely length, especially if the pretrained is a local path nested in many directories, which leads to
OSError: [Errno 36] File name too long:
Either this should be truncated, or user should be allowed to specify a path_name to override args.model_args, or only set output_name to task_name and let output_path be the differentiator.
output_name = "{}_{}".format(
re.sub("/|=", "__", args.model_args), task_name
)