EleutherAI/lm-evaluation-harness

log_samples File name too long. Need truncation or override

Open

#1,454 opened on Feb 21, 2024

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Python (3,306 forks)auto 404
feature requestgood first issuehelp wanted

Repository metrics

Stars
 (12,755 stars)
PR merge metrics
 (Avg merge 15d 7h) (11 merged PRs in 30d)

Description

When setting --log_samples, the default logic for lm_eval.main is the following:

            if args.log_samples:
                for task_name, config in results["configs"].items():
                    output_name = "{}_{}".format(
                        re.sub("/|=", "__", args.model_args), task_name
                    )
                    filename = path.joinpath(f"{output_name}.jsonl")
                    samples_dumped = json.dumps(
                        samples[task_name],
                        indent=2,
                        default=_handle_non_serializable,
                        ensure_ascii=False,
                    )
                    filename.write_text(samples_dumped, encoding="utf-8")

If a user passes in a lengthy --model_args such as pretrained=${model_path},tokenizer=${tokenizer},tensor_parallel_size=1,dtype=float16,gpu_memory_utilization=0.8,data_parallel_size=1

The filename can become extremely length, especially if the pretrained is a local path nested in many directories, which leads to

OSError: [Errno 36] File name too long: 

Either this should be truncated, or user should be allowed to specify a path_name to override args.model_args, or only set output_name to task_name and let output_path be the differentiator.

                    output_name = "{}_{}".format(
                        re.sub("/|=", "__", args.model_args), task_name
                    )

Contributor guide