## Describe the bug Using `hf_avail_splits=["test"]` with `LightevalTaskConfig` you are forced to hit: ```none [2025-06-06 18:21:10,501] [ WARNING]: Careful, the task community|foo is using evaluation data to build the few shot examples. (lighteval_task.py:260) ``` ## To Reproduce Since `"test"` isn't a value in `allowed_splits` at https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/tasks/lighteval_task.py#L251. ## Expected behavior Not forced to hit a warning with `hf_avail_splits=["test"]`, since `test` is a commonly used split. ## Version info `lighteval==0.10.0`
Describe the bug
Using
hf_avail_splits=["test"]withLightevalTaskConfigyou are forced to hit:To Reproduce
Since
"test"isn't a value inallowed_splitsat https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/tasks/lighteval_task.py#L251.Expected behavior
Not forced to hit a warning with
hf_avail_splits=["test"], sincetestis a commonly used split.Version info
lighteval==0.10.0