Skip to content

fix: use total denoising steps for Humaneval dual cache evaluation - #82

Open
Ashleyzm wants to merge 1 commit into
NVlabs:mainfrom
Ashleyzm:fix-humaneval-dual-cache-steps
Open

Ashleyzm wants to merge 1 commit into
NVlabs:mainfrom
Ashleyzm:fix-humaneval-dual-cache-steps

Conversation

@Ashleyzm

@Ashleyzm Ashleyzm commented Aug 31, 2026

Copy link
Copy Markdown

This is also inconsistent with the corresponding dual-cache configuration in eval_gsm8k.sh, which passes steps=${length}. For length=256 and block_length=32, the current Humaneval command passes steps=8. Since generate_with_dual_cache computes steps_per_block = steps // num_blocks, this reduces to 1 step per block. This may also address the behavior reported in #66, where dual-cache evaluation on HumanEval produced 0 accuracy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant