Uh oh!
There was an error while loading. Please reload this page.
Fix kernel_static_checker.py false positives on try/except and pass statements - #161
Open
cubeerea wants to merge 1 commit into
Open
Conversation
kernel_static_checker.py's try-except and pass-statement bypass checks currently match any occurrence of `try:`/`except:`/`pass` anywhere in the file via blanket regex, with no awareness of code structure. This flags legitimate defensive patterns as reward-hacking: - A try/except guarding a load_inline() compile call (falling back to an unfused-but-still-real code path only if the CUDA extension fails to build) trips the try-except check, even though the except handler never computes or returns an alternative result. - A trivial `try: cfg.use_cache = False / except: pass` attribute guard trips both checks. Replaces both with an AST-based classification: - Try/except is only flagged if an except handler itself produces a computed alternative answer (a `return` of a non-trivial expression, or an assignment whose value involves a call) - the actual "wrap the kernel call and fall back to a different computation" shape. An inert handler (pass, bare raise, a flag assignment, a log call) is left alone. - The pass-statement check is scoped to a function/method whose entire body (after an optional docstring) is just `pass` - the actual "inherits and does nothing" bypass - instead of any `pass` anywhere in the file, which also matches inert except-handler bodies. Both fall back to the original regex behavior if the code doesn't parse as valid Python. Verified against all existing correct_kernels samples in runs/ (no regressions) plus synthetic cases for both the false-positive patterns above and genuine reward-hacking shapes (which still correctly flag). The pass-statement false positive was independently reported in ScalingIntelligence#155 as a minor aside ("the checker flags any `pass` statement... which false-positives on legitimate empty `__init__` / control-flow `pass`"); this fixes it along with the related try/except case. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
kernel_static_checker.py's try-except and pass-statement bypass checks match those keywords anywhere in the file via blanket regex, with no code-structure awareness. This flags legitimate defensive code as reward-hacking:load_inline()compile call — falling back to an unfused-but-still-real code path only if the CUDA extension fails to build — trips the try-except check, even though the except handler never computes or returns an alternative result.try: cfg.use_cache = False / except: passattribute guard trips both the try-except and pass-statement checks.Both false positives showed up on real, non-reward-hacking completions (custom CUDA kernels with genuine
__global__implementations) generated while testing this repo's generation pipeline against a large reasoning model, which tends to wrap extension-compile calls defensively.The pass-statement false positive was also independently reported in #155 as a minor aside: "the checker flags any
passstatement as 'inheritance bypass'... which false-positives on legitimate empty__init__/ control-flowpass." This PR fixes that case too, along with the related try/except one.Change
Replaces both blanket regex checks with AST-based classification (
ast.walk, no new dependencies):returnof a non-trivial expression, or an assignment whose value involves aCall— i.e. the actual "wrap the kernel call and fall back to a different computation" shape. An inert handler (pass, bareraise, a flag assignment, a log call) is left alone.pass— the actual "inherits and does nothing" bypass — instead of matchingpassanywhere in the file, which also matches inert except-handler bodies.This is in the spirit of the checker's own docstring, which already lists AST-based detection as a planned direction: "In the future we can add - AST-based detections."
Test plan
load_inlinecompile guard, trivial attribute-guard)except: return F.linear(x, weight)), and aforward()method whose entire body ispass(with or without a leading docstring)