Skip to content

Fix kernel_static_checker.py false positives on try/except and pass statements - #161

Open
cubeerea wants to merge 1 commit into
ScalingIntelligence:mainfrom
cubeerea:fix/static-checker-ast-false-positives
Open

Fix kernel_static_checker.py false positives on try/except and pass statements#161
cubeerea wants to merge 1 commit into
ScalingIntelligence:mainfrom
cubeerea:fix/static-checker-ast-false-positives

Conversation

@cubeerea

@cubeereacubeerea commented Aug 19, 2026

Copy link
Copy Markdown

Summary

kernel_static_checker.py's try-except and pass-statement bypass checks match those keywords anywhere in the file via blanket regex, with no code-structure awareness. This flags legitimate defensive code as reward-hacking:

  • A try/except guarding a load_inline() compile call — falling back to an unfused-but-still-real code path only if the CUDA extension fails to build — trips the try-except check, even though the except handler never computes or returns an alternative result.
  • A trivial try: cfg.use_cache = False / except: pass attribute guard trips both the try-except and pass-statement checks.

Both false positives showed up on real, non-reward-hacking completions (custom CUDA kernels with genuine __global__ implementations) generated while testing this repo's generation pipeline against a large reasoning model, which tends to wrap extension-compile calls defensively.

The pass-statement false positive was also independently reported in #155 as a minor aside: "the checker flags any pass statement as 'inheritance bypass'... which false-positives on legitimate empty __init__ / control-flow pass." This PR fixes that case too, along with the related try/except one.

Change

Replaces both blanket regex checks with AST-based classification (ast.walk, no new dependencies):

  • Try/except: only flagged if an except handler itself produces a computed alternative answer — a return of a non-trivial expression, or an assignment whose value involves a Call — i.e. the actual "wrap the kernel call and fall back to a different computation" shape. An inert handler (pass, bare raise, a flag assignment, a log call) is left alone.
  • Pass statement: scoped to a function/method whose entire body (after an optional docstring) is just pass — the actual "inherits and does nothing" bypass — instead of matching pass anywhere in the file, which also matches inert except-handler bodies.
  • Both fall back to the original regex behavior if the code fails to parse as valid Python (e.g. a malformed/partial completion), preserving existing behavior for that edge case.

This is in the spirit of the checker's own docstring, which already lists AST-based detection as a planned direction: "In the future we can add - AST-based detections."

Test plan

  • Verified the two false-positive patterns above now pass (load_inline compile guard, trivial attribute-guard)
  • Verified synthetic genuine reward-hacking shapes still correctly flag: an except handler that returns a fallback computation (e.g. except: return F.linear(x, weight)), and a forward() method whose entire body is pass (with or without a leading docstring)
  • Regression-tested against 189 known-correct, previously-passing generated kernels across KernelBench levels 1-4 — no new false positives, no missed detections

kernel_static_checker.py's try-except and pass-statement bypass checks
currently match any occurrence of `try:`/`except:`/`pass` anywhere in the
file via blanket regex, with no awareness of code structure. This flags
legitimate defensive patterns as reward-hacking:
- A try/except guarding a load_inline() compile call (falling back to an
unfused-but-still-real code path only if the CUDA extension fails to
build) trips the try-except check, even though the except handler never
computes or returns an alternative result.
- A trivial `try: cfg.use_cache = False / except: pass` attribute guard
trips both checks.
Replaces both with an AST-based classification:
- Try/except is only flagged if an except handler itself produces a
computed alternative answer (a `return` of a non-trivial expression, or
an assignment whose value involves a call) - the actual "wrap the kernel
call and fall back to a different computation" shape. An inert handler
(pass, bare raise, a flag assignment, a log call) is left alone.
- The pass-statement check is scoped to a function/method whose entire
body (after an optional docstring) is just `pass` - the actual
"inherits and does nothing" bypass - instead of any `pass` anywhere in
the file, which also matches inert except-handler bodies.
Both fall back to the original regex behavior if the code doesn't parse
as valid Python. Verified against all existing correct_kernels samples in
runs/ (no regressions) plus synthetic cases for both the false-positive
patterns above and genuine reward-hacking shapes (which still correctly
flag).
The pass-statement false positive was independently reported in ScalingIntelligence#155 as
a minor aside ("the checker flags any `pass` statement... which
false-positives on legitimate empty `__init__` / control-flow `pass`");
this fixes it along with the related try/except case.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@cubeerea