Seed Thompson-sampling bandit with informative priors from static target profiling - #1
Merged
Merged
Conversation
Copilotcreated this pull request from a session on behalf of daedalusJuly 15, 2026 21:25 View session
daedalus
marked this pull request as ready for review
July 15, 2026 21:25
Reviewer's GuideIntroduces informative Beta priors for the Thompson-sampling MonteCarloScheduler using static target profiling, wires those priors through arm registration for schedulers that support them, and documents and tests the new behavior. Sequence diagram for registering arms with informative priorssequenceDiagram
participant Fuzzer
participant TargetProfiler
participant MonteCarloScheduler
participant MOptScheduler
Fuzzer->>TargetProfiler: format_operator_priors(profile)
TargetProfiler-->>Fuzzer: priors
Fuzzer->>MonteCarloScheduler: _register_arms(scheduler=mc, priors=priors)
activate MonteCarloScheduler
MonteCarloScheduler-->>MonteCarloScheduler: supports_priors = True
MonteCarloScheduler-->>MonteCarloScheduler: init_arm(op, prior_alpha, prior_beta)
deactivate MonteCarloScheduler
Fuzzer->>MOptScheduler: _register_arms(scheduler=mopt, priors=priors)
activate MOptScheduler
MOptScheduler-->>MOptScheduler: supports_priors (missing or False)
MOptScheduler-->>MOptScheduler: init_arm(op)
deactivate MOptScheduler
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've found 3 issues, and left some high level feedback:
- In
_register_arms, invalidpriorsentries (e.g. wrong-length tuples or non-numeric values) are silently ignored and fall back to default behavior; consider adding explicit validation/logging so configuration mistakes are surfaced rather than quietly degraded. - When clamping
prior_alpha/prior_betainMonteCarloScheduler.init_arm, you currently modify bad priors silently; adding a debug log (or raising in strict modes) when values are clamped would make it easier to detect misconfigured priors without affecting the default behavior.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments- In `_register_arms`, invalid `priors` entries (e.g. wrong-length tuples or non-numeric values) are silently ignored and fall back to default behavior; consider adding explicit validation/logging so configuration mistakes are surfaced rather than quietly degraded.
- When clamping `prior_alpha`/`prior_beta` in `MonteCarloScheduler.init_arm`, you currently modify bad priors silently; adding a debug log (or raising in strict modes) when values are clamped would make it easier to detect misconfigured priors without affecting the default behavior.
## Individual Comments### Comment 1
<locationpath="src/fuzzer_tool/core/target_profiler.py"line_range="619-626" />
<code_context>
+
+# Format signature -> structure-aware mutation operators that are almost
+# certainly useful for that format (see core.mutations.FORMAT_MUTATIONS).
+_FORMAT_OPERATOR_HINTS: dict[str, tuple[str, ...]] = {
+ "png": ("png_chunk_mutate", "png_crc_fix"),
+ "jpeg": ("jpeg_chunk_mutate", "jpeg_crc_fix"),
+ "gzip": ("gzip_chunk_mutate",),
+ "bz2": ("gzip_chunk_mutate",),
+ "xz": ("gzip_chunk_mutate",),
+ "zlib": ("zlib_chunk_mutate",),
+ "riff": ("bmp_chunk_mutate",),
+}
+
</code_context>
<issue_to_address>
**suggestion:** Consider normalizing/relaxing format_signature matching to reduce brittleness.
The lookup in `_FORMAT_OPERATOR_HINTS` uses `profile.format_signature` verbatim, so case or representation differences (e.g. `PNG`, `image/png`, `riff/wav`) will prevent matches and skip the boosted priors. If the canonical form isn’t already enforced upstream, consider normalizing `format_signature` before lookup (e.g. lowercasing, stripping MIME prefixes, alias mapping) or clearly documenting the expected canonical format here.
Suggested implementation:
```python# Format signature -> structure-aware mutation operators that are almost# certainly useful for that format (see core.mutations.FORMAT_MUTATIONS).## The lookup below expects a *normalized* format signature. Call# `_normalize_format_signature(profile.format_signature)` before using# `_FORMAT_OPERATOR_HINTS` to avoid brittleness due to case, MIME-style# prefixes (e.g. `image/png`), or common aliases (`jpg` vs `jpeg`,# `riff/wav` vs `riff`, etc.).def_normalize_format_signature(signature: str) -> str:
"""Normalize a format signature into the canonical key used in `_FORMAT_OPERATOR_HINTS`. This is intentionally conservative; it lowercases, strips common MIME-style prefixes, and applies a small alias map for widely-used variants. If a format is not recognized, the original (lowercased) signature is returned, so adding new formats remains straightforward."""
sig = signature.strip().lower()
ifnot sig:
return sig
# Common alias mappings for well-known formats.
alias_map = {
# PNG"png": "png",
"image/png": "png",
# JPEG"jpeg": "jpeg",
"jpg": "jpeg",
"image/jpeg": "jpeg",
# RIFF container (e.g. WAV)"riff": "riff",
"riff/wav": "riff",
"wav": "riff",
# Compression formats"gzip": "gzip",
"application/gzip": "gzip",
"bz2": "bz2",
"bzip2": "bz2",
"xz": "xz",
"application/x-xz": "xz",
"zlib": "zlib",
}
# Direct alias match first.if sig in alias_map:
return alias_map[sig]
# For MIME-like strings, prefer the subtype as the canonical key.if"/"in sig:
main, subtype = sig.split("/", 1)
subtype = subtype.strip().lower()
if subtype in alias_map:
return alias_map[subtype]
# Fall back to subtype if we don't know it explicitly.return subtype
return sig
_FORMAT_OPERATOR_HINTS: dict[str, tuple[str, ...]] = {
"png": ("png_chunk_mutate", "png_crc_fix"),
"jpeg": ("jpeg_chunk_mutate", "jpeg_crc_fix"),
"gzip": ("gzip_chunk_mutate",),
"bz2": ("gzip_chunk_mutate",),
"xz": ("gzip_chunk_mutate",),
"zlib": ("zlib_chunk_mutate",),
}
```1. Wherever `_FORMAT_OPERATOR_HINTS` is consulted (likely something like `_FORMAT_OPERATOR_HINTS.get(profile.format_signature, ...)`), update the lookup to normalize first, e.g.:
- Replace `profile.format_signature` with `_normalize_format_signature(profile.format_signature)` in the key used for `_FORMAT_OPERATOR_HINTS`.
2. If the profiler or `Profile` type has documentation/comments about `format_signature`, update them to mention that signatures are normalized via `_normalize_format_signature` and give examples of accepted forms (`PNG`, `image/png`, `riff/wav`, `jpg`).
</issue_to_address>
### Comment 2
<locationpath="src/fuzzer_tool/core/montecarlo.py"line_range="93-102" />
<code_context>
+ def init_arm(self, name: str, prior_alpha: float = 1.0, prior_beta: float = 1.0) -> None:
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Clarify the documented contract for prior parameters vs. the clamping behavior.
The docstring states prior_alpha/prior_beta "must be > 0", but the implementation clamps them with max(prior_*, MIN_BETA_PARAM). This lets callers pass 0 or negative values that are silently coerced to MIN_BETA_PARAM. Please either document this clamping behavior explicitly or consider raising/logging on non-positive inputs to surface configuration errors earlier.
Suggested implementation:
```pythondefinit_arm(self, name: str, prior_alpha: float=1.0, prior_beta: float=1.0) -> None:
"""Register a mutation operator arm with a Beta prior. Defaults to the uninformative Beta(1, 1) prior. Callers with prior knowledge about an operator's likely usefulness (e.g. static target profiling indicating a specific file format) can pass a stronger prior to bias early Thompson sampling before any evidence has been observed. A no-op if the arm is already registered — the prior only applies at first registration and is never overwritten by later calls, matching the existing idempotent behavior of this method. The `prior_alpha` and `prior_beta` parameters are expected to be strictly positive. For robustness, the implementation clamps non‑positive values up to the internal minimum Beta parameter `MIN_BETA_PARAM` via `max(prior_*, MIN_BETA_PARAM)`. This means that passing 0 or negative values will not raise, but will instead behave as if `prior_* == MIN_BETA_PARAM`.```If elsewhere in this docstring (or in parameter docs not shown in the snippet) you explicitly state that `prior_alpha` and `prior_beta` "must be > 0" without mentioning clamping, update that text to reference the clamping behavior described here to keep the contract consistent.</issue_to_address>### Comment 3<location path="tests/test_target_profiler.py" line_range="194-203" /><code_context>+class TestFormatOperatorPriors:</code_context><issue_to_address>**suggestion (testing):** Add a test that `interesting_strings` alone also boost dictionary-aware operatorsYou currently test `magic_bytes` and `boundary_markers` boosting `_DICT_OPERATORS`, but not `interesting_strings`, which is also part of the condition. Please add a test like:```pythondef test_interesting_strings_boost_dict_operators(self): p = TargetProfile() p.interesting_strings = [b"token"] priors = format_operator_priors(p) assert "dict_insert" in priors assert "dict_append" in priors```to exercise the `interesting_strings` branch of `if profile.magic_bytes or profile.boundary_markers or profile.interesting_strings:`.</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+619
to
+626
| _FORMAT_OPERATOR_HINTS: dict[str, tuple[str, ...]] = { | ||
| "png": ("png_chunk_mutate", "png_crc_fix"), | ||
| "jpeg": ("jpeg_chunk_mutate", "jpeg_crc_fix"), | ||
| "gzip": ("gzip_chunk_mutate",), | ||
| "bz2": ("gzip_chunk_mutate",), | ||
| "xz": ("gzip_chunk_mutate",), | ||
| "zlib": ("zlib_chunk_mutate",), | ||
| "riff": ("bmp_chunk_mutate",), |
There was a problem hiding this comment.
suggestion: Consider normalizing/relaxing format_signature matching to reduce brittleness.
The lookup in _FORMAT_OPERATOR_HINTS uses profile.format_signature verbatim, so case or representation differences (e.g. PNG, image/png, riff/wav) will prevent matches and skip the boosted priors. If the canonical form isn’t already enforced upstream, consider normalizing format_signature before lookup (e.g. lowercasing, stripping MIME prefixes, alias mapping) or clearly documenting the expected canonical format here.
Suggested implementation:
# Format signature -> structure-aware mutation operators that are almost# certainly useful for that format (see core.mutations.FORMAT_MUTATIONS).## The lookup below expects a *normalized* format signature. Call# `_normalize_format_signature(profile.format_signature)` before using# `_FORMAT_OPERATOR_HINTS` to avoid brittleness due to case, MIME-style# prefixes (e.g. `image/png`), or common aliases (`jpg` vs `jpeg`,# `riff/wav` vs `riff`, etc.).def_normalize_format_signature(signature: str) ->str:
"""Normalize a format signature into the canonical key used in `_FORMAT_OPERATOR_HINTS`. This is intentionally conservative; it lowercases, strips common MIME-style prefixes, and applies a small alias map for widely-used variants. If a format is not recognized, the original (lowercased) signature is returned, so adding new formats remains straightforward. """sig=signature.strip().lower()
ifnotsig:
returnsig# Common alias mappings for well-known formats.alias_map= {
# PNG"png": "png",
"image/png": "png",
# JPEG"jpeg": "jpeg",
"jpg": "jpeg",
"image/jpeg": "jpeg",
# RIFF container (e.g. WAV)"riff": "riff",
"riff/wav": "riff",
"wav": "riff",
# Compression formats"gzip": "gzip",
"application/gzip": "gzip",
"bz2": "bz2",
"bzip2": "bz2",
"xz": "xz",
"application/x-xz": "xz",
"zlib": "zlib",
}
# Direct alias match first.ifsiginalias_map:
returnalias_map[sig]
# For MIME-like strings, prefer the subtype as the canonical key.if"/"insig:
main, subtype=sig.split("/", 1)
subtype=subtype.strip().lower()
ifsubtypeinalias_map:
returnalias_map[subtype]
# Fall back to subtype if we don't know it explicitly.returnsubtypereturnsig_FORMAT_OPERATOR_HINTS: dict[str, tuple[str, ...]] = {
"png": ("png_chunk_mutate", "png_crc_fix"),
"jpeg": ("jpeg_chunk_mutate", "jpeg_crc_fix"),
"gzip": ("gzip_chunk_mutate",),
"bz2": ("gzip_chunk_mutate",),
"xz": ("gzip_chunk_mutate",),
"zlib": ("zlib_chunk_mutate",),
}- Wherever
_FORMAT_OPERATOR_HINTSis consulted (likely something like_FORMAT_OPERATOR_HINTS.get(profile.format_signature, ...)), update the lookup to normalize first, e.g.:- Replace
profile.format_signaturewith_normalize_format_signature(profile.format_signature)in the key used for_FORMAT_OPERATOR_HINTS.
- Replace
- If the profiler or
Profiletype has documentation/comments aboutformat_signature, update them to mention that signatures are normalized via_normalize_format_signatureand give examples of accepted forms (PNG,image/png,riff/wav,jpg).
Comment on lines
+93
to
+102
| def init_arm(self, name: str, prior_alpha: float = 1.0, prior_beta: float = 1.0) -> None: | ||
| """Register a mutation operator arm with a Beta prior. | ||
| Defaults to the uninformative Beta(1, 1) prior. Callers with prior | ||
| knowledge about an operator's likely usefulness (e.g. static target | ||
| profiling indicating a specific file format) can pass a stronger | ||
| prior to bias early Thompson sampling before any evidence has been | ||
| observed. A no-op if the arm is already registered — the prior only | ||
| applies at first registration and is never overwritten by later | ||
| calls, matching the existing idempotent behavior of this method. |
There was a problem hiding this comment.
suggestion (bug_risk): Clarify the documented contract for prior parameters vs. the clamping behavior.
The docstring states prior_alpha/prior_beta "must be > 0", but the implementation clamps them with max(prior_*, MIN_BETA_PARAM). This lets callers pass 0 or negative values that are silently coerced to MIN_BETA_PARAM. Please either document this clamping behavior explicitly or consider raising/logging on non-positive inputs to surface configuration errors earlier.
Suggested implementation:
definit_arm(self, name: str, prior_alpha: float=1.0, prior_beta: float=1.0) ->None:
"""RegisteramutationoperatorarmwithaBetaprior.
DefaultstotheuninformativeBeta(1, 1) prior. Callerswithpriorknowledgeaboutanoperator'slikelyusefulness (e.g. statictargetprofilingindicatingaspecificfileformat) canpassastrongerpriortobiasearlyThompsonsamplingbeforeanyevidencehasbeenobserved. Ano-opifthearmisalreadyregistered — theprioronlyappliesatfirstregistrationandisneveroverwrittenbylatercalls, matchingtheexistingidempotentbehaviorofthismethod.
The`prior_alpha`and`prior_beta`parametersareexpectedtobestrictlypositive. Forrobustness, theimplementationclampsnon‑positivevaluesuptotheinternalminimumBetaparameter`MIN_BETA_PARAM`via`max(prior_*, MIN_BETA_PARAM)`. Thismeansthatpassing0ornegativevalueswillnotraise, butwillinsteadbehaveasif`prior_* == MIN_BETA_PARAM`.If elsewhere in this docstring (or in parameter docs not shown in the snippet) you explicitly state that prior_alpha and prior_beta "must be > 0" without mentioning clamping, update that text to reference the clamping behavior described here to keep the contract consistent.
Comment on lines
+194
to
+203
| class TestFormatOperatorPriors: | ||
| def test_no_hints_when_profile_empty(self): | ||
| p = TargetProfile() | ||
| assert format_operator_priors(p) == {} | ||
| def test_png_format_boosts_png_operators(self): | ||
| p = TargetProfile() | ||
| p.format_signature = "png" | ||
| priors = format_operator_priors(p) | ||
| assert priors["png_chunk_mutate"] == (2.0, 1.0) |
There was a problem hiding this comment.
suggestion (testing): Add a test that interesting_strings alone also boost dictionary-aware operators
You currently test magic_bytes and boundary_markers boosting _DICT_OPERATORS, but not interesting_strings, which is also part of the condition. Please add a test like:
deftest_interesting_strings_boost_dict_operators(self):
p=TargetProfile()
p.interesting_strings= [b"token"]
priors=format_operator_priors(p)
assert"dict_insert"inpriorsassert"dict_append"inpriorsto exercise the interesting_strings branch of if profile.magic_bytes or profile.boundary_markers or profile.interesting_strings:.
There was a problem hiding this comment.
Pull request overview
This PR seeds the Thompson-sampling operator scheduler (MonteCarloScheduler) with informative Beta priors derived from static target profiling, so operator selection can start biased toward format- and token-relevant mutations instead of always starting from Beta(1, 1).
Changes:
- Added
format_operator_priors()to derive per-operator(alpha, beta)overrides fromTargetProfileformat/token hints. - Extended
MonteCarloScheduler.init_arm()to accept optional Beta prior parameters (with clamping) and exposed asupports_priorscapability flag. - Wired the fuzzer’s arm registration to pass priors only to schedulers that advertise prior support; updated README and added unit tests.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_target_profiler.py | Adds unit tests for format_operator_priors() output and basic prior sanity checks. |
| tests/test_montecarlo.py | Adds unit tests verifying informative priors, idempotency, and clamping behavior in init_arm(). |
| src/fuzzer_tool/services/fuzzer.py | Passes derived operator priors into arm registration for schedulers that support priors (MC bandit). |
| src/fuzzer_tool/core/target_profiler.py | Introduces operator-prior derivation and format/operator hint tables used to seed the bandit. |
| src/fuzzer_tool/core/montecarlo.py | Updates the Thompson scheduler to accept per-arm Beta priors and clamps invalid parameters. |
| README.md | Documents the new informative-prior behavior under static target analysis. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+619
to
+627
| _FORMAT_OPERATOR_HINTS: dict[str, tuple[str, ...]] = { | ||
| "png": ("png_chunk_mutate", "png_crc_fix"), | ||
| "jpeg": ("jpeg_chunk_mutate", "jpeg_crc_fix"), | ||
| "gzip": ("gzip_chunk_mutate",), | ||
| "bz2": ("gzip_chunk_mutate",), | ||
| "xz": ("gzip_chunk_mutate",), | ||
| "zlib": ("zlib_chunk_mutate",), | ||
| "riff": ("bmp_chunk_mutate",), | ||
| } |
Comment on lines
+640
to
+643
| # Beta prior for operators the profile suggests are relevant: same total | ||
| # "pseudo-observation" mass as the uninformative default (1, 1), but shifted | ||
| # toward success so Thompson sampling favors them before real evidence | ||
| # arrives. Weak enough that a handful of real failures will correct it. |
daedalus added a commit
that referenced
this pull request
Jul 16, 2026
Seed Thompson-sampling bandit with informative priors from static target profiling
daedalus added a commit
that referenced
this pull request
Aug 2, 2026
The x86 instruction-stream decoder (_decode_insns) was the #1 CPU consumer (0.216s/1000 iters) on targets where x86 mutations are irrelevant (PNG fuzzing): gating the operator behind a CLI flag cuts throughput cost to a single _AVAILABLE predicate check. Follows the enable_regex_bomb template: argparse flag + Fuzzer kwarg + registry availability predicate; no changes to build_dispatch/build_ops/ _register_arms. Removes the arm hint from _FORMAT_OPERATOR_HINTS so the profiler no longer auto-suggests it. Throughput 526 -> 664 eps (+26%) on the 2000-iter PNG hotpath run.
daedalus pushed a commit
that referenced
this pull request
Aug 20, 2026
…ession Records the four perf changes in this series, but weights the writeup toward the method errors, since those are not captured in the commit messages and cost the most time. Three findings worth carrying forward: - Profiling against a target with uninstrumented dependencies inverts the hotpath ranking. 90 edges (system libav) vs 4,248 (vendored, coverage built) moved two sites from absent to #1. - Fuzzer-level before/after A/B does not work. At a fixed 1,025 execs with -s 42, runs discovered 4,514 / 4,308 / 5,000 edges; allocation volume tracks edges and corpus size, not execs. An entire round of before/after numbers had to be discarded. - memray's total_bytes_allocated is churn, not pressure. The largest site by churn (13.4 GB) never touched peak RSS (~370 MB), and "fixing" it measured 2x slower. Also documents that --disable-x86asm does not disable inline asm, so --disable-inline-asm is required to link the vendored archives into a .so, and four predictions that measurement refuted.
daedalus added a commit
that referenced
this pull request
Aug 21, 2026
Bug report 2026-08-21, CRITICAL #1 and #4. adapters/persistent.py and services/minimize.py (twice) each built a private ctypes handle on libc and called shmat() without declaring restype. ctypes defaults to c_int, so the attach address was truncated to 32 bits and sign-extended: real 0x7fa8adb96000 observed 0xffffffffadba6000 persistent.py then memmove()d input data through that pointer on the first use of PersistentRunner, and minimize.py string_at()'d edge bitmaps through it on every replay of `minimize -c`. Reintroducing either site segfaults the pytest process outright. The truncation was also masking the failure path. shmat() reports failure as (void *) -1, which under the accidental c_int restype arrives as Python -1 -- so the `if ptr == -1` guards at all three sites were correct *because* of the bug. Under c_void_p the same value arrives as 0xffffffffffffffff, which is not equal to -1. Declaring the restype alone would have converted every attach failure into a silent one, with garbage read back from an unmapped address. Both halves change together. adapters/shm.py had the bindings right all along, which is what makes this a copy-the-neighbour bug. Rather than write the incantation a fourth time, the SysV calls now live in adapters/libc_shm.py, whose shmat() returns None on failure so the call-site check is a falsiness test that cannot be got wrong the way `== -1` can. shm.py keeps its own correct bindings; it is load-bearing and was left alone. Also corrected in passing: minimize.py passed 0o2000 to shmget with the comment "IPC_PRIVATE". 0o2000 is IPC_EXCL; IPC_CREAT is 0o1000. Harmless because key=IPC_PRIVATE creates unconditionally, but wrong as written. CRITICAL #5 is independent and still open: minimize -c against an uninstrumented target reads all-zero bitmaps and set-cover still prunes the entire corpus. #4 was necessary but not sufficient for cmin. Verification: 364 passed / 43 skipped across the shm, adapter, persistent, minimize, coverage and integration slices; 11 new tests in tests/test_regression_shmat_restype.py, including a package scan that fails if any module re-binds shmat without a restype. Confirmed the new tests catch the bug by reverting site 1 (pytest dies with SIGSEGV).
daedalus added a commit
that referenced
this pull request
Aug 21, 2026
Bug report 2026-08-21, CRITICAL #5, plus the discovery bug that was hiding it. #1/#4 are already fixed by 1f6a296; this builds on that. The masking bug -- minimize could not see a real corpus. minimize used a flat iterdir() on the directory it was handed, but save_to_corpus writes seeds/<hh>/id_<hash>. Against any real corpus it found nothing, printed "Corpus is empty" and exited 0. That is why #5 was unreachable in normal use, and why the #4 segfault only reproduced after flattening the files by hand. _discover_corpus_files accepts the sharded layout and a flat directory, and skips pruned/ at every level so a re-minimize cannot resurrect entries that were already pruned. This is the third writer/reader layout disagreement in this codebase, after load_corpus missing deltas/ and rehydrate_by_hash expecting sharded deltas. Same shape each time: one side shards, the other reads flat. A shared corpus-layout helper would be cheaper than a fourth instance. #5 -- a coverage blackout deleted the entire corpus. With discovery fixed, the reported behaviour appears: set cover picks files by the edges they contribute, so an all-zero bitmap set makes every file look redundant and all of them move to pruned/. before: [+] Minimized: 5 -> 0 files (5 removed) corpus wiped after: [+] Minimized: 3 -> 3 files (0 removed) refused A blackout means the measurement failed -- uninstrumented target, failed attach, or a child that never wrote the segment -- not that the seeds are worthless. _read_shm_bitmap's docstring (1f6a296) already states callers must read all-zero as "no coverage information" rather than "covers nothing"; this is that check. Bail out and name the likely cause instead. Legitimate pruning is unaffected: against a trace-pc-guard target, 6 -> 1 with pruned/ left intact. 5 of the 7 tests fail against 1f6a296. One note on the blackout test: it stubs the target runner as well as the attach. Patching the attach alone reproduces the blackout, but minimize_corpus then forks a real child, and under the full suite that fork happens in a multithreaded interpreter and segfaults the run outright -- finding E3 of the same report, which is live enough to kill a suite. The guard keys off the bitmaps, so stubbing the runner keeps the test hermetic rather than depending on a known landmine.
daedalus added a commit
that referenced
this pull request
Aug 21, 2026
Bug report 2026-08-21, CRITICAL #1 and #4. adapters/persistent.py and services/minimize.py (twice) each built a private ctypes handle on libc and called shmat() without declaring restype. ctypes defaults to c_int, so the attach address was truncated to 32 bits and sign-extended: real 0x7fa8adb96000 observed 0xffffffffadba6000 persistent.py then memmove()d input data through that pointer on the first use of PersistentRunner, and minimize.py string_at()'d edge bitmaps through it on every replay of `minimize -c`. Reintroducing either site segfaults the pytest process outright. The truncation was also masking the failure path. shmat() reports failure as (void *) -1, which under the accidental c_int restype arrives as Python -1 -- so the `if ptr == -1` guards at all three sites were correct *because* of the bug. Under c_void_p the same value arrives as 0xffffffffffffffff, which is not equal to -1. Declaring the restype alone would have converted every attach failure into a silent one, with garbage read back from an unmapped address. Both halves change together. adapters/shm.py had the bindings right all along, which is what makes this a copy-the-neighbour bug. Rather than write the incantation a fourth time, the SysV calls now live in adapters/libc_shm.py, whose shmat() returns None on failure so the call-site check is a falsiness test that cannot be got wrong the way `== -1` can. shm.py keeps its own correct bindings; it is load-bearing and was left alone. Also corrected in passing: minimize.py passed 0o2000 to shmget with the comment "IPC_PRIVATE". 0o2000 is IPC_EXCL; IPC_CREAT is 0o1000. Harmless because key=IPC_PRIVATE creates unconditionally, but wrong as written. CRITICAL #5 is independent and still open: minimize -c against an uninstrumented target reads all-zero bitmaps and set-cover still prunes the entire corpus. #4 was necessary but not sufficient for cmin. Verification: 364 passed / 43 skipped across the shm, adapter, persistent, minimize, coverage and integration slices; 11 new tests in tests/test_regression_shmat_restype.py, including a package scan that fails if any module re-binds shmat without a restype. Confirmed the new tests catch the bug by reverting site 1 (pytest dies with SIGSEGV).
daedalus added a commit
that referenced
this pull request
Aug 21, 2026
Bug report 2026-08-21, CRITICAL #5, plus the discovery bug that was hiding it. #1/#4 are already fixed by 1f6a296; this builds on that. The masking bug -- minimize could not see a real corpus. minimize used a flat iterdir() on the directory it was handed, but save_to_corpus writes seeds/<hh>/id_<hash>. Against any real corpus it found nothing, printed "Corpus is empty" and exited 0. That is why #5 was unreachable in normal use, and why the #4 segfault only reproduced after flattening the files by hand. _discover_corpus_files accepts the sharded layout and a flat directory, and skips pruned/ at every level so a re-minimize cannot resurrect entries that were already pruned. This is the third writer/reader layout disagreement in this codebase, after load_corpus missing deltas/ and rehydrate_by_hash expecting sharded deltas. Same shape each time: one side shards, the other reads flat. A shared corpus-layout helper would be cheaper than a fourth instance. #5 -- a coverage blackout deleted the entire corpus. With discovery fixed, the reported behaviour appears: set cover picks files by the edges they contribute, so an all-zero bitmap set makes every file look redundant and all of them move to pruned/. before: [+] Minimized: 5 -> 0 files (5 removed) corpus wiped after: [+] Minimized: 3 -> 3 files (0 removed) refused A blackout means the measurement failed -- uninstrumented target, failed attach, or a child that never wrote the segment -- not that the seeds are worthless. _read_shm_bitmap's docstring (1f6a296) already states callers must read all-zero as "no coverage information" rather than "covers nothing"; this is that check. Bail out and name the likely cause instead. Legitimate pruning is unaffected: against a trace-pc-guard target, 6 -> 1 with pruned/ left intact. 5 of the 7 tests fail against 1f6a296. One note on the blackout test: it stubs the target runner as well as the attach. Patching the attach alone reproduces the blackout, but minimize_corpus then forks a real child, and under the full suite that fork happens in a multithreaded interpreter and segfaults the run outright -- finding E3 of the same report, which is live enough to kill a suite. The guard keys off the bitmaps, so stubbing the runner keeps the test hermetic rather than depending on a known landmine.
daedalus pushed a commit
that referenced
this pull request
Aug 22, 2026
Annotates the bug report's status column, which had drifted badly: only #1 was marked FIXED, but #2 (6f7a866), #5 (c6fa0ce) and #7/#8/#9/#14/#21/#40 (3712812) had all been closed by earlier commits without the doc being updated. Anyone working the backlog was one step from re-fixing them. The learnings note covers the part worth keeping: that the obvious pytest timeout traded away a property the codebase depends on, and that both of the first two attempts to verify process-group cleanup produced a false FAIL -- a pgrep matching the harness's own command line, then a ps count including zombies that this container's PID 1 never reaps. When a cleanup assertion fails, suspect the observation before the code.
daedalus added a commit
that referenced
this pull request
Aug 31, 2026
…(arXiv:2603.03257) - Add literature-update note citing Diskin-Easo-Radhakrishnan-Sudakov-Tassion, 'Supercritical sharpness of percolation', proving exponential cluster-size decay via the isoperimetric function Phi(n) for every infinite transitive graph (no degree-distribution assumption needed). - Revise Module 3 (Target Difficulty Estimation): replace the Erdos-Renyi p_c = 1/<k> plan with estimating Phi(n) directly from the CFG, since Phi needs no assumption about the coverage graph's degree distribution. - Revise Module 5 (First Passage Percolation): note Theorem 3's explicit v_n growth-rate formula (inverting the integral of 1/Phi) as a principled replacement for ad hoc discovery-rate extrapolation. - Close falsifier #1 in section 9 (heavy-tailed/scale-free coverage graphs) as largely addressed, with two remaining caveats (finite/non-transitive graph, cost of computing Phi exactly). - Add References section.
daedalus added a commit
that referenced
this pull request
Sep 4, 2026
…point A campaign against ffmpeg_read reported an ASAN heap-buffer-overflow in mov_read_header. The top two frames were ours: #0 __afl_put_hexbytes afl_shim.c:1313 #1 __afl_cmplog_bytes afl_shim.c:1419 #2 afl_cmp_strncmp afl_shim.c:1698 #3 strncmp #4 url_find_protocol Our instrumentation crashing, filed as a crash in the target. The length an interceptor passes is a semantic bound -- how many bytes the real call was permitted to look at -- not a readable one. strncmp stops at the first NUL or mismatch, so char *p = malloc(2); p[0] = 'h'; p[1] = 'x'; strncmp(p, "http:", 5); is legal C: two bytes are compared and byte 2 is never touched. The logger read all five. Fix it at __afl_cmplog_bytes rather than in strncmp. There are 18 direct call sites, each open-coding its own length with a different convention (n, n + 1, k, a hand-rolled CMPLOG_MAX_OPERAND clamp). strcmp was written carefully, the memmem site already carries a comment about this exact class of read, and strncmp and strncasecmp were the ones that were missed. One guard at the only place that dereferences the operands means the next interceptor cannot forget it. Two layers, covering different builds: - Page clamp, unconditional. A read that stays inside the page holding p cannot fault, because the caller just read from that page. This is the only guard a non-ASAN target gets, and it is where it matters most: a fault in the nosan .so is an in-process SIGSEGV with our frame on top and no redzone diagnostic to say whose it was. - __asan_region_is_poisoned when the weak symbol resolves, tightening the page bound to the exact object bound. NULL in a non-ASAN build, so that path costs one null check. Not a truncation to the first mismatch, which was the first thing I tried. It is provably safe and it guts the record: with buf all 'A', strncmp(buf, "PREFIX_", 7) mismatches at byte 0, so the literal never reaches the pool -- and the token is the thing redqueen injects. test_regression_cmplog_shim_merge catches exactly that, and clamping by readability instead keeps it green: before fix: CMP 41414141414141 5052454649585f -15 7 after fix: CMP 41414141414141 5052454649585f -15 7 (unchanged) overread: CMP 6878 6874 4 5 (was: 5 bytes, ASAN abort) Adds tests/test_regression_cmplog_strncmp_overread.py, which fails without the change. It covers strncmp, strncasecmp and a fully readable compare, so a fix that clamps too aggressively fails the third case. Two tracecmp_visibility failures in the suite pre-date this change; the same two fail on a clean tree.
daedalus added a commit
that referenced
this pull request
Sep 4, 2026
…point A campaign against ffmpeg_read reported an ASAN heap-buffer-overflow in mov_read_header. The top two frames were ours: #0 __afl_put_hexbytes afl_shim.c:1313 #1 __afl_cmplog_bytes afl_shim.c:1419 #2 afl_cmp_strncmp afl_shim.c:1698 #3 strncmp #4 url_find_protocol Our instrumentation crashing, filed as a crash in the target. The length an interceptor passes is a semantic bound -- how many bytes the real call was permitted to look at -- not a readable one. strncmp stops at the first NUL or mismatch, so char *p = malloc(2); p[0] = 'h'; p[1] = 'x'; strncmp(p, "http:", 5); is legal C: two bytes are compared and byte 2 is never touched. The logger read all five. Fix it at __afl_cmplog_bytes rather than in strncmp. There are 18 direct call sites, each open-coding its own length with a different convention (n, n + 1, k, a hand-rolled CMPLOG_MAX_OPERAND clamp). strcmp was written carefully, the memmem site already carries a comment about this exact class of read, and strncmp and strncasecmp were the ones that were missed. One guard at the only place that dereferences the operands means the next interceptor cannot forget it. Two layers, covering different builds: - Page clamp, unconditional. A read that stays inside the page holding p cannot fault, because the caller just read from that page. This is the only guard a non-ASAN target gets, and it is where it matters most: a fault in the nosan .so is an in-process SIGSEGV with our frame on top and no redzone diagnostic to say whose it was. - __asan_region_is_poisoned when the weak symbol resolves, tightening the page bound to the exact object bound. NULL in a non-ASAN build, so that path costs one null check. Not a truncation to the first mismatch, which was the first thing I tried. It is provably safe and it guts the record: with buf all 'A', strncmp(buf, "PREFIX_", 7) mismatches at byte 0, so the literal never reaches the pool -- and the token is the thing redqueen injects. test_regression_cmplog_shim_merge catches exactly that, and clamping by readability instead keeps it green: before fix: CMP 41414141414141 5052454649585f -15 7 after fix: CMP 41414141414141 5052454649585f -15 7 (unchanged) overread: CMP 6878 6874 4 5 (was: 5 bytes, ASAN abort) Adds tests/test_regression_cmplog_strncmp_overread.py, which fails without the change. It covers strncmp, strncasecmp and a fully readable compare, so a fix that clamps too aggressively fails the third case. Two tracecmp_visibility failures in the suite pre-date this change; the same two fail on a clean tree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every mutation operator arm in the Thompson-sampling bandit (
MonteCarloScheduler) started from the same uninformative Beta(1, 1) prior, even when static analysis already knows the target's input format (e.g. PNG, gzip) or has extracted dictionary tokens — evidence the bandit would otherwise have to rediscover through executions.Bandit
MonteCarloScheduler.init_arm()accepts optionalprior_alpha/prior_beta; defaults unchanged (Beta(1,1))MIN_BETA_PARAMconstant to keepbetavariate()sampling stablesupports_priorscapability flag so only Bayesian schedulers accept priors (MOpt/Replicator/Elo use non-Beta internal state and are unaffected)Static profiling → priors
format_operator_priors()incore/target_profiler.py: derives{operator: (alpha, beta)}overridesFORMAT_MUTATIONSoperatorsDICT_MUTATIONSoperatorsWiring
services/fuzzer.py's_register_arms()computesformat_operator_priors(self._profile)and passes it through to the bandit only, with defensive validation before unpackingREADME updated to document the new prior-derivation behavior under Static Target Analysis.
Summary by Sourcery
Seed the Thompson-sampling scheduler with informative priors derived from static target profiling and wire these priors into scheduler arm registration.
New Features:
Enhancements:
Tests: