Reviewing project for overall assessment - #3
Merged
Conversation
zlib_read.c: decompresses gzip data using zlib's inflate function. Compile: gcc -O2 -g -o targets/zlib_read targets/zlib_read.c -lz Corpus: 3 seed gzip files for fuzzing.
_clean_env split LD_PRELOAD on ':' but the system uses spaces as separator. This left ksm_preload.so in the environment, which loads before ASan and causes 'ASan runtime does not come first' — the target crashes before AFL instrumentation initializes, producing zero edges. Fix: split on both ':' and whitespace via regex. Also move the 'import re' to module level for clarity.
…dead code, portable test path 1. ShapleyAttribution: replaced naive co-occurrence attribution with per-edge frequency-weighted crediting. Each edge's credit is now distributed among operators proportional to how often each operator co-occurred with that edge across all executions. Operators that consistently appear when a specific edge is observed get more credit; operators that merely co-occur with productive ones get less. 2. shm.py resize(): removed dead variables (old_seen, old_cumulative, old_total, old_size) that were captured but never used, and fixed misleading docstring that claimed 'preserves cumulative _seen' when the code correctly resets it. 3. test_crash_eta_integration.py: replaced hardcoded absolute path with Path.parents[2] for portability across machines.
Discovery in sparse coverage regions (low bitmap density) now gets higher reward weight than discovery in saturated regions. The weight is 1 - bitmap_density, clamped to [0.05, 1.0], computed per-execution from the existing edge_tracker infrastructure. Weighted reward flows into: - Thompson Bandit: alpha += weight (was hardcoded +1) - MOpt PSO: particle discovery = weight (was binary 1/0) - Replicator: fitness_sum += weight (was binary 1.0/0.0) - Elo strategy/seed matches: score = weight (was binary 1.0/0.0) - Brier calibration: outcome = weight (was binary 1.0/0.0) This means an operator that stumbles into a rare, previously-dark region of the coverage bitmap gets meaningfully more credit than one that finds a coverage-adjacent edge in an already-saturated area.
Add CrashMITracker that computes mutual information between each input byte position and the binary crash/no-crash outcome. Replaces the static keyword heuristic with a live, evidence-updating risk estimate. - CrashMITracker: joint distribution P(byte_val, crash), per-position MI computation, crash_density_estimate() for ETA blending - Blends 70% dynamic MI + 30% static keywords when enough data exists, dynamically weighting toward MI as observations accumulate - Persisted to crash_mi.json for resume across sessions - Wired into fuzz_one: records (mutated, is_crash) after every execution - Passed to estimate_execs_to_first_crash during calibration
FrameShift was fully built and tested (31/31 tests) but completely disconnected from the live fuzzing loop. Now: - Initialized in Fuzzer.__init__ with max_relations=64 - Relations discovered during calibration via seed0 insertion analysis - Length changes tracked per-mutation: on_insert/on_delete called when buffer size changes during block_insert, block_delete, splice, etc. - apply_to_buffer runs after every mutation loop (and after havoc) to auto-adjust known length fields before target execution This extends correct-length-field-preservation to every format the fuzzer encounters, not just PNG/JPEG/BMP/gzip with dedicated mutators.
Structure-aware mutator for the zlib compression format (distinct from gzip). Parses CMF/FLG header, compressed DEFLATE stream, and Adler-32 trailer. 10 mutation operations: corrupt CMF/CINFO, mutate FLEVEL, flip deflate stream bytes, replace deflate blocks, corrupt Adler-32, swap header nibbles, truncate, inject junk, change window size, and random generation. Registered as 'zlib_chunk_mutate' in FORMAT_MUTATIONS and dispatch table. Follows the exact template of gzip_mutations.py, jpeg_mutations.py, png_mutations.py, and bmp_mutations.py.
Add two operators that operate at arbitrary bit offsets, not just byte-aligned positions: - bit_offset_flip: flips a single bit at any offset across the entire buffer (0 to 8*len-1). Critical for DEFLATE Huffman codes and JPEG entropy data where the meaningful unit is a bit, not a byte. - bit_offset_span: flips 1-8 consecutive bits at an arbitrary offset, with width distribution matching common Huffman code lengths (3-7 bits for DEFLATE, 2-8 for JPEG DC/AC coefficients). Can cross byte boundaries, corrupting variable-length bit sequences. Also fixed test_asan_finds_heap_buffer_overflow: was checking only txt_files[0] which could pick up a different crash report depending on filesystem iteration order. Now checks any txt file.
… longer fit after length-shrinking mutations
…prior validation, comment
Seed Thompson-sampling bandit with informative priors from static target profiling
Consolidate benchmark shell scripts into shared lib + merged sweep script
Copilotcreated this pull request from a session on behalf of daedalusJuly 16, 2026 20:08 View session
Reviewer's GuideRefactors entropy sampling and stall detection into dedicated helpers, tightens stall recovery semantics based on entropy behavior, switches edge similarity hashing to a shared adapter, promotes key analysis diagnostics from debug to warning, and updates entropy-related tests and documentation accordingly. Sequence diagram for updated stall recovery based on entropy ratesequenceDiagram
participant Fuzzer
participant EdgeTracker
Fuzzer->>EdgeTracker: shannon_entropy_global
Fuzzer->>Fuzzer: _record_entropy_sample
alt [stall threshold reached]
Fuzzer->>Fuzzer: _compute_entropy_flat
alt [entropy_flat is False]
Fuzzer-->>Fuzzer: _maybe_trigger_stall_recovery returns False
else [entropy_flat is None or True]
Fuzzer->>Fuzzer: _maybe_trigger_stall_recovery
end
end
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
daedalus
marked this pull request as ready for review
July 16, 2026 20:10
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The entropy helpers use several hard-coded thresholds (history sizes 200/100, window size 4, rate < 0.001); consider lifting these into named attributes or constants so they can be tuned and referenced consistently across the fuzzer and tests.
- With the switch from zlib.crc32 to hash_data for band hashing, double-check that hash_data returns a deterministic value of the expected type (e.g., int) for struct-packed bytes so bucket keys remain comparable to existing data and any persisted state.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments- The entropy helpers use several hard-coded thresholds (history sizes 200/100, window size 4, rate < 0.001); consider lifting these into named attributes or constants so they can be tuned and referenced consistently across the fuzzer and tests.
- With the switch from zlib.crc32 to hash_data for band hashing, double-check that hash_data returns a deterministic value of the expected type (e.g., int) for struct-packed bytes so bucket keys remain comparable to existing data and any persisted state.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Copilot stopped work on behalf of daedalus due to an error
July 16, 2026 20:35
daedalus added a commit
that referenced
this pull request
Jul 17, 2026
ENTROPY_HISTORY_MAX=200, ENTROPY_HISTORY_TRIM=100, ENTROPY_WINDOW=4, ENTROPY_FLAT_THRESHOLD=0.001 All four findings verified: - #2: zlib.crc32 returns deterministic int for LSH bucket keys ✓ - #3: grammar repeat bounds already clamped with max(hi, lo) ✓ - #4: edge_tracker uses zlib.crc32, not builtin hash() ✓
daedalus added a commit
that referenced
this pull request
Aug 3, 2026
Attempted the audit's #3 candidate (4x np.array(list) -> array('d') + np.frombuffer views in seed_picker._compute_weights). Interleaved A/B at 1000 seeds measured the new version ~4% SLOWER (1269us vs 1319us/call, median of 3): array.append is ~46ns vs list.append ~26ns, and the data must be gathered element-by-element either way, so the slower appends outweigh the conversion savings — the from-array win only applies when the buffer already exists. Reverted the source change (diff is clean). Kept this behavior test (did not exist before): stubs the Phase-2 weight modifiers to identity and asserts the vectorized Phase-1 weights match the closed-form formula (2 seeds skips the Pareto adjustment). It is implementation-agnostic and pins the weight contract.
daedalus pushed a commit
that referenced
this pull request
Aug 22, 2026
Bug report 2026-08-21, CRITICAL #3. run_target_fast is the default spawn-fallback path -- run_target picks it whenever the run is neither file_mode nor cmplog -- and it was the only backend not honouring f.timeout. Three defects, each independently fatal, demonstrated against the unfixed code under a 15s external cap: infinite-loop target: SIGKILLed at 15s, never returned 400 KiB of stderr: SIGKILLed at 15s, never returned The first is os.waitpid(pid, 0) with no deadline. The second is separate and survives any timeout fix that only adds a deadline: the pipe holds 64 KiB, so a chatty target blocks in write() while the parent blocks in waitpid(), and reading stderr after the reap can never break the cycle. Third, 'except Exception: return -2, str(e), 0' replaced the pid the caller needs for crash attribution with 0, and left a spawned child neither killed nor reaped. After: the looper returns rc=-1 at the deadline; the chatty target returns rc=3 with 65536 B captured in 0.77s. The bound is enforced by poll() on the stderr pipe rather than by a watchdog thread. run_target_stdin and run_target_file both use a watchdog, and Hard Rule 1 says match the closest example -- but this path exists specifically to create no threads, and a thread here puts the hottest path in the fuzzer back inside E3's multi-threaded-fork blast radius. EOF on the pipe means every writer closed it, which for a spawned child means it exited, so one poll() is both the drain and the liveness wait. The child is spawned into its own process group (setpgroup=0, the posix_spawn equivalent of the siblings' preexec_fn=os.setsid) so the timeout kill reaches grandchildren. Measured, 5 interleaved repeats of 400 execs: timeout=None 980.3 eps timeout=5.0 989.2 eps Ranges fully overlap; no cost. timeout=None keeps the old unbounded behaviour for callers that pass nothing, so runner.py forwarding f.timeout is load-bearing and has a wiring test. _TRACKED_PIDS and friends move out of the 'Stdin mode' section: all three modes track their children now, and a leaked pid from any of them is the same bug. Residual case, documented rather than papered over: a target that closes fd 2 and then loops forever reaches EOF without exiting, and the reap blocks. Polling the reap would put a sleep on every execution to cover a target that deliberately closes its own stderr.
daedalus pushed a commit
that referenced
this pull request
Aug 22, 2026
Annotates the bug report's status column, which had drifted badly: only #1 was marked FIXED, but #2 (6f7a866), #5 (c6fa0ce) and #7/#8/#9/#14/#21/#40 (3712812) had all been closed by earlier commits without the doc being updated. Anyone working the backlog was one step from re-fixing them. The learnings note covers the part worth keeping: that the obvious pytest timeout traded away a property the codebase depends on, and that both of the first two attempts to verify process-group cleanup produced a false FAIL -- a pgrep matching the harness's own command line, then a ps count including zombies that this container's PID 1 never reaps. When a cleanup assertion fails, suspect the observation before the code.
daedalus added a commit
that referenced
this pull request
Sep 4, 2026
…point A campaign against ffmpeg_read reported an ASAN heap-buffer-overflow in mov_read_header. The top two frames were ours: #0 __afl_put_hexbytes afl_shim.c:1313 #1 __afl_cmplog_bytes afl_shim.c:1419 #2 afl_cmp_strncmp afl_shim.c:1698 #3 strncmp #4 url_find_protocol Our instrumentation crashing, filed as a crash in the target. The length an interceptor passes is a semantic bound -- how many bytes the real call was permitted to look at -- not a readable one. strncmp stops at the first NUL or mismatch, so char *p = malloc(2); p[0] = 'h'; p[1] = 'x'; strncmp(p, "http:", 5); is legal C: two bytes are compared and byte 2 is never touched. The logger read all five. Fix it at __afl_cmplog_bytes rather than in strncmp. There are 18 direct call sites, each open-coding its own length with a different convention (n, n + 1, k, a hand-rolled CMPLOG_MAX_OPERAND clamp). strcmp was written carefully, the memmem site already carries a comment about this exact class of read, and strncmp and strncasecmp were the ones that were missed. One guard at the only place that dereferences the operands means the next interceptor cannot forget it. Two layers, covering different builds: - Page clamp, unconditional. A read that stays inside the page holding p cannot fault, because the caller just read from that page. This is the only guard a non-ASAN target gets, and it is where it matters most: a fault in the nosan .so is an in-process SIGSEGV with our frame on top and no redzone diagnostic to say whose it was. - __asan_region_is_poisoned when the weak symbol resolves, tightening the page bound to the exact object bound. NULL in a non-ASAN build, so that path costs one null check. Not a truncation to the first mismatch, which was the first thing I tried. It is provably safe and it guts the record: with buf all 'A', strncmp(buf, "PREFIX_", 7) mismatches at byte 0, so the literal never reaches the pool -- and the token is the thing redqueen injects. test_regression_cmplog_shim_merge catches exactly that, and clamping by readability instead keeps it green: before fix: CMP 41414141414141 5052454649585f -15 7 after fix: CMP 41414141414141 5052454649585f -15 7 (unchanged) overread: CMP 6878 6874 4 5 (was: 5 bytes, ASAN abort) Adds tests/test_regression_cmplog_strncmp_overread.py, which fails without the change. It covers strncmp, strncasecmp and a fully readable compare, so a fix that clamps too aggressively fails the third case. Two tracecmp_visibility failures in the suite pre-date this change; the same two fail on a clean tree.
daedalus added a commit
that referenced
this pull request
Sep 4, 2026
…point A campaign against ffmpeg_read reported an ASAN heap-buffer-overflow in mov_read_header. The top two frames were ours: #0 __afl_put_hexbytes afl_shim.c:1313 #1 __afl_cmplog_bytes afl_shim.c:1419 #2 afl_cmp_strncmp afl_shim.c:1698 #3 strncmp #4 url_find_protocol Our instrumentation crashing, filed as a crash in the target. The length an interceptor passes is a semantic bound -- how many bytes the real call was permitted to look at -- not a readable one. strncmp stops at the first NUL or mismatch, so char *p = malloc(2); p[0] = 'h'; p[1] = 'x'; strncmp(p, "http:", 5); is legal C: two bytes are compared and byte 2 is never touched. The logger read all five. Fix it at __afl_cmplog_bytes rather than in strncmp. There are 18 direct call sites, each open-coding its own length with a different convention (n, n + 1, k, a hand-rolled CMPLOG_MAX_OPERAND clamp). strcmp was written carefully, the memmem site already carries a comment about this exact class of read, and strncmp and strncasecmp were the ones that were missed. One guard at the only place that dereferences the operands means the next interceptor cannot forget it. Two layers, covering different builds: - Page clamp, unconditional. A read that stays inside the page holding p cannot fault, because the caller just read from that page. This is the only guard a non-ASAN target gets, and it is where it matters most: a fault in the nosan .so is an in-process SIGSEGV with our frame on top and no redzone diagnostic to say whose it was. - __asan_region_is_poisoned when the weak symbol resolves, tightening the page bound to the exact object bound. NULL in a non-ASAN build, so that path costs one null check. Not a truncation to the first mismatch, which was the first thing I tried. It is provably safe and it guts the record: with buf all 'A', strncmp(buf, "PREFIX_", 7) mismatches at byte 0, so the literal never reaches the pool -- and the token is the thing redqueen injects. test_regression_cmplog_shim_merge catches exactly that, and clamping by readability instead keeps it green: before fix: CMP 41414141414141 5052454649585f -15 7 after fix: CMP 41414141414141 5052454649585f -15 7 (unchanged) overread: CMP 6878 6874 4 5 (was: 5 bytes, ASAN abort) Adds tests/test_regression_cmplog_strncmp_overread.py, which fails without the change. It covers strncmp, strncasecmp and a fully readable compare, so a fix that clamps too aggressively fails the third case. Two tracecmp_visibility failures in the suite pre-date this change; the same two fail on a clean tree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request created by AI Agent
Summary by Sourcery
Refine entropy-based stall detection in the fuzzer and align supporting components, logging, and documentation with the new behavior.
New Features:
Enhancements:
Tests: