Add script to generate fuzz coverage

[Prabhat1308]

This script generates the fuzz coverage in a html file when run .

Considerations

1. The script assumes a corpus directory in fuzz directory.
2. Also assumes the corpus for a particular target to be under the name of the target itself. for eg - corpus for base32_target.rs would be under corpus/base32_target.
3. The corpus I worked with was generated using libFuzzer it may/may not work with corpus generated from other fuzzers.

How to run

# from the root directory
chmod +x contrib/generate_fuzz_coverage.sh
contrib/generate_fuzz_coverage.sh

[ldk-reviews-bot]

👋 I see @jkczyz was un-assigned.
If you'd like another reviewer assignemnt, please click here.

[Prabhat1308]

If needed I can also push my script to generate corpus from libFuzzer that create the corpus according to the considerations if the reviewers deem it worthy .

[TheBlueMatt]

I wonder if we should include this coverage data in our codecov output in PRs...On the one hand its not "reliable" coverage data in that we aren't actually testing the behavior, but on the other hand we are at least hitting the code lines and presumable would find crashes. WDYT?

[TheBlueMatt]

Rather than running the fuzz target specifically, can we just use cargo llvm-cov to run the tests? For each binary in the fuzz tests we already support running cargo test and it'll pull all the files from fuzz/test_cases/X and run it through the fuzz processor. Seems easier than relying on the libfuzzer logic.

[Prabhat1308]

Not sure about that. Running on test_cases with cargo test would remove the fuzziness since now it will be run test_cases and wont run the fuzzer but since its coverage report we are basically doing the same thing. I would have to check on this.

[TheBlueMatt]

Someone doing fuzzing can presumably copy their corpus from wherever it is to test_cases. In the comment you note that we currently expect a corpus in corpus, but it could well be any folder name?

[Prabhat1308]

I have changed the script to use cargo test instance now. Surprisingly reduced the code by 99%. Although I don't see the test_cases in any branch . Maybe you can try running the script and provide some feedback since I dont know the format on how test_cases directory has the inputs.

[TheBlueMatt]

Right, test_cases isn't automatically created anywhere, its just read by the tests if it exists. Not sure what to do about that wrt the script here, maybe we should have it look for the llvm-libfuzzer, afl, and honggfuzz dirs, copy them to test (if its empty) and then run cargo test?

[TheBlueMatt]

Did you want to go ahead and do this?

[Prabhat1308]

Hi . sorry for the late response . Have been busy recently with my academics . Would implement this by the end of this month hopefully.

[Prabhat1308]

I updated the PR according to the comment , finding the corpus directories is not feasible since there is no clear indication/factor that can catch that this is a corpus directory apart from the object format which might also be different depending on the fuzzer. Also the location of corpus can be out of the project and some people tend to keep their project directory clean . Instead what I have done is to check for test_cases and if its not present/empty it will throw a message and instruction to copy paste (but not halt the fuzz run).

[ldk-reviews-bot]

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

[Prabhat1308]

I wonder if we should include this coverage data in our codecov output in PRs...On the one hand its not "reliable" coverage data in that we aren't actually testing the behavior, but on the other hand we are at least hitting the code lines and presumable would find crashes. WDYT?

I would prefer to not add it in codecov until we have a good public corpora of fuzz inputs to run the report on.

[TheBlueMatt]

I would prefer to not add it in codecov until we have a good public corpora of fuzz inputs to run the report on.

One reason to do this would be to only include "trivially-reachable" things in the codecov output. We don't really want to include stuff as coverage if the fuzzer happened to reach it only once, but if the fuzzer found its way into it in a minute of CI time, its probably decent coverage :)

[Prabhat1308]

I would prefer to not add it in codecov until we have a good public corpora of fuzz inputs to run the report on.

One reason to do this would be to only include "trivially-reachable" things in the codecov output. We don't really want to include stuff as coverage if the fuzzer happened to reach it only once, but if the fuzzer found its way into it in a minute of CI time, its probably decent coverage :)

Makes sense if thats the case.

[TheBlueMatt]

Do you intend to go ahead and run this in CI? Or would you prefer not to?

[Prabhat1308]

added this script to run in CI .

[Prabhat1308]

1.63 versions are failing because llvm-cov requires 1.65 , other 2 failures are because of storage issue on the CI instance.

[TheBlueMatt]

Oops, so sorry I missed your comments here.

[TheBlueMatt]

Ah, rather than doing this in ci-tests.sh let's do this in a new script which is called from the coverage job in CI.

[graphite-app]

The script currently checks for the existence of test corpus files but doesn't appear to use them when generating coverage with cargo llvm-cov. For accurate fuzz coverage reporting, the command should incorporate these corpus files as test inputs.

Consider modifying the cargo llvm-cov command to explicitly use the corpus files, perhaps with something like:

# For each target with a corpusfortarget_dirin test_cases/*;doif [ -d"$target_dir" ];then
target_name=$(basename "$target_dir")
cargo llvm-cov --html --test-dir "$target_dir" --target "$target_name" ...
fidone

This would ensure the coverage report reflects code paths exercised by the fuzzing corpus rather than just standard test execution.

Suggested change

	export RUSTFLAGS="--cfg=fuzzing --cfg=secp256k1_fuzz --cfg=hashes_fuzz"
	# ignore anything in fuzz directory since we don't want coverage of targets
	cargo llvm-cov --html --ignore-filename-regex "fuzz/" --output-dir "$OUTPUT_DIR"
	export RUSTFLAGS="--cfg=fuzzing --cfg=secp256k1_fuzz --cfg=hashes_fuzz"
	# Process each fuzz target with its corpus
	mkdir -p "$OUTPUT_DIR"
	fortarget_dirin fuzz/corpus/*;do
	if [ -d"$target_dir" ];then
	target_name=$(basename "$target_dir")
	echo"Generating coverage for target: $target_name"
	# Run coverage for this specific target with its corpus
	cargo llvm-cov --html --ignore-filename-regex "fuzz/" \
	--fuzz-target "$target_name" \
	--corpus-dir "$target_dir" \
	--output-path "$OUTPUT_DIR/$target_name"
	fi
	done
	# Generate a combined report
	cargo llvm-cov --html --ignore-filename-regex "fuzz/" --output-dir "$OUTPUT_DIR/combined"

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

[Prabhat1308]

Hmm I dont think this is the correct way ? Cargo-llvm will use all the tests_cases (if present) and all targets in this scenario. Individually calling them seems redundant work

[Prabhat1308]

Failure caused because of no storage left on the ci infra

[TheBlueMatt]

Man I hate github CI. I think if we add a deletion of some of the files in target after the first cargo llvm-cov in the CI build.yml it should pass.

[Prabhat1308]

Working locally when run with the CI workflow.

[Prabhat1308]

CI passed . had to nuke the entire target directory

[tnull]

Might be worth giving CI a kick now that we fixed the No space left on device issue (for now) with #3916.

[TheBlueMatt]

This could be cleaned up a bit, but it looks like it works, so just gonna land. I'll open a followup to trim down some fat.

[TheBlueMatt]

Ha, we now have a fuzz/ci-fuzz.sh and a ci/ci-fuzz.sh, that's confusing :)