Reduce CI fuzz iterations as we're now timing out

[TheBlueMatt]

I think github has slowed down the runners so now our fuzz tests are timing out. Here we just reduce iteration count a bit.

[ldk-reviews-bot]

I've assigned @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

ACK, seems we hit yet another failing case in the router target now: #3692

[joostjager]

Any specific reasoning behind the 10x, 10x and 2x reductions?

[TheBlueMatt]

No. I probably could have benchmarked, but in general there's not a lot of value in full_stack_target in CI because its just too dense to make any real progress, and similar for chanmon_consistency_target.

[joostjager]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Maybe I am not interpreting correctly though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

base32_target.rs
Summary iterations:1000002 time:198 speed:5050 crashes_count:0 timeout_count:0 new_units_added:449 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bech32_parse_target.rs
Summary iterations:1000002 time:22 speed:45454 crashes_count:0 timeout_count:0 new_units_added:897 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bolt11_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:211 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
chanmon_consistency_target.rs
Summary iterations:100002 time:9769 speed:10 crashes_count:0 timeout_count:87 new_units_added:3435 slowest_unit_ms:1139 guard_nb:566357 branch_coverage_percent:2 peak_rss_mb:42
chanmon_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:963 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
fromstr_to_netaddress_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:160 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
full_stack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:797 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
indexedmap_target.rs
Summary iterations:1000002 time:648 speed:1543 crashes_count:0 timeout_count:0 new_units_added:698 slowest_unit_ms:19 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_deser_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:2955 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_request_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:2244 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_target.rs
Summary iterations:1000002 time:23 speed:43478 crashes_count:0 timeout_count:0 new_units_added:1115 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_v2_target.rs
Summary iterations:1000002 time:246 speed:4065 crashes_count:0 timeout_count:0 new_units_added:1031 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_announcement_signatures_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_announcement_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:413 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_details_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:993 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_ready_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:374 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_reestablish_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:434 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_update_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:210 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_closing_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:380 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_commitment_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:697 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_decoded_onion_error_packet_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:178 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_error_message_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_created_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:342 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:304 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_gossip_timestamp_filter_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:294 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_init_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:1006 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_node_announcement_target.rs
Summary iterations:1000002 time:21 speed:47619 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_target.rs
Summary iterations:1000002 time:35 speed:28571 crashes_count:0 timeout_count:0 new_units_added:1163 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_v2_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1289 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_ping_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:91 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_pong_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:83 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:334 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_short_channel_ids_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:158 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_short_channel_ids_end_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:337 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_revoke_and_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:360 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_shutdown_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:387 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_init_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:401 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_locked_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_stfu_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_abort_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:403 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_ack_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:366 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_input_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1270 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_output_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:390 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_complete_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:285 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_init_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:393 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_input_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:297 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_output_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_signatures_target.rs
Summary iterations:1000002 time:52 speed:19230 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_add_htlc_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:430 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:480 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_malformed_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:321 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fee_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:308 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fulfill_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:309 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
offer_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:1333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_hop_data_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:981 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_message_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:395 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
peer_crypt_target.rs
Summary iterations:1000002 time:27 speed:37037 crashes_count:0 timeout_count:0 new_units_added:315 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
process_network_graph_target.rs
Summary iterations:735922 time:8434 speed:87 crashes_count:0 timeout_count:0 new_units_added:30 slowest_unit_ms:40 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:161

[TheBlueMatt]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

Ah, thanks for doing that. Yea, I was going on the results when I fuzz with real corpuses, where full_stack_target can get quite slow. But, given we currently don't initialize our CI fuzzers with a real corpus so it really doesn't get into it too much. I did go ahead and update the CI fuzzer to use the hard-coded full_stack_target seeds, but it still doesn't spend much time in complicated paths.

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Huh, interesting, I went ahead and slowed this one down though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

Yep!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.92%. Comparing base (c4d23bc) to head (11dde05).
Report is 38 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #3691 +/- ##
==========================================
+ Coverage 89.05% 89.92% +0.86% 
==========================================
Files 155 156 +1 Lines 122019 129760 +7741 Branches 122019 129760 +7741 ==========================================
+ Hits 108666 116683 +8017 + Misses 10695 10441 -254 + Partials 2658 2636 -22 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tnull]

LGTM

Fuzz failure is #3708

[joostjager]

Great that fuzzing again found an issue. This is awesome.

Before merging this, I think we still want to see a successful fuzz run well within the timeout?

[TheBlueMatt]

Yea, happy to wait until we can at least run a fuzzing run and check that all the timings make sense.

[TheBlueMatt]

It passed, but taking five hours for CI is excessive, so I reduced it further.

[tnull]

It passed, but taking five hours for CI is excessive, so I reduced it further.

Now finished in 273min ~ 4.55 hours. Is this acceptable?

[TheBlueMatt]

Mmm, that's still pretty aggressive. Divided a few jobs by two so hopefully this time's the charm.

[TheBlueMatt]

Okay, its now "only" 2h, which I think is kinda reasonable, or at least its in line with what some of our other jobs take, so happy to land this now.

[joostjager]

Distribution looks much better now:

test	iterations	time
base32_target.rs	2500002	278
bech32_parse_target.rs	2500002	58
bolt11_deser_target.rs	2500002	45
chanmon_consistency_target.rs	5002	852
chanmon_deser_target.r	2500002	55
fromstr_to_netaddress_target.r	2500002	45
full_stack_target.rs	50002	554
indexedmap_target.rs	500002	394
invoice_deser_target.r	2500002	51
invoice_request_deser_target.r	2500002	50
msg_accept_channel_target.rs	2500002	181
msg_accept_channel_v2_target.r	2500002	258
msg_announcement_signatures_target.rs	2500002	53
msg_channel_announcement_target.rs	2500002	73
msg_channel_details_target.rs	2500002	47
msg_channel_ready_target.r	2500002	49
msg_channel_reestablish_target.rs	2500002	52
msg_channel_update_target.rs	2500002	54
msg_closing_signed_target.rs	2500002	50
msg_commitment_signed_target.r	2500002	55
msg_decoded_onion_error_packet_target.rs	2500002	50
msg_error_message_target.r	2500002	51
msg_funding_created_target.rs	2500002	52
msg_funding_signed_target.rs	2500002	50
msg_gossip_timestamp_filter_target.rs	2500002	47
msg_init_target.rs	2500002	48
msg_node_announcement_target.r	2500002	60
msg_open_channel_target.rs	2500002	71
msg_open_channel_v2_target.rs	2500002	66
msg_ping_target.rs	2500002	48
msg_pong_target.rs	2500002	48
msg_query_channel_range_target.rs	2500002	47
msg_query_short_channel_ids_target.rs	2500002	52
msg_reply_channel_range_target.rs	2500002	48
msg_reply_short_channel_ids_end_target.rs	2500002	47
msg_revoke_and_ack_target.rs	2500002	50
msg_shutdown_target.rs	2500002	47
msg_splice_ack_target.rs	2500002	50
msg_splice_init_target.rs	2500002	50
msg_splice_locked_target.r	2500002	48
msg_stfu_target.rs	2500002	47
msg_tx_abort_target.rs	2500002	48
msg_tx_ack_rbf_target.rs	2500002	47
msg_tx_add_input_target.rs	2500002	73
msg_tx_add_output_target.r	2500002	48
msg_tx_complete_target.rs	2500002	47
msg_tx_init_rbf_target.rs	2500002	48
msg_tx_remove_input_target.rs	2500002	47
msg_tx_remove_output_target.rs	2500002	48
msg_tx_signatures_target.r	2500002	100
msg_update_add_htlc_target.rs	2500002	99
msg_update_fail_htlc_target.rs	2500002	52
msg_update_fail_malformed_htlc_target.rs	2500002	49
msg_update_fee_target.rs	2500002	48
msg_update_fulfill_htlc_target.rs	2500002	49
offer_deser_target.rs	2500002	48
onion_hop_data_target.rs	2500002	49
onion_message_target.r	2500002	64
peer_crypt_target.rs	2500002	108
process_network_graph_target.r	50002	568
process_onion_failure_target.r	2500002	165
refund_deser_target.rs	2500002	49
router_target.rs	50002	606
zbase32_target.rs	2500002	119

[joostjager]

Why these two additions?

[TheBlueMatt]

Because RUSTFLAGS aren't applied to doctests, so we have to restrict tests to only run library and binary tests.

[joostjager]

Could add comments when making changes like this. Make life easier for future devs.

[joostjager]

I wanted to verify that using the hardcoded seed indeed increases coverage, but it seems that hongfuzz reports in the log always

branch_coverage_percent:0

Not sure why that is? Otherwise it would be easy to compare with and without this hardcoded seed.

Or maybe there is another way to verify that it works indeed?

[TheBlueMatt]

The fact that the full_stack_target got much slower is pretty good evidence :)

[joostjager]

Okay, indirect evidence. But why is hongfuzz reporting 0?

[joostjager]

Is it not using coverage to direct fuzzing?

[TheBlueMatt]

Yea, that I don't know, it certainly does locally and there's nothing different locally vs in CI. Also CI does turn up (fairly shallow) fuzz bugs as well, so I'd be surprised to learn it was able to do that with just general patterns and no instrumentation...

[joostjager]

Does it show you branch_coverage_percent as non-zero locally?

[TheBlueMatt]

I'm not 100% sure about branch coverage specifically, but what I get from a running instance is: Coverage : edge: 16,485/568,547 [2%] pc: 738 cmp: 794,641. I assume "cmp" is the branch coverage.

[TheBlueMatt]

Went ahead and squashed the addition of a comment in the fuzz script.

[joostjager]

Great to have a more reasonable run time for the fuzzer!

Still interested to find out what's going on with that coverage indicator at zero in CI.

[TheBlueMatt]

I think github has slowed down the runners so now our fuzz tests are timing out. Here we just reduce iteration count a bit.

[ldk-reviews-bot]

I've assigned @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

ACK, seems we hit yet another failing case in the router target now: #3692

[joostjager]

Any specific reasoning behind the 10x, 10x and 2x reductions?

[TheBlueMatt]

No. I probably could have benchmarked, but in general there's not a lot of value in full_stack_target in CI because its just too dense to make any real progress, and similar for chanmon_consistency_target.

[joostjager]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Maybe I am not interpreting correctly though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

base32_target.rs
Summary iterations:1000002 time:198 speed:5050 crashes_count:0 timeout_count:0 new_units_added:449 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bech32_parse_target.rs
Summary iterations:1000002 time:22 speed:45454 crashes_count:0 timeout_count:0 new_units_added:897 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bolt11_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:211 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
chanmon_consistency_target.rs
Summary iterations:100002 time:9769 speed:10 crashes_count:0 timeout_count:87 new_units_added:3435 slowest_unit_ms:1139 guard_nb:566357 branch_coverage_percent:2 peak_rss_mb:42
chanmon_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:963 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
fromstr_to_netaddress_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:160 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
full_stack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:797 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
indexedmap_target.rs
Summary iterations:1000002 time:648 speed:1543 crashes_count:0 timeout_count:0 new_units_added:698 slowest_unit_ms:19 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_deser_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:2955 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_request_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:2244 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_target.rs
Summary iterations:1000002 time:23 speed:43478 crashes_count:0 timeout_count:0 new_units_added:1115 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_v2_target.rs
Summary iterations:1000002 time:246 speed:4065 crashes_count:0 timeout_count:0 new_units_added:1031 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_announcement_signatures_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_announcement_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:413 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_details_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:993 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_ready_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:374 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_reestablish_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:434 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_update_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:210 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_closing_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:380 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_commitment_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:697 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_decoded_onion_error_packet_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:178 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_error_message_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_created_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:342 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:304 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_gossip_timestamp_filter_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:294 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_init_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:1006 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_node_announcement_target.rs
Summary iterations:1000002 time:21 speed:47619 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_target.rs
Summary iterations:1000002 time:35 speed:28571 crashes_count:0 timeout_count:0 new_units_added:1163 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_v2_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1289 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_ping_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:91 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_pong_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:83 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:334 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_short_channel_ids_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:158 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_short_channel_ids_end_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:337 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_revoke_and_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:360 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_shutdown_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:387 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_init_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:401 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_locked_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_stfu_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_abort_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:403 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_ack_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:366 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_input_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1270 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_output_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:390 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_complete_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:285 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_init_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:393 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_input_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:297 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_output_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_signatures_target.rs
Summary iterations:1000002 time:52 speed:19230 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_add_htlc_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:430 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:480 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_malformed_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:321 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fee_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:308 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fulfill_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:309 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
offer_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:1333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_hop_data_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:981 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_message_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:395 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
peer_crypt_target.rs
Summary iterations:1000002 time:27 speed:37037 crashes_count:0 timeout_count:0 new_units_added:315 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
process_network_graph_target.rs
Summary iterations:735922 time:8434 speed:87 crashes_count:0 timeout_count:0 new_units_added:30 slowest_unit_ms:40 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:161

[TheBlueMatt]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

Ah, thanks for doing that. Yea, I was going on the results when I fuzz with real corpuses, where full_stack_target can get quite slow. But, given we currently don't initialize our CI fuzzers with a real corpus so it really doesn't get into it too much. I did go ahead and update the CI fuzzer to use the hard-coded full_stack_target seeds, but it still doesn't spend much time in complicated paths.

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Huh, interesting, I went ahead and slowed this one down though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

Yep!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.92%. Comparing base (c4d23bc) to head (11dde05).
Report is 38 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #3691 +/- ##
==========================================
+ Coverage 89.05% 89.92% +0.86% 
==========================================
Files 155 156 +1 Lines 122019 129760 +7741 Branches 122019 129760 +7741 ==========================================
+ Hits 108666 116683 +8017 + Misses 10695 10441 -254 + Partials 2658 2636 -22 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tnull]

LGTM

Fuzz failure is #3708

[joostjager]

Great that fuzzing again found an issue. This is awesome.

Before merging this, I think we still want to see a successful fuzz run well within the timeout?

[TheBlueMatt]

Yea, happy to wait until we can at least run a fuzzing run and check that all the timings make sense.

[TheBlueMatt]

It passed, but taking five hours for CI is excessive, so I reduced it further.

[tnull]

It passed, but taking five hours for CI is excessive, so I reduced it further.

Now finished in 273min ~ 4.55 hours. Is this acceptable?

[TheBlueMatt]

Mmm, that's still pretty aggressive. Divided a few jobs by two so hopefully this time's the charm.

[TheBlueMatt]

Okay, its now "only" 2h, which I think is kinda reasonable, or at least its in line with what some of our other jobs take, so happy to land this now.

[joostjager]

Distribution looks much better now:

test	iterations	time
base32_target.rs	2500002	278
bech32_parse_target.rs	2500002	58
bolt11_deser_target.rs	2500002	45
chanmon_consistency_target.rs	5002	852
chanmon_deser_target.r	2500002	55
fromstr_to_netaddress_target.r	2500002	45
full_stack_target.rs	50002	554
indexedmap_target.rs	500002	394
invoice_deser_target.r	2500002	51
invoice_request_deser_target.r	2500002	50
msg_accept_channel_target.rs	2500002	181
msg_accept_channel_v2_target.r	2500002	258
msg_announcement_signatures_target.rs	2500002	53
msg_channel_announcement_target.rs	2500002	73
msg_channel_details_target.rs	2500002	47
msg_channel_ready_target.r	2500002	49
msg_channel_reestablish_target.rs	2500002	52
msg_channel_update_target.rs	2500002	54
msg_closing_signed_target.rs	2500002	50
msg_commitment_signed_target.r	2500002	55
msg_decoded_onion_error_packet_target.rs	2500002	50
msg_error_message_target.r	2500002	51
msg_funding_created_target.rs	2500002	52
msg_funding_signed_target.rs	2500002	50
msg_gossip_timestamp_filter_target.rs	2500002	47
msg_init_target.rs	2500002	48
msg_node_announcement_target.r	2500002	60
msg_open_channel_target.rs	2500002	71
msg_open_channel_v2_target.rs	2500002	66
msg_ping_target.rs	2500002	48
msg_pong_target.rs	2500002	48
msg_query_channel_range_target.rs	2500002	47
msg_query_short_channel_ids_target.rs	2500002	52
msg_reply_channel_range_target.rs	2500002	48
msg_reply_short_channel_ids_end_target.rs	2500002	47
msg_revoke_and_ack_target.rs	2500002	50
msg_shutdown_target.rs	2500002	47
msg_splice_ack_target.rs	2500002	50
msg_splice_init_target.rs	2500002	50
msg_splice_locked_target.r	2500002	48
msg_stfu_target.rs	2500002	47
msg_tx_abort_target.rs	2500002	48
msg_tx_ack_rbf_target.rs	2500002	47
msg_tx_add_input_target.rs	2500002	73
msg_tx_add_output_target.r	2500002	48
msg_tx_complete_target.rs	2500002	47
msg_tx_init_rbf_target.rs	2500002	48
msg_tx_remove_input_target.rs	2500002	47
msg_tx_remove_output_target.rs	2500002	48
msg_tx_signatures_target.r	2500002	100
msg_update_add_htlc_target.rs	2500002	99
msg_update_fail_htlc_target.rs	2500002	52
msg_update_fail_malformed_htlc_target.rs	2500002	49
msg_update_fee_target.rs	2500002	48
msg_update_fulfill_htlc_target.rs	2500002	49
offer_deser_target.rs	2500002	48
onion_hop_data_target.rs	2500002	49
onion_message_target.r	2500002	64
peer_crypt_target.rs	2500002	108
process_network_graph_target.r	50002	568
process_onion_failure_target.r	2500002	165
refund_deser_target.rs	2500002	49
router_target.rs	50002	606
zbase32_target.rs	2500002	119

[joostjager]

Why these two additions?

[TheBlueMatt]

Because RUSTFLAGS aren't applied to doctests, so we have to restrict tests to only run library and binary tests.

[joostjager]

Could add comments when making changes like this. Make life easier for future devs.

[joostjager]

I wanted to verify that using the hardcoded seed indeed increases coverage, but it seems that hongfuzz reports in the log always

branch_coverage_percent:0

Not sure why that is? Otherwise it would be easy to compare with and without this hardcoded seed.

Or maybe there is another way to verify that it works indeed?

[TheBlueMatt]

The fact that the full_stack_target got much slower is pretty good evidence :)

[joostjager]

Okay, indirect evidence. But why is hongfuzz reporting 0?

[joostjager]

Is it not using coverage to direct fuzzing?

[TheBlueMatt]

Yea, that I don't know, it certainly does locally and there's nothing different locally vs in CI. Also CI does turn up (fairly shallow) fuzz bugs as well, so I'd be surprised to learn it was able to do that with just general patterns and no instrumentation...

[joostjager]

Does it show you branch_coverage_percent as non-zero locally?

[TheBlueMatt]

I'm not 100% sure about branch coverage specifically, but what I get from a running instance is: Coverage : edge: 16,485/568,547 [2%] pc: 738 cmp: 794,641. I assume "cmp" is the branch coverage.

[TheBlueMatt]

Went ahead and squashed the addition of a comment in the fuzz script.

[joostjager]

Great to have a more reasonable run time for the fuzzer!

Still interested to find out what's going on with that coverage indicator at zero in CI.

[TheBlueMatt]

I think github has slowed down the runners so now our fuzz tests are timing out. Here we just reduce iteration count a bit.

[ldk-reviews-bot]

I've assigned @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

ACK, seems we hit yet another failing case in the router target now: #3692

[joostjager]

Any specific reasoning behind the 10x, 10x and 2x reductions?

[TheBlueMatt]

No. I probably could have benchmarked, but in general there's not a lot of value in full_stack_target in CI because its just too dense to make any real progress, and similar for chanmon_consistency_target.

[joostjager]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Maybe I am not interpreting correctly though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

base32_target.rs
Summary iterations:1000002 time:198 speed:5050 crashes_count:0 timeout_count:0 new_units_added:449 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bech32_parse_target.rs
Summary iterations:1000002 time:22 speed:45454 crashes_count:0 timeout_count:0 new_units_added:897 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bolt11_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:211 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
chanmon_consistency_target.rs
Summary iterations:100002 time:9769 speed:10 crashes_count:0 timeout_count:87 new_units_added:3435 slowest_unit_ms:1139 guard_nb:566357 branch_coverage_percent:2 peak_rss_mb:42
chanmon_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:963 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
fromstr_to_netaddress_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:160 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
full_stack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:797 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
indexedmap_target.rs
Summary iterations:1000002 time:648 speed:1543 crashes_count:0 timeout_count:0 new_units_added:698 slowest_unit_ms:19 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_deser_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:2955 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_request_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:2244 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_target.rs
Summary iterations:1000002 time:23 speed:43478 crashes_count:0 timeout_count:0 new_units_added:1115 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_v2_target.rs
Summary iterations:1000002 time:246 speed:4065 crashes_count:0 timeout_count:0 new_units_added:1031 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_announcement_signatures_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_announcement_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:413 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_details_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:993 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_ready_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:374 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_reestablish_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:434 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_update_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:210 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_closing_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:380 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_commitment_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:697 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_decoded_onion_error_packet_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:178 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_error_message_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_created_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:342 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:304 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_gossip_timestamp_filter_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:294 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_init_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:1006 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_node_announcement_target.rs
Summary iterations:1000002 time:21 speed:47619 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_target.rs
Summary iterations:1000002 time:35 speed:28571 crashes_count:0 timeout_count:0 new_units_added:1163 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_v2_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1289 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_ping_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:91 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_pong_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:83 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:334 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_short_channel_ids_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:158 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_short_channel_ids_end_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:337 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_revoke_and_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:360 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_shutdown_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:387 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_init_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:401 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_locked_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_stfu_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_abort_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:403 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_ack_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:366 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_input_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1270 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_output_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:390 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_complete_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:285 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_init_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:393 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_input_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:297 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_output_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_signatures_target.rs
Summary iterations:1000002 time:52 speed:19230 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_add_htlc_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:430 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:480 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_malformed_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:321 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fee_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:308 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fulfill_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:309 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
offer_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:1333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_hop_data_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:981 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_message_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:395 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
peer_crypt_target.rs
Summary iterations:1000002 time:27 speed:37037 crashes_count:0 timeout_count:0 new_units_added:315 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
process_network_graph_target.rs
Summary iterations:735922 time:8434 speed:87 crashes_count:0 timeout_count:0 new_units_added:30 slowest_unit_ms:40 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:161

[TheBlueMatt]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

Ah, thanks for doing that. Yea, I was going on the results when I fuzz with real corpuses, where full_stack_target can get quite slow. But, given we currently don't initialize our CI fuzzers with a real corpus so it really doesn't get into it too much. I did go ahead and update the CI fuzzer to use the hard-coded full_stack_target seeds, but it still doesn't spend much time in complicated paths.

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Huh, interesting, I went ahead and slowed this one down though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

Yep!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.92%. Comparing base (c4d23bc) to head (11dde05).
Report is 38 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #3691 +/- ##
==========================================
+ Coverage 89.05% 89.92% +0.86% 
==========================================
Files 155 156 +1 Lines 122019 129760 +7741 Branches 122019 129760 +7741 ==========================================
+ Hits 108666 116683 +8017 + Misses 10695 10441 -254 + Partials 2658 2636 -22 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tnull]

LGTM

Fuzz failure is #3708

[joostjager]

Great that fuzzing again found an issue. This is awesome.

Before merging this, I think we still want to see a successful fuzz run well within the timeout?

[TheBlueMatt]

Yea, happy to wait until we can at least run a fuzzing run and check that all the timings make sense.

[TheBlueMatt]

It passed, but taking five hours for CI is excessive, so I reduced it further.

[tnull]

It passed, but taking five hours for CI is excessive, so I reduced it further.

Now finished in 273min ~ 4.55 hours. Is this acceptable?

[TheBlueMatt]

Mmm, that's still pretty aggressive. Divided a few jobs by two so hopefully this time's the charm.

[TheBlueMatt]

Okay, its now "only" 2h, which I think is kinda reasonable, or at least its in line with what some of our other jobs take, so happy to land this now.

[joostjager]

Distribution looks much better now:

test	iterations	time
base32_target.rs	2500002	278
bech32_parse_target.rs	2500002	58
bolt11_deser_target.rs	2500002	45
chanmon_consistency_target.rs	5002	852
chanmon_deser_target.r	2500002	55
fromstr_to_netaddress_target.r	2500002	45
full_stack_target.rs	50002	554
indexedmap_target.rs	500002	394
invoice_deser_target.r	2500002	51
invoice_request_deser_target.r	2500002	50
msg_accept_channel_target.rs	2500002	181
msg_accept_channel_v2_target.r	2500002	258
msg_announcement_signatures_target.rs	2500002	53
msg_channel_announcement_target.rs	2500002	73
msg_channel_details_target.rs	2500002	47
msg_channel_ready_target.r	2500002	49
msg_channel_reestablish_target.rs	2500002	52
msg_channel_update_target.rs	2500002	54
msg_closing_signed_target.rs	2500002	50
msg_commitment_signed_target.r	2500002	55
msg_decoded_onion_error_packet_target.rs	2500002	50
msg_error_message_target.r	2500002	51
msg_funding_created_target.rs	2500002	52
msg_funding_signed_target.rs	2500002	50
msg_gossip_timestamp_filter_target.rs	2500002	47
msg_init_target.rs	2500002	48
msg_node_announcement_target.r	2500002	60
msg_open_channel_target.rs	2500002	71
msg_open_channel_v2_target.rs	2500002	66
msg_ping_target.rs	2500002	48
msg_pong_target.rs	2500002	48
msg_query_channel_range_target.rs	2500002	47
msg_query_short_channel_ids_target.rs	2500002	52
msg_reply_channel_range_target.rs	2500002	48
msg_reply_short_channel_ids_end_target.rs	2500002	47
msg_revoke_and_ack_target.rs	2500002	50
msg_shutdown_target.rs	2500002	47
msg_splice_ack_target.rs	2500002	50
msg_splice_init_target.rs	2500002	50
msg_splice_locked_target.r	2500002	48
msg_stfu_target.rs	2500002	47
msg_tx_abort_target.rs	2500002	48
msg_tx_ack_rbf_target.rs	2500002	47
msg_tx_add_input_target.rs	2500002	73
msg_tx_add_output_target.r	2500002	48
msg_tx_complete_target.rs	2500002	47
msg_tx_init_rbf_target.rs	2500002	48
msg_tx_remove_input_target.rs	2500002	47
msg_tx_remove_output_target.rs	2500002	48
msg_tx_signatures_target.r	2500002	100
msg_update_add_htlc_target.rs	2500002	99
msg_update_fail_htlc_target.rs	2500002	52
msg_update_fail_malformed_htlc_target.rs	2500002	49
msg_update_fee_target.rs	2500002	48
msg_update_fulfill_htlc_target.rs	2500002	49
offer_deser_target.rs	2500002	48
onion_hop_data_target.rs	2500002	49
onion_message_target.r	2500002	64
peer_crypt_target.rs	2500002	108
process_network_graph_target.r	50002	568
process_onion_failure_target.r	2500002	165
refund_deser_target.rs	2500002	49
router_target.rs	50002	606
zbase32_target.rs	2500002	119

[joostjager]

Why these two additions?

[TheBlueMatt]

Because RUSTFLAGS aren't applied to doctests, so we have to restrict tests to only run library and binary tests.

[joostjager]

Could add comments when making changes like this. Make life easier for future devs.

[joostjager]

I wanted to verify that using the hardcoded seed indeed increases coverage, but it seems that hongfuzz reports in the log always

branch_coverage_percent:0

Not sure why that is? Otherwise it would be easy to compare with and without this hardcoded seed.

Or maybe there is another way to verify that it works indeed?

[TheBlueMatt]

The fact that the full_stack_target got much slower is pretty good evidence :)

[joostjager]

Okay, indirect evidence. But why is hongfuzz reporting 0?

[joostjager]

Is it not using coverage to direct fuzzing?

[TheBlueMatt]

Yea, that I don't know, it certainly does locally and there's nothing different locally vs in CI. Also CI does turn up (fairly shallow) fuzz bugs as well, so I'd be surprised to learn it was able to do that with just general patterns and no instrumentation...

[joostjager]

Does it show you branch_coverage_percent as non-zero locally?

[TheBlueMatt]

I'm not 100% sure about branch coverage specifically, but what I get from a running instance is: Coverage : edge: 16,485/568,547 [2%] pc: 738 cmp: 794,641. I assume "cmp" is the branch coverage.

[TheBlueMatt]

Went ahead and squashed the addition of a comment in the fuzz script.

[joostjager]

Great to have a more reasonable run time for the fuzzer!

Still interested to find out what's going on with that coverage indicator at zero in CI.

[TheBlueMatt]

I think github has slowed down the runners so now our fuzz tests are timing out. Here we just reduce iteration count a bit.

[ldk-reviews-bot]

I've assigned @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

ACK, seems we hit yet another failing case in the router target now: #3692

[joostjager]

Any specific reasoning behind the 10x, 10x and 2x reductions?

[TheBlueMatt]

No. I probably could have benchmarked, but in general there's not a lot of value in full_stack_target in CI because its just too dense to make any real progress, and similar for chanmon_consistency_target.

[joostjager]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Maybe I am not interpreting correctly though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

base32_target.rs
Summary iterations:1000002 time:198 speed:5050 crashes_count:0 timeout_count:0 new_units_added:449 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bech32_parse_target.rs
Summary iterations:1000002 time:22 speed:45454 crashes_count:0 timeout_count:0 new_units_added:897 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bolt11_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:211 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
chanmon_consistency_target.rs
Summary iterations:100002 time:9769 speed:10 crashes_count:0 timeout_count:87 new_units_added:3435 slowest_unit_ms:1139 guard_nb:566357 branch_coverage_percent:2 peak_rss_mb:42
chanmon_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:963 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
fromstr_to_netaddress_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:160 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
full_stack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:797 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
indexedmap_target.rs
Summary iterations:1000002 time:648 speed:1543 crashes_count:0 timeout_count:0 new_units_added:698 slowest_unit_ms:19 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_deser_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:2955 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_request_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:2244 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_target.rs
Summary iterations:1000002 time:23 speed:43478 crashes_count:0 timeout_count:0 new_units_added:1115 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_v2_target.rs
Summary iterations:1000002 time:246 speed:4065 crashes_count:0 timeout_count:0 new_units_added:1031 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_announcement_signatures_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_announcement_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:413 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_details_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:993 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_ready_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:374 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_reestablish_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:434 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_update_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:210 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_closing_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:380 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_commitment_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:697 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_decoded_onion_error_packet_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:178 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_error_message_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_created_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:342 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:304 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_gossip_timestamp_filter_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:294 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_init_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:1006 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_node_announcement_target.rs
Summary iterations:1000002 time:21 speed:47619 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_target.rs
Summary iterations:1000002 time:35 speed:28571 crashes_count:0 timeout_count:0 new_units_added:1163 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_v2_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1289 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_ping_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:91 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_pong_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:83 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:334 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_short_channel_ids_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:158 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_short_channel_ids_end_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:337 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_revoke_and_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:360 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_shutdown_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:387 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_init_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:401 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_locked_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_stfu_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_abort_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:403 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_ack_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:366 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_input_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1270 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_output_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:390 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_complete_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:285 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_init_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:393 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_input_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:297 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_output_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_signatures_target.rs
Summary iterations:1000002 time:52 speed:19230 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_add_htlc_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:430 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:480 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_malformed_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:321 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fee_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:308 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fulfill_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:309 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
offer_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:1333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_hop_data_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:981 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_message_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:395 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
peer_crypt_target.rs
Summary iterations:1000002 time:27 speed:37037 crashes_count:0 timeout_count:0 new_units_added:315 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
process_network_graph_target.rs
Summary iterations:735922 time:8434 speed:87 crashes_count:0 timeout_count:0 new_units_added:30 slowest_unit_ms:40 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:161

[TheBlueMatt]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

Ah, thanks for doing that. Yea, I was going on the results when I fuzz with real corpuses, where full_stack_target can get quite slow. But, given we currently don't initialize our CI fuzzers with a real corpus so it really doesn't get into it too much. I did go ahead and update the CI fuzzer to use the hard-coded full_stack_target seeds, but it still doesn't spend much time in complicated paths.

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Huh, interesting, I went ahead and slowed this one down though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

Yep!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.92%. Comparing base (c4d23bc) to head (11dde05).
Report is 38 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #3691 +/- ##
==========================================
+ Coverage 89.05% 89.92% +0.86% 
==========================================
Files 155 156 +1 Lines 122019 129760 +7741 Branches 122019 129760 +7741 ==========================================
+ Hits 108666 116683 +8017 + Misses 10695 10441 -254 + Partials 2658 2636 -22 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tnull]

LGTM

Fuzz failure is #3708

[joostjager]

Great that fuzzing again found an issue. This is awesome.

Before merging this, I think we still want to see a successful fuzz run well within the timeout?

[TheBlueMatt]

Yea, happy to wait until we can at least run a fuzzing run and check that all the timings make sense.

[TheBlueMatt]

It passed, but taking five hours for CI is excessive, so I reduced it further.

[tnull]

It passed, but taking five hours for CI is excessive, so I reduced it further.

Now finished in 273min ~ 4.55 hours. Is this acceptable?

[TheBlueMatt]

Mmm, that's still pretty aggressive. Divided a few jobs by two so hopefully this time's the charm.

[TheBlueMatt]

Okay, its now "only" 2h, which I think is kinda reasonable, or at least its in line with what some of our other jobs take, so happy to land this now.

[joostjager]

Distribution looks much better now:

test	iterations	time
base32_target.rs	2500002	278
bech32_parse_target.rs	2500002	58
bolt11_deser_target.rs	2500002	45
chanmon_consistency_target.rs	5002	852
chanmon_deser_target.r	2500002	55
fromstr_to_netaddress_target.r	2500002	45
full_stack_target.rs	50002	554
indexedmap_target.rs	500002	394
invoice_deser_target.r	2500002	51
invoice_request_deser_target.r	2500002	50
msg_accept_channel_target.rs	2500002	181
msg_accept_channel_v2_target.r	2500002	258
msg_announcement_signatures_target.rs	2500002	53
msg_channel_announcement_target.rs	2500002	73
msg_channel_details_target.rs	2500002	47
msg_channel_ready_target.r	2500002	49
msg_channel_reestablish_target.rs	2500002	52
msg_channel_update_target.rs	2500002	54
msg_closing_signed_target.rs	2500002	50
msg_commitment_signed_target.r	2500002	55
msg_decoded_onion_error_packet_target.rs	2500002	50
msg_error_message_target.r	2500002	51
msg_funding_created_target.rs	2500002	52
msg_funding_signed_target.rs	2500002	50
msg_gossip_timestamp_filter_target.rs	2500002	47
msg_init_target.rs	2500002	48
msg_node_announcement_target.r	2500002	60
msg_open_channel_target.rs	2500002	71
msg_open_channel_v2_target.rs	2500002	66
msg_ping_target.rs	2500002	48
msg_pong_target.rs	2500002	48
msg_query_channel_range_target.rs	2500002	47
msg_query_short_channel_ids_target.rs	2500002	52
msg_reply_channel_range_target.rs	2500002	48
msg_reply_short_channel_ids_end_target.rs	2500002	47
msg_revoke_and_ack_target.rs	2500002	50
msg_shutdown_target.rs	2500002	47
msg_splice_ack_target.rs	2500002	50
msg_splice_init_target.rs	2500002	50
msg_splice_locked_target.r	2500002	48
msg_stfu_target.rs	2500002	47
msg_tx_abort_target.rs	2500002	48
msg_tx_ack_rbf_target.rs	2500002	47
msg_tx_add_input_target.rs	2500002	73
msg_tx_add_output_target.r	2500002	48
msg_tx_complete_target.rs	2500002	47
msg_tx_init_rbf_target.rs	2500002	48
msg_tx_remove_input_target.rs	2500002	47
msg_tx_remove_output_target.rs	2500002	48
msg_tx_signatures_target.r	2500002	100
msg_update_add_htlc_target.rs	2500002	99
msg_update_fail_htlc_target.rs	2500002	52
msg_update_fail_malformed_htlc_target.rs	2500002	49
msg_update_fee_target.rs	2500002	48
msg_update_fulfill_htlc_target.rs	2500002	49
offer_deser_target.rs	2500002	48
onion_hop_data_target.rs	2500002	49
onion_message_target.r	2500002	64
peer_crypt_target.rs	2500002	108
process_network_graph_target.r	50002	568
process_onion_failure_target.r	2500002	165
refund_deser_target.rs	2500002	49
router_target.rs	50002	606
zbase32_target.rs	2500002	119

[joostjager]

Why these two additions?

[TheBlueMatt]

Because RUSTFLAGS aren't applied to doctests, so we have to restrict tests to only run library and binary tests.

[joostjager]

Could add comments when making changes like this. Make life easier for future devs.

[joostjager]

I wanted to verify that using the hardcoded seed indeed increases coverage, but it seems that hongfuzz reports in the log always

branch_coverage_percent:0

Not sure why that is? Otherwise it would be easy to compare with and without this hardcoded seed.

Or maybe there is another way to verify that it works indeed?

[TheBlueMatt]

The fact that the full_stack_target got much slower is pretty good evidence :)

[joostjager]

Okay, indirect evidence. But why is hongfuzz reporting 0?

[joostjager]

Is it not using coverage to direct fuzzing?

[TheBlueMatt]

Yea, that I don't know, it certainly does locally and there's nothing different locally vs in CI. Also CI does turn up (fairly shallow) fuzz bugs as well, so I'd be surprised to learn it was able to do that with just general patterns and no instrumentation...

[joostjager]

Does it show you branch_coverage_percent as non-zero locally?

[TheBlueMatt]

I'm not 100% sure about branch coverage specifically, but what I get from a running instance is: Coverage : edge: 16,485/568,547 [2%] pc: 738 cmp: 794,641. I assume "cmp" is the branch coverage.

[TheBlueMatt]

Went ahead and squashed the addition of a comment in the fuzz script.

[joostjager]

Great to have a more reasonable run time for the fuzzer!

Still interested to find out what's going on with that coverage indicator at zero in CI.

[TheBlueMatt]

I think github has slowed down the runners so now our fuzz tests are timing out. Here we just reduce iteration count a bit.

[ldk-reviews-bot]

I've assigned @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

ACK, seems we hit yet another failing case in the router target now: #3692

[joostjager]

Any specific reasoning behind the 10x, 10x and 2x reductions?

[TheBlueMatt]

No. I probably could have benchmarked, but in general there's not a lot of value in full_stack_target in CI because its just too dense to make any real progress, and similar for chanmon_consistency_target.

[joostjager]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Maybe I am not interpreting correctly though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

base32_target.rs
Summary iterations:1000002 time:198 speed:5050 crashes_count:0 timeout_count:0 new_units_added:449 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bech32_parse_target.rs
Summary iterations:1000002 time:22 speed:45454 crashes_count:0 timeout_count:0 new_units_added:897 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bolt11_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:211 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
chanmon_consistency_target.rs
Summary iterations:100002 time:9769 speed:10 crashes_count:0 timeout_count:87 new_units_added:3435 slowest_unit_ms:1139 guard_nb:566357 branch_coverage_percent:2 peak_rss_mb:42
chanmon_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:963 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
fromstr_to_netaddress_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:160 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
full_stack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:797 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
indexedmap_target.rs
Summary iterations:1000002 time:648 speed:1543 crashes_count:0 timeout_count:0 new_units_added:698 slowest_unit_ms:19 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_deser_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:2955 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_request_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:2244 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_target.rs
Summary iterations:1000002 time:23 speed:43478 crashes_count:0 timeout_count:0 new_units_added:1115 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_v2_target.rs
Summary iterations:1000002 time:246 speed:4065 crashes_count:0 timeout_count:0 new_units_added:1031 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_announcement_signatures_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_announcement_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:413 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_details_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:993 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_ready_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:374 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_reestablish_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:434 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_update_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:210 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_closing_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:380 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_commitment_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:697 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_decoded_onion_error_packet_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:178 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_error_message_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_created_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:342 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:304 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_gossip_timestamp_filter_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:294 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_init_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:1006 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_node_announcement_target.rs
Summary iterations:1000002 time:21 speed:47619 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_target.rs
Summary iterations:1000002 time:35 speed:28571 crashes_count:0 timeout_count:0 new_units_added:1163 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_v2_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1289 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_ping_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:91 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_pong_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:83 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:334 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_short_channel_ids_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:158 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_short_channel_ids_end_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:337 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_revoke_and_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:360 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_shutdown_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:387 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_init_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:401 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_locked_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_stfu_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_abort_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:403 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_ack_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:366 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_input_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1270 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_output_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:390 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_complete_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:285 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_init_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:393 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_input_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:297 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_output_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_signatures_target.rs
Summary iterations:1000002 time:52 speed:19230 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_add_htlc_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:430 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:480 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_malformed_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:321 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fee_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:308 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fulfill_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:309 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
offer_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:1333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_hop_data_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:981 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_message_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:395 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
peer_crypt_target.rs
Summary iterations:1000002 time:27 speed:37037 crashes_count:0 timeout_count:0 new_units_added:315 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
process_network_graph_target.rs
Summary iterations:735922 time:8434 speed:87 crashes_count:0 timeout_count:0 new_units_added:30 slowest_unit_ms:40 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:161

[TheBlueMatt]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

Ah, thanks for doing that. Yea, I was going on the results when I fuzz with real corpuses, where full_stack_target can get quite slow. But, given we currently don't initialize our CI fuzzers with a real corpus so it really doesn't get into it too much. I did go ahead and update the CI fuzzer to use the hard-coded full_stack_target seeds, but it still doesn't spend much time in complicated paths.

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Huh, interesting, I went ahead and slowed this one down though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

Yep!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.92%. Comparing base (c4d23bc) to head (11dde05).
Report is 38 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #3691 +/- ##
==========================================
+ Coverage 89.05% 89.92% +0.86% 
==========================================
Files 155 156 +1 Lines 122019 129760 +7741 Branches 122019 129760 +7741 ==========================================
+ Hits 108666 116683 +8017 + Misses 10695 10441 -254 + Partials 2658 2636 -22 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tnull]

LGTM

Fuzz failure is #3708

[joostjager]

Great that fuzzing again found an issue. This is awesome.

Before merging this, I think we still want to see a successful fuzz run well within the timeout?

[TheBlueMatt]

Yea, happy to wait until we can at least run a fuzzing run and check that all the timings make sense.

[TheBlueMatt]

It passed, but taking five hours for CI is excessive, so I reduced it further.

[tnull]

It passed, but taking five hours for CI is excessive, so I reduced it further.

Now finished in 273min ~ 4.55 hours. Is this acceptable?

[TheBlueMatt]

Mmm, that's still pretty aggressive. Divided a few jobs by two so hopefully this time's the charm.

[TheBlueMatt]

Okay, its now "only" 2h, which I think is kinda reasonable, or at least its in line with what some of our other jobs take, so happy to land this now.

[joostjager]

Distribution looks much better now:

test	iterations	time
base32_target.rs	2500002	278
bech32_parse_target.rs	2500002	58
bolt11_deser_target.rs	2500002	45
chanmon_consistency_target.rs	5002	852
chanmon_deser_target.r	2500002	55
fromstr_to_netaddress_target.r	2500002	45
full_stack_target.rs	50002	554
indexedmap_target.rs	500002	394
invoice_deser_target.r	2500002	51
invoice_request_deser_target.r	2500002	50
msg_accept_channel_target.rs	2500002	181
msg_accept_channel_v2_target.r	2500002	258
msg_announcement_signatures_target.rs	2500002	53
msg_channel_announcement_target.rs	2500002	73
msg_channel_details_target.rs	2500002	47
msg_channel_ready_target.r	2500002	49
msg_channel_reestablish_target.rs	2500002	52
msg_channel_update_target.rs	2500002	54
msg_closing_signed_target.rs	2500002	50
msg_commitment_signed_target.r	2500002	55
msg_decoded_onion_error_packet_target.rs	2500002	50
msg_error_message_target.r	2500002	51
msg_funding_created_target.rs	2500002	52
msg_funding_signed_target.rs	2500002	50
msg_gossip_timestamp_filter_target.rs	2500002	47
msg_init_target.rs	2500002	48
msg_node_announcement_target.r	2500002	60
msg_open_channel_target.rs	2500002	71
msg_open_channel_v2_target.rs	2500002	66
msg_ping_target.rs	2500002	48
msg_pong_target.rs	2500002	48
msg_query_channel_range_target.rs	2500002	47
msg_query_short_channel_ids_target.rs	2500002	52
msg_reply_channel_range_target.rs	2500002	48
msg_reply_short_channel_ids_end_target.rs	2500002	47
msg_revoke_and_ack_target.rs	2500002	50
msg_shutdown_target.rs	2500002	47
msg_splice_ack_target.rs	2500002	50
msg_splice_init_target.rs	2500002	50
msg_splice_locked_target.r	2500002	48
msg_stfu_target.rs	2500002	47
msg_tx_abort_target.rs	2500002	48
msg_tx_ack_rbf_target.rs	2500002	47
msg_tx_add_input_target.rs	2500002	73
msg_tx_add_output_target.r	2500002	48
msg_tx_complete_target.rs	2500002	47
msg_tx_init_rbf_target.rs	2500002	48
msg_tx_remove_input_target.rs	2500002	47
msg_tx_remove_output_target.rs	2500002	48
msg_tx_signatures_target.r	2500002	100
msg_update_add_htlc_target.rs	2500002	99
msg_update_fail_htlc_target.rs	2500002	52
msg_update_fail_malformed_htlc_target.rs	2500002	49
msg_update_fee_target.rs	2500002	48
msg_update_fulfill_htlc_target.rs	2500002	49
offer_deser_target.rs	2500002	48
onion_hop_data_target.rs	2500002	49
onion_message_target.r	2500002	64
peer_crypt_target.rs	2500002	108
process_network_graph_target.r	50002	568
process_onion_failure_target.r	2500002	165
refund_deser_target.rs	2500002	49
router_target.rs	50002	606
zbase32_target.rs	2500002	119

[joostjager]

Why these two additions?

[TheBlueMatt]

Because RUSTFLAGS aren't applied to doctests, so we have to restrict tests to only run library and binary tests.

[joostjager]

Could add comments when making changes like this. Make life easier for future devs.

[joostjager]

I wanted to verify that using the hardcoded seed indeed increases coverage, but it seems that hongfuzz reports in the log always

branch_coverage_percent:0

Not sure why that is? Otherwise it would be easy to compare with and without this hardcoded seed.

Or maybe there is another way to verify that it works indeed?

[TheBlueMatt]

The fact that the full_stack_target got much slower is pretty good evidence :)

[joostjager]

Okay, indirect evidence. But why is hongfuzz reporting 0?

[joostjager]

Is it not using coverage to direct fuzzing?

[TheBlueMatt]

Yea, that I don't know, it certainly does locally and there's nothing different locally vs in CI. Also CI does turn up (fairly shallow) fuzz bugs as well, so I'd be surprised to learn it was able to do that with just general patterns and no instrumentation...

[joostjager]

Does it show you branch_coverage_percent as non-zero locally?

[TheBlueMatt]

I'm not 100% sure about branch coverage specifically, but what I get from a running instance is: Coverage : edge: 16,485/568,547 [2%] pc: 738 cmp: 794,641. I assume "cmp" is the branch coverage.

[TheBlueMatt]

Went ahead and squashed the addition of a comment in the fuzz script.

[joostjager]

Great to have a more reasonable run time for the fuzzer!

Still interested to find out what's going on with that coverage indicator at zero in CI.

[TheBlueMatt]

I think github has slowed down the runners so now our fuzz tests are timing out. Here we just reduce iteration count a bit.

[ldk-reviews-bot]

I've assigned @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

ACK, seems we hit yet another failing case in the router target now: #3692

[joostjager]

Any specific reasoning behind the 10x, 10x and 2x reductions?

[TheBlueMatt]

No. I probably could have benchmarked, but in general there's not a lot of value in full_stack_target in CI because its just too dense to make any real progress, and similar for chanmon_consistency_target.

[joostjager]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Maybe I am not interpreting correctly though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

base32_target.rs
Summary iterations:1000002 time:198 speed:5050 crashes_count:0 timeout_count:0 new_units_added:449 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bech32_parse_target.rs
Summary iterations:1000002 time:22 speed:45454 crashes_count:0 timeout_count:0 new_units_added:897 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bolt11_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:211 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
chanmon_consistency_target.rs
Summary iterations:100002 time:9769 speed:10 crashes_count:0 timeout_count:87 new_units_added:3435 slowest_unit_ms:1139 guard_nb:566357 branch_coverage_percent:2 peak_rss_mb:42
chanmon_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:963 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
fromstr_to_netaddress_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:160 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
full_stack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:797 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
indexedmap_target.rs
Summary iterations:1000002 time:648 speed:1543 crashes_count:0 timeout_count:0 new_units_added:698 slowest_unit_ms:19 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_deser_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:2955 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_request_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:2244 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_target.rs
Summary iterations:1000002 time:23 speed:43478 crashes_count:0 timeout_count:0 new_units_added:1115 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_v2_target.rs
Summary iterations:1000002 time:246 speed:4065 crashes_count:0 timeout_count:0 new_units_added:1031 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_announcement_signatures_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_announcement_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:413 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_details_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:993 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_ready_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:374 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_reestablish_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:434 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_update_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:210 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_closing_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:380 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_commitment_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:697 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_decoded_onion_error_packet_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:178 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_error_message_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_created_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:342 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:304 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_gossip_timestamp_filter_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:294 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_init_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:1006 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_node_announcement_target.rs
Summary iterations:1000002 time:21 speed:47619 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_target.rs
Summary iterations:1000002 time:35 speed:28571 crashes_count:0 timeout_count:0 new_units_added:1163 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_v2_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1289 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_ping_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:91 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_pong_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:83 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:334 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_short_channel_ids_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:158 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_short_channel_ids_end_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:337 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_revoke_and_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:360 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_shutdown_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:387 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_init_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:401 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_locked_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_stfu_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_abort_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:403 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_ack_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:366 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_input_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1270 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_output_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:390 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_complete_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:285 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_init_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:393 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_input_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:297 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_output_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_signatures_target.rs
Summary iterations:1000002 time:52 speed:19230 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_add_htlc_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:430 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:480 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_malformed_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:321 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fee_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:308 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fulfill_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:309 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
offer_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:1333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_hop_data_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:981 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_message_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:395 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
peer_crypt_target.rs
Summary iterations:1000002 time:27 speed:37037 crashes_count:0 timeout_count:0 new_units_added:315 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
process_network_graph_target.rs
Summary iterations:735922 time:8434 speed:87 crashes_count:0 timeout_count:0 new_units_added:30 slowest_unit_ms:40 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:161

[TheBlueMatt]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

Ah, thanks for doing that. Yea, I was going on the results when I fuzz with real corpuses, where full_stack_target can get quite slow. But, given we currently don't initialize our CI fuzzers with a real corpus so it really doesn't get into it too much. I did go ahead and update the CI fuzzer to use the hard-coded full_stack_target seeds, but it still doesn't spend much time in complicated paths.

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Huh, interesting, I went ahead and slowed this one down though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

Yep!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.92%. Comparing base (c4d23bc) to head (11dde05).
Report is 38 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #3691 +/- ##
==========================================
+ Coverage 89.05% 89.92% +0.86% 
==========================================
Files 155 156 +1 Lines 122019 129760 +7741 Branches 122019 129760 +7741 ==========================================
+ Hits 108666 116683 +8017 + Misses 10695 10441 -254 + Partials 2658 2636 -22 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tnull]

LGTM

Fuzz failure is #3708

[joostjager]

Great that fuzzing again found an issue. This is awesome.

Before merging this, I think we still want to see a successful fuzz run well within the timeout?

[TheBlueMatt]

Yea, happy to wait until we can at least run a fuzzing run and check that all the timings make sense.

[TheBlueMatt]

It passed, but taking five hours for CI is excessive, so I reduced it further.

[tnull]

It passed, but taking five hours for CI is excessive, so I reduced it further.

Now finished in 273min ~ 4.55 hours. Is this acceptable?

[TheBlueMatt]

Mmm, that's still pretty aggressive. Divided a few jobs by two so hopefully this time's the charm.

[TheBlueMatt]

Okay, its now "only" 2h, which I think is kinda reasonable, or at least its in line with what some of our other jobs take, so happy to land this now.

[joostjager]

Distribution looks much better now:

test	iterations	time
base32_target.rs	2500002	278
bech32_parse_target.rs	2500002	58
bolt11_deser_target.rs	2500002	45
chanmon_consistency_target.rs	5002	852
chanmon_deser_target.r	2500002	55
fromstr_to_netaddress_target.r	2500002	45
full_stack_target.rs	50002	554
indexedmap_target.rs	500002	394
invoice_deser_target.r	2500002	51
invoice_request_deser_target.r	2500002	50
msg_accept_channel_target.rs	2500002	181
msg_accept_channel_v2_target.r	2500002	258
msg_announcement_signatures_target.rs	2500002	53
msg_channel_announcement_target.rs	2500002	73
msg_channel_details_target.rs	2500002	47
msg_channel_ready_target.r	2500002	49
msg_channel_reestablish_target.rs	2500002	52
msg_channel_update_target.rs	2500002	54
msg_closing_signed_target.rs	2500002	50
msg_commitment_signed_target.r	2500002	55
msg_decoded_onion_error_packet_target.rs	2500002	50
msg_error_message_target.r	2500002	51
msg_funding_created_target.rs	2500002	52
msg_funding_signed_target.rs	2500002	50
msg_gossip_timestamp_filter_target.rs	2500002	47
msg_init_target.rs	2500002	48
msg_node_announcement_target.r	2500002	60
msg_open_channel_target.rs	2500002	71
msg_open_channel_v2_target.rs	2500002	66
msg_ping_target.rs	2500002	48
msg_pong_target.rs	2500002	48
msg_query_channel_range_target.rs	2500002	47
msg_query_short_channel_ids_target.rs	2500002	52
msg_reply_channel_range_target.rs	2500002	48
msg_reply_short_channel_ids_end_target.rs	2500002	47
msg_revoke_and_ack_target.rs	2500002	50
msg_shutdown_target.rs	2500002	47
msg_splice_ack_target.rs	2500002	50
msg_splice_init_target.rs	2500002	50
msg_splice_locked_target.r	2500002	48
msg_stfu_target.rs	2500002	47
msg_tx_abort_target.rs	2500002	48
msg_tx_ack_rbf_target.rs	2500002	47
msg_tx_add_input_target.rs	2500002	73
msg_tx_add_output_target.r	2500002	48
msg_tx_complete_target.rs	2500002	47
msg_tx_init_rbf_target.rs	2500002	48
msg_tx_remove_input_target.rs	2500002	47
msg_tx_remove_output_target.rs	2500002	48
msg_tx_signatures_target.r	2500002	100
msg_update_add_htlc_target.rs	2500002	99
msg_update_fail_htlc_target.rs	2500002	52
msg_update_fail_malformed_htlc_target.rs	2500002	49
msg_update_fee_target.rs	2500002	48
msg_update_fulfill_htlc_target.rs	2500002	49
offer_deser_target.rs	2500002	48
onion_hop_data_target.rs	2500002	49
onion_message_target.r	2500002	64
peer_crypt_target.rs	2500002	108
process_network_graph_target.r	50002	568
process_onion_failure_target.r	2500002	165
refund_deser_target.rs	2500002	49
router_target.rs	50002	606
zbase32_target.rs	2500002	119

[joostjager]

Why these two additions?

[TheBlueMatt]

Because RUSTFLAGS aren't applied to doctests, so we have to restrict tests to only run library and binary tests.

[joostjager]

Could add comments when making changes like this. Make life easier for future devs.

[joostjager]

I wanted to verify that using the hardcoded seed indeed increases coverage, but it seems that hongfuzz reports in the log always

branch_coverage_percent:0

Not sure why that is? Otherwise it would be easy to compare with and without this hardcoded seed.

Or maybe there is another way to verify that it works indeed?

[TheBlueMatt]

The fact that the full_stack_target got much slower is pretty good evidence :)

[joostjager]

Okay, indirect evidence. But why is hongfuzz reporting 0?

[joostjager]

Is it not using coverage to direct fuzzing?

[TheBlueMatt]

Yea, that I don't know, it certainly does locally and there's nothing different locally vs in CI. Also CI does turn up (fairly shallow) fuzz bugs as well, so I'd be surprised to learn it was able to do that with just general patterns and no instrumentation...

[joostjager]

Does it show you branch_coverage_percent as non-zero locally?

[TheBlueMatt]

I'm not 100% sure about branch coverage specifically, but what I get from a running instance is: Coverage : edge: 16,485/568,547 [2%] pc: 738 cmp: 794,641. I assume "cmp" is the branch coverage.

[TheBlueMatt]

Went ahead and squashed the addition of a comment in the fuzz script.

[joostjager]

Great to have a more reasonable run time for the fuzzer!

Still interested to find out what's going on with that coverage indicator at zero in CI.

[TheBlueMatt]

I think github has slowed down the runners so now our fuzz tests are timing out. Here we just reduce iteration count a bit.

[ldk-reviews-bot]

I've assigned @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

ACK, seems we hit yet another failing case in the router target now: #3692

[joostjager]

Any specific reasoning behind the 10x, 10x and 2x reductions?

[TheBlueMatt]

No. I probably could have benchmarked, but in general there's not a lot of value in full_stack_target in CI because its just too dense to make any real progress, and similar for chanmon_consistency_target.

[joostjager]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Maybe I am not interpreting correctly though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

base32_target.rs
Summary iterations:1000002 time:198 speed:5050 crashes_count:0 timeout_count:0 new_units_added:449 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bech32_parse_target.rs
Summary iterations:1000002 time:22 speed:45454 crashes_count:0 timeout_count:0 new_units_added:897 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bolt11_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:211 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
chanmon_consistency_target.rs
Summary iterations:100002 time:9769 speed:10 crashes_count:0 timeout_count:87 new_units_added:3435 slowest_unit_ms:1139 guard_nb:566357 branch_coverage_percent:2 peak_rss_mb:42
chanmon_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:963 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
fromstr_to_netaddress_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:160 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
full_stack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:797 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
indexedmap_target.rs
Summary iterations:1000002 time:648 speed:1543 crashes_count:0 timeout_count:0 new_units_added:698 slowest_unit_ms:19 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_deser_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:2955 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_request_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:2244 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_target.rs
Summary iterations:1000002 time:23 speed:43478 crashes_count:0 timeout_count:0 new_units_added:1115 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_v2_target.rs
Summary iterations:1000002 time:246 speed:4065 crashes_count:0 timeout_count:0 new_units_added:1031 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_announcement_signatures_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_announcement_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:413 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_details_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:993 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_ready_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:374 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_reestablish_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:434 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_update_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:210 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_closing_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:380 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_commitment_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:697 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_decoded_onion_error_packet_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:178 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_error_message_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_created_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:342 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:304 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_gossip_timestamp_filter_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:294 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_init_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:1006 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_node_announcement_target.rs
Summary iterations:1000002 time:21 speed:47619 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_target.rs
Summary iterations:1000002 time:35 speed:28571 crashes_count:0 timeout_count:0 new_units_added:1163 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_v2_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1289 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_ping_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:91 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_pong_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:83 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:334 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_short_channel_ids_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:158 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_short_channel_ids_end_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:337 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_revoke_and_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:360 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_shutdown_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:387 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_init_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:401 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_locked_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_stfu_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_abort_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:403 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_ack_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:366 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_input_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1270 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_output_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:390 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_complete_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:285 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_init_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:393 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_input_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:297 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_output_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_signatures_target.rs
Summary iterations:1000002 time:52 speed:19230 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_add_htlc_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:430 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:480 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_malformed_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:321 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fee_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:308 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fulfill_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:309 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
offer_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:1333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_hop_data_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:981 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_message_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:395 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
peer_crypt_target.rs
Summary iterations:1000002 time:27 speed:37037 crashes_count:0 timeout_count:0 new_units_added:315 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
process_network_graph_target.rs
Summary iterations:735922 time:8434 speed:87 crashes_count:0 timeout_count:0 new_units_added:30 slowest_unit_ms:40 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:161

[TheBlueMatt]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

Ah, thanks for doing that. Yea, I was going on the results when I fuzz with real corpuses, where full_stack_target can get quite slow. But, given we currently don't initialize our CI fuzzers with a real corpus so it really doesn't get into it too much. I did go ahead and update the CI fuzzer to use the hard-coded full_stack_target seeds, but it still doesn't spend much time in complicated paths.

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Huh, interesting, I went ahead and slowed this one down though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

Yep!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.92%. Comparing base (c4d23bc) to head (11dde05).
Report is 38 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #3691 +/- ##
==========================================
+ Coverage 89.05% 89.92% +0.86% 
==========================================
Files 155 156 +1 Lines 122019 129760 +7741 Branches 122019 129760 +7741 ==========================================
+ Hits 108666 116683 +8017 + Misses 10695 10441 -254 + Partials 2658 2636 -22 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tnull]

LGTM

Fuzz failure is #3708

[joostjager]

Great that fuzzing again found an issue. This is awesome.

Before merging this, I think we still want to see a successful fuzz run well within the timeout?

[TheBlueMatt]

Yea, happy to wait until we can at least run a fuzzing run and check that all the timings make sense.

[TheBlueMatt]

It passed, but taking five hours for CI is excessive, so I reduced it further.

[tnull]

It passed, but taking five hours for CI is excessive, so I reduced it further.

Now finished in 273min ~ 4.55 hours. Is this acceptable?

[TheBlueMatt]

Mmm, that's still pretty aggressive. Divided a few jobs by two so hopefully this time's the charm.

[TheBlueMatt]

Okay, its now "only" 2h, which I think is kinda reasonable, or at least its in line with what some of our other jobs take, so happy to land this now.

[joostjager]

Distribution looks much better now:

test	iterations	time
base32_target.rs	2500002	278
bech32_parse_target.rs	2500002	58
bolt11_deser_target.rs	2500002	45
chanmon_consistency_target.rs	5002	852
chanmon_deser_target.r	2500002	55
fromstr_to_netaddress_target.r	2500002	45
full_stack_target.rs	50002	554
indexedmap_target.rs	500002	394
invoice_deser_target.r	2500002	51
invoice_request_deser_target.r	2500002	50
msg_accept_channel_target.rs	2500002	181
msg_accept_channel_v2_target.r	2500002	258
msg_announcement_signatures_target.rs	2500002	53
msg_channel_announcement_target.rs	2500002	73
msg_channel_details_target.rs	2500002	47
msg_channel_ready_target.r	2500002	49
msg_channel_reestablish_target.rs	2500002	52
msg_channel_update_target.rs	2500002	54
msg_closing_signed_target.rs	2500002	50
msg_commitment_signed_target.r	2500002	55
msg_decoded_onion_error_packet_target.rs	2500002	50
msg_error_message_target.r	2500002	51
msg_funding_created_target.rs	2500002	52
msg_funding_signed_target.rs	2500002	50
msg_gossip_timestamp_filter_target.rs	2500002	47
msg_init_target.rs	2500002	48
msg_node_announcement_target.r	2500002	60
msg_open_channel_target.rs	2500002	71
msg_open_channel_v2_target.rs	2500002	66
msg_ping_target.rs	2500002	48
msg_pong_target.rs	2500002	48
msg_query_channel_range_target.rs	2500002	47
msg_query_short_channel_ids_target.rs	2500002	52
msg_reply_channel_range_target.rs	2500002	48
msg_reply_short_channel_ids_end_target.rs	2500002	47
msg_revoke_and_ack_target.rs	2500002	50
msg_shutdown_target.rs	2500002	47
msg_splice_ack_target.rs	2500002	50
msg_splice_init_target.rs	2500002	50
msg_splice_locked_target.r	2500002	48
msg_stfu_target.rs	2500002	47
msg_tx_abort_target.rs	2500002	48
msg_tx_ack_rbf_target.rs	2500002	47
msg_tx_add_input_target.rs	2500002	73
msg_tx_add_output_target.r	2500002	48
msg_tx_complete_target.rs	2500002	47
msg_tx_init_rbf_target.rs	2500002	48
msg_tx_remove_input_target.rs	2500002	47
msg_tx_remove_output_target.rs	2500002	48
msg_tx_signatures_target.r	2500002	100
msg_update_add_htlc_target.rs	2500002	99
msg_update_fail_htlc_target.rs	2500002	52
msg_update_fail_malformed_htlc_target.rs	2500002	49
msg_update_fee_target.rs	2500002	48
msg_update_fulfill_htlc_target.rs	2500002	49
offer_deser_target.rs	2500002	48
onion_hop_data_target.rs	2500002	49
onion_message_target.r	2500002	64
peer_crypt_target.rs	2500002	108
process_network_graph_target.r	50002	568
process_onion_failure_target.r	2500002	165
refund_deser_target.rs	2500002	49
router_target.rs	50002	606
zbase32_target.rs	2500002	119

[joostjager]

Why these two additions?

[TheBlueMatt]

Because RUSTFLAGS aren't applied to doctests, so we have to restrict tests to only run library and binary tests.

[joostjager]

Could add comments when making changes like this. Make life easier for future devs.

[joostjager]

I wanted to verify that using the hardcoded seed indeed increases coverage, but it seems that hongfuzz reports in the log always

branch_coverage_percent:0

Not sure why that is? Otherwise it would be easy to compare with and without this hardcoded seed.

Or maybe there is another way to verify that it works indeed?

[TheBlueMatt]

The fact that the full_stack_target got much slower is pretty good evidence :)

[joostjager]

Okay, indirect evidence. But why is hongfuzz reporting 0?

[joostjager]

Is it not using coverage to direct fuzzing?

[TheBlueMatt]

Yea, that I don't know, it certainly does locally and there's nothing different locally vs in CI. Also CI does turn up (fairly shallow) fuzz bugs as well, so I'd be surprised to learn it was able to do that with just general patterns and no instrumentation...

[joostjager]

Does it show you branch_coverage_percent as non-zero locally?

[TheBlueMatt]

I'm not 100% sure about branch coverage specifically, but what I get from a running instance is: Coverage : edge: 16,485/568,547 [2%] pc: 738 cmp: 794,641. I assume "cmp" is the branch coverage.

[TheBlueMatt]

Went ahead and squashed the addition of a comment in the fuzz script.

[joostjager]

Great to have a more reasonable run time for the fuzzer!

Still interested to find out what's going on with that coverage indicator at zero in CI.

[TheBlueMatt]

I think github has slowed down the runners so now our fuzz tests are timing out. Here we just reduce iteration count a bit.

[ldk-reviews-bot]

I've assigned @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

ACK, seems we hit yet another failing case in the router target now: #3692

[joostjager]

Any specific reasoning behind the 10x, 10x and 2x reductions?

[TheBlueMatt]

No. I probably could have benchmarked, but in general there's not a lot of value in full_stack_target in CI because its just too dense to make any real progress, and similar for chanmon_consistency_target.

[joostjager]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Maybe I am not interpreting correctly though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

base32_target.rs
Summary iterations:1000002 time:198 speed:5050 crashes_count:0 timeout_count:0 new_units_added:449 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bech32_parse_target.rs
Summary iterations:1000002 time:22 speed:45454 crashes_count:0 timeout_count:0 new_units_added:897 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
bolt11_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:211 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
chanmon_consistency_target.rs
Summary iterations:100002 time:9769 speed:10 crashes_count:0 timeout_count:87 new_units_added:3435 slowest_unit_ms:1139 guard_nb:566357 branch_coverage_percent:2 peak_rss_mb:42
chanmon_deser_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:963 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
fromstr_to_netaddress_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:160 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
full_stack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:797 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
indexedmap_target.rs
Summary iterations:1000002 time:648 speed:1543 crashes_count:0 timeout_count:0 new_units_added:698 slowest_unit_ms:19 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_deser_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:2955 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
invoice_request_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:2244 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_target.rs
Summary iterations:1000002 time:23 speed:43478 crashes_count:0 timeout_count:0 new_units_added:1115 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_accept_channel_v2_target.rs
Summary iterations:1000002 time:246 speed:4065 crashes_count:0 timeout_count:0 new_units_added:1031 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_announcement_signatures_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_announcement_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:413 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_details_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:993 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_ready_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:374 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_reestablish_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:434 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_channel_update_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:210 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_closing_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:380 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_commitment_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:697 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_decoded_onion_error_packet_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:178 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_error_message_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_created_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:342 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_funding_signed_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:304 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_gossip_timestamp_filter_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:294 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_init_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:1006 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_node_announcement_target.rs
Summary iterations:1000002 time:21 speed:47619 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_target.rs
Summary iterations:1000002 time:35 speed:28571 crashes_count:0 timeout_count:0 new_units_added:1163 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_open_channel_v2_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1289 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_ping_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:91 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_pong_target.rs
Summary iterations:1000002 time:20 speed:50000 crashes_count:0 timeout_count:0 new_units_added:83 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:334 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_query_short_channel_ids_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:108 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_channel_range_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:158 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_reply_short_channel_ids_end_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:337 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_revoke_and_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:360 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_shutdown_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_ack_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:387 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_init_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:401 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_splice_locked_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_stfu_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:319 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_abort_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:403 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_ack_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:366 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_input_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:1270 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_add_output_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:390 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_complete_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:285 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_init_rbf_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:393 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_input_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:297 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_remove_output_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:322 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_tx_signatures_target.rs
Summary iterations:1000002 time:52 speed:19230 crashes_count:0 timeout_count:0 new_units_added:851 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_add_htlc_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:430 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:480 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fail_malformed_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:321 slowest_unit_ms:17 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fee_target.rs
Summary iterations:1000002 time:17 speed:58823 crashes_count:0 timeout_count:0 new_units_added:308 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
msg_update_fulfill_htlc_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:309 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
offer_deser_target.rs
Summary iterations:1000002 time:19 speed:52631 crashes_count:0 timeout_count:0 new_units_added:1333 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_hop_data_target.rs
Summary iterations:1000002 time:18 speed:55555 crashes_count:0 timeout_count:0 new_units_added:981 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
onion_message_target.rs
Summary iterations:1000002 time:24 speed:41666 crashes_count:0 timeout_count:0 new_units_added:395 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
peer_crypt_target.rs
Summary iterations:1000002 time:27 speed:37037 crashes_count:0 timeout_count:0 new_units_added:315 slowest_unit_ms:16 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:42
process_network_graph_target.rs
Summary iterations:735922 time:8434 speed:87 crashes_count:0 timeout_count:0 new_units_added:30 slowest_unit_ms:40 guard_nb:566357 branch_coverage_percent:0 peak_rss_mb:161

[TheBlueMatt]

I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed chanmon_consistency_target takes nearly 3 hours. But the other one you mention, full_stack_target is only 18 seconds?

Ah, thanks for doing that. Yea, I was going on the results when I fuzz with real corpuses, where full_stack_target can get quite slow. But, given we currently don't initialize our CI fuzzers with a real corpus so it really doesn't get into it too much. I did go ahead and update the CI fuzzer to use the hard-coded full_stack_target seeds, but it still doesn't spend much time in complicated paths.

process_network_graph_target is also slow (2+ hours), and looking at the fuzz log in #3687, it doesn't seem to be resolved with that change.

Huh, interesting, I went ahead and slowed this one down though.

Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed.

Yep!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.92%. Comparing base (c4d23bc) to head (11dde05).
Report is 38 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #3691 +/- ##
==========================================
+ Coverage 89.05% 89.92% +0.86% 
==========================================
Files 155 156 +1 Lines 122019 129760 +7741 Branches 122019 129760 +7741 ==========================================
+ Hits 108666 116683 +8017 + Misses 10695 10441 -254 + Partials 2658 2636 -22 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tnull]

LGTM

Fuzz failure is #3708

[joostjager]

Great that fuzzing again found an issue. This is awesome.

Before merging this, I think we still want to see a successful fuzz run well within the timeout?

[TheBlueMatt]

Yea, happy to wait until we can at least run a fuzzing run and check that all the timings make sense.

[TheBlueMatt]

It passed, but taking five hours for CI is excessive, so I reduced it further.

[tnull]

It passed, but taking five hours for CI is excessive, so I reduced it further.

Now finished in 273min ~ 4.55 hours. Is this acceptable?

[TheBlueMatt]

Mmm, that's still pretty aggressive. Divided a few jobs by two so hopefully this time's the charm.

[TheBlueMatt]

Okay, its now "only" 2h, which I think is kinda reasonable, or at least its in line with what some of our other jobs take, so happy to land this now.

[joostjager]

Distribution looks much better now:

test	iterations	time
base32_target.rs	2500002	278
bech32_parse_target.rs	2500002	58
bolt11_deser_target.rs	2500002	45
chanmon_consistency_target.rs	5002	852
chanmon_deser_target.r	2500002	55
fromstr_to_netaddress_target.r	2500002	45
full_stack_target.rs	50002	554
indexedmap_target.rs	500002	394
invoice_deser_target.r	2500002	51
invoice_request_deser_target.r	2500002	50
msg_accept_channel_target.rs	2500002	181
msg_accept_channel_v2_target.r	2500002	258
msg_announcement_signatures_target.rs	2500002	53
msg_channel_announcement_target.rs	2500002	73
msg_channel_details_target.rs	2500002	47
msg_channel_ready_target.r	2500002	49
msg_channel_reestablish_target.rs	2500002	52
msg_channel_update_target.rs	2500002	54
msg_closing_signed_target.rs	2500002	50
msg_commitment_signed_target.r	2500002	55
msg_decoded_onion_error_packet_target.rs	2500002	50
msg_error_message_target.r	2500002	51
msg_funding_created_target.rs	2500002	52
msg_funding_signed_target.rs	2500002	50
msg_gossip_timestamp_filter_target.rs	2500002	47
msg_init_target.rs	2500002	48
msg_node_announcement_target.r	2500002	60
msg_open_channel_target.rs	2500002	71
msg_open_channel_v2_target.rs	2500002	66
msg_ping_target.rs	2500002	48
msg_pong_target.rs	2500002	48
msg_query_channel_range_target.rs	2500002	47
msg_query_short_channel_ids_target.rs	2500002	52
msg_reply_channel_range_target.rs	2500002	48
msg_reply_short_channel_ids_end_target.rs	2500002	47
msg_revoke_and_ack_target.rs	2500002	50
msg_shutdown_target.rs	2500002	47
msg_splice_ack_target.rs	2500002	50
msg_splice_init_target.rs	2500002	50
msg_splice_locked_target.r	2500002	48
msg_stfu_target.rs	2500002	47
msg_tx_abort_target.rs	2500002	48
msg_tx_ack_rbf_target.rs	2500002	47
msg_tx_add_input_target.rs	2500002	73
msg_tx_add_output_target.r	2500002	48
msg_tx_complete_target.rs	2500002	47
msg_tx_init_rbf_target.rs	2500002	48
msg_tx_remove_input_target.rs	2500002	47
msg_tx_remove_output_target.rs	2500002	48
msg_tx_signatures_target.r	2500002	100
msg_update_add_htlc_target.rs	2500002	99
msg_update_fail_htlc_target.rs	2500002	52
msg_update_fail_malformed_htlc_target.rs	2500002	49
msg_update_fee_target.rs	2500002	48
msg_update_fulfill_htlc_target.rs	2500002	49
offer_deser_target.rs	2500002	48
onion_hop_data_target.rs	2500002	49
onion_message_target.r	2500002	64
peer_crypt_target.rs	2500002	108
process_network_graph_target.r	50002	568
process_onion_failure_target.r	2500002	165
refund_deser_target.rs	2500002	49
router_target.rs	50002	606
zbase32_target.rs	2500002	119

[joostjager]

Why these two additions?

[TheBlueMatt]

Because RUSTFLAGS aren't applied to doctests, so we have to restrict tests to only run library and binary tests.

[joostjager]

Could add comments when making changes like this. Make life easier for future devs.

[joostjager]

I wanted to verify that using the hardcoded seed indeed increases coverage, but it seems that hongfuzz reports in the log always

branch_coverage_percent:0

Not sure why that is? Otherwise it would be easy to compare with and without this hardcoded seed.

Or maybe there is another way to verify that it works indeed?

[TheBlueMatt]

The fact that the full_stack_target got much slower is pretty good evidence :)

[joostjager]

Okay, indirect evidence. But why is hongfuzz reporting 0?

[joostjager]

Is it not using coverage to direct fuzzing?

[TheBlueMatt]

Yea, that I don't know, it certainly does locally and there's nothing different locally vs in CI. Also CI does turn up (fairly shallow) fuzz bugs as well, so I'd be surprised to learn it was able to do that with just general patterns and no instrumentation...

[joostjager]

Does it show you branch_coverage_percent as non-zero locally?

[TheBlueMatt]

I'm not 100% sure about branch coverage specifically, but what I get from a running instance is: Coverage : edge: 16,485/568,547 [2%] pc: 738 cmp: 794,641. I assume "cmp" is the branch coverage.

[TheBlueMatt]

Went ahead and squashed the addition of a comment in the fuzz script.

[joostjager]

Great to have a more reasonable run time for the fuzzer!

Still interested to find out what's going on with that coverage indicator at zero in CI.