Context
Found during review of #48.
Problem
HcMatchGenerator stores absolute positions as u32. When a single-frame encode exceeds ~4 GiB, insert_position stops inserting (guard at abs_pos >= u32::MAX), and all existing chain candidates become stale within one window slide. Better level degrades to repcode-only matching for the remainder of the stream.
Suggested approach
Periodically rebase stored positions by subtracting a base offset when the u32 range fills up. Keep a position_epoch counter; on rebase, subtract epoch from all live hash/chain entries and update history_abs_start accordingly. This matches C zstd's approach where positions are relative to a sliding base.
Files involved
zstd/src/encoding/match_generator.rs (HcMatchGenerator::insert_position, chain_candidates)
Context
Found during review of #48.
Problem
HcMatchGenerator stores absolute positions as u32. When a single-frame encode exceeds ~4 GiB,
insert_positionstops inserting (guard atabs_pos >= u32::MAX), and all existing chain candidates become stale within one window slide. Better level degrades to repcode-only matching for the remainder of the stream.Suggested approach
Periodically rebase stored positions by subtracting a base offset when the u32 range fills up. Keep a
position_epochcounter; on rebase, subtract epoch from all live hash/chain entries and updatehistory_abs_startaccordingly. This matches C zstd's approach where positions are relative to a sliding base.Files involved
zstd/src/encoding/match_generator.rs(HcMatchGenerator::insert_position, chain_candidates)