Skip to content

fix(c-api): ZSTD_sizeof_CCtx underreports dictionary-backed contexts #388

Description

@polaz

Context

Found during review of #387.

Problem

ZSTD_sizeof_CCtx() sums only the inline ZSTD_CCtx struct and the scratch
buffer. After the first ZSTD_compress_usingCDict() call, the context also
caches a FrameCompressor (dict_compressor) whose primed match-finder tables
and dictionary snapshot dominate the real footprint, so the reported size can
miss most of the context's heap. Upstream ZSTD_sizeof_CCtx includes the
CDict-copied working tables.

Suggested approach

Add a heap-size accessor to the encoder that sums its owned allocations, then
include it in ZSTD_sizeof_CCtx when dict_compressor is Some:

  • MatchGeneratorDriver::heap_size() — sum vec_pool capacities + the active
    MatcherStorage variant's owned buffers + the primed snapshot.
  • A size method on each backend generator (FastKernelMatcher,
    DfastMatchGenerator, RowMatchGenerator, HcMatchGenerator) summing its
    hash/chain/history table capacities.
  • FrameCompressor::heap_size() = matcher size + its own scratch/pending/FSE
    buffers + dictionary snapshot.

Mirrors the decoder-side FrameDecoder::workspace_size() already used by
ZSTD_sizeof_DCtx.

Files involved

  • c-api/src/context.rsZSTD_sizeof_CCtx
  • zstd/src/encoding/frame_compressor.rs — new heap_size
  • zstd/src/encoding/match_generator.rs — driver heap_size
  • zstd/src/encoding/simple/fast_matcher.rs, zstd/src/encoding/dfast/mod.rs,
    zstd/src/encoding/row/mod.rs — per-backend size methods

Why not in #387

A correct accounting requires new size methods across all four backend
generator modules (none touched by #387) plus snapshot recursion — a
self-contained feature, not a one-line fix. ZSTD_sizeof_CCtx is an estimate
API (no correctness/round-trip impact), so an incomplete hand-rolled sum would
be more misleading than tracking it properly here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2-mediumMedium priority — important improvementenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions