fix: field stats parquet aggregation

[nikhilsinhaparseable]

calculate field stats directly from parquet record batches remove old datafusion table path logic

add bounded high-cardinality handling
per-file stats IDs for correct flattened-row aggregation

improve logging and keep partial stats when individual parquet batches fail

Summary by CodeRabbit

• Refactor

  • Enhanced field statistics calculation pipeline for improved performance and accuracy in Parquet data processing.

• Tests

  • Updated unit tests for field statistics functionality.

[coderabbitai]

Walkthrough

This PR replaces SQL-based per-field statistics with a single-pass Arrow/Parquet reader pipeline. It introduces bounded distinct value tracking using HyperLogLog approximation, semaphore-controlled concurrency, and updates statistics query generation with new aggregation CTEs.

Changes

Field Statistics Computation Refactor

Layer / File(s)	Summary
Imports and Concurrency Infrastructure src/storage/field_stats.rs:33–38, 67–100	Removes query session state imports and adds storage/stream dependencies; introduces global semaphore and Rayon thread pool constants for parallel Parquet processing, plus distinct-tracking configuration (MAX_DISTINCT_VALUES, HyperLogLog parameters).
DatasetStats Extension src/storage/field_stats.rs:116–121	Adds stats_id field to DatasetStats struct, derived from Parquet file path to uniquely identify each computed statistics snapshot.
Async Entrypoint and Timing src/storage/field_stats.rs:132–152, 217–255	Refactors calculate_field_stats entrypoint to delegate to inner implementation and wrap with execution timing logs; introduces ensure_dataset_stats_stream helper to guarantee internal stream existence before ingestion.
Distinct Value Tracking with HyperLogLog src/storage/field_stats.rs:367–508, 510–570	Implements FieldCountState for bounded exact counting of top distinct values with automatic candidate eviction when cap is reached; includes HyperLogLog register-based approximation and xxh3 hashing for field-value cardinality estimation.
Single-Pass Parquet Statistics Collection src/storage/field_stats.rs:161–173, 180, 257–366	New collect_all_field_stats_from_parquet replaces SQL-based approach: acquires semaphore permit, spawns blocking work, reads Parquet record batches in memory, computes per-field counts serially or in parallel (Rayon) based on field cardinality, merges into FieldCountState, and yields statistics for non-zero fields.
Query Generation Updates src/storage/field_stats.rs:945–949, 968–1015	Rewrites build_stats_sql to introduce field_distincts and field_totals CTEs aggregating stored statistics; joins aggregates with paginated top_values results and updates window query pagination structure.
Test Suite Overhaul src/storage/field_stats.rs:1016–1031, 1184–1298	Replaces SQL/DataFusion-oriented tests with unit tests validating build_stats_sql CTE/aggregate structure, single-pass Parquet computation (field counts, distinct approximations, formatted output), and FieldCountState eviction under high cardinality.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• parseablehq/parseable#1555: Modifies build_stats_sql query generation and CTE structure for dataset field statistics computation and return format.
• parseablehq/parseable#1529: Adds dataset stats collection/retrieval APIs and initial build_stats_sql implementation that this PR refactors to support single-pass Parquet computation.

Suggested reviewers

• nitisht
• parmesant

Poem

🐰 A parser hops through Parquet files with glee,
Counting fields and distincts in one pass, you see!
No SQL needed, HyperLogLog's the key—
Bounded tracking keeps memory light and free.
Statistics flow swift, from batch to stream so bright! ✨

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description covers key changes (direct Parquet calculation, high-cardinality handling, per-file stats IDs, improved logging) but does not follow the template structure with required sections like testing confirmation or documentation notes.	Add template sections: confirm testing of log ingestion/query, document code comments explaining intent, and note any new documentation added for modified features.
Docstring Coverage	⚠️ Warning	Docstring coverage is 39.29% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'fix: field stats parquet aggregation' directly summarizes the main change - replacing SQL-based field statistics with direct Parquet aggregation.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/storage/field_stats.rs (1)

929-929: ⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

SQL injection vulnerability: incorrect escaping for string literal context.

The dataset_name is escaped for double-quoted identifiers (" → ""), but it's used in a single-quoted string literal at line 952. If dataset_name contains a single quote (e.g., test'--), it will break out of the SQL string and allow injection.

🔒 Proposed fix

- let dataset_name = dataset_name.replace('"', "\"\"");+ let dataset_name = dataset_name.replace('\'', "''");

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: fb92efa7-3d3c-4b10-bf5b-3bd43577692f

📥 Commits

Reviewing files that changed from the base of the PR and between d61d73b and 3a2c67b.

📒 Files selected for processing (1)

• src/storage/field_stats.rs