Skip to content

perf: fast-path ASCII field matching - #2

Closed
ErikBPF wants to merge 1 commit into
dwsmith1983:feature/delta-native-scanfrom
ErikBPF:ascii-case-insensitive-fast-path
Closed

perf: fast-path ASCII field matching#2
ErikBPF wants to merge 1 commit into
dwsmith1983:feature/delta-native-scanfrom
ErikBPF:ascii-case-insensitive-fast-path

Conversation

@ErikBPF

@ErikBPF ErikBPF commented Sep 2, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Part of apache#174 and apache#5365.

Addresses the ASCII fast-path review findings in
apache#5365 (comment) and
apache#5365 (comment).

This targets feature/delta-native-scan because the JVM case-table matcher is
currently part of that unmerged branch.

Rationale for this change

names_equal_ignore_case_java currently allocates two Strings and performs
per-character case-table lookups for every comparison, including pure ASCII
field names. The matcher is used by the shared Parquet scan path.

Reviewer-reported release measurements found 2.3x to 12.0x end-to-end scan
slowdowns for representative many-file and many-column shapes. A separate
allocation probe reported 160,800 allocation or reallocation calls for 20,100
ASCII comparisons, versus zero with this guard.

What changes are included in this PR?

  • Use str::eq_ignore_ascii_case when both names are ASCII.
  • Preserve JVM-compatible lowercasing whenever either name is non-ASCII.
  • Add a focused test proving ASCII matching does not depend on JVM case tables.

How are these changes tested?

  • RED: the focused test failed before the guard at the intended assertion.
  • GREEN: focused test passed: 1 passed, 319 filtered out.
  • cargo test -p datafusion-comet: 316 passed, 19 ignored.
  • cargo clippy -p datafusion-comet --lib -- -D warnings: clean.
  • cargo fmt --all --check: clean.
  • git diff --check: clean.

Independent microbenchmark

Measured on a local server with
Rust 1.97.1, and the repository's default release profile (optimized + debuginfo).

The temporary probe compared the exact previous expression
(java_lowercase(a, tables) == java_lowercase(b, tables)) with
names_equal_ignore_case_java. It used 200 already-lowercase ASCII names,
the lower-triangular 20,100 comparisons repeated 50 times (1,005,000
comparisons per sample), five samples per path, and reported the median.
black_box protected inputs/results, and both paths were required to return
the same match count.

process run previous path ASCII fast path speedup
cold process 800.003 ms 19.965 ms 40.1x
warm repeat 830.755 ms 20.121 ms 41.3x

The temporary benchmark was removed after measurement and is not part of the
production diff.

Results and limitations

The local release-mode matcher result independently confirms the expected
hot-path improvement. It is not an end-to-end DataSourceExec benchmark and
does not independently measure allocation counts; those figures remain the
reviewers' measurements.

Non-ASCII comparisons still allocate and perform JVM case-table lookups. This
does not implement the separately suggested schema-wide lowercase hoisting.

AI assistance

OpenAI Codex assisted with implementation, test execution, benchmarking,
review, and this description. The contributor reviewed the change and remains
responsible for it.

Avoid JVM case-table allocations and lookups for the common ASCII path.
@ErikBPF

ErikBPF commented Sep 3, 2026

Copy link
Copy Markdown
Author

Superseded by apache#5602, which merged the broader fix on main: ASCII fast path, schema-wide pre-folding, exact JVM non-ASCII folding, and caching. Closing to avoid duplicate work.

@ErikBPF ErikBPF closed this Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant