Skip to content

[C++][Parquet] Using length to optimize bloom filter read #38860

Description

@mapleFU

Describe the enhancement requested

Parquet supports a bloom_filter_length in 2.10[1]. We'd like to using this length for read.

The current implemention [2] using the code below:

  1. Using a "guessed" header length to read the header. The header is likely to be 40B, but we use a larger value to avoid it evolves
  2. From the header, we get the bloom filter length, and load it from input.

Now, we can directly load the whole bloom-filter, without reading twice. We shouldn't remove the stale code because we need to read the stale file.

We also need to generate a new parquet-testing file ( I can do this ASAP )

[1] https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L824
[2] https://github.com/apache/arrow/blob/main/cpp/src/parquet/bloom_filter.cc#L117

Component(s)

C++, Parquet

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions