Consolidate Parquet Metadata handling into its own module and struct `DFParquetMetadata` by alamb · Pull Request #17127 · apache/datafusion

alamb · 2025-08-11T18:42:22Z

Which issue does this PR close?

Follow on to feat: Use Cached Metadata for ListingTable Statistics #17022
part of [Epic] Enable parquet metadata cache by default #17000

Rationale for this change

As suggested by @nuno-faria here: #17022 (comment)

The number of options and flags that are being passed around to the various metadata handling
function in the parquet code is getting somewhat out of hand

For example in #17022 from @shehabgamin a significant portion
of the PR is adding new options to existing functions to thread through the new options
and the tests. If we had this code organized better it would be easier to maintain and extend.

Also, as we use the caching more it is important to ensure it is used in all the right places.

What changes are included in this PR?

Proposal:

Extract the options into a struct DFParquetMetadata
Deprecate the old functions
Update the functions / tests to create the struct

Are these changes tested?

yes, it is all covered by existing unit tests (changed in this PR)

Are there any user-facing changes?

alamb · 2025-08-11T18:44:09Z

- Some(ctx.runtime_env().cache_manager.get_file_metadata_cache()),
- )
- .await?;
+ let file_metadata_cache =


this shows the key API difference -- instead of calling a bunch of free functions, you now construct a DFParquetMetadata and call methods on that struct instead

It looks way cleaner now.

alamb · 2025-08-11T18:54:30Z

+ df_meta.with_file_metadata_cache(Some(Arc::clone(&file_metadata_cache)));
+
 // Increases by 3 because cache has no entries yet
- fetch_parquet_metadata(


I think the new struct makes it much clearer what is being tested vs what is test setup functionality and I find the updated tests to be much easier to read

alamb · 2025-08-11T18:55:38Z

 })
 }

-async fn fetch_schema_with_location(


Most of this PR is moving code in this module into metadata.rs

alamb · 2025-08-11T18:56:18Z

 /// [`ParquetFileReaderFactory`]: crate::ParquetFileReaderFactory
-pub async fn fetch_parquet_metadata<F: MetadataFetch>(
- fetch: F,
+#[deprecated(


I left all the existing public APIs and deprecated them, and updated them to call the new DFParquetMetadata structure

alamb · 2025-08-11T18:56:46Z

 Ok(file_metadata)
 }

-/// Min/max aggregation can take Dictionary encode input but always produces unpacked


I am quite please that most of the statistics handling is now consolidated into its own module

alamb · 2025-08-11T18:57:39Z

- file_meta.object_meta.location,
- ))
- })
+ // TODO should there be metadata prefetch hint here?


The metadata prefetch hint isn't passed here (it isn't on main either) but this refactor leads me to believe it might be helpful to do so 🤔

From a user's perspective, I think it makes sense that the metadata prefetch option should apply everywhere metadata is fetched. It can be quite confusing when you change an option and either see no change at all (positive, negative, system resource usage etc.), or perhaps even worse, inconsistent change based on a specific workflow (e.g. "Why do queries for table X use twice the network hops, but table Y uses 50% more bandwidth?")

yeah, in theory it should be controlled by a config option: https://datafusion.apache.org/user-guide/configs.html
datafusion.execution.parquet.metadata_size_hint NULL
I haven't traced down why that one is not used here

At the time I only set the hint in inner = inner.with_footer_size_hint(hint), and then in get_metadata we would read it like so: reader.try_load(&mut self.inner, object_meta.size).await?;. Yes its better if we pass it to DFParquetMetadata.

I don't think this is a change / regression in this PR, so can we open an issue to follow up and let it go here?
I do agree it should be passed down from the config at least in ListingTable and other "default" uses that have access to the config.

Filed CachedParquetFileReader should respect the metadata prefetch hint #17279

nuno-faria · 2025-08-12T08:46:52Z

LGTM, its a much cleaner API.

…_metadata_handling

alamb · 2025-08-20T19:30:33Z

I merged up to fix some conflicts, largely caused by

fix: respect inexact flags in row group metadata #16412

alamb · 2025-08-20T19:32:24Z

@adriangb I wonder if you might be able to review this PR? @jonathanc-n and @nuno-faria have already approved it but neither of them are committers so I can't merge this PR yet unfortunately

adriangb · 2025-08-20T20:26:27Z

Yes will put it on my queue

adriangb

Overall looks good! Some nits and a request for a followup issue.

adriangb · 2025-08-20T22:26:03Z

+
+ let fetch = ObjectStoreFetch::new(*store, object_meta);
+
+ // implementation to fetch parquet metadata


I'm not reviewing the implementation, I assume it was largely copied over

Yes, it was entirely copied over

adriangb · 2025-08-20T22:27:19Z

+ }
+
+ if cache_metadata && file_metadata_cache.is_some() {
+ // Need to retrieve the entire metadata for the caching to be effective.


Does this mean that even if I have page indexes disabled if I use a metadata cache it will still retrieve (and decode?) them?

Yes, I think that is exactly what it it means, which is confusing.

adriangb · 2025-08-20T22:27:54Z

+ /// Read and parse the schema of the Parquet file
+ pub async fn fetch_schema(&self) -> Result<Schema> {
+ let metadata = self.fetch_metadata().await?;


Related to question above: how much work is fetching the schema? Does it also fetch row group stats? Page indexes?

TLDR is I am not 100% sure, but I am to find out with @BlakeOrth
POC: datafusion-cli instrumented object store #17266

adriangb · 2025-08-20T22:28:35Z

+ Ok((loc_path, schema))
+ }
+
+ pub async fn fetch_statistics(&self, table_schema: &SchemaRef) -> Result<Statistics> {


Needs a docstring, details about what this operation involves

adriangb · 2025-08-20T22:36:03Z

- file_meta.object_meta.location,
- ))
- })
+ // TODO should there be metadata prefetch hint here?


I don't think this is a change / regression in this PR, so can we open an issue to follow up and let it go here?
I do agree it should be passed down from the config at least in ListingTable and other "default" uses that have access to the config.

…_metadata_handling

Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

…alamb/datafusion into alamb/extract_parquet_metadata_handling

Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

…alamb/datafusion into alamb/extract_parquet_metadata_handling

github-actionsBot added core Core DataFusion crate datasource Changes to the datasource crate labels Aug 11, 2025

alamb changed the title ~~Consolidate Parquet Metadata handling~~Consolidate Parquet Metadata handling into its own module and struct DFParquetMetadataAug 11, 2025

alamb force-pushed the alamb/extract_parquet_metadata_handling branch from 8c2a99f to d993b04Compare August 11, 2025 18:49

alamb commented Aug 11, 2025
View reviewed changes

Consolidate Parquet Metadata handling
ef90d05

alamb force-pushed the alamb/extract_parquet_metadata_handling branch from d993b04 to ef90d05Compare August 11, 2025 19:03

github-actionsBot added the common Related to common crate label Aug 11, 2025

alamb mentioned this pull request Aug 11, 2025
feat: Use Cached Metadata for ListingTable Statistics #17022
Merged

alamb marked this pull request as ready for review August 11, 2025 19:07

Merge remote-tracking branch 'apache/main' into alamb/extract_parquet…
8033362
…_metadata_handling

adriangb self-requested a review August 20, 2025 20:26

adriangb approved these changes Aug 20, 2025
View reviewed changes

alamband others added 5 commits August 21, 2025 15:24

Merge remote-tracking branch 'apache/main' into alamb/extract_parquet…
f313e14
…_metadata_handling

Apply suggestions from code review
e0a49f0
Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

Merge branch 'alamb/extract_parquet_metadata_handling' of github.com:…
3f31ad3
…alamb/datafusion into alamb/extract_parquet_metadata_handling

Update datafusion/datasource-parquet/src/metadata.rs
8b402f8
Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

Merge branch 'alamb/extract_parquet_metadata_handling' of github.com:…
f345778
…alamb/datafusion into alamb/extract_parquet_metadata_handling

alamb mentioned this pull request Aug 21, 2025
CachedParquetFileReader should respect the metadata prefetch hint #17279
Closed

Improved comments
b0d1e5d

alamb merged commit f363e38 into apache:mainAug 22, 2025
27 checks passed

petern48 mentioned this pull request Oct 31, 2025
Use metadata cache when fetching metadata in geoparquet reader apache/sedona-db#250
Closed


		let fetch = ObjectStoreFetch::new(*store, object_meta);

		// implementation to fetch parquet metadata

Uh oh!

Conversation

alamb commented Aug 11, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alambAug 11, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nuno-faria commented Aug 12, 2025

Uh oh!

alamb commented Aug 20, 2025

Uh oh!

alamb commented Aug 20, 2025

Uh oh!

adriangb commented Aug 20, 2025

Uh oh!

adriangb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alambAug 11, 2025•
edited
Loading