Add example for building an external secondary index for parquet files

[alamb]

Note: While this PR looks very large (715 lines) around half of the content is comments / docstrings

Which issue does this PR close?

Closes#10546

Rationale for this change

See #10546

Building and using external indexes in DataFusion is an important feature. Adding an example of how to do so will help drive the design and APIs

What changes are included in this PR?

New Example

Are these changes tested?

CI

Are there any user-facing changes?

No -- just an example

TODOs

• Propose a nicer API for extracting the statistics
• Connect pruning predicate into scan to avoid scanning files
• File tickets / PRs to make creating ParquetExec easier
• Try and make some PRs / documentation upstream in the parquet crate to make it easier to work with parquet statistics

[alamb]

I think the example speaks for itself in terms of comments, so I don't plan to add additional ones on the PR unless something is unclear

[alamb]

This PR is now ready for review

[alamb]

@crepererum and @NGA-TRAN -- here is a PR ready for your review that shows how to do file level pruning with statistics.

I will make an example of how to do row group level / page level pruning next

[NGA-TRAN]

I start reviewing this

[crepererum]

The example is a bit long (in lines of code) but I also don't have any concrete recommendation on how to make it shorter w/o sacrificing its good readability. Maybe -- but that's personal taste -- collapse the use statements a bit, e.g. instead of

use foo::Bar;use foo::Baz;

use

use foo::{Bar,Baz};

[alamb]

I collapsed this a bit but paradoxically when I tried to collapse them even more the number of lines actually increased

For example, this goes from 4 lines

use datafusion::datasource::listing::PartitionedFile;use datafusion::datasource::physical_plan::parquet::{RequestedStatistics,StatisticsConverter,};

to 6 lines (though it is less dense)

use datafusion::datasource::{
listing::PartitionedFile;
physical_plan::parquet::{RequestedStatistics,StatisticsConverter,}};

[crepererum]

I personally find the grouped ones easier to parse though.

[alamb]

Another reason I have heard for using the single line includes is that it reduces merge conflicts when people change / update the includes. I am not sure how relevant this is in this case

FWIW the includes were automatically created by my editor (rust rover).

[alamb]

Thank you very much for the review @crepererum