[feat] Introduce cacheManager in session ctx and make StatisticsCache share in session by Ted-Jiang · Pull Request #7570 · apache/datafusion

Ted-Jiang · 2023-09-15T10:05:38Z

Which issue does this PR close?

Closes#7556 .

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Ted-Jiang · 2023-09-15T10:06:59Z


+ #[tokio::test]
+ async fn load_table_stats_with_session_level_cache() -> Result<()> {
+ let testdata = crate::test_util::parquet_test_data();


Add test to check cache share in session level

Nice -- thank you. Since this is an end to end test, I recommend moving it to somewhere in core_integration: datafusion/core/tests/core_integration.rs perhaps

alamb

Thank you @Ted-Jiang -- this is looking really nice. I left some various clean up comments but I think this PR looks very nice and is well commented and structured 🏆

alamb · 2023-09-15T16:45:01Z

+// The cache accessor, users usually working on this interface while manipulating caches
+pub trait CacheAccessor<K, V>: Send + Sync {
+ // Extra info but not part of the cache key or cache value.
+ type Extra: Clone;


Can you explain what the usecase for Extra is? Specifically I wonder why such information could not be added as a field to the Value

Like in default FileStatisticsCache, get func need check last_modified from ObjectMeta which not impl Hash so can not be part of the key, we need put this info in Extra.

I see -- that makes sense -- it might help to document the rationale in statistics

alamb · 2023-09-15T16:46:07Z

+ }
+
+ /// Get `Statistics` for file location. Returns None if file has changed or not found.
+ fn get_with_extra(&self, k: &Path, e: &Self::Extra) -> Option<Statistics> {


as written this is going to copy the statistics (though I realize that is what this PR did previously) -- maybe we could use something like Arc<Statistics> to store the statistics.

Yes , thanks for point this out

alamb · 2023-09-15T17:18:31Z


+ #[tokio::test]
+ async fn load_table_stats_with_session_level_cache() -> Result<()> {
+ let testdata = crate::test_util::parquet_test_data();


Nice -- thank you. Since this is an end to end test, I recommend moving it to somewhere in core_integration: datafusion/core/tests/core_integration.rs perhaps

alamb · 2023-09-15T17:24:34Z

+}
+
+impl CacheManagerConfig {
+ pub fn enable_table_files_statistics_cache(mut self, cache: FileStaticCache) -> Self {


Can we possibly have the field names match -- here it is called table_files_statistics_cache but on the CacheManager it is called file_statistics_cache -- I think they should be the same in both places (I like file_statistics_cache best as it matches the type name)

ops, this make sense

alamb · 2023-09-15T17:26:29Z

+pub mod cache_unit;
+
+// The cache accessor, users usually working on this interface while manipulating caches
+pub trait CacheAccessor<K, V>: Send + Sync {


Another classic API to add here would be "clear()" to clear all the values

alamb · 2023-09-15T17:30:07Z

 .with_schema(resolved_schema);
- let table =
- ListingTable::try_new(config)?.with_definition(cmd.definition.clone());
+ let provider;


If you implement the with_cache API, this can look like
let provider = ListingTable::try_new(config)? .with_cache(state.runtime_env().cache_manager.get_file_statistic_cache())

More clear way ! 👍

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Ted-Jiang · 2023-09-17T11:11:32Z

@alamb PTAL

alamb

Thanks @Ted-Jiang -- I think this looks great -- I had a suggestion to improve the comments but I think we can do that as a follow on PR as well. Nice work!

alamb · 2023-09-17T13:02:42Z

 use tempfile::NamedTempFile;

 mod custom_reader;
+mod file_statistics;


alamb · 2023-09-17T13:05:33Z

cc @Dandandan / @thinkharderdev @liukun4515 and @mateuszkj who appears to have added this feature originally in 85c11c1 / #3649

liukun4515

LGTM for file statistics trait api

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Ted-Jiang · 2023-09-18T05:29:07Z

@alamb @liukun4515 Thanks for the review.

Ted-Jiang added 3 commits September 15, 2023 15:31

add extensible cache manager in session ctx.
be03887

move FileStatisticCache into cache manager
04a2f06

add config and test
b81230d

github-actionsBot added the core Core DataFusion crate label Sep 15, 2023

Ted-Jiang commented Sep 15, 2023
View reviewed changes

Ted-Jiang requested review from alamb, liukun4515 and tustvold September 15, 2023 10:09

Ted-Jiang added 2 commits September 15, 2023 21:25

fix cargo check
382888f

fix cargo clippy
a66ee12

alamb reviewed Sep 15, 2023
View reviewed changes

Ted-Jiangand others added 4 commits September 16, 2023 09:08

Update datafusion/core/src/datasource/listing/table.rs
0160108
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Update datafusion/execution/src/cache/mod.rs
8060a76
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

fix comments
aaa912d

imply debug for CacheManager
c5c162a

Ted-Jiang force-pushed the cache branch from 9e471bd to c5c162aCompare September 16, 2023 03:00

Ted-Jiang added 4 commits September 16, 2023 11:01

imply debug for CacheManager2
83ba66a

fix doc
b09479e

fix clippy
650d419

fix clippy2
784666f

Ted-Jiang requested a review from alamb September 16, 2023 07:45

alamb mentioned this pull request Sep 16, 2023
Parallelize Parquet Serialization #7562
Merged

alamb approved these changes Sep 17, 2023
View reviewed changes

liukun4515 reviewed Sep 17, 2023
View reviewed changes

Comment threaddatafusion/execution/src/cache/mod.rs

liukun4515 approved these changes Sep 17, 2023
View reviewed changes

Ted-Jiangand others added 2 commits September 18, 2023 10:33

Update datafusion/core/src/datasource/listing/table.rs
d77ad00
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

fix fmt
cac0180

Ted-Jiang merged commit 678d27a into apache:mainSep 18, 2023

This was referenced Sep 18, 2023
Should parallel collecting statistics like infer schema? #7573
Closed
Use file statistics in query planning to avoid sorting when unecessary #7490
Closed

alamb mentioned this pull request Apr 5, 2024
[EPIC] Improve the performance of ListingTable #9964
Closed
4 tasks

Uh oh!

Conversation

Ted-Jiang commented Sep 15, 2023

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ted-JiangSep 16, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ted-Jiang commented Sep 17, 2023

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Sep 17, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

liukun4515 left a comment

Choose a reason for hiding this comment

Uh oh!

Ted-Jiang commented Sep 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Ted-JiangSep 16, 2023•
edited
Loading

alamb commented Sep 17, 2023•
edited
Loading