Skip to content

File partitioning for ListingTable - #1141

Merged
alamb merged 18 commits into
apache:masterfrom
rdettai:file-partitioning
Nov 1, 2021
Merged

File partitioning for ListingTable#1141
alamb merged 18 commits into
apache:masterfrom
rdettai:file-partitioning

Conversation

@rdettai

@rdettairdettai commented Oct 18, 2021

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes#1139.

Rationale for this change

Adding capability to parse file partitioning and prune unnecessary files

What changes are included in this PR?

  • add partition_values: Vec<ScalarValue> to PartitionedFile
  • implement pruned_partition_list
  • add extra column for the partition dimensions to execute() record batch result in file format execution plans
    • avro/csv/json
    • parquet
  • add the proper TableProviderFilterPushDown value to supports_filter_pushdown() to avoid re-evaluation of the partition pruning [1]]

What changes are planned in further issues/PRs?

Are there any user-facing changes?

Rename the ListingOptions.partitions to ListingOptions.table_partition_cols to make it a bit more explicit.


[1] re-evaluating the filters on the the partition column would be expensive:

  • it requires the the column to be pushed down, thus materialized in the source execution plan. This is acceptable if we use DictionaryArray<uint8> which is pretty cheap.
  • when applying the filtering expression the dictionary needs to be expanded because many kernel ops are not supported on Dictionaries for now (link and Add better and faster support for dictionary types #87). This could be very expensive!!!!

Loading
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api changeChanges the API exposed to users of the crateenhancementNew feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement partitioned read in listing table provider

5 participants

@rdettai@alamb@houqp@Dandandan@jimexist