Skip to content

ARROW-12441: [Rust][DataFusion] Support cartesian join - #10092

Closed
Dandandan wants to merge 14 commits into
apache:masterfrom
Dandandan:cartesian_join
Closed

ARROW-12441: [Rust][DataFusion] Support cartesian join#10092
Dandandan wants to merge 14 commits into
apache:masterfrom
Dandandan:cartesian_join

Conversation

@Dandandan

@DandandanDandandan commented Apr 18, 2021

Copy link
Copy Markdown
Contributor

This is a first (naive, but probably not that bad) implementation of the cartesian join and CROSS JOIN syntax.

The left side gets loaded into memory and the right side is streamed and gets combined with the left side.

Memory consumption could be improved, the current implementation results in large batches if both of the sides are big, which could be solved by keeping a "cursor" of the left side and producing the batches one by one instead of concatenating the result of the full cartesian product.

FYI @andygrove@alamb@jorgecarleitao

This also makes query 9 run in DataFusion (though performance is not OK, but I believe that should be not related to the cross join itself, but is caused by another issue).

@github-actions

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW

Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@codecov-commenter

codecov-commenter commented Apr 18, 2021

Copy link
Copy Markdown

Codecov Report

Merging #10092 (edcedb0) into master (9a4ef46) will increase coverage by 0.00%.
The diff coverage is 77.22%.

❗ Current head edcedb0 differs from pull request most recent head f93a7d3. Consider uploading reports for the commit f93a7d3 to get more accurate results
Impacted file tree graph

@@ Coverage Diff @@## master #10092 +/- ##
========================================
Coverage 78.87% 78.87% ========================================
Files 286 287 +1 Lines 64808 64974 +166 ========================================
+ Hits 51119 51250 +131 - Misses 13689 13724 +35 
Impacted FilesCoverage Δ
...lista/rust/core/src/serde/logical_plan/to_proto.rs0.00% <ø> (ø)
rust/datafusion/src/optimizer/constant_folding.rs91.95% <0.00%> (-0.36%)⬇️
...t/datafusion/src/optimizer/projection_push_down.rs98.66% <ø> (ø)
rust/datafusion/src/physical_plan/hash_utils.rs100.00% <ø> (+2.89%)⬆️
rust/datafusion/src/physical_plan/mod.rs87.09% <ø> (ø)
...datafusion/src/optimizer/hash_build_probe_order.rs55.65% <55.55%> (-0.02%)⬇️
rust/datafusion/src/logical_plan/plan.rs79.83% <61.53%> (-0.56%)⬇️
...ust/datafusion/src/physical_plan/cartesian_join.rs75.47% <75.47%> (ø)
rust/benchmarks/src/bin/tpch.rs38.86% <100.00%> (+0.52%)⬆️
rust/datafusion/src/logical_plan/builder.rs89.33% <100.00%> (+0.44%)⬆️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a4ef46...f93a7d3. Read the comment docs.

@DandandanDandandan changed the title WIP cartesian joinARROW-12441: Support cartesian joinApr 18, 2021
@github-actions

Copy link
Copy Markdown

@DandandanDandandan changed the title ARROW-12441: Support cartesian joinARROW-12441: [Rust][DataFusion] Support cartesian joinApr 18, 2021
@Dandandan

Copy link
Copy Markdown
ContributorAuthor

Moved to apache/datafusion#11

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Dandandan@codecov-commenter