Skip to content

WIP: Add tokomak optimizer - #441

Closed
Dandandan wants to merge 14 commits into
apache:masterfrom
Dandandan:tokomak_optimizer
Closed

WIP: Add tokomak optimizer#441
Dandandan wants to merge 14 commits into
apache:masterfrom
Dandandan:tokomak_optimizer

Conversation

@Dandandan

@DandandanDandandan commented May 29, 2021

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes#440

TODO:

  • Decide whether to make this (and the egg dependency) an optional (off by default) feature.
  • Add some more tests
  • Add support for more expressions to support more queries:
    • dates
    • in
    • between
    • aggregations
    • cast
  • Some cleanup

Rationale for this change

See #440 for a longer explanation.
I believe using a optimization library is a great way to accelerate developing & increase the number of optimizations possible.

Also, the more optimizations we add, the more we end up building a own (probably worse) version of a optimization framework.

What changes are included in this PR?

Adds a new optimizer Tokomak that does expression simplification based on rewrite rules.

Are there any user-facing changes?

@Dandandan
Dandandan marked this pull request as draft May 29, 2021 20:51
@codecov-commenter

codecov-commenter commented May 29, 2021

Copy link
Copy Markdown

Codecov Report

Merging #441 (cad2aae) into master (c9ed34c) will decrease coverage by 0.02%.
The diff coverage is 70.78%.

Impacted file tree graph

@@ Coverage Diff @@## master #441 +/- ##
==========================================
- Coverage 75.28% 75.26% -0.03% 
==========================================
Files 147 153 +6 Lines 24868 25542 +674 ==========================================
+ Hits 18723 19223 +500 - Misses 6145 6319 +174 
Impacted FilesCoverage Δ
datafusion/src/execution/context.rs92.08% <ø> (ø)
datafusion/src/optimizer/simplification.rs70.78% <70.78%> (ø)
datafusion/tests/sql.rs99.89% <0.00%> (ø)
datafusion-cli/src/main.rs0.00% <0.00%> (ø)
ballista/rust/core/src/utils.rs0.00% <0.00%> (ø)
datafusion/src/datasource/csv.rs72.81% <0.00%> (ø)
ballista/rust/executor/src/main.rs0.00% <0.00%> (ø)
datafusion/src/physical_plan/mod.rs78.70% <0.00%> (ø)
ballista/rust/executor/src/execution_loop.rs0.00% <0.00%> (ø)
ballista/rust/executor/src/flight_service.rs0.00% <0.00%> (ø)
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c9ed34c...cad2aae. Read the comment docs.

@alamb

Copy link
Copy Markdown
Contributor

@Dandandan do you still plan to work on this PR?

@houqp

Copy link
Copy Markdown
Member

@Dandandan let me know if you need help with some of the todo list items, happy to help push this PR forward.

@alamb

Copy link
Copy Markdown
Contributor

Closing stale PRs to keep PR review list manageable. Please reopen if that is a mistake

@alambalamb closed this Aug 20, 2021
unkloud pushed a commit to unkloud/datafusion that referenced this pull request Mar 23, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Future of experimental optimizer datafusion-tokomak

4 participants

@Dandandan@codecov-commenter@alamb@houqp