Implement Display for Expr, improve operator display by matthewmturner · Pull Request #971 · apache/datafusion

matthewmturner · 2021-09-06T04:08:40Z

Which issue does this PR close?

Closes#347

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

matthewmturner · 2021-09-06T04:20:43Z

@alamb @Dandandan FYI i just started on this. I added the impl for Display on BinaryExpr and updated some of the logical_plan tests to confirm it worked. If you think this is an ok approach i can work on updating the other tests.

alamb

Thank you @matthewmturner !

I think this looks like a good approach. Thank you

alamb · 2021-09-09T18:19:29Z

+ ref right,
+ ref op,
+ } => write!(f, "{} {} {}", left, op, right),
+ _ => write!(f, "{}", ""),


Ideally, this code would cover all the variants of Expr -- however, that doesn't have to be done in one PR.
How about as a potential interim solution we could do something like the following and use the debug display until someone has time to add a better display implementation?
Suggested change
_ => write!(f,"{}",""),
_ => write!(f,"{:?}",self),

alamb · 2021-09-09T18:19:39Z

 assert_eq!(
 format!("{:?}", lit(1u32) + lit(2u32)),
- "UInt32(1) Plus UInt32(2)"
+ "UInt32(1) + UInt32(2)"


that sure looks better

alamb · 2021-09-09T18:20:21Z

Note the python test failure has been resolved on master so if you rebase this branch against apache/master that test should pass

matthewmturner · 2021-09-10T15:00:52Z

 let expected = "Projection: #person.first_name, #MAX(person.age)\
- \n Filter: #MAX(person.age) Gt Int64(100) And #MIN(person.id Minus Int64(2)) Lt Int64(50)\
- \n Aggregate: groupBy=[[#person.first_name]], aggr=[[MAX(#person.age), MIN(#person.id Minus Int64(2))]]\
+ \n Filter: #MAX(person.age) > Int64(100) AND #MIN(person.id Minus Int64(2)) < Int64(50)\


@alamb Im a bit confused. The minus operator seems to be formatted differently depending on the part of the plan its in - is that expected? The test passes right now but i had to use Minus in the Filter section and - in the Aggregate section.

One difference is that MIN is a aggregate function which is a different type of Expr -- perhaps it formats its arguments using {:?}

matthewmturner · 2021-09-15T15:21:54Z

 fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
 // uppercase of the debug.
- write!(f, "{}", format!("{:?}", self).to_uppercase())
+ write!(f, "{}", self)


@alamb FYI after making this update I get a stack overflow error when running the test select_aggregate_with_group_by_with_having_using_derived_column_aggreagate_not_in_select. I think i saw on some other issues/PRs a stack related issue, and im not sure if it could be related to this, but wanted to run it by you.

#910 (comment) is a perhaps relevant comment. It adds RUST_MIN_STACK_SIZE when the tests are run. Perhaps you can try rebasing to pick up the changes in #910 and see if the problem you were seeing is resolved?

Dandandan · 2021-09-15T17:49:56Z

+ ref left,
+ ref right,
+ ref op,
+ } => write!(f, "{} {} {}", left, op, right),


This is not used at the moment, as the implementation uses debug.

Dandandan · 2021-09-15T17:51:27Z

 fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
 // uppercase of the debug.
- write!(f, "{}", format!("{:?}", self).to_uppercase())
+ write!(f, "{}", self)


Isn't this doing infinite recursion, as it will try to use the same method to display self?

matthewmturner · 2021-09-16T15:12:21Z

 Expr::BinaryExpr { left, op, right } => {
- write!(f, "{:?} {:?} {:?}", left, op, right)
+ write!(f, "{:?} {} {:?}", left, op, right)


@Dandandan FYI i updated here to use display and i think thats what made it work. i was able to get the expected results on some tests after this as well.
Are you ok with this approach?

I would be ok with this as intermediate solution. Ideally we'll move the formatting to Display instead and use that for printing @alamb wdyt?

I agree with @Dandandan

Dandandan · 2021-09-16T15:39:33Z

Nice, some tests to go ;)

matthewmturner · 2021-09-17T19:30:27Z

@Dandandan @alamb I updated and got all tests to pass. let me know if anything else needed :)

Dandandan

LGTM, great quality of life improvement 👍💯

alamb

Looks much nicer to me. Thank you @matthewmturner !

alamb · 2021-09-18T11:04:20Z

 Expr::BinaryExpr { left, op, right } => {
- write!(f, "{:?} {:?} {:?}", left, op, right)
+ write!(f, "{:?} {} {:?}", left, op, right)


I agree with @Dandandan

alamb · 2021-09-18T11:06:09Z

I am going to test this branch locally against main to make sure there are no logical conficts in the tests after merging #792 and if not will merge it in

alamb · 2021-09-18T11:11:05Z

unfortunately when I merged apache/master into this branch locally, several tests failed. It looks like a few newly added tests also need to be updated


failures:
---- execution::context::tests::window_partition_by_order_by stdout ----
thread 'execution::context::tests::window_partition_by_order_by' panicked at 'assertion failed: `(left == right)`
left: `["+----+----+--------------+-----------------------------------+----------------------------------+------------------------------------------+--------------+----------------+--------------+--------------+--------------+", "| c1 | c2 | ROW_NUMBER() | FIRST_VALUE(test.c2 Plus test.c1) | LAST_VALUE(test.c2 Plus test.c1) | NTH_VALUE(test.c2 Plus test.c1,Int64(1)) | SUM(test.c2) | COUNT(test.c2) | MAX(test.c2) | MIN(test.c2) | AVG(test.c2) |", "+----+----+--------------+-----------------------------------+----------------------------------+------------------------------------------+--------------+----------------+--------------+--------------+--------------+", "| 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |", "| 0 | 2 | 1 | 2 | 2 | 2 | 2 | 1 | 2 | 2 | 2 |", "| 0 | 3 | 1 | 3 | 3 | 3 | 3 | 1 | 3 | 3 | 3 |", "| 0 | 4 | 1 | 4 | 4 | 4 | 4 | 1 | 4 | 4 | 4 |", "| 0 | 5 | 1 | 5 | 5 | 5 | 5 | 1 | 5 | 5 | 5 |", "+----+----+--------------+-----------------------------------+----------------------------------+------------------------------------------+--------------+----------------+--------------+--------------+--------------+"]`,
right: `["+----+----+--------------+--------------------------------+-------------------------------+---------------------------------------+--------------+----------------+--------------+--------------+--------------+", "| c1 | c2 | ROW_NUMBER() | FIRST_VALUE(test.c2 + test.c1) | LAST_VALUE(test.c2 + test.c1) | NTH_VALUE(test.c2 + test.c1,Int64(1)) | SUM(test.c2) | COUNT(test.c2) | MAX(test.c2) | MIN(test.c2) | AVG(test.c2) |", "+----+----+--------------+--------------------------------+-------------------------------+---------------------------------------+--------------+----------------+--------------+--------------+--------------+", "| 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |", "| 0 | 2 | 1 | 2 | 2 | 2 | 2 | 1 | 2 | 2 | 2 |", "| 0 | 3 | 1 | 3 | 3 | 3 | 3 | 1 | 3 | 3 | 3 |", "| 0 | 4 | 1 | 4 | 4 | 4 | 4 | 1 | 4 | 4 | 4 |", "| 0 | 5 | 1 | 5 | 5 | 5 | 5 | 1 | 5 | 5 | 5 |", "+----+----+--------------+--------------------------------+-------------------------------+---------------------------------------+--------------+----------------+--------------+--------------+--------------+"]`: expected:
[
"+----+----+--------------+-----------------------------------+----------------------------------+------------------------------------------+--------------+----------------+--------------+--------------+--------------+",
"| c1 | c2 | ROW_NUMBER() | FIRST_VALUE(test.c2 Plus test.c1) | LAST_VALUE(test.c2 Plus test.c1) | NTH_VALUE(test.c2 Plus test.c1,Int64(1)) | SUM(test.c2) | COUNT(test.c2) | MAX(test.c2) | MIN(test.c2) | AVG(test.c2) |",
"+----+----+--------------+-----------------------------------+----------------------------------+------------------------------------------+--------------+----------------+--------------+--------------+--------------+",
"| 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |",
"| 0 | 2 | 1 | 2 | 2 | 2 | 2 | 1 | 2 | 2 | 2 |",
"| 0 | 3 | 1 | 3 | 3 | 3 | 3 | 1 | 3 | 3 | 3 |",
"| 0 | 4 | 1 | 4 | 4 | 4 | 4 | 1 | 4 | 4 | 4 |",
"| 0 | 5 | 1 | 5 | 5 | 5 | 5 | 1 | 5 | 5 | 5 |",
"+----+----+--------------+-----------------------------------+----------------------------------+------------------------------------------+--------------+----------------+--------------+--------------+--------------+",
]
actual:
[
"+----+----+--------------+--------------------------------+-------------------------------+---------------------------------------+--------------+----------------+--------------+--------------+--------------+",
"| c1 | c2 | ROW_NUMBER() | FIRST_VALUE(test.c2 + test.c1) | LAST_VALUE(test.c2 + test.c1) | NTH_VALUE(test.c2 + test.c1,Int64(1)) | SUM(test.c2) | COUNT(test.c2) | MAX(test.c2) | MIN(test.c2) | AVG(test.c2) |",
"+----+----+--------------+--------------------------------+-------------------------------+---------------------------------------+--------------+----------------+--------------+--------------+--------------+",
"| 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |",
"| 0 | 2 | 1 | 2 | 2 | 2 | 2 | 1 | 2 | 2 | 2 |",
"| 0 | 3 | 1 | 3 | 3 | 3 | 3 | 1 | 3 | 3 | 3 |",
"| 0 | 4 | 1 | 4 | 4 | 4 | 4 | 1 | 4 | 4 | 4 |",
"| 0 | 5 | 1 | 5 | 5 | 5 | 5 | 1 | 5 | 5 | 5 |",
"+----+----+--------------+--------------------------------+-------------------------------+---------------------------------------+--------------+----------------+--------------+--------------+--------------+",
]
', datafusion/src/execution/context.rs:1688:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- optimizer::common_subexpr_eliminate::test::subexpr_in_different_order stdout ----
thread 'optimizer::common_subexpr_eliminate::test::subexpr_in_different_order' panicked at 'assertion failed: `(left == right)`
left: `"Projection: Int32(1) + #test.a, #test.a + Int32(1)\n TableScan: test projection=None"`,
right: `"Projection: Int32(1) Plus #test.a, #test.a Plus Int32(1)\n TableScan: test projection=None"`', datafusion/src/optimizer/common_subexpr_eliminate.rs:647:9
---- optimizer::common_subexpr_eliminate::test::cross_plans_subexpr stdout ----
thread 'optimizer::common_subexpr_eliminate::test::cross_plans_subexpr' panicked at 'assertion failed: `(left == right)`
left: `"Projection: #Int32(1) + test.a\n Projection: Int32(1) + #test.a\n TableScan: test projection=None"`,
right: `"Projection: #Int32(1) Plus test.a\n Projection: Int32(1) Plus #test.a\n TableScan: test projection=None"`', datafusion/src/optimizer/common_subexpr_eliminate.rs:647:9
---- optimizer::common_subexpr_eliminate::test::subexpr_in_same_order stdout ----
thread 'optimizer::common_subexpr_eliminate::test::subexpr_in_same_order' panicked at 'assertion failed: `(left == right)`
left: `"Projection: #BinaryExpr-+Column-test.aLiteral1 AS Int32(1) + test.a AS first, #BinaryExpr-+Column-test.aLiteral1 AS Int32(1) + test.a AS second\n Projection: Int32(1) + #test.a AS BinaryExpr-+Column-test.aLiteral1, #test.a, #test.b, #test.c\n TableScan: test projection=None"`,
right: `"Projection: #BinaryExpr-+Column-test.aLiteral1 AS Int32(1) Plus test.a AS first, #BinaryExpr-+Column-test.aLiteral1 AS Int32(1) Plus test.a AS second\n Projection: Int32(1) Plus #test.a AS BinaryExpr-+Column-test.aLiteral1, #test.a, #test.b, #test.c\n TableScan: test projection=None"`', datafusion/src/optimizer/common_subexpr_eliminate.rs:647:9
---- optimizer::common_subexpr_eliminate::test::aggregate stdout ----
thread 'optimizer::common_subexpr_eliminate::test::aggregate' panicked at 'assertion failed: `(left == right)`
left: `"Aggregate: groupBy=[[]], aggr=[[Int32(1) + #AggregateFunction-AVGfalseColumn-test.a AS AVG(test.a), Int32(1) - #AggregateFunction-AVGfalseColumn-test.a AS AVG(test.a)]]\n Projection: AVG(#test.a) AS AggregateFunction-AVGfalseColumn-test.a, #test.a, #test.b, #test.c\n TableScan: test projection=None"`,
right: `"Aggregate: groupBy=[[]], aggr=[[Int32(1) Plus #AggregateFunction-AVGfalseColumn-test.a AS AVG(test.a), Int32(1) Minus #AggregateFunction-AVGfalseColumn-test.a AS AVG(test.a)]]\n Projection: AVG(#test.a) AS AggregateFunction-AVGfalseColumn-test.a, #test.a, #test.b, #test.c\n TableScan: test projection=None"`', datafusion/src/optimizer/common_subexpr_eliminate.rs:647:9
---- optimizer::common_subexpr_eliminate::test::tpch_q1_simplified stdout ----
thread 'optimizer::common_subexpr_eliminate::test::tpch_q1_simplified' panicked at 'assertion failed: `(left == right)`
left: `"Aggregate: groupBy=[[]], aggr=[[SUM(#BinaryExpr-*BinaryExpr--Column-test.bLiteral1Column-test.a AS test.a * Int32(1) - test.b), SUM(#BinaryExpr-*BinaryExpr--Column-test.bLiteral1Column-test.a AS test.a * Int32(1) - test.b * Int32(1) + #test.c)]]\n Projection: #test.a * Int32(1) - #test.b AS BinaryExpr-*BinaryExpr--Column-test.bLiteral1Column-test.a, #test.a, #test.b, #test.c\n TableScan: test projection=None"`,
right: `"Aggregate: groupBy=[[]], aggr=[[SUM(#BinaryExpr-*BinaryExpr--Column-test.bLiteral1Column-test.a AS test.a Multiply Int32(1) Minus test.b), SUM(#BinaryExpr-*BinaryExpr--Column-test.bLiteral1Column-test.a AS test.a Multiply Int32(1) Minus test.b Multiply Int32(1) Plus #test.c)]]\n Projection: #test.a Multiply Int32(1) Minus #test.b AS BinaryExpr-*BinaryExpr--Column-test.bLiteral1Column-test.a, #test.a, #test.b, #test.c\n TableScan: test projection=None"`', datafusion/src/optimizer/common_subexpr_eliminate.rs:647:9
failures:
execution::context::tests::window_partition_by_order_by
optimizer::common_subexpr_eliminate::test::aggregate
optimizer::common_subexpr_eliminate::test::cross_plans_subexpr
optimizer::common_subexpr_eliminate::test::subexpr_in_different_order
optimizer::common_subexpr_eliminate::test::subexpr_in_same_order
optimizer::common_subexpr_eliminate::test::tpch_q1_simplified

Dandandan · 2021-09-20T19:51:02Z

@matthewmturner let us know when you can have a look at the tests, I think only a rebase and update of the tests is what's needed here

matthewmturner · 2021-09-20T20:37:46Z

@matthewmturner let us know when you can have a look at the tests, I think only a rebase and update of the tests is what's needed here

sry, somehow missed the prior comments. will work on this.

matthewmturner · 2021-09-20T21:00:14Z

@alamb @Dandandan

as im relatively new to having to rebase, and ive had some issues in the past with it, i just want to confirm im doing it the right way.

i was going to do git pull -rebase=interactive upstream master and then squash all commits except the first so that all other commits arent included.

can you confirm if this is ok or if there is a better way?

thanks in advance!

alamb · 2021-09-20T21:57:40Z

can you confirm if this is ok or if there is a better way?

I normally do it by:

git fetch apache
git rebase -i origin/apache

where I have previously set apache to be

git remote add apache git@github.com:apache/arrow-datafusion.git

But I suspect there are other workflows that work too

houqp · 2021-09-21T03:56:52Z

@matthewmturner if you are not comfortable doing rebase, you can also do a simple git pull too to pull in the latest master change and resolve the conflicts locally on merge. We do squash merge on every PR, so it doesn't really matter if you do a rebase pull or merge pull in your local branch.

matthewmturner · 2021-09-21T14:03:26Z

@alamb thank you
@houqp i should be ok with doing rebase - i just saw there were a bunch of options for it and wanted to make sure i wasnt messing up any commit history etc. thanks! will have this done shortly

# This is the 1st commit message: Add Display for Expr::BinaryExpr # This is the commit message #2: Update logical_plan/operators tests # This is the commit message #3: rebase and debug display for non binary expr

Update logical_plan/operators tests rebase and debug display for non binary expr Add Display for Expr::BinaryExpr Update logical_plan/operators tests Updating tests Update aggregate display Updating tests without aggregate More tests Working on agg/scalar functions Fix binary_expr in create_name function and attendant tests More tests More tests Doc tests Rebase and update new tests

matthewmturner · 2021-09-21T17:38:03Z

@alamb @houqp im failing on a number of avro related tests now that dont seem impacted by this PR. any thoughts?

alamb · 2021-09-21T17:46:56Z

@alamb @houqp im failing on a number of avro related tests now that dont seem impacted by this PR. any thoughts?

It looks to me like somehow the submodule pin of testing has changed (likely inadvertently) as part of the PR:

matthewmturner · 2021-09-21T17:51:28Z

@alamb @houqp im failing on a number of avro related tests now that dont seem impacted by this PR. any thoughts?
It looks to me like somehow the submodule pin of testing has changed (likely inadvertently) as part of the PR:

gah, guess i messed up the rebase. although im confused how the submodule was impacted - i didnt do anything for that (maybe that was the issue). ill look into it.

matthewmturner · 2021-09-22T15:14:41Z

@alamb all good now i think!

alamb

Thanks @matthewmturner -- something is still messed up with the submodules as this PR shows a change to them.

Let me take a shot at cleaning up the PR and see if I can get rid of them...

matthewmturner · 2021-09-22T15:43:51Z

@alamb ugh sry for the trouble. its not clear to me the right flow i should have performed. my understanding was that i was missing the updates to the submodules (avro in particular) so i updated them to the latest. what is the correct way to update submodules then?

alamb

I pushed a fix in af86048. For the record, here are the commands I used (cargo culted from past experience):

git reset apache/master testing
git restore testing
git reset apache/master parquet-testing
git restore parquet-testing
git commit -a -m 'Restore submodule references from master'
git push

Thanks for sticking with this @matthewmturner -- I plan to merge this PR when the tests have completed

matthewmturner · 2021-09-22T15:47:02Z

I pushed a fix in af86048. For the record, here are the commands I used (cargo culted from past experience):
git reset apache/master testing
git restore testing
git reset apache/master parquet-testing
git restore parquet-testing
git commit -a -m 'Restore submodule references from master'
git push
Thanks for sticking with this @matthewmturner -- I plan to merge this PR when the tests have completed

Ok - thank you for providing that.

alamb · 2021-09-22T17:00:16Z

ugh sry for the trouble. its not clear to me the right flow i should have performed

No worries! I personally find working with submodules with git also quite confusing -- part of the problem is that changes to them are picked up if you do git commit -a -m even if you didn't explicitly want to change the references. 🤷 hopefully it won't hit us again

alamb · 2021-09-22T17:00:40Z

Thanks again @matthewmturner !

matthewmturner · 2021-09-22T18:12:07Z

@alamb np. appreciate your guidance and patience.

github-actionsBot added the datafusion label Sep 6, 2021

houqp requested review from Dandandan and alamb September 6, 2021 04:34

alamb reviewed Sep 9, 2021
View reviewed changes

alamb mentioned this pull request Sep 9, 2021
Improve display of operators in Explain output #347
Closed

alamb reviewed Sep 9, 2021
View reviewed changes

github-actionsBot added ballista development-process Related to development process of DataFusion sql SQL Planner labels Sep 10, 2021

matthewmturner commented Sep 10, 2021
View reviewed changes

matthewmturner force-pushed the improve_operator_display branch from efaba4c to ef99adfCompare September 15, 2021 15:05

matthewmturner commented Sep 15, 2021
View reviewed changes

Dandandan reviewed Sep 15, 2021
View reviewed changes

matthewmturner commented Sep 16, 2021
View reviewed changes

Dandandan approved these changes Sep 17, 2021
View reviewed changes

houqp approved these changes Sep 17, 2021
View reviewed changes

alamb changed the title ~~Improve operator display~~Implement Display for Expr, improve operator displaySep 18, 2021

alamb approved these changes Sep 18, 2021
View reviewed changes

houqp added enhancement New feature or request and removed sql SQL Planner development-process Related to development process of DataFusion python labels Sep 21, 2021

matthewmturner added 2 commits September 21, 2021 10:07

# This is a combination of 3 commits.
ddbfc74
# This is the 1st commit message: Add Display for Expr::BinaryExpr # This is the commit message #2: Update logical_plan/operators tests # This is the commit message #3: rebase and debug display for non binary expr

matthewmturner force-pushed the improve_operator_display branch from 818b1d2 to c856388Compare September 21, 2021 14:39

github-actionsBot added the sql SQL Planner label Sep 21, 2021

Submodule update
5972850

alamb reviewed Sep 22, 2021
View reviewed changes

Restore submodule references from master
af86048

alamb approved these changes Sep 22, 2021
View reviewed changes

alamb merged commit a02d5e1 into apache:masterSep 22, 2021

Uh oh!

Conversation

matthewmturner commented Sep 6, 2021

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

matthewmturner commented Sep 6, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Sep 9, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Dandandan commented Sep 16, 2021

Uh oh!

matthewmturner commented Sep 17, 2021

Uh oh!

Dandandan left a comment

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Sep 18, 2021

Uh oh!

alamb commented Sep 18, 2021

Uh oh!

Dandandan commented Sep 20, 2021

Uh oh!

matthewmturner commented Sep 20, 2021

Uh oh!

matthewmturner commented Sep 20, 2021

Uh oh!

alamb commented Sep 20, 2021

Uh oh!

houqp commented Sep 21, 2021

Uh oh!

matthewmturner commented Sep 21, 2021

Uh oh!

matthewmturner commented Sep 21, 2021

Uh oh!

alamb commented Sep 21, 2021

Uh oh!

matthewmturner commented Sep 21, 2021

Uh oh!

matthewmturner commented Sep 22, 2021

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

matthewmturner commented Sep 22, 2021

Uh oh!

alamb left a comment

matthewmturner commented Sep 6, 2021•
edited
Loading