Skip to content

chore: Improvements of fallback info - #2450

Merged
mbutrovich merged 1 commit into
apache:mainfrom
wForget:minor2
Sep 24, 2025
Merged

mbutrovich merged 1 commit into
apache:mainfrom
wForget:minor2

Conversation

@wForget

@wForget wForget commented Sep 24, 2025

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #2449.

Rationale for this change

Some improvements of fallback info:

  • Ignore unsupported info for CometSparkRowToColumnar: CometSparkRowToColumnar is not supported
  • ReusedExchangeExec will break the children extended Infos

What changes are included in this PR?

How are these changes tested?

@codecov-commenter

codecov-commenter commented Sep 24, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.06%. Comparing base (f09f8af) to head (3521ccb).
⚠️ Report is 537 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2450      +/-   ##
============================================
+ Coverage     56.12%   58.06%   +1.93%     
- Complexity      976     1401     +425     
============================================
  Files           119      146      +27     
  Lines         11743    13466    +1723     
  Branches       2251     2342      +91     
============================================
+ Hits           6591     7819    +1228     
- Misses         4012     4413     +401     
- Partials       1140     1234      +94     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mbutrovich

Copy link
Copy Markdown
Contributor

Thanks @wForget! I'm sure this will be helpful for everyone trying to understand how to get more Comet plans running natively. Do you have a before/after example of the output?

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wForget. The changes look reasonable to me. It would be nice to have a test to prevent regressions, but I'm happy to merge these changes without it.

@mbutrovich
mbutrovich merged commit 657cac8 into apache:main Sep 24, 2025
104 checks passed
@wForget

wForget commented Sep 24, 2025

Copy link
Copy Markdown
Member Author

@mbutrovich @andygrove Thank you for your review, and sorry for missing the How are these changes tested? information.

This improvements come from two unexpected checks in the failing CometExpressionSuite.'explain comet' test case:

Set("Comet shuffle is not enabled: spark.comet.exec.shuffle.enabled is not enabled")),

Set(
"Cast from CalendarIntervalType to StringType is not supported",
"Comet shuffle is not enabled: spark.comet.exec.shuffle.enabled is not enabled")),

  1. The first failure occurs because of the extra message CometSparkRowToColumnar is not supported. The error looked like this:
- explain comet *** FAILED *** (720 milliseconds)
  Comet shuffle is not enabled: spark.comet.exec.shuffle.enabled is not enabled
  CometSparkRowToColumnar is not supported != Comet shuffle is not enabled: spark.comet.exec.shuffle.enabled is not enabled (case-insensitive comparison) (CometTestBase.scala:299)
  1. The second failure happens because ReusedExchangeExec exists in the SparkPlan, which causes the expected Comet shuffle is not enabled... message to be missing. The error looked like this:
Cast from CalendarIntervalType to StringType is not supported != Cast from CalendarIntervalType to StringType is not supported
Comet shuffle is not enabled: spark.comet.exec.shuffle.enabled is not enabled

After this fix, these checks work as expected

coderfender pushed a commit to coderfender/datafusion-comet that referenced this pull request Dec 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve fallback info

4 participants