Skip to content

branch-4.1:[improvement](fe) Infer set operator distinct by NDV #64618 - #66859

Merged
yiguolei merged 3 commits into
apache:branch-4.1from
feiniaofeiafei:pick-infer-set-op-distinct
Aug 20, 2026
Merged

branch-4.1:[improvement](fe) Infer set operator distinct by NDV #64618#66859
yiguolei merged 3 commits into
apache:branch-4.1from
feiniaofeiafei:pick-infer-set-op-distinct

Conversation

@feiniaofeiafei

@feiniaofeiafeifeiniaofeiafei commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

picked from #64618

@feiniaofeiafei
[improvement](fe) Infer set operator distinct by NDV (apache#64618)
4a430ba
Problem Summary:
`InferSetOperatorDistinct` previously relied on a cost-based rewrite job
to decide whether to add local distinct aggregates under set operations.
This introduces extra optimizer cost.
This PR changes `InferSetOperatorDistinct` to run as a normal top-down
rewrite rule and makes the rule decide whether to generate local
distinct aggregates by child NDV/cardinality statistics. The NDV
heuristic follows the existing logic in `EagerAggRewriter#checkStats`,
but is kept local to `InferSetOperatorDistinct` because set-operation
local distinct inference and eager aggregation pushdown have different
optimization boundaries.
Each child of a DISTINCT set operation is judged independently, so one
child can get a local distinct aggregate while another child can remain
unchanged when its NDV indicates poor deduplication benefit.
@feiniaofeiafei
fix test shape
c08924d
@feiniaofeiafei

feiniaofeiafei commented Aug 18, 2026

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@hello-stephen

hello-stephen commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@feiniaofeiafei

feiniaofeiafei commented Aug 18, 2026

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@feiniaofeiafei

feiniaofeiafei commented Aug 19, 2026

Copy link
Copy Markdown
CollaboratorAuthor

run external

@feiniaofeiafei

feiniaofeiafei commented Aug 19, 2026

Copy link
Copy Markdown
CollaboratorAuthor

run feut

@feiniaofeiafei

feiniaofeiafei commented Aug 19, 2026

Copy link
Copy Markdown
CollaboratorAuthor

run external

@hello-stephen

hello-stephen commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 54.84% (34/62) 🎉
Increment coverage report
Complete coverage report

@feiniaofeiafei

feiniaofeiafei commented Aug 19, 2026

Copy link
Copy Markdown
CollaboratorAuthor

run buildall

@yiguolei
yiguolei merged commit 3258380 into apache:branch-4.1Aug 20, 2026
28 of 32 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguoleiyiguoleiyiguolei approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@feiniaofeiafei@hello-stephen@yiguolei