refactor(hash-aggr): treat soft limit as hint

[2010YOUY01]

Which issue does this PR close?

Part of #22710
Depends on #24259

Only the last commit belongs to this PR. I’ll rebase onto main after the dependent PR is merged.

Rationale for this change

For the background of 'soft limit optimization' in aggregation, see:

datafusion/datafusion/physical-plan/src/aggregates/hash_stream.rs

Line 81 in 8cb7c84

/// # Optimization: DISTINCT LIMIT Soft Limit

'soft' in the term means it's only a hint for the operator, it can try to produce outputs less than the 'soft limit' for efficiency, if it ignores this soft limit, there will be no correctness issue, there will be a downstream LimitExec for the final processing.

The original implementation only targets partial/final mode for this optimization: #8038, but somehow this soft limit hint can be pushed to other aggregation modes (like single aggregation)

For simplicity, this PR allows soft limit in all aggregation modes, since

• no correctness issue
• Should have no performance regression, as previously this optimization is only applicable to partial/final aggregation.
• [] TODO for myself: double check the benchmark

Later we can try to apply this optimization more widely, or trying to implement the related optimizer rule more defensively.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[codecov-commenter]

Codecov Report

❌ Patch coverage is 86.91099% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.00%. Comparing base (634f0b3) to head (0fc3e59).
⚠️ Report is 42 commits behind head on main.

Files with missing lines	Patch %	Lines
...sical-plan/src/aggregates/ordered_single_stream.rs	86.74%	49 Missing and 10 partials ⚠️
datafusion/physical-plan/src/aggregates/mod.rs	84.81%	4 Missing and 8 partials ⚠️
.../aggregates/aggregate_hash_table/common_ordered.rs	60.00%	0 Missing and 2 partials ⚠️
...gates/aggregate_hash_table/ordered_single_table.rs	94.44%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #24260 +/- ##
==========================================
- Coverage 81.06% 81.00% -0.06% 
==========================================
Files 1107 1108 +1 Lines 382191 384589 +2398 Branches 382191 384589 +2398 ==========================================
+ Hits 309805 311531 +1726 - Misses 54083 54700 +617 - Partials 18303 18358 +55 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[alamb]

Now that #24259 is merged, perhaps this one is ready for review (along with others marked on #22710 (comment)) "