Skip to content

branch-4.0: [feature](score) support BM25 scoring in inverted index query_v2 #59847 - #61470

Merged
yiguolei merged 4 commits into
apache:branch-4.0from
airborne12:pick/branch-4.0/59847
Mar 19, 2026
Merged

branch-4.0: [feature](score) support BM25 scoring in inverted index query_v2 #59847#61470
yiguolei merged 4 commits into
apache:branch-4.0from
airborne12:pick/branch-4.0/59847

Conversation

@airborne12

@airborne12airborne12 commented Mar 18, 2026

Copy link
Copy Markdown
Member

Proposed changes

Cherry-pick #59847 to branch-4.0.

Original PR: #59847

Further comments

Resolved cherry-pick conflicts caused by directory renames (be/src/storage/ → be/src/olap/, be/src/exprs/ → be/src/vec/exprs/) in branch-4.0. Key conflict resolutions:

  • Updated all include paths for branch-4.0 directory structure
  • Added vectorized:: namespace qualifiers for types (VExprSPtr, VSlotRef, etc.)
  • Added IndexExecContext constructor parameters for segment/iter_opts
  • Disabled nested group search (not available in branch-4.0)
  • Excluded unrelated fields (enable_adjust_conjunct_order_by_cost, enable_paimon_cpp_reader, single_backend_query, cteMaxRecursionDepth) from thrift/SessionVariable

@hello-stephen

hello-stephen commented Mar 18, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12

airborne12 commented Mar 18, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@airborne12
airborne12force-pushed the pick/branch-4.0/59847 branch from d731bf0 to b1c77d7CompareMarch 18, 2026 08:03
@airborne12

airborne12 commented Mar 18, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@airborne12
airborne12force-pushed the pick/branch-4.0/59847 branch from b1c77d7 to 36f6c68CompareMarch 18, 2026 08:11
@airborne12

airborne12 commented Mar 18, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented Mar 18, 2026

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage79.15% (1788/2259)
Line Coverage64.40% (31930/49580)
Region Coverage65.20% (15972/24496)
Branch Coverage55.82% (8503/15234)

@airborne12
[fix](score) Update clucene submodule for BM25 block-level scoring APIs
d03c2f7
Update contrib/clucene from 8b57674 to c51b5cc to include:
- ac9475a: block max WAND algorithm with BM25 similarity
- aef5c9c: Fix GCC -Werror compilation errors
- c51b5cc: Fix GCC -Werror=overloaded-virtual in FieldForMerge
Required for readBlock, getMaxBlockFreq, getMaxBlockNorm,
getLastDocInBlock APIs used by segment_postings.h
@airborne12
[fix](score) Fix CI failures for BM25 scoring PR
61b4970
1. Fix BE compilation error: TermWeight::for_each_pruning hides base
class virtual methods. Added using declaration to bring base class
overloads into scope.
2. Fix doc_set_collector: When context.readers is empty (e.g., AllQuery
for MATCH_ALL_DOCS), for_each_index_segment returns immediately
without iterating. Added fallback to create scorer directly from
weight, which allows AllScorer to work without an IndexReader.
3. Update test_bm25_score error message to match new FE message that
includes "MATCH or SEARCH function".
@airborne12

airborne12 commented Mar 18, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented Mar 18, 2026

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage79.15% (1788/2259)
Line Coverage64.43% (31945/49580)
Region Coverage65.23% (15979/24496)
Branch Coverage55.82% (8504/15234)

@hello-stephen

hello-stephen commented Mar 18, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 33.33% (2/6) 🎉
Increment coverage report
Complete coverage report

@airborne12
[fix](score) Add forward declaration for IndexExecContext in function…
41825bc
…_search.h
function_search.h references vectorized::IndexExecContext but neither
includes vexpr_context.h nor forward-declares the class, causing
compilation failure on CI.
@airborne12

airborne12 commented Mar 19, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented Mar 19, 2026

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage79.15% (1788/2259)
Line Coverage64.42% (31938/49580)
Region Coverage65.26% (15986/24496)
Branch Coverage55.79% (8499/15234)

@hello-stephen

hello-stephen commented Mar 19, 2026

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 26.21% (201/767) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage52.96% (19213/36278)
Line Coverage36.11% (178888/495379)
Region Coverage32.75% (138694/423490)
Branch Coverage33.70% (60226/178695)

@yiguolei
yiguolei merged commit d1a904d into apache:branch-4.0Mar 19, 2026
24 of 28 checks passed
@hello-stephen

hello-stephen commented Mar 19, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 34.72% (268/772) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.34% (25336/35513)
Line Coverage54.02% (267115/494493)
Region Coverage51.45% (220066/427764)
Branch Coverage53.00% (95055/179338)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguoleiyiguoleiyiguolei approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@airborne12@hello-stephen@yiguolei@zzzxl1993