Skip to content

[fix](ann-index) skip ANN index build for segments with insufficient training rows - #64955

Open
yx-keith wants to merge 1 commit into
apache:branch-4.0from
yx-keith:fix-ann-train-min-rows-4.0
Open

[fix](ann-index) skip ANN index build for segments with insufficient training rows#64955
yx-keith wants to merge 1 commit into
apache:branch-4.0from
yx-keith:fix-ann-train-min-rows-4.0

Conversation

@yx-keith

@yx-keithyx-keith commented Jun 29, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@github-actions

github-actionsBot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Possible file(s) that should be tracked in LFS detected: 🚨

The following file(s) exceeds the file size limit: 1048576 bytes, as set in the .yml configuration files:

  • be/dict/pinyin/polyphone.txt

Consider using git-lfs to manage large files.

@github-actionsgithub-actionsBot added the lfs-detected! Warning Label for use when LFS is detected in the commits of a Pull Request label Jun 29, 2026
@yx-keith
yx-keith changed the base branch from master to branch-4.0June 29, 2026 09:59
@yx-keithyx-keith changed the title Fix ann train min rows 4.0[fix](ann-index) skip ANN index build for segments with insufficient training rows)Jun 29, 2026
@yx-keithyx-keith changed the title [fix](ann-index) skip ANN index build for segments with insufficient training rows)[fix](ann-index) skip ANN index build for segments with insufficient training rowsJun 29, 2026
@yx-keith
[fix](ann-index) skip ANN index build for segments with insufficient …
ee78f08
…training rows
When a segment has fewer rows than an IVF index needs (nlist) or a PQ
quantizer needs (2^pq_nbits * 100), FAISS k-means training fails with
'nx >= k', which aborts segment finalize and fails the load. With group
commit this makes WAL replay keep failing on the same segment.
Skip building the ANN index for such segments instead of failing the
load. The query path already falls back to brute-force TopN for segments
without an ANN index. Also avoid retraining on the sub-chunk remainder
in finish() once a full chunk has already trained the index, which would
otherwise hit the same 'nx >= k' error for segments larger than one
build chunk.
This backports the crash-fix part of apache#64082 to branch-4.0.
@yx-keith
yx-keithforce-pushed the fix-ann-train-min-rows-4.0 branch from b0af7aa to ee78f08CompareJune 30, 2026 03:15
@github-actionsgithub-actionsBot removed the lfs-detected! Warning Label for use when LFS is detected in the commits of a Pull Request label Jun 30, 2026
@yx-keith

yx-keith commented Jul 6, 2026

Copy link
Copy Markdown
ContributorAuthor

run buildall

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01liaoxin01Awaiting requested review from liaoxin01
@luwei16luwei16Awaiting requested review from luwei16
@gavinchougavinchouAwaiting requested review from gavinchou
@deardengdeardengAwaiting requested review from deardeng
@morningmanmorningmanAwaiting requested review from morningmanmorningman is a code owner
@Gabriel39Gabriel39Awaiting requested review from Gabriel39
@eldenmooneldenmoonAwaiting requested review from eldenmoon
@csun5285csun5285Awaiting requested review from csun5285
@zclllyybbzclllyybbAwaiting requested review from zclllyybb
@BiteTheDDDDtBiteTheDDDDtAwaiting requested review from BiteTheDDDDt
@CalvinKirsCalvinKirsAwaiting requested review from CalvinKirs

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@yx-keith@hello-stephen