Uh oh!
There was an error while loading. Please reload this page.
branch-4.1: [fix](inverted index) update clucene to reject terms containing an embedded NUL byte #67179 - #67185
Merged
Merged
Conversation
…bedded NUL byte (#67179) Bumps contrib/clucene from abe2b71a1c1 to 08d45e541fe, picking up apache/doris-thirdparty#368. Problem ------- BE dies with SIGSEGV while writing an inverted index when a term contains a raw NUL byte in the middle (JSON \u0000 decodes to this). The crash reproduces on any table with an inverted index over such data, and restarts in a loop. Root cause ---------- In SDocumentWriter.cpp the two paths disagree on what a term is: * addPosition() hashes and dedups with eq(p->term_, term), i.e. the FULL string_view length, so "svc\0A" and "svc\0B" are two distinct postings; * comparePostings() stops at CLUCENE_END_OF_WORD, so the very same two postings compare EQUAL. A run of mutually "equal" but distinct postings makes quickSort()'s partition loop peel exactly one element per call, so the recursion depth becomes O(n) (measured depth/N = 1.000). Each frame is 128 bytes, so ~65k levels exhaust the 8 MiB thread stack. The stack is so far gone that the kernel cannot build a signal frame and force-kills the process (si_code=SI_KERNEL), which is why no handler output or stack trace ever reaches the log. Fix --- doris-thirdparty#368 rejects such terms in addPosition(), before they can enter the postings hash, so the degenerate equal-run can no longer form. The row now fails with a clear CLuceneError, which InvertedIndexColumnWriter::add_document() already converts into INVERTED_INDEX_CLUCENE_ERROR, instead of taking the BE down. Verification ------------ Reproducer: 80k terms shaped "svc\0<i>", untokenized inverted index, 8 MiB stack. before: SIGSEGV, exit 139 after : 80000 rows rejected, exit 0, 0 postings reach the sort Performance on clean data (the guard is one compare + a never-taken branch): perf instructions, 1M tokens : 10,116,997,256 -> 10,118,997,132 = +2.00 instructions/token (+0.020%) index output (.fnm/.frq/.nrm/.tii/.tis) : byte-identical
hello-stephen
commented
Aug 27, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
hello-stephen
commented
Aug 27, 2026
Contributor
run buildall |
hello-stephen
commented
Aug 27, 2026
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
hello-stephen
commented
Aug 27, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Aug 27, 2026
Uh oh!
There was an error while loading. Please reload this page.
ContributorAuthor
PR approved by anyone and no changes requested. |
ContributorAuthor
PR approved by at least one committer and no changes requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #67179