Change the default search sort tiebreaker to _shard_doc for PIT search - #4378

Merged
LantaoJin merged 11 commits into
opensearch-project:mainfrom
LantaoJin:pr/issues/3064
Oct 1, 2025
Merged

Change the default search sort tiebreaker to _shard_doc for PIT search#4378
LantaoJin merged 11 commits into
opensearch-project:mainfrom
LantaoJin:pr/issues/3064

Conversation

@LantaoJin

@LantaoJinLantaoJin commented Sep 25, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Description

Change the default search sort tiebreaker of PIT search requests to _shard_doc after opensearch-project/OpenSearch#18924 being merged.

Main changes:

  1. Remove sort field _doc in pushDownFilter(). (_doc is a default hits ordering if no specific orders are set)
  2. Explicitly add _shard_doc as sort tiebreaker of PIT search (not 100% sure the current implementation of _shard_doc is an implicit sort in PIT search, seequestion).
  3. Keep the legacy engine unchanged.

Related Issues

Resolves#3064

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@LantaoJin
Change the default search sorting to _shard_doc
8151247
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin added v3.3.0 enhancement New feature or request labels Sep 25, 2025
@LantaoJin

LantaoJin commented Sep 26, 2025

Copy link
Copy Markdown
MemberAuthor

Is it a must have for 3.3.0?

dai-chen
dai-chen reviewed Sep 26, 2025
View reviewed changes

@dai-chendai-chen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?

@LantaoJin

LantaoJin commented Sep 28, 2025•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?

I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.

I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.

@LantaoJin
refactor
c43d0c3
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
typo
fe0efac
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin changed the title Change the default search sorting to _shard_docChange the default search sort tiebreaker to _shard_doc for PIT searchSep 28, 2025
@LantaoJin
Fix ITs
d8140cf
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3064
e885030
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Fix explain IT
7fe7015
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin marked this pull request as ready for review September 29, 2025 11:26
LantaoJin
LantaoJin commented Sep 30, 2025
View reviewed changes
Comment threadopensearch/src/main/java/org/opensearch/sql/opensearch/request/OpenSearchQueryRequest.java
// https://github.com/opensearch-project/OpenSearch/pull/18924#issuecomment-3342365950
if (this.sourceBuilder.sorts() == null
|| this.sourceBuilder.sorts().stream().noneMatch(ShardDocSortBuilder.class::isInstance)) {
this.sourceBuilder.sort(SortBuilders.shardDocSort());

@LantaoJinLantaoJinSep 30, 2025•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is the primary change of this PR.
Currently, the sort tiebreaker _shard_doc is not implicit added in PIT search request. So we explicit sort with SortBuilders.shardDocSort().

Swiddis
Swiddis previously approved these changes Sep 30, 2025
View reviewed changes
@SwiddisSwiddis mentioned this pull request Sep 30, 2025
Closed
8 tasks
@Swiddis

Swiddis commented Sep 30, 2025

Copy link
Copy Markdown
Collaborator

#4420 still fails tests after merge

@LantaoJin

LantaoJin commented Oct 1, 2025

Copy link
Copy Markdown
MemberAuthor

#4420 still fails tests after merge

Seems #4420 contains some changes that are not in this PR. Let me update from upstream and run CI again.

@LantaoJin
Fix conflicts
6f4ba2b
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@RyanL1997

RyanL1997 commented Oct 1, 2025

Copy link
Copy Markdown
Collaborator

https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481

CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)

@RyanL1997

RyanL1997 commented Oct 1, 2025

Copy link
Copy Markdown
Collaborator

this is the same failure on #4344

@RyanL1997RyanL1997 mentioned this pull request Oct 1, 2025
Merged
8 tasks
Swiddis
Swiddis previously approved these changes Oct 1, 2025
View reviewed changes
RyanL1997
RyanL1997 previously approved these changes Oct 1, 2025
View reviewed changes
@LantaoJin
Ignore the incorrect IT
0d93c14
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin dismissed stale reviews from RyanL1997 and Swiddis via 0d93c14October 1, 2025 01:59
@LantaoJin

LantaoJin commented Oct 1, 2025

Copy link
Copy Markdown
MemberAuthor

https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481

CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)

@RyanL1997@Swiddis , confirmed with @yuancu , seems a test bug triggered at Oct 1st. To unblock our release, I have ignored it in the last commit and @yuancu will look into it offline.

@LantaoJin

LantaoJin commented Oct 1, 2025•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

Reminder: Do not backport it to 2.19-dev since it calls an API only provided in OS core 3.3.0.

@LantaoJin
LantaoJin enabled auto-merge (squash) October 1, 2025 02:28
@LantaoJin
LantaoJin merged commit 3e95147 into opensearch-project:mainOct 1, 2025
33 checks passed
@penghuo

penghuo commented Oct 1, 2025•
edited
Loading

Copy link
Copy Markdown
Collaborator

Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?

I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.

I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.

For search=index query, What is performance difference sort[_doc, _id], vs sort[_shard_doc] ?
Should we use _shard_docs as tiebreaker only?

@ahkcsahkcs mentioned this pull request Oct 3, 2025
Merged
asifabashar added a commit to asifabashar/sql that referenced this pull request Oct 10, 2025
@asifabashar
Merge branch 'main-apple' into sync-mirror-update
6505a4e
* main-apple: (218 commits)
Add ignorePrometheus Flag for integTest and docTest (opensearch-project#4442)
Create fab-radar.yml
PPL `fillnull` command enhancement (opensearch-project#4421)
reverting to _doc + _id (opensearch-project#4435)
Support `multisearch` command in calcite (opensearch-project#4332)
Add 3.3 release notes (opensearch-project#4422) (opensearch-project#4423)
[SQL/PPL] Fix the `count(*)` and `dc(field)` to be capped at MAX_INTEGER opensearch-project#4416 (opensearch-project#4418)
Change the default search sort tiebreaker to `_shard_doc` for PIT search (opensearch-project#4378)
[Enhancement] Add error handling for known limitation of sql `JOIN` (opensearch-project#4344)
Bugfix: SQL type mapping for legacy JDBC output (opensearch-project#3613)
Version bump: 3.3 (opensearch-project#4417)
Add max/min eval functions (opensearch-project#4333)
Support time modifiers in search command (opensearch-project#4224)
Fix numbered token bug and make it optional output in patterns command (opensearch-project#4402)
refactor span (opensearch-project#4334)
Move release notes categories (opensearch-project#3818)
[Doc] Enable doctest with Calcite (opensearch-project#4379)
Mod function should return decimal instead of float when handle the operands are decimal literal (opensearch-project#4407)
Scale of decimal literal should always be positive in Calcite (opensearch-project#4401)
Enable Calcite by default and implicit fallback the unsupported commands (opensearch-project#4372)
...
@LantaoJinLantaoJin mentioned this pull request Oct 15, 2025
Merged
8 tasks
LantaoJin added a commit to LantaoJin/search-plugins-sql that referenced this pull request Oct 15, 2025
@LantaoJin
Partially backportopensearch-project#4378
3cd0b23
* Partially backport opensearch-project#4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
@LantaoJinLantaoJin mentioned this pull request Oct 15, 2025
Merged
yuancu pushed a commit that referenced this pull request Oct 16, 2025
@LantaoJin
[Backport 2.19-dev] Partially backport#4378excludingshard_doc(#…
6b8f420
…4573)
* Partially backport #4378
* Partially backport #4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
* fix doctest
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin added the backport-manually Filed a PR to backport manually. label Oct 16, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dai-chendai-chendai-chen left review comments
@yuancuyuancuyuancu approved these changes
@SwiddisSwiddisSwiddis approved these changes
@RyanL1997RyanL1997RyanL1997 left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@vamsimanoharvamsimanoharAwaiting requested review from vamsimanoharvamsimanohar is a code owner
@penghuopenghuoAwaiting requested review from penghuopenghuo is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees

No one assigned

Labels

backport-manuallyFiled a PR to backport manually.enhancementNew feature or requestv3.3.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[FEATURE] Stabilize preferred SQL implicit sorting method

6 participants

@LantaoJin@Swiddis@RyanL1997@penghuo@yuancu@dai-chen
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Change the default search sort tiebreaker to _shard_doc for PIT search - #4378
Merged
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Oct 1, 2025
Merged
Change the default search sort tiebreaker to _shard_doc for PIT search#4378
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Sep 25, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
Change the default search sort tiebreaker of PIT search requests to _shard_doc after opensearch-project/OpenSearch#18924 being merged.
Main changes:
  1. Remove sort field _doc in pushDownFilter(). (_doc is a default hits ordering if no specific orders are set)
  2. Explicitly add _shard_doc as sort tiebreaker of PIT search (not 100% sure the current implementation of _shard_doc is an implicit sort in PIT search, seequestion).
  3. Keep the legacy engine unchanged.
Related Issues
Resolves#3064
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  New PPL command checklist all confirmed.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff or -s.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Change the default search sorting to _shard_doc
8151247
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added

v3.3.0

enhancement
New feature or request
labels
Sep 25, 2025
@LantaoJin
LantaoJin
commented
Sep 26, 2025

Copy link


Copy Markdown

MemberAuthor
Is it a must have for 3.3.0?
dai-chen
dai-chen
reviewed
Sep 26, 2025
View reviewed changes
@dai-chendai-chen
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
@LantaoJin
LantaoJin
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
@LantaoJin
refactor
c43d0c3
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
typo
fe0efac
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Change the default search sorting to _shard_docChange the default search sort tiebreaker to _shard_doc for PIT searchSep 28, 2025
@LantaoJin
Fix ITs
d8140cf
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3064
e885030
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Fix explain IT
7fe7015
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
September 29, 2025 11:26
LantaoJin
LantaoJin
commented
Sep 30, 2025
View reviewed changes
Comment threadopensearch/src/main/java/org/opensearch/sql/opensearch/request/OpenSearchQueryRequest.java
// https://github.com/opensearch-project/OpenSearch/pull/18924#issuecomment-3342365950
if (this.sourceBuilder.sorts() == null
 || this.sourceBuilder.sorts().stream().noneMatch(ShardDocSortBuilder.class::isInstance)) {
 this.sourceBuilder.sort(SortBuilders.shardDocSort());
@LantaoJinLantaoJinSep 30, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is the primary change of this PR.

Currently, the sort tiebreaker _shard_doc is not implicit added in PIT search request. So we explicit sort with SortBuilders.shardDocSort().
Swiddis
Swiddis
previously approved these changes
Sep 30, 2025
View reviewed changes
@SwiddisSwiddis
mentioned this pull request
Sep 30, 2025
 Closed

8 tasks
@Swiddis
Swiddis
commented
Sep 30, 2025

Copy link


Copy Markdown

Collaborator
#4420 still fails tests after merge
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
#4420 still fails tests after merge
Seems #4420 contains some changes that are not in this PR. Let me update from upstream and run CI again.
@LantaoJin
Fix conflicts
6f4ba2b
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
this is the same failure on #4344
@RyanL1997RyanL1997
mentioned this pull request
Oct 1, 2025
 Merged

8 tasks
Swiddis
Swiddis
previously approved these changes
Oct 1, 2025
View reviewed changes
RyanL1997
RyanL1997
previously approved these changes
Oct 1, 2025
View reviewed changes
@LantaoJin
Ignore the incorrect IT
0d93c14
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
dismissed stale reviews from RyanL1997 and Swiddis
via
0d93c14October 1, 2025 01:59
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997@Swiddis , confirmed with @yuancu , seems a test bug triggered at Oct 1st. To unblock our release, I have ignored it in the last commit and @yuancu will look into it offline.
@LantaoJin
LantaoJin
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Reminder: Do not backport it to 2.19-dev since it calls an API only provided in OS core 3.3.0.
@LantaoJin
LantaoJin
enabled auto-merge (squash)
October 1, 2025 02:28
@LantaoJin
LantaoJin
merged commit 3e95147
into
opensearch-project:mainOct 1, 2025

33 checks passed

@penghuo
penghuo
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
For search=index query, What is performance difference sort[_doc, _id], vs sort[_shard_doc] ?

Should we use _shard_docs as tiebreaker only?
@ahkcsahkcs
mentioned this pull request
Oct 3, 2025
 Merged

asifabashar added a commit
to asifabashar/sql
that referenced
this pull request
Oct 10, 2025
@asifabashar
Merge branch 'main-apple' into sync-mirror-update
6505a4e
* main-apple: (218 commits)
Add ignorePrometheus Flag for integTest and docTest (opensearch-project#4442)
Create fab-radar.yml
PPL `fillnull` command enhancement (opensearch-project#4421)
reverting to _doc + _id (opensearch-project#4435)
Support `multisearch` command in calcite (opensearch-project#4332)
Add 3.3 release notes (opensearch-project#4422) (opensearch-project#4423)
[SQL/PPL] Fix the `count(*)` and `dc(field)` to be capped at MAX_INTEGER opensearch-project#4416 (opensearch-project#4418)
Change the default search sort tiebreaker to `_shard_doc` for PIT search (opensearch-project#4378)
[Enhancement] Add error handling for known limitation of sql `JOIN` (opensearch-project#4344)
Bugfix: SQL type mapping for legacy JDBC output (opensearch-project#3613)
Version bump: 3.3 (opensearch-project#4417)
Add max/min eval functions (opensearch-project#4333)
Support time modifiers in search command (opensearch-project#4224)
Fix numbered token bug and make it optional output in patterns command (opensearch-project#4402)
refactor span (opensearch-project#4334)
Move release notes categories (opensearch-project#3818)
[Doc] Enable doctest with Calcite (opensearch-project#4379)
Mod function should return decimal instead of float when handle the operands are decimal literal (opensearch-project#4407)
Scale of decimal literal should always be positive in Calcite (opensearch-project#4401)
Enable Calcite by default and implicit fallback the unsupported commands (opensearch-project#4372)
...
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

8 tasks
LantaoJin added a commit
to LantaoJin/search-plugins-sql
that referenced
this pull request
Oct 15, 2025
@LantaoJin
Partially backportopensearch-project#4378
3cd0b23
* Partially backport opensearch-project#4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

yuancu pushed a commit
that referenced
this pull request
Oct 16, 2025
@LantaoJin
[Backport 2.19-dev] Partially backport#4378excludingshard_doc(#…
6b8f420
…4573)
* Partially backport #4378
* Partially backport #4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
* fix doctest
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added
the 
backport-manually
Filed a PR to backport manually.
label
Oct 16, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@dai-chendai-chendai-chen left review comments
@yuancuyuancuyuancu approved these changes
@SwiddisSwiddisSwiddis approved these changes
@RyanL1997RyanL1997RyanL1997 left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@vamsimanoharvamsimanoharAwaiting requested review from vamsimanoharvamsimanohar is a code owner
@penghuopenghuoAwaiting requested review from penghuopenghuo is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport-manuallyFiled a PR to backport manually.enhancementNew feature or requestv3.3.0

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[FEATURE] Stabilize preferred SQL implicit sorting method


6 participants

@LantaoJin@Swiddis@RyanL1997@penghuo@yuancu@dai-chen

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Change the default search sort tiebreaker to _shard_doc for PIT search - #4378
Merged
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Oct 1, 2025
Merged
Change the default search sort tiebreaker to _shard_doc for PIT search#4378
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Sep 25, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
Change the default search sort tiebreaker of PIT search requests to _shard_doc after opensearch-project/OpenSearch#18924 being merged.
Main changes:
  1. Remove sort field _doc in pushDownFilter(). (_doc is a default hits ordering if no specific orders are set)
  2. Explicitly add _shard_doc as sort tiebreaker of PIT search (not 100% sure the current implementation of _shard_doc is an implicit sort in PIT search, seequestion).
  3. Keep the legacy engine unchanged.
Related Issues
Resolves#3064
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  New PPL command checklist all confirmed.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff or -s.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Change the default search sorting to _shard_doc
8151247
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added

v3.3.0

enhancement
New feature or request
labels
Sep 25, 2025
@LantaoJin
LantaoJin
commented
Sep 26, 2025

Copy link


Copy Markdown

MemberAuthor
Is it a must have for 3.3.0?
dai-chen
dai-chen
reviewed
Sep 26, 2025
View reviewed changes
@dai-chendai-chen
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
@LantaoJin
LantaoJin
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
@LantaoJin
refactor
c43d0c3
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
typo
fe0efac
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Change the default search sorting to _shard_docChange the default search sort tiebreaker to _shard_doc for PIT searchSep 28, 2025
@LantaoJin
Fix ITs
d8140cf
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3064
e885030
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Fix explain IT
7fe7015
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
September 29, 2025 11:26
LantaoJin
LantaoJin
commented
Sep 30, 2025
View reviewed changes
Comment threadopensearch/src/main/java/org/opensearch/sql/opensearch/request/OpenSearchQueryRequest.java
// https://github.com/opensearch-project/OpenSearch/pull/18924#issuecomment-3342365950
if (this.sourceBuilder.sorts() == null
 || this.sourceBuilder.sorts().stream().noneMatch(ShardDocSortBuilder.class::isInstance)) {
 this.sourceBuilder.sort(SortBuilders.shardDocSort());
@LantaoJinLantaoJinSep 30, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is the primary change of this PR.

Currently, the sort tiebreaker _shard_doc is not implicit added in PIT search request. So we explicit sort with SortBuilders.shardDocSort().
Swiddis
Swiddis
previously approved these changes
Sep 30, 2025
View reviewed changes
@SwiddisSwiddis
mentioned this pull request
Sep 30, 2025
 Closed

8 tasks
@Swiddis
Swiddis
commented
Sep 30, 2025

Copy link


Copy Markdown

Collaborator
#4420 still fails tests after merge
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
#4420 still fails tests after merge
Seems #4420 contains some changes that are not in this PR. Let me update from upstream and run CI again.
@LantaoJin
Fix conflicts
6f4ba2b
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
this is the same failure on #4344
@RyanL1997RyanL1997
mentioned this pull request
Oct 1, 2025
 Merged

8 tasks
Swiddis
Swiddis
previously approved these changes
Oct 1, 2025
View reviewed changes
RyanL1997
RyanL1997
previously approved these changes
Oct 1, 2025
View reviewed changes
@LantaoJin
Ignore the incorrect IT
0d93c14
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
dismissed stale reviews from RyanL1997 and Swiddis
via
0d93c14October 1, 2025 01:59
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997@Swiddis , confirmed with @yuancu , seems a test bug triggered at Oct 1st. To unblock our release, I have ignored it in the last commit and @yuancu will look into it offline.
@LantaoJin
LantaoJin
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Reminder: Do not backport it to 2.19-dev since it calls an API only provided in OS core 3.3.0.
@LantaoJin
LantaoJin
enabled auto-merge (squash)
October 1, 2025 02:28
@LantaoJin
LantaoJin
merged commit 3e95147
into
opensearch-project:mainOct 1, 2025

33 checks passed

@penghuo
penghuo
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
For search=index query, What is performance difference sort[_doc, _id], vs sort[_shard_doc] ?

Should we use _shard_docs as tiebreaker only?
@ahkcsahkcs
mentioned this pull request
Oct 3, 2025
 Merged

asifabashar added a commit
to asifabashar/sql
that referenced
this pull request
Oct 10, 2025
@asifabashar
Merge branch 'main-apple' into sync-mirror-update
6505a4e
* main-apple: (218 commits)
Add ignorePrometheus Flag for integTest and docTest (opensearch-project#4442)
Create fab-radar.yml
PPL `fillnull` command enhancement (opensearch-project#4421)
reverting to _doc + _id (opensearch-project#4435)
Support `multisearch` command in calcite (opensearch-project#4332)
Add 3.3 release notes (opensearch-project#4422) (opensearch-project#4423)
[SQL/PPL] Fix the `count(*)` and `dc(field)` to be capped at MAX_INTEGER opensearch-project#4416 (opensearch-project#4418)
Change the default search sort tiebreaker to `_shard_doc` for PIT search (opensearch-project#4378)
[Enhancement] Add error handling for known limitation of sql `JOIN` (opensearch-project#4344)
Bugfix: SQL type mapping for legacy JDBC output (opensearch-project#3613)
Version bump: 3.3 (opensearch-project#4417)
Add max/min eval functions (opensearch-project#4333)
Support time modifiers in search command (opensearch-project#4224)
Fix numbered token bug and make it optional output in patterns command (opensearch-project#4402)
refactor span (opensearch-project#4334)
Move release notes categories (opensearch-project#3818)
[Doc] Enable doctest with Calcite (opensearch-project#4379)
Mod function should return decimal instead of float when handle the operands are decimal literal (opensearch-project#4407)
Scale of decimal literal should always be positive in Calcite (opensearch-project#4401)
Enable Calcite by default and implicit fallback the unsupported commands (opensearch-project#4372)
...
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

8 tasks
LantaoJin added a commit
to LantaoJin/search-plugins-sql
that referenced
this pull request
Oct 15, 2025
@LantaoJin
Partially backportopensearch-project#4378
3cd0b23
* Partially backport opensearch-project#4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

yuancu pushed a commit
that referenced
this pull request
Oct 16, 2025
@LantaoJin
[Backport 2.19-dev] Partially backport#4378excludingshard_doc(#…
6b8f420
…4573)
* Partially backport #4378
* Partially backport #4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
* fix doctest
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added
the 
backport-manually
Filed a PR to backport manually.
label
Oct 16, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@dai-chendai-chendai-chen left review comments
@yuancuyuancuyuancu approved these changes
@SwiddisSwiddisSwiddis approved these changes
@RyanL1997RyanL1997RyanL1997 left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@vamsimanoharvamsimanoharAwaiting requested review from vamsimanoharvamsimanohar is a code owner
@penghuopenghuoAwaiting requested review from penghuopenghuo is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport-manuallyFiled a PR to backport manually.enhancementNew feature or requestv3.3.0

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[FEATURE] Stabilize preferred SQL implicit sorting method


6 participants

@LantaoJin@Swiddis@RyanL1997@penghuo@yuancu@dai-chen

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Change the default search sort tiebreaker to _shard_doc for PIT search - #4378
Merged
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Oct 1, 2025
Merged
Change the default search sort tiebreaker to _shard_doc for PIT search#4378
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Sep 25, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
Change the default search sort tiebreaker of PIT search requests to _shard_doc after opensearch-project/OpenSearch#18924 being merged.
Main changes:
  1. Remove sort field _doc in pushDownFilter(). (_doc is a default hits ordering if no specific orders are set)
  2. Explicitly add _shard_doc as sort tiebreaker of PIT search (not 100% sure the current implementation of _shard_doc is an implicit sort in PIT search, seequestion).
  3. Keep the legacy engine unchanged.
Related Issues
Resolves#3064
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  New PPL command checklist all confirmed.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff or -s.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Change the default search sorting to _shard_doc
8151247
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added

v3.3.0

enhancement
New feature or request
labels
Sep 25, 2025
@LantaoJin
LantaoJin
commented
Sep 26, 2025

Copy link


Copy Markdown

MemberAuthor
Is it a must have for 3.3.0?
dai-chen
dai-chen
reviewed
Sep 26, 2025
View reviewed changes
@dai-chendai-chen
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
@LantaoJin
LantaoJin
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
@LantaoJin
refactor
c43d0c3
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
typo
fe0efac
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Change the default search sorting to _shard_docChange the default search sort tiebreaker to _shard_doc for PIT searchSep 28, 2025
@LantaoJin
Fix ITs
d8140cf
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3064
e885030
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Fix explain IT
7fe7015
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
September 29, 2025 11:26
LantaoJin
LantaoJin
commented
Sep 30, 2025
View reviewed changes
Comment threadopensearch/src/main/java/org/opensearch/sql/opensearch/request/OpenSearchQueryRequest.java
// https://github.com/opensearch-project/OpenSearch/pull/18924#issuecomment-3342365950
if (this.sourceBuilder.sorts() == null
 || this.sourceBuilder.sorts().stream().noneMatch(ShardDocSortBuilder.class::isInstance)) {
 this.sourceBuilder.sort(SortBuilders.shardDocSort());
@LantaoJinLantaoJinSep 30, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is the primary change of this PR.

Currently, the sort tiebreaker _shard_doc is not implicit added in PIT search request. So we explicit sort with SortBuilders.shardDocSort().
Swiddis
Swiddis
previously approved these changes
Sep 30, 2025
View reviewed changes
@SwiddisSwiddis
mentioned this pull request
Sep 30, 2025
 Closed

8 tasks
@Swiddis
Swiddis
commented
Sep 30, 2025

Copy link


Copy Markdown

Collaborator
#4420 still fails tests after merge
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
#4420 still fails tests after merge
Seems #4420 contains some changes that are not in this PR. Let me update from upstream and run CI again.
@LantaoJin
Fix conflicts
6f4ba2b
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
this is the same failure on #4344
@RyanL1997RyanL1997
mentioned this pull request
Oct 1, 2025
 Merged

8 tasks
Swiddis
Swiddis
previously approved these changes
Oct 1, 2025
View reviewed changes
RyanL1997
RyanL1997
previously approved these changes
Oct 1, 2025
View reviewed changes
@LantaoJin
Ignore the incorrect IT
0d93c14
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
dismissed stale reviews from RyanL1997 and Swiddis
via
0d93c14October 1, 2025 01:59
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997@Swiddis , confirmed with @yuancu , seems a test bug triggered at Oct 1st. To unblock our release, I have ignored it in the last commit and @yuancu will look into it offline.
@LantaoJin
LantaoJin
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Reminder: Do not backport it to 2.19-dev since it calls an API only provided in OS core 3.3.0.
@LantaoJin
LantaoJin
enabled auto-merge (squash)
October 1, 2025 02:28
@LantaoJin
LantaoJin
merged commit 3e95147
into
opensearch-project:mainOct 1, 2025

33 checks passed

@penghuo
penghuo
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
For search=index query, What is performance difference sort[_doc, _id], vs sort[_shard_doc] ?

Should we use _shard_docs as tiebreaker only?
@ahkcsahkcs
mentioned this pull request
Oct 3, 2025
 Merged

asifabashar added a commit
to asifabashar/sql
that referenced
this pull request
Oct 10, 2025
@asifabashar
Merge branch 'main-apple' into sync-mirror-update
6505a4e
* main-apple: (218 commits)
Add ignorePrometheus Flag for integTest and docTest (opensearch-project#4442)
Create fab-radar.yml
PPL `fillnull` command enhancement (opensearch-project#4421)
reverting to _doc + _id (opensearch-project#4435)
Support `multisearch` command in calcite (opensearch-project#4332)
Add 3.3 release notes (opensearch-project#4422) (opensearch-project#4423)
[SQL/PPL] Fix the `count(*)` and `dc(field)` to be capped at MAX_INTEGER opensearch-project#4416 (opensearch-project#4418)
Change the default search sort tiebreaker to `_shard_doc` for PIT search (opensearch-project#4378)
[Enhancement] Add error handling for known limitation of sql `JOIN` (opensearch-project#4344)
Bugfix: SQL type mapping for legacy JDBC output (opensearch-project#3613)
Version bump: 3.3 (opensearch-project#4417)
Add max/min eval functions (opensearch-project#4333)
Support time modifiers in search command (opensearch-project#4224)
Fix numbered token bug and make it optional output in patterns command (opensearch-project#4402)
refactor span (opensearch-project#4334)
Move release notes categories (opensearch-project#3818)
[Doc] Enable doctest with Calcite (opensearch-project#4379)
Mod function should return decimal instead of float when handle the operands are decimal literal (opensearch-project#4407)
Scale of decimal literal should always be positive in Calcite (opensearch-project#4401)
Enable Calcite by default and implicit fallback the unsupported commands (opensearch-project#4372)
...
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

8 tasks
LantaoJin added a commit
to LantaoJin/search-plugins-sql
that referenced
this pull request
Oct 15, 2025
@LantaoJin
Partially backportopensearch-project#4378
3cd0b23
* Partially backport opensearch-project#4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

yuancu pushed a commit
that referenced
this pull request
Oct 16, 2025
@LantaoJin
[Backport 2.19-dev] Partially backport#4378excludingshard_doc(#…
6b8f420
…4573)
* Partially backport #4378
* Partially backport #4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
* fix doctest
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added
the 
backport-manually
Filed a PR to backport manually.
label
Oct 16, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@dai-chendai-chendai-chen left review comments
@yuancuyuancuyuancu approved these changes
@SwiddisSwiddisSwiddis approved these changes
@RyanL1997RyanL1997RyanL1997 left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@vamsimanoharvamsimanoharAwaiting requested review from vamsimanoharvamsimanohar is a code owner
@penghuopenghuoAwaiting requested review from penghuopenghuo is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport-manuallyFiled a PR to backport manually.enhancementNew feature or requestv3.3.0

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[FEATURE] Stabilize preferred SQL implicit sorting method


6 participants

@LantaoJin@Swiddis@RyanL1997@penghuo@yuancu@dai-chen

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Change the default search sort tiebreaker to _shard_doc for PIT search - #4378
Merged
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Oct 1, 2025
Merged
Change the default search sort tiebreaker to _shard_doc for PIT search#4378
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Sep 25, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
Change the default search sort tiebreaker of PIT search requests to _shard_doc after opensearch-project/OpenSearch#18924 being merged.
Main changes:
  1. Remove sort field _doc in pushDownFilter(). (_doc is a default hits ordering if no specific orders are set)
  2. Explicitly add _shard_doc as sort tiebreaker of PIT search (not 100% sure the current implementation of _shard_doc is an implicit sort in PIT search, seequestion).
  3. Keep the legacy engine unchanged.
Related Issues
Resolves#3064
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  New PPL command checklist all confirmed.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff or -s.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Change the default search sorting to _shard_doc
8151247
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added

v3.3.0

enhancement
New feature or request
labels
Sep 25, 2025
@LantaoJin
LantaoJin
commented
Sep 26, 2025

Copy link


Copy Markdown

MemberAuthor
Is it a must have for 3.3.0?
dai-chen
dai-chen
reviewed
Sep 26, 2025
View reviewed changes
@dai-chendai-chen
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
@LantaoJin
LantaoJin
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
@LantaoJin
refactor
c43d0c3
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
typo
fe0efac
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Change the default search sorting to _shard_docChange the default search sort tiebreaker to _shard_doc for PIT searchSep 28, 2025
@LantaoJin
Fix ITs
d8140cf
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3064
e885030
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Fix explain IT
7fe7015
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
September 29, 2025 11:26
LantaoJin
LantaoJin
commented
Sep 30, 2025
View reviewed changes
Comment threadopensearch/src/main/java/org/opensearch/sql/opensearch/request/OpenSearchQueryRequest.java
// https://github.com/opensearch-project/OpenSearch/pull/18924#issuecomment-3342365950
if (this.sourceBuilder.sorts() == null
 || this.sourceBuilder.sorts().stream().noneMatch(ShardDocSortBuilder.class::isInstance)) {
 this.sourceBuilder.sort(SortBuilders.shardDocSort());
@LantaoJinLantaoJinSep 30, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is the primary change of this PR.

Currently, the sort tiebreaker _shard_doc is not implicit added in PIT search request. So we explicit sort with SortBuilders.shardDocSort().
Swiddis
Swiddis
previously approved these changes
Sep 30, 2025
View reviewed changes
@SwiddisSwiddis
mentioned this pull request
Sep 30, 2025
 Closed

8 tasks
@Swiddis
Swiddis
commented
Sep 30, 2025

Copy link


Copy Markdown

Collaborator
#4420 still fails tests after merge
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
#4420 still fails tests after merge
Seems #4420 contains some changes that are not in this PR. Let me update from upstream and run CI again.
@LantaoJin
Fix conflicts
6f4ba2b
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
this is the same failure on #4344
@RyanL1997RyanL1997
mentioned this pull request
Oct 1, 2025
 Merged

8 tasks
Swiddis
Swiddis
previously approved these changes
Oct 1, 2025
View reviewed changes
RyanL1997
RyanL1997
previously approved these changes
Oct 1, 2025
View reviewed changes
@LantaoJin
Ignore the incorrect IT
0d93c14
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
dismissed stale reviews from RyanL1997 and Swiddis
via
0d93c14October 1, 2025 01:59
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997@Swiddis , confirmed with @yuancu , seems a test bug triggered at Oct 1st. To unblock our release, I have ignored it in the last commit and @yuancu will look into it offline.
@LantaoJin
LantaoJin
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Reminder: Do not backport it to 2.19-dev since it calls an API only provided in OS core 3.3.0.
@LantaoJin
LantaoJin
enabled auto-merge (squash)
October 1, 2025 02:28
@LantaoJin
LantaoJin
merged commit 3e95147
into
opensearch-project:mainOct 1, 2025

33 checks passed

@penghuo
penghuo
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
For search=index query, What is performance difference sort[_doc, _id], vs sort[_shard_doc] ?

Should we use _shard_docs as tiebreaker only?
@ahkcsahkcs
mentioned this pull request
Oct 3, 2025
 Merged

asifabashar added a commit
to asifabashar/sql
that referenced
this pull request
Oct 10, 2025
@asifabashar
Merge branch 'main-apple' into sync-mirror-update
6505a4e
* main-apple: (218 commits)
Add ignorePrometheus Flag for integTest and docTest (opensearch-project#4442)
Create fab-radar.yml
PPL `fillnull` command enhancement (opensearch-project#4421)
reverting to _doc + _id (opensearch-project#4435)
Support `multisearch` command in calcite (opensearch-project#4332)
Add 3.3 release notes (opensearch-project#4422) (opensearch-project#4423)
[SQL/PPL] Fix the `count(*)` and `dc(field)` to be capped at MAX_INTEGER opensearch-project#4416 (opensearch-project#4418)
Change the default search sort tiebreaker to `_shard_doc` for PIT search (opensearch-project#4378)
[Enhancement] Add error handling for known limitation of sql `JOIN` (opensearch-project#4344)
Bugfix: SQL type mapping for legacy JDBC output (opensearch-project#3613)
Version bump: 3.3 (opensearch-project#4417)
Add max/min eval functions (opensearch-project#4333)
Support time modifiers in search command (opensearch-project#4224)
Fix numbered token bug and make it optional output in patterns command (opensearch-project#4402)
refactor span (opensearch-project#4334)
Move release notes categories (opensearch-project#3818)
[Doc] Enable doctest with Calcite (opensearch-project#4379)
Mod function should return decimal instead of float when handle the operands are decimal literal (opensearch-project#4407)
Scale of decimal literal should always be positive in Calcite (opensearch-project#4401)
Enable Calcite by default and implicit fallback the unsupported commands (opensearch-project#4372)
...
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

8 tasks
LantaoJin added a commit
to LantaoJin/search-plugins-sql
that referenced
this pull request
Oct 15, 2025
@LantaoJin
Partially backportopensearch-project#4378
3cd0b23
* Partially backport opensearch-project#4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

yuancu pushed a commit
that referenced
this pull request
Oct 16, 2025
@LantaoJin
[Backport 2.19-dev] Partially backport#4378excludingshard_doc(#…
6b8f420
…4573)
* Partially backport #4378
* Partially backport #4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
* fix doctest
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added
the 
backport-manually
Filed a PR to backport manually.
label
Oct 16, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@dai-chendai-chendai-chen left review comments
@yuancuyuancuyuancu approved these changes
@SwiddisSwiddisSwiddis approved these changes
@RyanL1997RyanL1997RyanL1997 left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@vamsimanoharvamsimanoharAwaiting requested review from vamsimanoharvamsimanohar is a code owner
@penghuopenghuoAwaiting requested review from penghuopenghuo is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport-manuallyFiled a PR to backport manually.enhancementNew feature or requestv3.3.0

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[FEATURE] Stabilize preferred SQL implicit sorting method


6 participants

@LantaoJin@Swiddis@RyanL1997@penghuo@yuancu@dai-chen

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Change the default search sort tiebreaker to _shard_doc for PIT search - #4378
Merged
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Oct 1, 2025
Merged
Change the default search sort tiebreaker to _shard_doc for PIT search#4378
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Sep 25, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
Change the default search sort tiebreaker of PIT search requests to _shard_doc after opensearch-project/OpenSearch#18924 being merged.
Main changes:
  1. Remove sort field _doc in pushDownFilter(). (_doc is a default hits ordering if no specific orders are set)
  2. Explicitly add _shard_doc as sort tiebreaker of PIT search (not 100% sure the current implementation of _shard_doc is an implicit sort in PIT search, seequestion).
  3. Keep the legacy engine unchanged.
Related Issues
Resolves#3064
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  New PPL command checklist all confirmed.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff or -s.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Change the default search sorting to _shard_doc
8151247
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added

v3.3.0

enhancement
New feature or request
labels
Sep 25, 2025
@LantaoJin
LantaoJin
commented
Sep 26, 2025

Copy link


Copy Markdown

MemberAuthor
Is it a must have for 3.3.0?
dai-chen
dai-chen
reviewed
Sep 26, 2025
View reviewed changes
@dai-chendai-chen
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
@LantaoJin
LantaoJin
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
@LantaoJin
refactor
c43d0c3
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
typo
fe0efac
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Change the default search sorting to _shard_docChange the default search sort tiebreaker to _shard_doc for PIT searchSep 28, 2025
@LantaoJin
Fix ITs
d8140cf
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3064
e885030
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Fix explain IT
7fe7015
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
September 29, 2025 11:26
LantaoJin
LantaoJin
commented
Sep 30, 2025
View reviewed changes
Comment threadopensearch/src/main/java/org/opensearch/sql/opensearch/request/OpenSearchQueryRequest.java
// https://github.com/opensearch-project/OpenSearch/pull/18924#issuecomment-3342365950
if (this.sourceBuilder.sorts() == null
 || this.sourceBuilder.sorts().stream().noneMatch(ShardDocSortBuilder.class::isInstance)) {
 this.sourceBuilder.sort(SortBuilders.shardDocSort());
@LantaoJinLantaoJinSep 30, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is the primary change of this PR.

Currently, the sort tiebreaker _shard_doc is not implicit added in PIT search request. So we explicit sort with SortBuilders.shardDocSort().
Swiddis
Swiddis
previously approved these changes
Sep 30, 2025
View reviewed changes
@SwiddisSwiddis
mentioned this pull request
Sep 30, 2025
 Closed

8 tasks
@Swiddis
Swiddis
commented
Sep 30, 2025

Copy link


Copy Markdown

Collaborator
#4420 still fails tests after merge
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
#4420 still fails tests after merge
Seems #4420 contains some changes that are not in this PR. Let me update from upstream and run CI again.
@LantaoJin
Fix conflicts
6f4ba2b
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
this is the same failure on #4344
@RyanL1997RyanL1997
mentioned this pull request
Oct 1, 2025
 Merged

8 tasks
Swiddis
Swiddis
previously approved these changes
Oct 1, 2025
View reviewed changes
RyanL1997
RyanL1997
previously approved these changes
Oct 1, 2025
View reviewed changes
@LantaoJin
Ignore the incorrect IT
0d93c14
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
dismissed stale reviews from RyanL1997 and Swiddis
via
0d93c14October 1, 2025 01:59
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997@Swiddis , confirmed with @yuancu , seems a test bug triggered at Oct 1st. To unblock our release, I have ignored it in the last commit and @yuancu will look into it offline.
@LantaoJin
LantaoJin
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Reminder: Do not backport it to 2.19-dev since it calls an API only provided in OS core 3.3.0.
@LantaoJin
LantaoJin
enabled auto-merge (squash)
October 1, 2025 02:28
@LantaoJin
LantaoJin
merged commit 3e95147
into
opensearch-project:mainOct 1, 2025

33 checks passed

@penghuo
penghuo
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
For search=index query, What is performance difference sort[_doc, _id], vs sort[_shard_doc] ?

Should we use _shard_docs as tiebreaker only?
@ahkcsahkcs
mentioned this pull request
Oct 3, 2025
 Merged

asifabashar added a commit
to asifabashar/sql
that referenced
this pull request
Oct 10, 2025
@asifabashar
Merge branch 'main-apple' into sync-mirror-update
6505a4e
* main-apple: (218 commits)
Add ignorePrometheus Flag for integTest and docTest (opensearch-project#4442)
Create fab-radar.yml
PPL `fillnull` command enhancement (opensearch-project#4421)
reverting to _doc + _id (opensearch-project#4435)
Support `multisearch` command in calcite (opensearch-project#4332)
Add 3.3 release notes (opensearch-project#4422) (opensearch-project#4423)
[SQL/PPL] Fix the `count(*)` and `dc(field)` to be capped at MAX_INTEGER opensearch-project#4416 (opensearch-project#4418)
Change the default search sort tiebreaker to `_shard_doc` for PIT search (opensearch-project#4378)
[Enhancement] Add error handling for known limitation of sql `JOIN` (opensearch-project#4344)
Bugfix: SQL type mapping for legacy JDBC output (opensearch-project#3613)
Version bump: 3.3 (opensearch-project#4417)
Add max/min eval functions (opensearch-project#4333)
Support time modifiers in search command (opensearch-project#4224)
Fix numbered token bug and make it optional output in patterns command (opensearch-project#4402)
refactor span (opensearch-project#4334)
Move release notes categories (opensearch-project#3818)
[Doc] Enable doctest with Calcite (opensearch-project#4379)
Mod function should return decimal instead of float when handle the operands are decimal literal (opensearch-project#4407)
Scale of decimal literal should always be positive in Calcite (opensearch-project#4401)
Enable Calcite by default and implicit fallback the unsupported commands (opensearch-project#4372)
...
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

8 tasks
LantaoJin added a commit
to LantaoJin/search-plugins-sql
that referenced
this pull request
Oct 15, 2025
@LantaoJin
Partially backportopensearch-project#4378
3cd0b23
* Partially backport opensearch-project#4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

yuancu pushed a commit
that referenced
this pull request
Oct 16, 2025
@LantaoJin
[Backport 2.19-dev] Partially backport#4378excludingshard_doc(#…
6b8f420
…4573)
* Partially backport #4378
* Partially backport #4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
* fix doctest
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added
the 
backport-manually
Filed a PR to backport manually.
label
Oct 16, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@dai-chendai-chendai-chen left review comments
@yuancuyuancuyuancu approved these changes
@SwiddisSwiddisSwiddis approved these changes
@RyanL1997RyanL1997RyanL1997 left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@vamsimanoharvamsimanoharAwaiting requested review from vamsimanoharvamsimanohar is a code owner
@penghuopenghuoAwaiting requested review from penghuopenghuo is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport-manuallyFiled a PR to backport manually.enhancementNew feature or requestv3.3.0

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[FEATURE] Stabilize preferred SQL implicit sorting method


6 participants

@LantaoJin@Swiddis@RyanL1997@penghuo@yuancu@dai-chen

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Change the default search sort tiebreaker to _shard_doc for PIT search - #4378
Merged
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Oct 1, 2025
Merged
Change the default search sort tiebreaker to _shard_doc for PIT search#4378
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Sep 25, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
Change the default search sort tiebreaker of PIT search requests to _shard_doc after opensearch-project/OpenSearch#18924 being merged.
Main changes:
  1. Remove sort field _doc in pushDownFilter(). (_doc is a default hits ordering if no specific orders are set)
  2. Explicitly add _shard_doc as sort tiebreaker of PIT search (not 100% sure the current implementation of _shard_doc is an implicit sort in PIT search, seequestion).
  3. Keep the legacy engine unchanged.
Related Issues
Resolves#3064
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  New PPL command checklist all confirmed.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff or -s.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Change the default search sorting to _shard_doc
8151247
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added

v3.3.0

enhancement
New feature or request
labels
Sep 25, 2025
@LantaoJin
LantaoJin
commented
Sep 26, 2025

Copy link


Copy Markdown

MemberAuthor
Is it a must have for 3.3.0?
dai-chen
dai-chen
reviewed
Sep 26, 2025
View reviewed changes
@dai-chendai-chen
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
@LantaoJin
LantaoJin
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
@LantaoJin
refactor
c43d0c3
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
typo
fe0efac
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Change the default search sorting to _shard_docChange the default search sort tiebreaker to _shard_doc for PIT searchSep 28, 2025
@LantaoJin
Fix ITs
d8140cf
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3064
e885030
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Fix explain IT
7fe7015
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
September 29, 2025 11:26
LantaoJin
LantaoJin
commented
Sep 30, 2025
View reviewed changes
Comment threadopensearch/src/main/java/org/opensearch/sql/opensearch/request/OpenSearchQueryRequest.java
// https://github.com/opensearch-project/OpenSearch/pull/18924#issuecomment-3342365950
if (this.sourceBuilder.sorts() == null
 || this.sourceBuilder.sorts().stream().noneMatch(ShardDocSortBuilder.class::isInstance)) {
 this.sourceBuilder.sort(SortBuilders.shardDocSort());
@LantaoJinLantaoJinSep 30, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is the primary change of this PR.

Currently, the sort tiebreaker _shard_doc is not implicit added in PIT search request. So we explicit sort with SortBuilders.shardDocSort().
Swiddis
Swiddis
previously approved these changes
Sep 30, 2025
View reviewed changes
@SwiddisSwiddis
mentioned this pull request
Sep 30, 2025
 Closed

8 tasks
@Swiddis
Swiddis
commented
Sep 30, 2025

Copy link


Copy Markdown

Collaborator
#4420 still fails tests after merge
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
#4420 still fails tests after merge
Seems #4420 contains some changes that are not in this PR. Let me update from upstream and run CI again.
@LantaoJin
Fix conflicts
6f4ba2b
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
this is the same failure on #4344
@RyanL1997RyanL1997
mentioned this pull request
Oct 1, 2025
 Merged

8 tasks
Swiddis
Swiddis
previously approved these changes
Oct 1, 2025
View reviewed changes
RyanL1997
RyanL1997
previously approved these changes
Oct 1, 2025
View reviewed changes
@LantaoJin
Ignore the incorrect IT
0d93c14
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
dismissed stale reviews from RyanL1997 and Swiddis
via
0d93c14October 1, 2025 01:59
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997@Swiddis , confirmed with @yuancu , seems a test bug triggered at Oct 1st. To unblock our release, I have ignored it in the last commit and @yuancu will look into it offline.
@LantaoJin
LantaoJin
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Reminder: Do not backport it to 2.19-dev since it calls an API only provided in OS core 3.3.0.
@LantaoJin
LantaoJin
enabled auto-merge (squash)
October 1, 2025 02:28
@LantaoJin
LantaoJin
merged commit 3e95147
into
opensearch-project:mainOct 1, 2025

33 checks passed

@penghuo
penghuo
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
For search=index query, What is performance difference sort[_doc, _id], vs sort[_shard_doc] ?

Should we use _shard_docs as tiebreaker only?
@ahkcsahkcs
mentioned this pull request
Oct 3, 2025
 Merged

asifabashar added a commit
to asifabashar/sql
that referenced
this pull request
Oct 10, 2025
@asifabashar
Merge branch 'main-apple' into sync-mirror-update
6505a4e
* main-apple: (218 commits)
Add ignorePrometheus Flag for integTest and docTest (opensearch-project#4442)
Create fab-radar.yml
PPL `fillnull` command enhancement (opensearch-project#4421)
reverting to _doc + _id (opensearch-project#4435)
Support `multisearch` command in calcite (opensearch-project#4332)
Add 3.3 release notes (opensearch-project#4422) (opensearch-project#4423)
[SQL/PPL] Fix the `count(*)` and `dc(field)` to be capped at MAX_INTEGER opensearch-project#4416 (opensearch-project#4418)
Change the default search sort tiebreaker to `_shard_doc` for PIT search (opensearch-project#4378)
[Enhancement] Add error handling for known limitation of sql `JOIN` (opensearch-project#4344)
Bugfix: SQL type mapping for legacy JDBC output (opensearch-project#3613)
Version bump: 3.3 (opensearch-project#4417)
Add max/min eval functions (opensearch-project#4333)
Support time modifiers in search command (opensearch-project#4224)
Fix numbered token bug and make it optional output in patterns command (opensearch-project#4402)
refactor span (opensearch-project#4334)
Move release notes categories (opensearch-project#3818)
[Doc] Enable doctest with Calcite (opensearch-project#4379)
Mod function should return decimal instead of float when handle the operands are decimal literal (opensearch-project#4407)
Scale of decimal literal should always be positive in Calcite (opensearch-project#4401)
Enable Calcite by default and implicit fallback the unsupported commands (opensearch-project#4372)
...
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

8 tasks
LantaoJin added a commit
to LantaoJin/search-plugins-sql
that referenced
this pull request
Oct 15, 2025
@LantaoJin
Partially backportopensearch-project#4378
3cd0b23
* Partially backport opensearch-project#4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

yuancu pushed a commit
that referenced
this pull request
Oct 16, 2025
@LantaoJin
[Backport 2.19-dev] Partially backport#4378excludingshard_doc(#…
6b8f420
…4573)
* Partially backport #4378
* Partially backport #4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
* fix doctest
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added
the 
backport-manually
Filed a PR to backport manually.
label
Oct 16, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@dai-chendai-chendai-chen left review comments
@yuancuyuancuyuancu approved these changes
@SwiddisSwiddisSwiddis approved these changes
@RyanL1997RyanL1997RyanL1997 left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@vamsimanoharvamsimanoharAwaiting requested review from vamsimanoharvamsimanohar is a code owner
@penghuopenghuoAwaiting requested review from penghuopenghuo is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport-manuallyFiled a PR to backport manually.enhancementNew feature or requestv3.3.0

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[FEATURE] Stabilize preferred SQL implicit sorting method


6 participants

@LantaoJin@Swiddis@RyanL1997@penghuo@yuancu@dai-chen

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Change the default search sort tiebreaker to _shard_doc for PIT search - #4378
Merged
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Oct 1, 2025
Merged
Change the default search sort tiebreaker to _shard_doc for PIT search#4378
LantaoJin merged 11 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3064
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Sep 25, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
Change the default search sort tiebreaker of PIT search requests to _shard_doc after opensearch-project/OpenSearch#18924 being merged.
Main changes:
  1. Remove sort field _doc in pushDownFilter(). (_doc is a default hits ordering if no specific orders are set)
  2. Explicitly add _shard_doc as sort tiebreaker of PIT search (not 100% sure the current implementation of _shard_doc is an implicit sort in PIT search, seequestion).
  3. Keep the legacy engine unchanged.
Related Issues
Resolves#3064
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  New PPL command checklist all confirmed.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff or -s.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Change the default search sorting to _shard_doc
8151247
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added

v3.3.0

enhancement
New feature or request
labels
Sep 25, 2025
@LantaoJin
LantaoJin
commented
Sep 26, 2025

Copy link


Copy Markdown

MemberAuthor
Is it a must have for 3.3.0?
dai-chen
dai-chen
reviewed
Sep 26, 2025
View reviewed changes
@dai-chendai-chen
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
@LantaoJin
LantaoJin
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
@LantaoJin
refactor
c43d0c3
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
typo
fe0efac
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Change the default search sorting to _shard_docChange the default search sort tiebreaker to _shard_doc for PIT searchSep 28, 2025
@LantaoJin
Fix ITs
d8140cf
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3064
e885030
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Fix explain IT
7fe7015
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
September 29, 2025 11:26
LantaoJin
LantaoJin
commented
Sep 30, 2025
View reviewed changes
Comment threadopensearch/src/main/java/org/opensearch/sql/opensearch/request/OpenSearchQueryRequest.java
// https://github.com/opensearch-project/OpenSearch/pull/18924#issuecomment-3342365950
if (this.sourceBuilder.sorts() == null
 || this.sourceBuilder.sorts().stream().noneMatch(ShardDocSortBuilder.class::isInstance)) {
 this.sourceBuilder.sort(SortBuilders.shardDocSort());
@LantaoJinLantaoJinSep 30, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is the primary change of this PR.

Currently, the sort tiebreaker _shard_doc is not implicit added in PIT search request. So we explicit sort with SortBuilders.shardDocSort().
Swiddis
Swiddis
previously approved these changes
Sep 30, 2025
View reviewed changes
@SwiddisSwiddis
mentioned this pull request
Sep 30, 2025
 Closed

8 tasks
@Swiddis
Swiddis
commented
Sep 30, 2025

Copy link


Copy Markdown

Collaborator
#4420 still fails tests after merge
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
#4420 still fails tests after merge
Seems #4420 contains some changes that are not in this PR. Let me update from upstream and run CI again.
@LantaoJin
Fix conflicts
6f4ba2b
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997
RyanL1997
commented
Oct 1, 2025

Copy link


Copy Markdown

Collaborator
this is the same failure on #4344
@RyanL1997RyanL1997
mentioned this pull request
Oct 1, 2025
 Merged

8 tasks
Swiddis
Swiddis
previously approved these changes
Oct 1, 2025
View reviewed changes
RyanL1997
RyanL1997
previously approved these changes
Oct 1, 2025
View reviewed changes
@LantaoJin
Ignore the incorrect IT
0d93c14
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
dismissed stale reviews from RyanL1997 and Swiddis
via
0d93c14October 1, 2025 01:59
@LantaoJin
LantaoJin
commented
Oct 1, 2025

Copy link


Copy Markdown

MemberAuthor
https://github.com/opensearch-project/sql/actions/runs/18148240816/job/51654012596?pr=4378#step:6:481
CalciteNoPushdownIT > org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions FAILED
REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests 'org.opensearch.sql.calcite.remote.CalciteSearchCommandIT.testSearchWithComplexChainedExpressions' -Dtests.seed=FD9C48B5A496F6A0 -Dtests.security.manager=false -Dtests.locale=ff-SN -Dtests.timezone=America/Santo_Domingo -Druntime.java=21
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([FD9C48B5A496F6A0:5AB6F276A0DA7BAD]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at org.opensearch.sql.util.MatcherUtils.verifyNumOfRows(MatcherUtils.java:188)
at org.opensearch.sql.ppl.SearchCommandIT.testSearchWithComplexChainedExpressions(SearchCommandIT.java:1316)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:1583)
@RyanL1997@Swiddis , confirmed with @yuancu , seems a test bug triggered at Oct 1st. To unblock our release, I have ignored it in the last commit and @yuancu will look into it offline.
@LantaoJin
LantaoJin
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Reminder: Do not backport it to 2.19-dev since it calls an API only provided in OS core 3.3.0.
@LantaoJin
LantaoJin
enabled auto-merge (squash)
October 1, 2025 02:28
@LantaoJin
LantaoJin
merged commit 3e95147
into
opensearch-project:mainOct 1, 2025

33 checks passed

@penghuo
penghuo
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
Just to confirm: Do all PIT search requests add _shard_doc as tiebreaker field implicitly if no sort provided?
I think all PIT search requests should add _shard_doc as tiebreaker, or the _score will used as implicit sort field in response. But this PR doesn't check all code base, it just changes the all places which use _doc + _id as tiebreaker to _shard_doc.
I checked PR of _shard_doc, seems only PIT search requests can sort by _shard_doc. Non-PIT search requests sort by _shard_doc throws exception. I will refactor this PR to force v2 and v3 only. The v1 (legacy) code is out of my knowledge, I won't touch them in this PR.
For search=index query, What is performance difference sort[_doc, _id], vs sort[_shard_doc] ?

Should we use _shard_docs as tiebreaker only?
@ahkcsahkcs
mentioned this pull request
Oct 3, 2025
 Merged

asifabashar added a commit
to asifabashar/sql
that referenced
this pull request
Oct 10, 2025
@asifabashar
Merge branch 'main-apple' into sync-mirror-update
6505a4e
* main-apple: (218 commits)
Add ignorePrometheus Flag for integTest and docTest (opensearch-project#4442)
Create fab-radar.yml
PPL `fillnull` command enhancement (opensearch-project#4421)
reverting to _doc + _id (opensearch-project#4435)
Support `multisearch` command in calcite (opensearch-project#4332)
Add 3.3 release notes (opensearch-project#4422) (opensearch-project#4423)
[SQL/PPL] Fix the `count(*)` and `dc(field)` to be capped at MAX_INTEGER opensearch-project#4416 (opensearch-project#4418)
Change the default search sort tiebreaker to `_shard_doc` for PIT search (opensearch-project#4378)
[Enhancement] Add error handling for known limitation of sql `JOIN` (opensearch-project#4344)
Bugfix: SQL type mapping for legacy JDBC output (opensearch-project#3613)
Version bump: 3.3 (opensearch-project#4417)
Add max/min eval functions (opensearch-project#4333)
Support time modifiers in search command (opensearch-project#4224)
Fix numbered token bug and make it optional output in patterns command (opensearch-project#4402)
refactor span (opensearch-project#4334)
Move release notes categories (opensearch-project#3818)
[Doc] Enable doctest with Calcite (opensearch-project#4379)
Mod function should return decimal instead of float when handle the operands are decimal literal (opensearch-project#4407)
Scale of decimal literal should always be positive in Calcite (opensearch-project#4401)
Enable Calcite by default and implicit fallback the unsupported commands (opensearch-project#4372)
...
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

8 tasks
LantaoJin added a commit
to LantaoJin/search-plugins-sql
that referenced
this pull request
Oct 15, 2025
@LantaoJin
Partially backportopensearch-project#4378
3cd0b23
* Partially backport opensearch-project#4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
@LantaoJinLantaoJin
mentioned this pull request
Oct 15, 2025
 Merged

yuancu pushed a commit
that referenced
this pull request
Oct 16, 2025
@LantaoJin
[Backport 2.19-dev] Partially backport#4378excludingshard_doc(#…
6b8f420
…4573)
* Partially backport #4378
* Partially backport #4378
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
(cherry picked from commit 3e95147)
* fix doctest
Signed-off-by: Lantao Jin <ltjin@amazon.com>
---------
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
added
the 
backport-manually
Filed a PR to backport manually.
label
Oct 16, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@dai-chendai-chendai-chen left review comments
@yuancuyuancuyuancu approved these changes
@SwiddisSwiddisSwiddis approved these changes
@RyanL1997RyanL1997RyanL1997 left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@vamsimanoharvamsimanoharAwaiting requested review from vamsimanoharvamsimanohar is a code owner
@penghuopenghuoAwaiting requested review from penghuopenghuo is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport-manuallyFiled a PR to backport manually.enhancementNew feature or requestv3.3.0

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[FEATURE] Stabilize preferred SQL implicit sorting method


6 participants

@LantaoJin@Swiddis@RyanL1997@penghuo@yuancu@dai-chen