Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance by LantaoJin · Pull Request #4569 · opensearch-project/sql

LantaoJin · 2025-10-15T08:48:44Z

Description

Before #4378, the sort in PIT search is
case 1: if no sort field specified, sort by _doc + _id (+ means "then"). (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)
case 3: if sort fields specified and query contains a filter, sort by _doc. (❎ paged results could miss or duplicate hits)

#4378 added the _shard_doc as sort tiebreaker with
case 1: if no sort field specified, sort by _shard_doc. (❎ performance regression)
case 2: if sort fields specified, sort by fields + _shard_doc.(❎ lower performance on low cardinality field)

#4435 found performance regression in case 1 and partially revert the changes to
case 1: if no sort field specified, sort by _doc + _id. (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)

After this PR, we change the sort in PIT search to
case 1: if no sort field specified, sort by _doc + _shard_doc. ✅
case 2: if sort fields specified, sort by fields + _doc + _shard_doc.✅

RCA of performance regression:
_shard_doc is not a stored field in index which will be generated in runtime when comparison. Computing _shard_doc per document is a high cost operation. But sorting by _doc then _shard_doc only generates _shard_doc when the _doc values are conflicted.
Even in the case of user specified sort fields, we should sort by fields then _doc then _shard_doc to reduce the computing of _shard_doc. For example, if the sort field is a low cardinality field, e.g. gender, sorting by gender then _doc then _shard_doc generates _shard_doc for comparison only if values of gender and _doc are both conflicted.

This PR is no needed to backport to 2.19-dev since shard_doc feature is only available since OS 3.3.0

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Swiddis · 2025-10-15T14:59:26Z

- // Workaround to preserve sort location more exactly,
- // see https://github.com/opensearch-project/sql/pull/3061
- this.sourceBuilder.sort(METADATA_FIELD_ID, ASC);
+ this.sourceBuilder.sort(SortBuilders.shardDocSort());


Does it matter if we duplicate fields in the sorting list? We could simplify/remove the below else logic by just always appending this, I would expect Lucene to optimize it in the background but I haven't measured it.

not sure will Lucene optimize duplicated fields or _doc in sorting, but for sure the duplicated _shard_doc is not allowed in OpenSearch Core. It is no harmful for restricted checker here.

anasalkouz · 2025-10-15T15:10:51Z

Can you share the performance benchmark for the 3 approaches?

LantaoJin · 2025-10-15T15:32:28Z

Can you share the performance benchmark for the 3 approaches?

I haven't run the benchmark, the RCA was made by reading the code of Luence and OS shard_doc feature.

The performance of _doc then _shard_doc is same as _doc then _id, provided by @ahkcs on Oct 1st. (the case 1)

For case 2, the current fields + _doc + _shard_doc is an further optimization upon fields + _shard_doc based on above benchmark result with inference.

Will rerun some benchmark to double confirm.

commit cba8d02 Author: Tomoyuki MORITA <moritato@amazon.com> Date: Wed Oct 15 13:08:05 2025 -0700 Add MAP_APPEND internal function to Calcite PPL (opensearch-project#4515) * Add MAP_APPEND internal function to Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comment Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Rebase and fix IT issue Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 3388dc7 Author: Lantao Jin <ltjin@amazon.com> Date: Thu Oct 16 01:45:29 2025 +0800 Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance (opensearch-project#4569) * Use _shard_doc as sort tiebreaker Signed-off-by: Lantao Jin <ltjin@amazon.com> * _doc as a part of tie-breaker have better performance Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 5630119 Author: qianheng <qianheng@amazon.com> Date: Wed Oct 15 16:40:41 2025 +0800 Fix sort push down into agg after project already pushed (opensearch-project#4546) * Fix sort push down into agg Signed-off-by: Heng Qian <qianheng@amazon.com> * Change some json files to yaml format Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 1e62fba Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 17:20:38 2025 -0700 Fix JsonExtractAllFunctionIT failure (opensearch-project#4556) Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 02ee33e Author: Kai Huang <105710027+ahkcs@users.noreply.github.com> Date: Tue Oct 14 14:28:53 2025 -0700 Add more examples to the `where` command doc (opensearch-project#4457) Co-authored-by: Manasvini B S <manasvis@amazon.com> commit 0b7e86c Author: Jialiang Liang <jiallian@amazon.com> Date: Tue Oct 14 10:46:01 2025 -0700 [Enhancement] Error handling for illegal character usage in java regex named capture group (opensearch-project#4434) Co-authored-by: Simeon Widdis <sawiddis@amazon.com> commit 9c97cfb Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 08:36:43 2025 -0700 Add JSON_EXTRACT_ALL internal function for Calcite PPL (opensearch-project#4489) * Add JSON_EXTRACT_ALL internal function for Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comments Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 89dbc31 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:24:52 2025 +0800 Check server status before starting Prometheus (opensearch-project#4537) * Check server status before starting Prometheus Signed-off-by: Lantao Jin <ltjin@amazon.com> * Change to func call Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix doc Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit fe62472 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:10:27 2025 +0800 Update request builder after pushdown sort into agg buckets (opensearch-project#4541) Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 42a415f Author: qianheng <qianheng@amazon.com> Date: Tue Oct 14 17:42:45 2025 +0800 Including metadata fields type when doing agg/filter script push down (opensearch-project#4522) * Including metadata fields type when doing agg/filter script push down Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 8de0386 Author: Xinyuan Lu <xinyual@amazon.com> Date: Tue Oct 14 16:41:08 2025 +0800 Fix percentile bug (opensearch-project#4539) * fix percentile bug Signed-off-by: xinyual <xinyual@amazon.com> * add IT Signed-off-by: xinyual <xinyual@amazon.com> * optimize it Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com> commit de2fdc8 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 12:29:03 2025 +0800 [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited (opensearch-project#4534) * [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix doctest Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix conflicts Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 977b7ab Author: Simeon Widdis <sawiddis@gmail.com> Date: Mon Oct 13 20:23:10 2025 -0700 Update stalled action (opensearch-project#4485) commit fddbb70 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 10:23:12 2025 +0800 Add configurable sytem limitations for `subsearch` and `join` command (opensearch-project#4501) * Add configurable sytem limitations for subsearch and join command Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * remove rollback in doc Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

LantaoJin · 2025-10-15T08:48:44Z

Description

Before #4378, the sort in PIT search is
case 1: if no sort field specified, sort by _doc + _id (+ means "then"). (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)
case 3: if sort fields specified and query contains a filter, sort by _doc. (❎ paged results could miss or duplicate hits)

#4378 added the _shard_doc as sort tiebreaker with
case 1: if no sort field specified, sort by _shard_doc. (❎ performance regression)
case 2: if sort fields specified, sort by fields + _shard_doc.(❎ lower performance on low cardinality field)

#4435 found performance regression in case 1 and partially revert the changes to
case 1: if no sort field specified, sort by _doc + _id. (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)

After this PR, we change the sort in PIT search to
case 1: if no sort field specified, sort by _doc + _shard_doc. ✅
case 2: if sort fields specified, sort by fields + _doc + _shard_doc.✅

RCA of performance regression:
_shard_doc is not a stored field in index which will be generated in runtime when comparison. Computing _shard_doc per document is a high cost operation. But sorting by _doc then _shard_doc only generates _shard_doc when the _doc values are conflicted.
Even in the case of user specified sort fields, we should sort by fields then _doc then _shard_doc to reduce the computing of _shard_doc. For example, if the sort field is a low cardinality field, e.g. gender, sorting by gender then _doc then _shard_doc generates _shard_doc for comparison only if values of gender and _doc are both conflicted.

This PR is no needed to backport to 2.19-dev since shard_doc feature is only available since OS 3.3.0

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Swiddis · 2025-10-15T14:59:26Z

- // Workaround to preserve sort location more exactly,
- // see https://github.com/opensearch-project/sql/pull/3061
- this.sourceBuilder.sort(METADATA_FIELD_ID, ASC);
+ this.sourceBuilder.sort(SortBuilders.shardDocSort());


Does it matter if we duplicate fields in the sorting list? We could simplify/remove the below else logic by just always appending this, I would expect Lucene to optimize it in the background but I haven't measured it.

not sure will Lucene optimize duplicated fields or _doc in sorting, but for sure the duplicated _shard_doc is not allowed in OpenSearch Core. It is no harmful for restricted checker here.

anasalkouz · 2025-10-15T15:10:51Z

Can you share the performance benchmark for the 3 approaches?

LantaoJin · 2025-10-15T15:32:28Z

Can you share the performance benchmark for the 3 approaches?

I haven't run the benchmark, the RCA was made by reading the code of Luence and OS shard_doc feature.

The performance of _doc then _shard_doc is same as _doc then _id, provided by @ahkcs on Oct 1st. (the case 1)

For case 2, the current fields + _doc + _shard_doc is an further optimization upon fields + _shard_doc based on above benchmark result with inference.

Will rerun some benchmark to double confirm.

commit cba8d02 Author: Tomoyuki MORITA <moritato@amazon.com> Date: Wed Oct 15 13:08:05 2025 -0700 Add MAP_APPEND internal function to Calcite PPL (opensearch-project#4515) * Add MAP_APPEND internal function to Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comment Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Rebase and fix IT issue Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 3388dc7 Author: Lantao Jin <ltjin@amazon.com> Date: Thu Oct 16 01:45:29 2025 +0800 Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance (opensearch-project#4569) * Use _shard_doc as sort tiebreaker Signed-off-by: Lantao Jin <ltjin@amazon.com> * _doc as a part of tie-breaker have better performance Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 5630119 Author: qianheng <qianheng@amazon.com> Date: Wed Oct 15 16:40:41 2025 +0800 Fix sort push down into agg after project already pushed (opensearch-project#4546) * Fix sort push down into agg Signed-off-by: Heng Qian <qianheng@amazon.com> * Change some json files to yaml format Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 1e62fba Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 17:20:38 2025 -0700 Fix JsonExtractAllFunctionIT failure (opensearch-project#4556) Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 02ee33e Author: Kai Huang <105710027+ahkcs@users.noreply.github.com> Date: Tue Oct 14 14:28:53 2025 -0700 Add more examples to the `where` command doc (opensearch-project#4457) Co-authored-by: Manasvini B S <manasvis@amazon.com> commit 0b7e86c Author: Jialiang Liang <jiallian@amazon.com> Date: Tue Oct 14 10:46:01 2025 -0700 [Enhancement] Error handling for illegal character usage in java regex named capture group (opensearch-project#4434) Co-authored-by: Simeon Widdis <sawiddis@amazon.com> commit 9c97cfb Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 08:36:43 2025 -0700 Add JSON_EXTRACT_ALL internal function for Calcite PPL (opensearch-project#4489) * Add JSON_EXTRACT_ALL internal function for Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comments Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 89dbc31 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:24:52 2025 +0800 Check server status before starting Prometheus (opensearch-project#4537) * Check server status before starting Prometheus Signed-off-by: Lantao Jin <ltjin@amazon.com> * Change to func call Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix doc Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit fe62472 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:10:27 2025 +0800 Update request builder after pushdown sort into agg buckets (opensearch-project#4541) Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 42a415f Author: qianheng <qianheng@amazon.com> Date: Tue Oct 14 17:42:45 2025 +0800 Including metadata fields type when doing agg/filter script push down (opensearch-project#4522) * Including metadata fields type when doing agg/filter script push down Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 8de0386 Author: Xinyuan Lu <xinyual@amazon.com> Date: Tue Oct 14 16:41:08 2025 +0800 Fix percentile bug (opensearch-project#4539) * fix percentile bug Signed-off-by: xinyual <xinyual@amazon.com> * add IT Signed-off-by: xinyual <xinyual@amazon.com> * optimize it Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com> commit de2fdc8 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 12:29:03 2025 +0800 [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited (opensearch-project#4534) * [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix doctest Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix conflicts Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 977b7ab Author: Simeon Widdis <sawiddis@gmail.com> Date: Mon Oct 13 20:23:10 2025 -0700 Update stalled action (opensearch-project#4485) commit fddbb70 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 10:23:12 2025 +0800 Add configurable sytem limitations for `subsearch` and `join` command (opensearch-project#4501) * Add configurable sytem limitations for subsearch and join command Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * remove rollback in doc Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

LantaoJin · 2025-10-15T08:48:44Z

Description

Before #4378, the sort in PIT search is
case 1: if no sort field specified, sort by _doc + _id (+ means "then"). (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)
case 3: if sort fields specified and query contains a filter, sort by _doc. (❎ paged results could miss or duplicate hits)

#4378 added the _shard_doc as sort tiebreaker with
case 1: if no sort field specified, sort by _shard_doc. (❎ performance regression)
case 2: if sort fields specified, sort by fields + _shard_doc.(❎ lower performance on low cardinality field)

#4435 found performance regression in case 1 and partially revert the changes to
case 1: if no sort field specified, sort by _doc + _id. (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)

After this PR, we change the sort in PIT search to
case 1: if no sort field specified, sort by _doc + _shard_doc. ✅
case 2: if sort fields specified, sort by fields + _doc + _shard_doc.✅

RCA of performance regression:
_shard_doc is not a stored field in index which will be generated in runtime when comparison. Computing _shard_doc per document is a high cost operation. But sorting by _doc then _shard_doc only generates _shard_doc when the _doc values are conflicted.
Even in the case of user specified sort fields, we should sort by fields then _doc then _shard_doc to reduce the computing of _shard_doc. For example, if the sort field is a low cardinality field, e.g. gender, sorting by gender then _doc then _shard_doc generates _shard_doc for comparison only if values of gender and _doc are both conflicted.

This PR is no needed to backport to 2.19-dev since shard_doc feature is only available since OS 3.3.0

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Swiddis · 2025-10-15T14:59:26Z

- // Workaround to preserve sort location more exactly,
- // see https://github.com/opensearch-project/sql/pull/3061
- this.sourceBuilder.sort(METADATA_FIELD_ID, ASC);
+ this.sourceBuilder.sort(SortBuilders.shardDocSort());


Does it matter if we duplicate fields in the sorting list? We could simplify/remove the below else logic by just always appending this, I would expect Lucene to optimize it in the background but I haven't measured it.

not sure will Lucene optimize duplicated fields or _doc in sorting, but for sure the duplicated _shard_doc is not allowed in OpenSearch Core. It is no harmful for restricted checker here.

anasalkouz · 2025-10-15T15:10:51Z

Can you share the performance benchmark for the 3 approaches?

LantaoJin · 2025-10-15T15:32:28Z

Can you share the performance benchmark for the 3 approaches?

I haven't run the benchmark, the RCA was made by reading the code of Luence and OS shard_doc feature.

The performance of _doc then _shard_doc is same as _doc then _id, provided by @ahkcs on Oct 1st. (the case 1)

For case 2, the current fields + _doc + _shard_doc is an further optimization upon fields + _shard_doc based on above benchmark result with inference.

Will rerun some benchmark to double confirm.

commit cba8d02 Author: Tomoyuki MORITA <moritato@amazon.com> Date: Wed Oct 15 13:08:05 2025 -0700 Add MAP_APPEND internal function to Calcite PPL (opensearch-project#4515) * Add MAP_APPEND internal function to Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comment Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Rebase and fix IT issue Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 3388dc7 Author: Lantao Jin <ltjin@amazon.com> Date: Thu Oct 16 01:45:29 2025 +0800 Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance (opensearch-project#4569) * Use _shard_doc as sort tiebreaker Signed-off-by: Lantao Jin <ltjin@amazon.com> * _doc as a part of tie-breaker have better performance Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 5630119 Author: qianheng <qianheng@amazon.com> Date: Wed Oct 15 16:40:41 2025 +0800 Fix sort push down into agg after project already pushed (opensearch-project#4546) * Fix sort push down into agg Signed-off-by: Heng Qian <qianheng@amazon.com> * Change some json files to yaml format Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 1e62fba Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 17:20:38 2025 -0700 Fix JsonExtractAllFunctionIT failure (opensearch-project#4556) Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 02ee33e Author: Kai Huang <105710027+ahkcs@users.noreply.github.com> Date: Tue Oct 14 14:28:53 2025 -0700 Add more examples to the `where` command doc (opensearch-project#4457) Co-authored-by: Manasvini B S <manasvis@amazon.com> commit 0b7e86c Author: Jialiang Liang <jiallian@amazon.com> Date: Tue Oct 14 10:46:01 2025 -0700 [Enhancement] Error handling for illegal character usage in java regex named capture group (opensearch-project#4434) Co-authored-by: Simeon Widdis <sawiddis@amazon.com> commit 9c97cfb Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 08:36:43 2025 -0700 Add JSON_EXTRACT_ALL internal function for Calcite PPL (opensearch-project#4489) * Add JSON_EXTRACT_ALL internal function for Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comments Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 89dbc31 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:24:52 2025 +0800 Check server status before starting Prometheus (opensearch-project#4537) * Check server status before starting Prometheus Signed-off-by: Lantao Jin <ltjin@amazon.com> * Change to func call Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix doc Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit fe62472 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:10:27 2025 +0800 Update request builder after pushdown sort into agg buckets (opensearch-project#4541) Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 42a415f Author: qianheng <qianheng@amazon.com> Date: Tue Oct 14 17:42:45 2025 +0800 Including metadata fields type when doing agg/filter script push down (opensearch-project#4522) * Including metadata fields type when doing agg/filter script push down Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 8de0386 Author: Xinyuan Lu <xinyual@amazon.com> Date: Tue Oct 14 16:41:08 2025 +0800 Fix percentile bug (opensearch-project#4539) * fix percentile bug Signed-off-by: xinyual <xinyual@amazon.com> * add IT Signed-off-by: xinyual <xinyual@amazon.com> * optimize it Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com> commit de2fdc8 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 12:29:03 2025 +0800 [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited (opensearch-project#4534) * [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix doctest Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix conflicts Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 977b7ab Author: Simeon Widdis <sawiddis@gmail.com> Date: Mon Oct 13 20:23:10 2025 -0700 Update stalled action (opensearch-project#4485) commit fddbb70 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 10:23:12 2025 +0800 Add configurable sytem limitations for `subsearch` and `join` command (opensearch-project#4501) * Add configurable sytem limitations for subsearch and join command Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * remove rollback in doc Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

LantaoJin · 2025-10-15T08:48:44Z

Description

Before #4378, the sort in PIT search is
case 1: if no sort field specified, sort by _doc + _id (+ means "then"). (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)
case 3: if sort fields specified and query contains a filter, sort by _doc. (❎ paged results could miss or duplicate hits)

#4378 added the _shard_doc as sort tiebreaker with
case 1: if no sort field specified, sort by _shard_doc. (❎ performance regression)
case 2: if sort fields specified, sort by fields + _shard_doc.(❎ lower performance on low cardinality field)

#4435 found performance regression in case 1 and partially revert the changes to
case 1: if no sort field specified, sort by _doc + _id. (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)

After this PR, we change the sort in PIT search to
case 1: if no sort field specified, sort by _doc + _shard_doc. ✅
case 2: if sort fields specified, sort by fields + _doc + _shard_doc.✅

RCA of performance regression:
_shard_doc is not a stored field in index which will be generated in runtime when comparison. Computing _shard_doc per document is a high cost operation. But sorting by _doc then _shard_doc only generates _shard_doc when the _doc values are conflicted.
Even in the case of user specified sort fields, we should sort by fields then _doc then _shard_doc to reduce the computing of _shard_doc. For example, if the sort field is a low cardinality field, e.g. gender, sorting by gender then _doc then _shard_doc generates _shard_doc for comparison only if values of gender and _doc are both conflicted.

This PR is no needed to backport to 2.19-dev since shard_doc feature is only available since OS 3.3.0

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Swiddis · 2025-10-15T14:59:26Z

- // Workaround to preserve sort location more exactly,
- // see https://github.com/opensearch-project/sql/pull/3061
- this.sourceBuilder.sort(METADATA_FIELD_ID, ASC);
+ this.sourceBuilder.sort(SortBuilders.shardDocSort());


Does it matter if we duplicate fields in the sorting list? We could simplify/remove the below else logic by just always appending this, I would expect Lucene to optimize it in the background but I haven't measured it.

not sure will Lucene optimize duplicated fields or _doc in sorting, but for sure the duplicated _shard_doc is not allowed in OpenSearch Core. It is no harmful for restricted checker here.

anasalkouz · 2025-10-15T15:10:51Z

Can you share the performance benchmark for the 3 approaches?

LantaoJin · 2025-10-15T15:32:28Z

Can you share the performance benchmark for the 3 approaches?

I haven't run the benchmark, the RCA was made by reading the code of Luence and OS shard_doc feature.

The performance of _doc then _shard_doc is same as _doc then _id, provided by @ahkcs on Oct 1st. (the case 1)

For case 2, the current fields + _doc + _shard_doc is an further optimization upon fields + _shard_doc based on above benchmark result with inference.

Will rerun some benchmark to double confirm.

commit cba8d02 Author: Tomoyuki MORITA <moritato@amazon.com> Date: Wed Oct 15 13:08:05 2025 -0700 Add MAP_APPEND internal function to Calcite PPL (opensearch-project#4515) * Add MAP_APPEND internal function to Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comment Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Rebase and fix IT issue Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 3388dc7 Author: Lantao Jin <ltjin@amazon.com> Date: Thu Oct 16 01:45:29 2025 +0800 Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance (opensearch-project#4569) * Use _shard_doc as sort tiebreaker Signed-off-by: Lantao Jin <ltjin@amazon.com> * _doc as a part of tie-breaker have better performance Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 5630119 Author: qianheng <qianheng@amazon.com> Date: Wed Oct 15 16:40:41 2025 +0800 Fix sort push down into agg after project already pushed (opensearch-project#4546) * Fix sort push down into agg Signed-off-by: Heng Qian <qianheng@amazon.com> * Change some json files to yaml format Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 1e62fba Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 17:20:38 2025 -0700 Fix JsonExtractAllFunctionIT failure (opensearch-project#4556) Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 02ee33e Author: Kai Huang <105710027+ahkcs@users.noreply.github.com> Date: Tue Oct 14 14:28:53 2025 -0700 Add more examples to the `where` command doc (opensearch-project#4457) Co-authored-by: Manasvini B S <manasvis@amazon.com> commit 0b7e86c Author: Jialiang Liang <jiallian@amazon.com> Date: Tue Oct 14 10:46:01 2025 -0700 [Enhancement] Error handling for illegal character usage in java regex named capture group (opensearch-project#4434) Co-authored-by: Simeon Widdis <sawiddis@amazon.com> commit 9c97cfb Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 08:36:43 2025 -0700 Add JSON_EXTRACT_ALL internal function for Calcite PPL (opensearch-project#4489) * Add JSON_EXTRACT_ALL internal function for Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comments Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 89dbc31 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:24:52 2025 +0800 Check server status before starting Prometheus (opensearch-project#4537) * Check server status before starting Prometheus Signed-off-by: Lantao Jin <ltjin@amazon.com> * Change to func call Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix doc Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit fe62472 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:10:27 2025 +0800 Update request builder after pushdown sort into agg buckets (opensearch-project#4541) Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 42a415f Author: qianheng <qianheng@amazon.com> Date: Tue Oct 14 17:42:45 2025 +0800 Including metadata fields type when doing agg/filter script push down (opensearch-project#4522) * Including metadata fields type when doing agg/filter script push down Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 8de0386 Author: Xinyuan Lu <xinyual@amazon.com> Date: Tue Oct 14 16:41:08 2025 +0800 Fix percentile bug (opensearch-project#4539) * fix percentile bug Signed-off-by: xinyual <xinyual@amazon.com> * add IT Signed-off-by: xinyual <xinyual@amazon.com> * optimize it Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com> commit de2fdc8 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 12:29:03 2025 +0800 [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited (opensearch-project#4534) * [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix doctest Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix conflicts Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 977b7ab Author: Simeon Widdis <sawiddis@gmail.com> Date: Mon Oct 13 20:23:10 2025 -0700 Update stalled action (opensearch-project#4485) commit fddbb70 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 10:23:12 2025 +0800 Add configurable sytem limitations for `subsearch` and `join` command (opensearch-project#4501) * Add configurable sytem limitations for subsearch and join command Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * remove rollback in doc Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

LantaoJin · 2025-10-15T08:48:44Z

Description

Before #4378, the sort in PIT search is
case 1: if no sort field specified, sort by _doc + _id (+ means "then"). (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)
case 3: if sort fields specified and query contains a filter, sort by _doc. (❎ paged results could miss or duplicate hits)

#4378 added the _shard_doc as sort tiebreaker with
case 1: if no sort field specified, sort by _shard_doc. (❎ performance regression)
case 2: if sort fields specified, sort by fields + _shard_doc.(❎ lower performance on low cardinality field)

#4435 found performance regression in case 1 and partially revert the changes to
case 1: if no sort field specified, sort by _doc + _id. (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)

After this PR, we change the sort in PIT search to
case 1: if no sort field specified, sort by _doc + _shard_doc. ✅
case 2: if sort fields specified, sort by fields + _doc + _shard_doc.✅

RCA of performance regression:
_shard_doc is not a stored field in index which will be generated in runtime when comparison. Computing _shard_doc per document is a high cost operation. But sorting by _doc then _shard_doc only generates _shard_doc when the _doc values are conflicted.
Even in the case of user specified sort fields, we should sort by fields then _doc then _shard_doc to reduce the computing of _shard_doc. For example, if the sort field is a low cardinality field, e.g. gender, sorting by gender then _doc then _shard_doc generates _shard_doc for comparison only if values of gender and _doc are both conflicted.

This PR is no needed to backport to 2.19-dev since shard_doc feature is only available since OS 3.3.0

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Swiddis · 2025-10-15T14:59:26Z

- // Workaround to preserve sort location more exactly,
- // see https://github.com/opensearch-project/sql/pull/3061
- this.sourceBuilder.sort(METADATA_FIELD_ID, ASC);
+ this.sourceBuilder.sort(SortBuilders.shardDocSort());


Does it matter if we duplicate fields in the sorting list? We could simplify/remove the below else logic by just always appending this, I would expect Lucene to optimize it in the background but I haven't measured it.

not sure will Lucene optimize duplicated fields or _doc in sorting, but for sure the duplicated _shard_doc is not allowed in OpenSearch Core. It is no harmful for restricted checker here.

anasalkouz · 2025-10-15T15:10:51Z

Can you share the performance benchmark for the 3 approaches?

LantaoJin · 2025-10-15T15:32:28Z

Can you share the performance benchmark for the 3 approaches?

I haven't run the benchmark, the RCA was made by reading the code of Luence and OS shard_doc feature.

The performance of _doc then _shard_doc is same as _doc then _id, provided by @ahkcs on Oct 1st. (the case 1)

For case 2, the current fields + _doc + _shard_doc is an further optimization upon fields + _shard_doc based on above benchmark result with inference.

Will rerun some benchmark to double confirm.

commit cba8d02 Author: Tomoyuki MORITA <moritato@amazon.com> Date: Wed Oct 15 13:08:05 2025 -0700 Add MAP_APPEND internal function to Calcite PPL (opensearch-project#4515) * Add MAP_APPEND internal function to Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comment Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Rebase and fix IT issue Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 3388dc7 Author: Lantao Jin <ltjin@amazon.com> Date: Thu Oct 16 01:45:29 2025 +0800 Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance (opensearch-project#4569) * Use _shard_doc as sort tiebreaker Signed-off-by: Lantao Jin <ltjin@amazon.com> * _doc as a part of tie-breaker have better performance Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 5630119 Author: qianheng <qianheng@amazon.com> Date: Wed Oct 15 16:40:41 2025 +0800 Fix sort push down into agg after project already pushed (opensearch-project#4546) * Fix sort push down into agg Signed-off-by: Heng Qian <qianheng@amazon.com> * Change some json files to yaml format Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 1e62fba Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 17:20:38 2025 -0700 Fix JsonExtractAllFunctionIT failure (opensearch-project#4556) Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 02ee33e Author: Kai Huang <105710027+ahkcs@users.noreply.github.com> Date: Tue Oct 14 14:28:53 2025 -0700 Add more examples to the `where` command doc (opensearch-project#4457) Co-authored-by: Manasvini B S <manasvis@amazon.com> commit 0b7e86c Author: Jialiang Liang <jiallian@amazon.com> Date: Tue Oct 14 10:46:01 2025 -0700 [Enhancement] Error handling for illegal character usage in java regex named capture group (opensearch-project#4434) Co-authored-by: Simeon Widdis <sawiddis@amazon.com> commit 9c97cfb Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 08:36:43 2025 -0700 Add JSON_EXTRACT_ALL internal function for Calcite PPL (opensearch-project#4489) * Add JSON_EXTRACT_ALL internal function for Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comments Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 89dbc31 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:24:52 2025 +0800 Check server status before starting Prometheus (opensearch-project#4537) * Check server status before starting Prometheus Signed-off-by: Lantao Jin <ltjin@amazon.com> * Change to func call Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix doc Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit fe62472 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:10:27 2025 +0800 Update request builder after pushdown sort into agg buckets (opensearch-project#4541) Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 42a415f Author: qianheng <qianheng@amazon.com> Date: Tue Oct 14 17:42:45 2025 +0800 Including metadata fields type when doing agg/filter script push down (opensearch-project#4522) * Including metadata fields type when doing agg/filter script push down Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 8de0386 Author: Xinyuan Lu <xinyual@amazon.com> Date: Tue Oct 14 16:41:08 2025 +0800 Fix percentile bug (opensearch-project#4539) * fix percentile bug Signed-off-by: xinyual <xinyual@amazon.com> * add IT Signed-off-by: xinyual <xinyual@amazon.com> * optimize it Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com> commit de2fdc8 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 12:29:03 2025 +0800 [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited (opensearch-project#4534) * [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix doctest Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix conflicts Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 977b7ab Author: Simeon Widdis <sawiddis@gmail.com> Date: Mon Oct 13 20:23:10 2025 -0700 Update stalled action (opensearch-project#4485) commit fddbb70 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 10:23:12 2025 +0800 Add configurable sytem limitations for `subsearch` and `join` command (opensearch-project#4501) * Add configurable sytem limitations for subsearch and join command Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * remove rollback in doc Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

LantaoJin · 2025-10-15T08:48:44Z

Description

Before #4378, the sort in PIT search is
case 1: if no sort field specified, sort by _doc + _id (+ means "then"). (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)
case 3: if sort fields specified and query contains a filter, sort by _doc. (❎ paged results could miss or duplicate hits)

#4378 added the _shard_doc as sort tiebreaker with
case 1: if no sort field specified, sort by _shard_doc. (❎ performance regression)
case 2: if sort fields specified, sort by fields + _shard_doc.(❎ lower performance on low cardinality field)

#4435 found performance regression in case 1 and partially revert the changes to
case 1: if no sort field specified, sort by _doc + _id. (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)

After this PR, we change the sort in PIT search to
case 1: if no sort field specified, sort by _doc + _shard_doc. ✅
case 2: if sort fields specified, sort by fields + _doc + _shard_doc.✅

RCA of performance regression:
_shard_doc is not a stored field in index which will be generated in runtime when comparison. Computing _shard_doc per document is a high cost operation. But sorting by _doc then _shard_doc only generates _shard_doc when the _doc values are conflicted.
Even in the case of user specified sort fields, we should sort by fields then _doc then _shard_doc to reduce the computing of _shard_doc. For example, if the sort field is a low cardinality field, e.g. gender, sorting by gender then _doc then _shard_doc generates _shard_doc for comparison only if values of gender and _doc are both conflicted.

This PR is no needed to backport to 2.19-dev since shard_doc feature is only available since OS 3.3.0

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Swiddis · 2025-10-15T14:59:26Z

- // Workaround to preserve sort location more exactly,
- // see https://github.com/opensearch-project/sql/pull/3061
- this.sourceBuilder.sort(METADATA_FIELD_ID, ASC);
+ this.sourceBuilder.sort(SortBuilders.shardDocSort());


Does it matter if we duplicate fields in the sorting list? We could simplify/remove the below else logic by just always appending this, I would expect Lucene to optimize it in the background but I haven't measured it.

not sure will Lucene optimize duplicated fields or _doc in sorting, but for sure the duplicated _shard_doc is not allowed in OpenSearch Core. It is no harmful for restricted checker here.

anasalkouz · 2025-10-15T15:10:51Z

Can you share the performance benchmark for the 3 approaches?

LantaoJin · 2025-10-15T15:32:28Z

Can you share the performance benchmark for the 3 approaches?

I haven't run the benchmark, the RCA was made by reading the code of Luence and OS shard_doc feature.

The performance of _doc then _shard_doc is same as _doc then _id, provided by @ahkcs on Oct 1st. (the case 1)

For case 2, the current fields + _doc + _shard_doc is an further optimization upon fields + _shard_doc based on above benchmark result with inference.

Will rerun some benchmark to double confirm.

commit cba8d02 Author: Tomoyuki MORITA <moritato@amazon.com> Date: Wed Oct 15 13:08:05 2025 -0700 Add MAP_APPEND internal function to Calcite PPL (opensearch-project#4515) * Add MAP_APPEND internal function to Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comment Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Rebase and fix IT issue Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 3388dc7 Author: Lantao Jin <ltjin@amazon.com> Date: Thu Oct 16 01:45:29 2025 +0800 Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance (opensearch-project#4569) * Use _shard_doc as sort tiebreaker Signed-off-by: Lantao Jin <ltjin@amazon.com> * _doc as a part of tie-breaker have better performance Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 5630119 Author: qianheng <qianheng@amazon.com> Date: Wed Oct 15 16:40:41 2025 +0800 Fix sort push down into agg after project already pushed (opensearch-project#4546) * Fix sort push down into agg Signed-off-by: Heng Qian <qianheng@amazon.com> * Change some json files to yaml format Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 1e62fba Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 17:20:38 2025 -0700 Fix JsonExtractAllFunctionIT failure (opensearch-project#4556) Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 02ee33e Author: Kai Huang <105710027+ahkcs@users.noreply.github.com> Date: Tue Oct 14 14:28:53 2025 -0700 Add more examples to the `where` command doc (opensearch-project#4457) Co-authored-by: Manasvini B S <manasvis@amazon.com> commit 0b7e86c Author: Jialiang Liang <jiallian@amazon.com> Date: Tue Oct 14 10:46:01 2025 -0700 [Enhancement] Error handling for illegal character usage in java regex named capture group (opensearch-project#4434) Co-authored-by: Simeon Widdis <sawiddis@amazon.com> commit 9c97cfb Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 08:36:43 2025 -0700 Add JSON_EXTRACT_ALL internal function for Calcite PPL (opensearch-project#4489) * Add JSON_EXTRACT_ALL internal function for Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comments Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 89dbc31 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:24:52 2025 +0800 Check server status before starting Prometheus (opensearch-project#4537) * Check server status before starting Prometheus Signed-off-by: Lantao Jin <ltjin@amazon.com> * Change to func call Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix doc Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit fe62472 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:10:27 2025 +0800 Update request builder after pushdown sort into agg buckets (opensearch-project#4541) Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 42a415f Author: qianheng <qianheng@amazon.com> Date: Tue Oct 14 17:42:45 2025 +0800 Including metadata fields type when doing agg/filter script push down (opensearch-project#4522) * Including metadata fields type when doing agg/filter script push down Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 8de0386 Author: Xinyuan Lu <xinyual@amazon.com> Date: Tue Oct 14 16:41:08 2025 +0800 Fix percentile bug (opensearch-project#4539) * fix percentile bug Signed-off-by: xinyual <xinyual@amazon.com> * add IT Signed-off-by: xinyual <xinyual@amazon.com> * optimize it Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com> commit de2fdc8 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 12:29:03 2025 +0800 [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited (opensearch-project#4534) * [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix doctest Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix conflicts Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 977b7ab Author: Simeon Widdis <sawiddis@gmail.com> Date: Mon Oct 13 20:23:10 2025 -0700 Update stalled action (opensearch-project#4485) commit fddbb70 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 10:23:12 2025 +0800 Add configurable sytem limitations for `subsearch` and `join` command (opensearch-project#4501) * Add configurable sytem limitations for subsearch and join command Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * remove rollback in doc Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

LantaoJin · 2025-10-15T08:48:44Z

Description

Before #4378, the sort in PIT search is
case 1: if no sort field specified, sort by _doc + _id (+ means "then"). (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)
case 3: if sort fields specified and query contains a filter, sort by _doc. (❎ paged results could miss or duplicate hits)

#4378 added the _shard_doc as sort tiebreaker with
case 1: if no sort field specified, sort by _shard_doc. (❎ performance regression)
case 2: if sort fields specified, sort by fields + _shard_doc.(❎ lower performance on low cardinality field)

#4435 found performance regression in case 1 and partially revert the changes to
case 1: if no sort field specified, sort by _doc + _id. (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)

After this PR, we change the sort in PIT search to
case 1: if no sort field specified, sort by _doc + _shard_doc. ✅
case 2: if sort fields specified, sort by fields + _doc + _shard_doc.✅

RCA of performance regression:
_shard_doc is not a stored field in index which will be generated in runtime when comparison. Computing _shard_doc per document is a high cost operation. But sorting by _doc then _shard_doc only generates _shard_doc when the _doc values are conflicted.
Even in the case of user specified sort fields, we should sort by fields then _doc then _shard_doc to reduce the computing of _shard_doc. For example, if the sort field is a low cardinality field, e.g. gender, sorting by gender then _doc then _shard_doc generates _shard_doc for comparison only if values of gender and _doc are both conflicted.

This PR is no needed to backport to 2.19-dev since shard_doc feature is only available since OS 3.3.0

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Swiddis · 2025-10-15T14:59:26Z

- // Workaround to preserve sort location more exactly,
- // see https://github.com/opensearch-project/sql/pull/3061
- this.sourceBuilder.sort(METADATA_FIELD_ID, ASC);
+ this.sourceBuilder.sort(SortBuilders.shardDocSort());


Does it matter if we duplicate fields in the sorting list? We could simplify/remove the below else logic by just always appending this, I would expect Lucene to optimize it in the background but I haven't measured it.

not sure will Lucene optimize duplicated fields or _doc in sorting, but for sure the duplicated _shard_doc is not allowed in OpenSearch Core. It is no harmful for restricted checker here.

anasalkouz · 2025-10-15T15:10:51Z

Can you share the performance benchmark for the 3 approaches?

LantaoJin · 2025-10-15T15:32:28Z

Can you share the performance benchmark for the 3 approaches?

I haven't run the benchmark, the RCA was made by reading the code of Luence and OS shard_doc feature.

The performance of _doc then _shard_doc is same as _doc then _id, provided by @ahkcs on Oct 1st. (the case 1)

For case 2, the current fields + _doc + _shard_doc is an further optimization upon fields + _shard_doc based on above benchmark result with inference.

Will rerun some benchmark to double confirm.

commit cba8d02 Author: Tomoyuki MORITA <moritato@amazon.com> Date: Wed Oct 15 13:08:05 2025 -0700 Add MAP_APPEND internal function to Calcite PPL (opensearch-project#4515) * Add MAP_APPEND internal function to Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comment Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Rebase and fix IT issue Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 3388dc7 Author: Lantao Jin <ltjin@amazon.com> Date: Thu Oct 16 01:45:29 2025 +0800 Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance (opensearch-project#4569) * Use _shard_doc as sort tiebreaker Signed-off-by: Lantao Jin <ltjin@amazon.com> * _doc as a part of tie-breaker have better performance Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 5630119 Author: qianheng <qianheng@amazon.com> Date: Wed Oct 15 16:40:41 2025 +0800 Fix sort push down into agg after project already pushed (opensearch-project#4546) * Fix sort push down into agg Signed-off-by: Heng Qian <qianheng@amazon.com> * Change some json files to yaml format Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 1e62fba Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 17:20:38 2025 -0700 Fix JsonExtractAllFunctionIT failure (opensearch-project#4556) Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 02ee33e Author: Kai Huang <105710027+ahkcs@users.noreply.github.com> Date: Tue Oct 14 14:28:53 2025 -0700 Add more examples to the `where` command doc (opensearch-project#4457) Co-authored-by: Manasvini B S <manasvis@amazon.com> commit 0b7e86c Author: Jialiang Liang <jiallian@amazon.com> Date: Tue Oct 14 10:46:01 2025 -0700 [Enhancement] Error handling for illegal character usage in java regex named capture group (opensearch-project#4434) Co-authored-by: Simeon Widdis <sawiddis@amazon.com> commit 9c97cfb Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 08:36:43 2025 -0700 Add JSON_EXTRACT_ALL internal function for Calcite PPL (opensearch-project#4489) * Add JSON_EXTRACT_ALL internal function for Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comments Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 89dbc31 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:24:52 2025 +0800 Check server status before starting Prometheus (opensearch-project#4537) * Check server status before starting Prometheus Signed-off-by: Lantao Jin <ltjin@amazon.com> * Change to func call Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix doc Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit fe62472 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:10:27 2025 +0800 Update request builder after pushdown sort into agg buckets (opensearch-project#4541) Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 42a415f Author: qianheng <qianheng@amazon.com> Date: Tue Oct 14 17:42:45 2025 +0800 Including metadata fields type when doing agg/filter script push down (opensearch-project#4522) * Including metadata fields type when doing agg/filter script push down Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 8de0386 Author: Xinyuan Lu <xinyual@amazon.com> Date: Tue Oct 14 16:41:08 2025 +0800 Fix percentile bug (opensearch-project#4539) * fix percentile bug Signed-off-by: xinyual <xinyual@amazon.com> * add IT Signed-off-by: xinyual <xinyual@amazon.com> * optimize it Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com> commit de2fdc8 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 12:29:03 2025 +0800 [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited (opensearch-project#4534) * [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix doctest Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix conflicts Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 977b7ab Author: Simeon Widdis <sawiddis@gmail.com> Date: Mon Oct 13 20:23:10 2025 -0700 Update stalled action (opensearch-project#4485) commit fddbb70 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 10:23:12 2025 +0800 Add configurable sytem limitations for `subsearch` and `join` command (opensearch-project#4501) * Add configurable sytem limitations for subsearch and join command Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * remove rollback in doc Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

LantaoJin · 2025-10-15T08:48:44Z

Description

Before #4378, the sort in PIT search is
case 1: if no sort field specified, sort by _doc + _id (+ means "then"). (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)
case 3: if sort fields specified and query contains a filter, sort by _doc. (❎ paged results could miss or duplicate hits)

#4378 added the _shard_doc as sort tiebreaker with
case 1: if no sort field specified, sort by _shard_doc. (❎ performance regression)
case 2: if sort fields specified, sort by fields + _shard_doc.(❎ lower performance on low cardinality field)

#4435 found performance regression in case 1 and partially revert the changes to
case 1: if no sort field specified, sort by _doc + _id. (❎ could cause high memory issue)
case 2: if sort fields specified, sort by fields. (❎ paged results could miss or duplicate hits)

After this PR, we change the sort in PIT search to
case 1: if no sort field specified, sort by _doc + _shard_doc. ✅
case 2: if sort fields specified, sort by fields + _doc + _shard_doc.✅

RCA of performance regression:
_shard_doc is not a stored field in index which will be generated in runtime when comparison. Computing _shard_doc per document is a high cost operation. But sorting by _doc then _shard_doc only generates _shard_doc when the _doc values are conflicted.
Even in the case of user specified sort fields, we should sort by fields then _doc then _shard_doc to reduce the computing of _shard_doc. For example, if the sort field is a low cardinality field, e.g. gender, sorting by gender then _doc then _shard_doc generates _shard_doc for comparison only if values of gender and _doc are both conflicted.

This PR is no needed to backport to 2.19-dev since shard_doc feature is only available since OS 3.3.0

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Swiddis · 2025-10-15T14:59:26Z

- // Workaround to preserve sort location more exactly,
- // see https://github.com/opensearch-project/sql/pull/3061
- this.sourceBuilder.sort(METADATA_FIELD_ID, ASC);
+ this.sourceBuilder.sort(SortBuilders.shardDocSort());


Does it matter if we duplicate fields in the sorting list? We could simplify/remove the below else logic by just always appending this, I would expect Lucene to optimize it in the background but I haven't measured it.

not sure will Lucene optimize duplicated fields or _doc in sorting, but for sure the duplicated _shard_doc is not allowed in OpenSearch Core. It is no harmful for restricted checker here.

anasalkouz · 2025-10-15T15:10:51Z

Can you share the performance benchmark for the 3 approaches?

LantaoJin · 2025-10-15T15:32:28Z

Can you share the performance benchmark for the 3 approaches?

I haven't run the benchmark, the RCA was made by reading the code of Luence and OS shard_doc feature.

The performance of _doc then _shard_doc is same as _doc then _id, provided by @ahkcs on Oct 1st. (the case 1)

For case 2, the current fields + _doc + _shard_doc is an further optimization upon fields + _shard_doc based on above benchmark result with inference.

Will rerun some benchmark to double confirm.

commit cba8d02 Author: Tomoyuki MORITA <moritato@amazon.com> Date: Wed Oct 15 13:08:05 2025 -0700 Add MAP_APPEND internal function to Calcite PPL (opensearch-project#4515) * Add MAP_APPEND internal function to Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comment Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Rebase and fix IT issue Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 3388dc7 Author: Lantao Jin <ltjin@amazon.com> Date: Thu Oct 16 01:45:29 2025 +0800 Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance (opensearch-project#4569) * Use _shard_doc as sort tiebreaker Signed-off-by: Lantao Jin <ltjin@amazon.com> * _doc as a part of tie-breaker have better performance Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 5630119 Author: qianheng <qianheng@amazon.com> Date: Wed Oct 15 16:40:41 2025 +0800 Fix sort push down into agg after project already pushed (opensearch-project#4546) * Fix sort push down into agg Signed-off-by: Heng Qian <qianheng@amazon.com> * Change some json files to yaml format Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 1e62fba Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 17:20:38 2025 -0700 Fix JsonExtractAllFunctionIT failure (opensearch-project#4556) Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 02ee33e Author: Kai Huang <105710027+ahkcs@users.noreply.github.com> Date: Tue Oct 14 14:28:53 2025 -0700 Add more examples to the `where` command doc (opensearch-project#4457) Co-authored-by: Manasvini B S <manasvis@amazon.com> commit 0b7e86c Author: Jialiang Liang <jiallian@amazon.com> Date: Tue Oct 14 10:46:01 2025 -0700 [Enhancement] Error handling for illegal character usage in java regex named capture group (opensearch-project#4434) Co-authored-by: Simeon Widdis <sawiddis@amazon.com> commit 9c97cfb Author: Tomoyuki MORITA <moritato@amazon.com> Date: Tue Oct 14 08:36:43 2025 -0700 Add JSON_EXTRACT_ALL internal function for Calcite PPL (opensearch-project#4489) * Add JSON_EXTRACT_ALL internal function for Calcite PPL Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address comments Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Minor fix Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com> commit 89dbc31 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:24:52 2025 +0800 Check server status before starting Prometheus (opensearch-project#4537) * Check server status before starting Prometheus Signed-off-by: Lantao Jin <ltjin@amazon.com> * Change to func call Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix doc Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit fe62472 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 18:10:27 2025 +0800 Update request builder after pushdown sort into agg buckets (opensearch-project#4541) Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 42a415f Author: qianheng <qianheng@amazon.com> Date: Tue Oct 14 17:42:45 2025 +0800 Including metadata fields type when doing agg/filter script push down (opensearch-project#4522) * Including metadata fields type when doing agg/filter script push down Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> commit 8de0386 Author: Xinyuan Lu <xinyual@amazon.com> Date: Tue Oct 14 16:41:08 2025 +0800 Fix percentile bug (opensearch-project#4539) * fix percentile bug Signed-off-by: xinyual <xinyual@amazon.com> * add IT Signed-off-by: xinyual <xinyual@amazon.com> * optimize it Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com> commit de2fdc8 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 12:29:03 2025 +0800 [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited (opensearch-project#4534) * [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix doctest Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix conflicts Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> commit 977b7ab Author: Simeon Widdis <sawiddis@gmail.com> Date: Mon Oct 13 20:23:10 2025 -0700 Update stalled action (opensearch-project#4485) commit fddbb70 Author: Lantao Jin <ltjin@amazon.com> Date: Tue Oct 14 10:23:12 2025 +0800 Add configurable sytem limitations for `subsearch` and `join` command (opensearch-project#4501) * Add configurable sytem limitations for subsearch and join command Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * remove rollback in doc Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix typo Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Use _shard_doc as sort tiebreaker
7e6da58
Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin added the enhancement New feature or request label Oct 15, 2025

LantaoJin changed the title ~~Use _shard_doc as sort tiebreaker to get better performance~~Use _doc + _shard_doc as sort tiebreaker to get better performanceOct 15, 2025

_doc as a part of tie-breaker have better performance
18456f5
Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin marked this pull request as ready for review October 15, 2025 14:21

Swiddis approved these changes Oct 15, 2025
View reviewed changes

penghuo approved these changes Oct 15, 2025
View reviewed changes

penghuo merged commit 3388dc7 into opensearch-project:mainOct 15, 2025
38 checks passed

LantaoJin mentioned this pull request Nov 4, 2025
Add allowed_warnings in yaml restful tests #4731
Merged
8 tasks