Add LogicalSystemLimit automatically for data-intensive operations - #3749

Closed
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom
LantaoJin:pr/issues/3731
Closed

Add LogicalSystemLimit automatically for data-intensive operations#3749
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom
LantaoJin:pr/issues/3731

Conversation

@LantaoJin

@LantaoJinLantaoJin commented Jun 9, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Description

From v3.0.0, PPL introduces commands that may increase data volume. To prevent out-of-memory problem, the system automatically enforces a LogicalSystemLimit operator for such commands.

plugins.query.system_limit: The size configures the maximum of rows in the subsearch to data-intensive operations against (e.g. join, lookup). The default value is: 50000. Value range is from 0 to 2147483647 (Int.MaxValue).

Update

Now, all PPL join/lookup/expand commands (data-bloat) will be affected by this PR. In future, we can add more command argument to control specific command.

For Join, when join type is

  • SEMI, ANTI: no affect
  • RIGHT: add a LogicalSystemLimit operator to left side (main-search)
  • Others: add a LogicalSystemLimit operator to right side (sub-search)

For Lookup

  • add a LogicalSystemLimit operator to right side (sub-search)

For expand

  • add a LogicalSystemLimit operator to right side (sub-search)

The results of impacted search (for example, the lookup table of lookup command, right side of inner join, etc.)
cannot exceed the limitation (50000 rows by default). If the actual number of rows in lookup table or right side
of inner join is greater then the system limit, only the number of rows specified by the configuration will be searched.
You can set the configuration to the maximum integer value (2147483647) if you are certain resources are not a concern.

Related Issues

Resolves#3731

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@LantaoJin
Pushdown system limit automatically for data-intensive operations
90eb072
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix TPCH IT
f55d8de
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin marked this pull request as ready for review June 9, 2025 10:10
@LantaoJinLantaoJin added the calcite calcite migration releated label Jun 9, 2025
penghuo
penghuo reviewed Jun 9, 2025
View reviewed changes
Comment threaddocs/user/ppl/admin/settings.rst Outdated
Comment threadcommon/src/main/java/org/opensearch/sql/common/setting/Settings.java
Comment thread...earch/src/main/java/org/opensearch/sql/opensearch/storage/scan/AbstractCalciteIndexScan.java
Comment thread...search/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java Outdated
Comment threaddocs/user/admin/settings.rst Outdated
Comment thread...arch/src/main/java/org/opensearch/sql/opensearch/planner/physical/SystemLimitRuleConfig.java Outdated
@LantaoJin
address comments
02c1386
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix typo
fddf273
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3731
e885f9c
@LantaoJin
LantaoJin requested a review from penghuoJune 10, 2025 09:04
@qianheng-aws

qianheng-aws commented Jun 10, 2025

Copy link
Copy Markdown
Collaborator

  1. Will this change leads to incorrect results?

  2. To avoid data bloating, why not adding a limit operator for each child of join operator? It should have similar effect.

penghuo
penghuo previously approved these changes Jun 10, 2025
View reviewed changes
@penghuo
penghuo dismissed their stale reviewJune 10, 2025 15:33

new comments

penghuo
penghuo reviewed Jun 10, 2025
View reviewed changes
Comment threaddocs/user/ppl/limitations/limitations.rst Outdated
@LantaoJin
LantaoJin marked this pull request as draft June 11, 2025 08:56
@LantaoJin
revert unused code
7a372f6
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
c6c7600
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
efde8b9
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin changed the title Pushdown system limit automatically for data-intensive operationsAdd LogicalSystemLimit automatically for data-intensive operationsJun 11, 2025
@LantaoJin
Fix IT
29ee469
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin

LantaoJin commented Jun 11, 2025

Copy link
Copy Markdown
MemberAuthor

@penghuo@qianheng-aws@dai-chen I have updated the description with new code refactor, and docs. please take another look.

@LantaoJin
LantaoJin marked this pull request as ready for review June 11, 2025 10:03
This was referenced Jun 12, 2025
Closed
Closed
@opensearch-trigger-bot

opensearch-trigger-botBot commented Jul 18, 2025

Copy link
Copy Markdown
Contributor

This PR is stalled because it has been open for 30 days with no activity.

@LantaoJinLantaoJin mentioned this pull request Jul 31, 2025
Merged
7 tasks
@RyanL1997

RyanL1997 commented Aug 19, 2025

Copy link
Copy Markdown
Collaborator

Hi @LantaoJin , is this still requred?

@opensearch-trigger-bot

opensearch-trigger-botBot commented Sep 21, 2025

Copy link
Copy Markdown
Contributor

This PR is stalled because it has been open for 30 days with no activity.

@LantaoJin

LantaoJin commented Oct 10, 2025

Copy link
Copy Markdown
MemberAuthor

Close as there is a new alternative PR #4501

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@penghuopenghuopenghuo left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@dai-chendai-chenAwaiting requested review from dai-chendai-chen is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@SwiddisSwiddisAwaiting requested review from SwiddisSwiddis is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees

No one assigned

Labels

backport 2.19-devcalcitecalcite migration releatedstalled

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[ENHANCEMENT] Set operator limitation for data-intensive operators

4 participants

@LantaoJin@qianheng-aws@RyanL1997@penghuo
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Add LogicalSystemLimit automatically for data-intensive operations - #3749
Closed
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Closed
Add LogicalSystemLimit automatically for data-intensive operations#3749
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Jun 9, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
From v3.0.0, PPL introduces commands that may increase data volume. To prevent out-of-memory problem, the system automatically enforces a LogicalSystemLimit operator for such commands.
plugins.query.system_limit: The size configures the maximum of rows in the subsearch to data-intensive operations against (e.g. join, lookup). The default value is: 50000. Value range is from 0 to 2147483647 (Int.MaxValue).
Update
Now, all PPL join/lookup/expand commands (data-bloat) will be affected by this PR. In future, we can add more command argument to control specific command.
For Join, when join type is
  • SEMI, ANTI: no affect
  • RIGHT: add a LogicalSystemLimit operator to left side (main-search)
  • Others: add a LogicalSystemLimit operator to right side (sub-search)
For Lookup
  • add a LogicalSystemLimit operator to right side (sub-search)
For expand
  • add a LogicalSystemLimit operator to right side (sub-search)
The results of impacted search (for example, the lookup table of lookup command, right side of inner join, etc.)

cannot exceed the limitation (50000 rows by default). If the actual number of rows in lookup table or right side

of inner join is greater then the system limit, only the number of rows specified by the configuration will be searched.

You can set the configuration to the maximum integer value (2147483647) if you are certain resources are not a concern.
Related Issues
Resolves#3731
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Pushdown system limit automatically for data-intensive operations
90eb072
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix TPCH IT
f55d8de
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 9, 2025 10:10
@LantaoJinLantaoJin
added
the 
calcite
calcite migration releated
label
Jun 9, 2025
penghuo
penghuo
reviewed
Jun 9, 2025
View reviewed changes
Comment threaddocs/user/ppl/admin/settings.rst
Outdated

Comment threadcommon/src/main/java/org/opensearch/sql/common/setting/Settings.java
Comment thread...earch/src/main/java/org/opensearch/sql/opensearch/storage/scan/AbstractCalciteIndexScan.java
Comment thread...search/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java
Outdated

Comment threaddocs/user/admin/settings.rst
Outdated

Comment thread...arch/src/main/java/org/opensearch/sql/opensearch/planner/physical/SystemLimitRuleConfig.java
Outdated

@LantaoJin
address comments
02c1386
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix typo
fddf273
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3731
e885f9c
@LantaoJin
LantaoJin
requested a review
from penghuoJune 10, 2025 09:04
@qianheng-aws
qianheng-aws
commented
Jun 10, 2025

Copy link


Copy Markdown

Collaborator
  1. Will this change leads to incorrect results?
  2. To avoid data bloating, why not adding a limit operator for each child of join operator? It should have similar effect.
penghuo
penghuo
previously approved these changes
Jun 10, 2025
View reviewed changes
@penghuo
penghuo
dismissed
their stale reviewJune 10, 2025 15:33
new comments
penghuo
penghuo
reviewed
Jun 10, 2025
View reviewed changes
Comment threaddocs/user/ppl/limitations/limitations.rst
Outdated

@LantaoJin
LantaoJin
marked this pull request as draft
June 11, 2025 08:56
@LantaoJin
revert unused code
7a372f6
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
c6c7600
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
efde8b9
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Pushdown system limit automatically for data-intensive operationsAdd LogicalSystemLimit automatically for data-intensive operationsJun 11, 2025
@LantaoJin
Fix IT
29ee469
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
commented
Jun 11, 2025

Copy link


Copy Markdown

MemberAuthor
@penghuo@qianheng-aws@dai-chen I have updated the description with new code refactor, and docs. please take another look.
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 11, 2025 10:03

This was referenced Jun 12, 2025

Closed


Closed

@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Jul 18, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJinLantaoJin
mentioned this pull request
Jul 31, 2025
 Merged

7 tasks
@RyanL1997
RyanL1997
commented
Aug 19, 2025

Copy link


Copy Markdown

Collaborator
Hi @LantaoJin , is this still requred?
@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Sep 21, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJin
LantaoJin
commented
Oct 10, 2025

Copy link


Copy Markdown

MemberAuthor
Close as there is a new alternative PR #4501
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@penghuopenghuopenghuo left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@dai-chendai-chenAwaiting requested review from dai-chendai-chen is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@SwiddisSwiddisAwaiting requested review from SwiddisSwiddis is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport 2.19-devcalcitecalcite migration releatedstalled

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[ENHANCEMENT] Set operator limitation for data-intensive operators


4 participants

@LantaoJin@qianheng-aws@RyanL1997@penghuo

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add LogicalSystemLimit automatically for data-intensive operations - #3749
Closed
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Closed
Add LogicalSystemLimit automatically for data-intensive operations#3749
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Jun 9, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
From v3.0.0, PPL introduces commands that may increase data volume. To prevent out-of-memory problem, the system automatically enforces a LogicalSystemLimit operator for such commands.
plugins.query.system_limit: The size configures the maximum of rows in the subsearch to data-intensive operations against (e.g. join, lookup). The default value is: 50000. Value range is from 0 to 2147483647 (Int.MaxValue).
Update
Now, all PPL join/lookup/expand commands (data-bloat) will be affected by this PR. In future, we can add more command argument to control specific command.
For Join, when join type is
  • SEMI, ANTI: no affect
  • RIGHT: add a LogicalSystemLimit operator to left side (main-search)
  • Others: add a LogicalSystemLimit operator to right side (sub-search)
For Lookup
  • add a LogicalSystemLimit operator to right side (sub-search)
For expand
  • add a LogicalSystemLimit operator to right side (sub-search)
The results of impacted search (for example, the lookup table of lookup command, right side of inner join, etc.)

cannot exceed the limitation (50000 rows by default). If the actual number of rows in lookup table or right side

of inner join is greater then the system limit, only the number of rows specified by the configuration will be searched.

You can set the configuration to the maximum integer value (2147483647) if you are certain resources are not a concern.
Related Issues
Resolves#3731
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Pushdown system limit automatically for data-intensive operations
90eb072
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix TPCH IT
f55d8de
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 9, 2025 10:10
@LantaoJinLantaoJin
added
the 
calcite
calcite migration releated
label
Jun 9, 2025
penghuo
penghuo
reviewed
Jun 9, 2025
View reviewed changes
Comment threaddocs/user/ppl/admin/settings.rst
Outdated

Comment threadcommon/src/main/java/org/opensearch/sql/common/setting/Settings.java
Comment thread...earch/src/main/java/org/opensearch/sql/opensearch/storage/scan/AbstractCalciteIndexScan.java
Comment thread...search/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java
Outdated

Comment threaddocs/user/admin/settings.rst
Outdated

Comment thread...arch/src/main/java/org/opensearch/sql/opensearch/planner/physical/SystemLimitRuleConfig.java
Outdated

@LantaoJin
address comments
02c1386
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix typo
fddf273
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3731
e885f9c
@LantaoJin
LantaoJin
requested a review
from penghuoJune 10, 2025 09:04
@qianheng-aws
qianheng-aws
commented
Jun 10, 2025

Copy link


Copy Markdown

Collaborator
  1. Will this change leads to incorrect results?
  2. To avoid data bloating, why not adding a limit operator for each child of join operator? It should have similar effect.
penghuo
penghuo
previously approved these changes
Jun 10, 2025
View reviewed changes
@penghuo
penghuo
dismissed
their stale reviewJune 10, 2025 15:33
new comments
penghuo
penghuo
reviewed
Jun 10, 2025
View reviewed changes
Comment threaddocs/user/ppl/limitations/limitations.rst
Outdated

@LantaoJin
LantaoJin
marked this pull request as draft
June 11, 2025 08:56
@LantaoJin
revert unused code
7a372f6
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
c6c7600
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
efde8b9
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Pushdown system limit automatically for data-intensive operationsAdd LogicalSystemLimit automatically for data-intensive operationsJun 11, 2025
@LantaoJin
Fix IT
29ee469
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
commented
Jun 11, 2025

Copy link


Copy Markdown

MemberAuthor
@penghuo@qianheng-aws@dai-chen I have updated the description with new code refactor, and docs. please take another look.
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 11, 2025 10:03

This was referenced Jun 12, 2025

Closed


Closed

@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Jul 18, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJinLantaoJin
mentioned this pull request
Jul 31, 2025
 Merged

7 tasks
@RyanL1997
RyanL1997
commented
Aug 19, 2025

Copy link


Copy Markdown

Collaborator
Hi @LantaoJin , is this still requred?
@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Sep 21, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJin
LantaoJin
commented
Oct 10, 2025

Copy link


Copy Markdown

MemberAuthor
Close as there is a new alternative PR #4501
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@penghuopenghuopenghuo left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@dai-chendai-chenAwaiting requested review from dai-chendai-chen is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@SwiddisSwiddisAwaiting requested review from SwiddisSwiddis is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport 2.19-devcalcitecalcite migration releatedstalled

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[ENHANCEMENT] Set operator limitation for data-intensive operators


4 participants

@LantaoJin@qianheng-aws@RyanL1997@penghuo

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add LogicalSystemLimit automatically for data-intensive operations - #3749
Closed
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Closed
Add LogicalSystemLimit automatically for data-intensive operations#3749
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Jun 9, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
From v3.0.0, PPL introduces commands that may increase data volume. To prevent out-of-memory problem, the system automatically enforces a LogicalSystemLimit operator for such commands.
plugins.query.system_limit: The size configures the maximum of rows in the subsearch to data-intensive operations against (e.g. join, lookup). The default value is: 50000. Value range is from 0 to 2147483647 (Int.MaxValue).
Update
Now, all PPL join/lookup/expand commands (data-bloat) will be affected by this PR. In future, we can add more command argument to control specific command.
For Join, when join type is
  • SEMI, ANTI: no affect
  • RIGHT: add a LogicalSystemLimit operator to left side (main-search)
  • Others: add a LogicalSystemLimit operator to right side (sub-search)
For Lookup
  • add a LogicalSystemLimit operator to right side (sub-search)
For expand
  • add a LogicalSystemLimit operator to right side (sub-search)
The results of impacted search (for example, the lookup table of lookup command, right side of inner join, etc.)

cannot exceed the limitation (50000 rows by default). If the actual number of rows in lookup table or right side

of inner join is greater then the system limit, only the number of rows specified by the configuration will be searched.

You can set the configuration to the maximum integer value (2147483647) if you are certain resources are not a concern.
Related Issues
Resolves#3731
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Pushdown system limit automatically for data-intensive operations
90eb072
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix TPCH IT
f55d8de
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 9, 2025 10:10
@LantaoJinLantaoJin
added
the 
calcite
calcite migration releated
label
Jun 9, 2025
penghuo
penghuo
reviewed
Jun 9, 2025
View reviewed changes
Comment threaddocs/user/ppl/admin/settings.rst
Outdated

Comment threadcommon/src/main/java/org/opensearch/sql/common/setting/Settings.java
Comment thread...earch/src/main/java/org/opensearch/sql/opensearch/storage/scan/AbstractCalciteIndexScan.java
Comment thread...search/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java
Outdated

Comment threaddocs/user/admin/settings.rst
Outdated

Comment thread...arch/src/main/java/org/opensearch/sql/opensearch/planner/physical/SystemLimitRuleConfig.java
Outdated

@LantaoJin
address comments
02c1386
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix typo
fddf273
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3731
e885f9c
@LantaoJin
LantaoJin
requested a review
from penghuoJune 10, 2025 09:04
@qianheng-aws
qianheng-aws
commented
Jun 10, 2025

Copy link


Copy Markdown

Collaborator
  1. Will this change leads to incorrect results?
  2. To avoid data bloating, why not adding a limit operator for each child of join operator? It should have similar effect.
penghuo
penghuo
previously approved these changes
Jun 10, 2025
View reviewed changes
@penghuo
penghuo
dismissed
their stale reviewJune 10, 2025 15:33
new comments
penghuo
penghuo
reviewed
Jun 10, 2025
View reviewed changes
Comment threaddocs/user/ppl/limitations/limitations.rst
Outdated

@LantaoJin
LantaoJin
marked this pull request as draft
June 11, 2025 08:56
@LantaoJin
revert unused code
7a372f6
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
c6c7600
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
efde8b9
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Pushdown system limit automatically for data-intensive operationsAdd LogicalSystemLimit automatically for data-intensive operationsJun 11, 2025
@LantaoJin
Fix IT
29ee469
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
commented
Jun 11, 2025

Copy link


Copy Markdown

MemberAuthor
@penghuo@qianheng-aws@dai-chen I have updated the description with new code refactor, and docs. please take another look.
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 11, 2025 10:03

This was referenced Jun 12, 2025

Closed


Closed

@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Jul 18, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJinLantaoJin
mentioned this pull request
Jul 31, 2025
 Merged

7 tasks
@RyanL1997
RyanL1997
commented
Aug 19, 2025

Copy link


Copy Markdown

Collaborator
Hi @LantaoJin , is this still requred?
@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Sep 21, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJin
LantaoJin
commented
Oct 10, 2025

Copy link


Copy Markdown

MemberAuthor
Close as there is a new alternative PR #4501
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@penghuopenghuopenghuo left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@dai-chendai-chenAwaiting requested review from dai-chendai-chen is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@SwiddisSwiddisAwaiting requested review from SwiddisSwiddis is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport 2.19-devcalcitecalcite migration releatedstalled

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[ENHANCEMENT] Set operator limitation for data-intensive operators


4 participants

@LantaoJin@qianheng-aws@RyanL1997@penghuo

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Add LogicalSystemLimit automatically for data-intensive operations - #3749
Closed
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Closed
Add LogicalSystemLimit automatically for data-intensive operations#3749
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Jun 9, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
From v3.0.0, PPL introduces commands that may increase data volume. To prevent out-of-memory problem, the system automatically enforces a LogicalSystemLimit operator for such commands.
plugins.query.system_limit: The size configures the maximum of rows in the subsearch to data-intensive operations against (e.g. join, lookup). The default value is: 50000. Value range is from 0 to 2147483647 (Int.MaxValue).
Update
Now, all PPL join/lookup/expand commands (data-bloat) will be affected by this PR. In future, we can add more command argument to control specific command.
For Join, when join type is
  • SEMI, ANTI: no affect
  • RIGHT: add a LogicalSystemLimit operator to left side (main-search)
  • Others: add a LogicalSystemLimit operator to right side (sub-search)
For Lookup
  • add a LogicalSystemLimit operator to right side (sub-search)
For expand
  • add a LogicalSystemLimit operator to right side (sub-search)
The results of impacted search (for example, the lookup table of lookup command, right side of inner join, etc.)

cannot exceed the limitation (50000 rows by default). If the actual number of rows in lookup table or right side

of inner join is greater then the system limit, only the number of rows specified by the configuration will be searched.

You can set the configuration to the maximum integer value (2147483647) if you are certain resources are not a concern.
Related Issues
Resolves#3731
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Pushdown system limit automatically for data-intensive operations
90eb072
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix TPCH IT
f55d8de
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 9, 2025 10:10
@LantaoJinLantaoJin
added
the 
calcite
calcite migration releated
label
Jun 9, 2025
penghuo
penghuo
reviewed
Jun 9, 2025
View reviewed changes
Comment threaddocs/user/ppl/admin/settings.rst
Outdated

Comment threadcommon/src/main/java/org/opensearch/sql/common/setting/Settings.java
Comment thread...earch/src/main/java/org/opensearch/sql/opensearch/storage/scan/AbstractCalciteIndexScan.java
Comment thread...search/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java
Outdated

Comment threaddocs/user/admin/settings.rst
Outdated

Comment thread...arch/src/main/java/org/opensearch/sql/opensearch/planner/physical/SystemLimitRuleConfig.java
Outdated

@LantaoJin
address comments
02c1386
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix typo
fddf273
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3731
e885f9c
@LantaoJin
LantaoJin
requested a review
from penghuoJune 10, 2025 09:04
@qianheng-aws
qianheng-aws
commented
Jun 10, 2025

Copy link


Copy Markdown

Collaborator
  1. Will this change leads to incorrect results?
  2. To avoid data bloating, why not adding a limit operator for each child of join operator? It should have similar effect.
penghuo
penghuo
previously approved these changes
Jun 10, 2025
View reviewed changes
@penghuo
penghuo
dismissed
their stale reviewJune 10, 2025 15:33
new comments
penghuo
penghuo
reviewed
Jun 10, 2025
View reviewed changes
Comment threaddocs/user/ppl/limitations/limitations.rst
Outdated

@LantaoJin
LantaoJin
marked this pull request as draft
June 11, 2025 08:56
@LantaoJin
revert unused code
7a372f6
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
c6c7600
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
efde8b9
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Pushdown system limit automatically for data-intensive operationsAdd LogicalSystemLimit automatically for data-intensive operationsJun 11, 2025
@LantaoJin
Fix IT
29ee469
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
commented
Jun 11, 2025

Copy link


Copy Markdown

MemberAuthor
@penghuo@qianheng-aws@dai-chen I have updated the description with new code refactor, and docs. please take another look.
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 11, 2025 10:03

This was referenced Jun 12, 2025

Closed


Closed

@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Jul 18, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJinLantaoJin
mentioned this pull request
Jul 31, 2025
 Merged

7 tasks
@RyanL1997
RyanL1997
commented
Aug 19, 2025

Copy link


Copy Markdown

Collaborator
Hi @LantaoJin , is this still requred?
@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Sep 21, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJin
LantaoJin
commented
Oct 10, 2025

Copy link


Copy Markdown

MemberAuthor
Close as there is a new alternative PR #4501
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@penghuopenghuopenghuo left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@dai-chendai-chenAwaiting requested review from dai-chendai-chen is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@SwiddisSwiddisAwaiting requested review from SwiddisSwiddis is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport 2.19-devcalcitecalcite migration releatedstalled

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[ENHANCEMENT] Set operator limitation for data-intensive operators


4 participants

@LantaoJin@qianheng-aws@RyanL1997@penghuo

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add LogicalSystemLimit automatically for data-intensive operations - #3749
Closed
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Closed
Add LogicalSystemLimit automatically for data-intensive operations#3749
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Jun 9, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
From v3.0.0, PPL introduces commands that may increase data volume. To prevent out-of-memory problem, the system automatically enforces a LogicalSystemLimit operator for such commands.
plugins.query.system_limit: The size configures the maximum of rows in the subsearch to data-intensive operations against (e.g. join, lookup). The default value is: 50000. Value range is from 0 to 2147483647 (Int.MaxValue).
Update
Now, all PPL join/lookup/expand commands (data-bloat) will be affected by this PR. In future, we can add more command argument to control specific command.
For Join, when join type is
  • SEMI, ANTI: no affect
  • RIGHT: add a LogicalSystemLimit operator to left side (main-search)
  • Others: add a LogicalSystemLimit operator to right side (sub-search)
For Lookup
  • add a LogicalSystemLimit operator to right side (sub-search)
For expand
  • add a LogicalSystemLimit operator to right side (sub-search)
The results of impacted search (for example, the lookup table of lookup command, right side of inner join, etc.)

cannot exceed the limitation (50000 rows by default). If the actual number of rows in lookup table or right side

of inner join is greater then the system limit, only the number of rows specified by the configuration will be searched.

You can set the configuration to the maximum integer value (2147483647) if you are certain resources are not a concern.
Related Issues
Resolves#3731
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Pushdown system limit automatically for data-intensive operations
90eb072
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix TPCH IT
f55d8de
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 9, 2025 10:10
@LantaoJinLantaoJin
added
the 
calcite
calcite migration releated
label
Jun 9, 2025
penghuo
penghuo
reviewed
Jun 9, 2025
View reviewed changes
Comment threaddocs/user/ppl/admin/settings.rst
Outdated

Comment threadcommon/src/main/java/org/opensearch/sql/common/setting/Settings.java
Comment thread...earch/src/main/java/org/opensearch/sql/opensearch/storage/scan/AbstractCalciteIndexScan.java
Comment thread...search/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java
Outdated

Comment threaddocs/user/admin/settings.rst
Outdated

Comment thread...arch/src/main/java/org/opensearch/sql/opensearch/planner/physical/SystemLimitRuleConfig.java
Outdated

@LantaoJin
address comments
02c1386
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix typo
fddf273
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3731
e885f9c
@LantaoJin
LantaoJin
requested a review
from penghuoJune 10, 2025 09:04
@qianheng-aws
qianheng-aws
commented
Jun 10, 2025

Copy link


Copy Markdown

Collaborator
  1. Will this change leads to incorrect results?
  2. To avoid data bloating, why not adding a limit operator for each child of join operator? It should have similar effect.
penghuo
penghuo
previously approved these changes
Jun 10, 2025
View reviewed changes
@penghuo
penghuo
dismissed
their stale reviewJune 10, 2025 15:33
new comments
penghuo
penghuo
reviewed
Jun 10, 2025
View reviewed changes
Comment threaddocs/user/ppl/limitations/limitations.rst
Outdated

@LantaoJin
LantaoJin
marked this pull request as draft
June 11, 2025 08:56
@LantaoJin
revert unused code
7a372f6
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
c6c7600
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
efde8b9
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Pushdown system limit automatically for data-intensive operationsAdd LogicalSystemLimit automatically for data-intensive operationsJun 11, 2025
@LantaoJin
Fix IT
29ee469
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
commented
Jun 11, 2025

Copy link


Copy Markdown

MemberAuthor
@penghuo@qianheng-aws@dai-chen I have updated the description with new code refactor, and docs. please take another look.
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 11, 2025 10:03

This was referenced Jun 12, 2025

Closed


Closed

@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Jul 18, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJinLantaoJin
mentioned this pull request
Jul 31, 2025
 Merged

7 tasks
@RyanL1997
RyanL1997
commented
Aug 19, 2025

Copy link


Copy Markdown

Collaborator
Hi @LantaoJin , is this still requred?
@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Sep 21, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJin
LantaoJin
commented
Oct 10, 2025

Copy link


Copy Markdown

MemberAuthor
Close as there is a new alternative PR #4501
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@penghuopenghuopenghuo left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@dai-chendai-chenAwaiting requested review from dai-chendai-chen is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@SwiddisSwiddisAwaiting requested review from SwiddisSwiddis is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport 2.19-devcalcitecalcite migration releatedstalled

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[ENHANCEMENT] Set operator limitation for data-intensive operators


4 participants

@LantaoJin@qianheng-aws@RyanL1997@penghuo

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add LogicalSystemLimit automatically for data-intensive operations - #3749
Closed
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Closed
Add LogicalSystemLimit automatically for data-intensive operations#3749
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Jun 9, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
From v3.0.0, PPL introduces commands that may increase data volume. To prevent out-of-memory problem, the system automatically enforces a LogicalSystemLimit operator for such commands.
plugins.query.system_limit: The size configures the maximum of rows in the subsearch to data-intensive operations against (e.g. join, lookup). The default value is: 50000. Value range is from 0 to 2147483647 (Int.MaxValue).
Update
Now, all PPL join/lookup/expand commands (data-bloat) will be affected by this PR. In future, we can add more command argument to control specific command.
For Join, when join type is
  • SEMI, ANTI: no affect
  • RIGHT: add a LogicalSystemLimit operator to left side (main-search)
  • Others: add a LogicalSystemLimit operator to right side (sub-search)
For Lookup
  • add a LogicalSystemLimit operator to right side (sub-search)
For expand
  • add a LogicalSystemLimit operator to right side (sub-search)
The results of impacted search (for example, the lookup table of lookup command, right side of inner join, etc.)

cannot exceed the limitation (50000 rows by default). If the actual number of rows in lookup table or right side

of inner join is greater then the system limit, only the number of rows specified by the configuration will be searched.

You can set the configuration to the maximum integer value (2147483647) if you are certain resources are not a concern.
Related Issues
Resolves#3731
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Pushdown system limit automatically for data-intensive operations
90eb072
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix TPCH IT
f55d8de
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 9, 2025 10:10
@LantaoJinLantaoJin
added
the 
calcite
calcite migration releated
label
Jun 9, 2025
penghuo
penghuo
reviewed
Jun 9, 2025
View reviewed changes
Comment threaddocs/user/ppl/admin/settings.rst
Outdated

Comment threadcommon/src/main/java/org/opensearch/sql/common/setting/Settings.java
Comment thread...earch/src/main/java/org/opensearch/sql/opensearch/storage/scan/AbstractCalciteIndexScan.java
Comment thread...search/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java
Outdated

Comment threaddocs/user/admin/settings.rst
Outdated

Comment thread...arch/src/main/java/org/opensearch/sql/opensearch/planner/physical/SystemLimitRuleConfig.java
Outdated

@LantaoJin
address comments
02c1386
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix typo
fddf273
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3731
e885f9c
@LantaoJin
LantaoJin
requested a review
from penghuoJune 10, 2025 09:04
@qianheng-aws
qianheng-aws
commented
Jun 10, 2025

Copy link


Copy Markdown

Collaborator
  1. Will this change leads to incorrect results?
  2. To avoid data bloating, why not adding a limit operator for each child of join operator? It should have similar effect.
penghuo
penghuo
previously approved these changes
Jun 10, 2025
View reviewed changes
@penghuo
penghuo
dismissed
their stale reviewJune 10, 2025 15:33
new comments
penghuo
penghuo
reviewed
Jun 10, 2025
View reviewed changes
Comment threaddocs/user/ppl/limitations/limitations.rst
Outdated

@LantaoJin
LantaoJin
marked this pull request as draft
June 11, 2025 08:56
@LantaoJin
revert unused code
7a372f6
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
c6c7600
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
efde8b9
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Pushdown system limit automatically for data-intensive operationsAdd LogicalSystemLimit automatically for data-intensive operationsJun 11, 2025
@LantaoJin
Fix IT
29ee469
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
commented
Jun 11, 2025

Copy link


Copy Markdown

MemberAuthor
@penghuo@qianheng-aws@dai-chen I have updated the description with new code refactor, and docs. please take another look.
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 11, 2025 10:03

This was referenced Jun 12, 2025

Closed


Closed

@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Jul 18, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJinLantaoJin
mentioned this pull request
Jul 31, 2025
 Merged

7 tasks
@RyanL1997
RyanL1997
commented
Aug 19, 2025

Copy link


Copy Markdown

Collaborator
Hi @LantaoJin , is this still requred?
@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Sep 21, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJin
LantaoJin
commented
Oct 10, 2025

Copy link


Copy Markdown

MemberAuthor
Close as there is a new alternative PR #4501
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@penghuopenghuopenghuo left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@dai-chendai-chenAwaiting requested review from dai-chendai-chen is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@SwiddisSwiddisAwaiting requested review from SwiddisSwiddis is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport 2.19-devcalcitecalcite migration releatedstalled

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[ENHANCEMENT] Set operator limitation for data-intensive operators


4 participants

@LantaoJin@qianheng-aws@RyanL1997@penghuo

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Add LogicalSystemLimit automatically for data-intensive operations - #3749
Closed
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Closed
Add LogicalSystemLimit automatically for data-intensive operations#3749
LantaoJin wants to merge 13 commits into
opensearch-project:mainfrom 
LantaoJin:pr/issues/3731
Conversation
@LantaoJin
@LantaoJinLantaoJin
commented
Jun 9, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Description
From v3.0.0, PPL introduces commands that may increase data volume. To prevent out-of-memory problem, the system automatically enforces a LogicalSystemLimit operator for such commands.
plugins.query.system_limit: The size configures the maximum of rows in the subsearch to data-intensive operations against (e.g. join, lookup). The default value is: 50000. Value range is from 0 to 2147483647 (Int.MaxValue).
Update
Now, all PPL join/lookup/expand commands (data-bloat) will be affected by this PR. In future, we can add more command argument to control specific command.
For Join, when join type is
  • SEMI, ANTI: no affect
  • RIGHT: add a LogicalSystemLimit operator to left side (main-search)
  • Others: add a LogicalSystemLimit operator to right side (sub-search)
For Lookup
  • add a LogicalSystemLimit operator to right side (sub-search)
For expand
  • add a LogicalSystemLimit operator to right side (sub-search)
The results of impacted search (for example, the lookup table of lookup command, right side of inner join, etc.)

cannot exceed the limitation (50000 rows by default). If the actual number of rows in lookup table or right side

of inner join is greater then the system limit, only the number of rows specified by the configuration will be searched.

You can set the configuration to the maximum integer value (2147483647) if you are certain resources are not a concern.
Related Issues
Resolves#3731
Check List
  •  New functionality includes testing.
  •  New functionality has been documented.
  •  New functionality has javadoc added.
  •  New functionality has a user manual doc added.
  •  API changes companion pull request created.
  •  Commits are signed per the DCO using --signoff.
  •  Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@LantaoJin
Pushdown system limit automatically for data-intensive operations
90eb072
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix TPCH IT
f55d8de
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 9, 2025 10:10
@LantaoJinLantaoJin
added
the 
calcite
calcite migration releated
label
Jun 9, 2025
penghuo
penghuo
reviewed
Jun 9, 2025
View reviewed changes
Comment threaddocs/user/ppl/admin/settings.rst
Outdated

Comment threadcommon/src/main/java/org/opensearch/sql/common/setting/Settings.java
Comment thread...earch/src/main/java/org/opensearch/sql/opensearch/storage/scan/AbstractCalciteIndexScan.java
Comment thread...search/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteLogicalIndexScan.java
Outdated

Comment threaddocs/user/admin/settings.rst
Outdated

Comment thread...arch/src/main/java/org/opensearch/sql/opensearch/planner/physical/SystemLimitRuleConfig.java
Outdated

@LantaoJin
address comments
02c1386
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix typo
fddf273
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Merge remote-tracking branch 'upstream/main' into issues/3731
e885f9c
@LantaoJin
LantaoJin
requested a review
from penghuoJune 10, 2025 09:04
@qianheng-aws
qianheng-aws
commented
Jun 10, 2025

Copy link


Copy Markdown

Collaborator
  1. Will this change leads to incorrect results?
  2. To avoid data bloating, why not adding a limit operator for each child of join operator? It should have similar effect.
penghuo
penghuo
previously approved these changes
Jun 10, 2025
View reviewed changes
@penghuo
penghuo
dismissed
their stale reviewJune 10, 2025 15:33
new comments
penghuo
penghuo
reviewed
Jun 10, 2025
View reviewed changes
Comment threaddocs/user/ppl/limitations/limitations.rst
Outdated

@LantaoJin
LantaoJin
marked this pull request as draft
June 11, 2025 08:56
@LantaoJin
revert unused code
7a372f6
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
c6c7600
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
revert unused code
efde8b9
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJinLantaoJin
changed the title
Pushdown system limit automatically for data-intensive operationsAdd LogicalSystemLimit automatically for data-intensive operationsJun 11, 2025
@LantaoJin
Fix IT
29ee469
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
LantaoJin
commented
Jun 11, 2025

Copy link


Copy Markdown

MemberAuthor
@penghuo@qianheng-aws@dai-chen I have updated the description with new code refactor, and docs. please take another look.
@LantaoJin
LantaoJin
marked this pull request as ready for review
June 11, 2025 10:03

This was referenced Jun 12, 2025

Closed


Closed

@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Jul 18, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJinLantaoJin
mentioned this pull request
Jul 31, 2025
 Merged

7 tasks
@RyanL1997
RyanL1997
commented
Aug 19, 2025

Copy link


Copy Markdown

Collaborator
Hi @LantaoJin , is this still requred?
@opensearch-trigger-bot
opensearch-trigger-botBot
commented
Sep 21, 2025

Copy link


Copy Markdown

Contributor
This PR is stalled because it has been open for 30 days with no activity.
@LantaoJin
LantaoJin
commented
Oct 10, 2025

Copy link


Copy Markdown

MemberAuthor
Close as there is a new alternative PR #4501
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@penghuopenghuopenghuo left review comments
@ps48ps48Awaiting requested review from ps48ps48 is a code owner
@kavithacmkavithacmAwaiting requested review from kavithacm
@derek-hoderek-hoAwaiting requested review from derek-ho
@joshuali925joshuali925Awaiting requested review from joshuali925joshuali925 is a code owner
@dai-chendai-chenAwaiting requested review from dai-chendai-chen is a code owner
@YANG-DBYANG-DBAwaiting requested review from YANG-DB
@mengweiericmengweiericAwaiting requested review from mengweiericmengweieric is a code owner
@SwiddisSwiddisAwaiting requested review from SwiddisSwiddis is a code owner
@seankao-azseankao-azAwaiting requested review from seankao-az
@MaxKsyunzMaxKsyunzAwaiting requested review from MaxKsyunz
@Yury-FridlyandYury-FridlyandAwaiting requested review from Yury-Fridlyand
@anirudhaanirudhaAwaiting requested review from anirudhaanirudha is a code owner
@forestmveyforestmveyAwaiting requested review from forestmvey
@acarbonettoacarbonettoAwaiting requested review from acarbonettoacarbonetto is a code owner
@GumpacGGumpacGAwaiting requested review from GumpacG
@ykmr1224ykmr1224Awaiting requested review from ykmr1224ykmr1224 is a code owner
@noChargernoChargerAwaiting requested review from noChargernoCharger is a code owner
@qianheng-awsqianheng-awsAwaiting requested review from qianheng-awsqianheng-aws is a code owner

Assignees


No one assigned


Labels

backport 2.19-devcalcitecalcite migration releatedstalled

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[ENHANCEMENT] Set operator limitation for data-intensive operators


4 participants

@LantaoJin@qianheng-aws@RyanL1997@penghuo