Add configurable sytem limitations for subsearch and join command

[LantaoJin]

Description

Add two configurable limitations for PPL.

1. plugins.ppl.subsearch.maxout (default value 10000, similar to maxout in [subsearch], ref)
2. plugins.ppl.join.subsearch_maxout (default value 50000, similar to subsearch_maxout in [join], ref)

Related Issues

Resolves#3731 and #4430

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[penghuo]

nit, is it possible to avoid access private method?

2cents, Add a frame in CalcitePlanContext, frame is boundary of subsearch, and define limit on frame. When visit subsearch, append LogicalSystemLimit to subsearch on each frame.

[LantaoJin]

nit, is it possible to avoid access private method?

I don't think so.

When visit the subsearch side (right in join for example), the right plan was pushed to stack.

public RelNode analyze(UnresolvedPlan unresolved, CalcitePlanContext context) {
return unresolved.accept(this, context);
}

RelBuilder.pop() is private either. So we don't have a way to replace it.

Here was my previous try code for join

 public RelNode visitJoin(Join node, CalcitePlanContext context) {
// visit the main side
analyze(node.getLeft(), context);
if (context.sysLimit.joinSubsearchLimit() >= 0) {
// add join.subsearch_maxout limit to subsearch side
RelNode withLimit = context.relBuilder.with(
analyze(node.getRight(), context),
r -> LogicalSystemLimit.create(
SystemLimitType.JOIN_SUBSEARCH_MAXOUT,
r.peek(),
r.literal(context.sysLimit.joinSubsearchLimit())));
context.relBuilder.push(withLimit); // push the new subsearch plan
} else {
// visit the subsearch side
analyze(node.getRight(), context);
}

The code use relBuilder.with(), but the first parameter analyze(node.getRight(), context) will push the subsearch to stack, and the with() method push it twice.

 /** Evaluates an expression with a relational expression temporarily on the
* stack. */
public <E> E with(RelNode r, Function<RelBuilder, E> fn) {
try {
push(r);
return fn.apply(this);
} finally {
stack.pop();
}
}

1. push left plan by analyze(node.getLeft(), context), stack size is 1
2. push right plan by the first parameter of with(analyze(node.getRight(), context)), stack size is 2
3. push duplicated right plan by push in with, stack size is 3
4. pop duplicated right plan by pop in with, stack size is 2
5. push new right plan by context.relBuilder.push(withLimit), stack size is 3 (incorrect)

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[LantaoJin]

Sync offline. We still cannot use relbuilder.build() + relbuilder.push(newTop) since it will empty the fields of Frame.

 private void replaceTop(RelNode node) {
final Frame frame = stack.pop();
stack.push(new Frame(node, frame.fields)); // <--- frame.fields will be kept all the time
}

[penghuo]

How does SQL Join translate to RelNode? It use the private method?

[penghuo]

What if set to 0? Join/Subquery will be optimzied by Calcite?

[LantaoJin]

For now, I followed the standard behaviour in database:

select * from t_outer where exists (select 1 from t_inner where t_outer.id = t_inner.id limit 0);
select * from t_outer where id in (select id from t_inner limit 0);
select * from t_outer where id = (select id from t_inner limit 0);
select * from t_outer where id = (select count(*) from t_inner limit 0);

All above queries return empty in SQL (postgresql).

The implementation is here https://github.com/opensearch-project/sql/pull/4501/files#diff-e5198d773af75bf3173ef25676a2803a0091cb51e32d6ae30241273519d30261R601-R605

[LantaoJin]

What's your thoughts, set both 0 and negative value to unlimited? @penghuo

[penghuo]

discussed offline, 0 and -1 means unlimited.

[LantaoJin]

discussed offline, 0 and -1 means unlimited.

sure, let me update the code and doc to
0 means unlimited, and minValue=0 in Settings

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[qianheng-aws]

[question]Will there be case that there is join or union in subsearch? In those case there will be more than 1 input for the specific operators? If so, the current code will construct incorrect plan.

[LantaoJin]

Fixed in latest commit. For BiRel or SetOp, just return.

[qianheng-aws]

SQL support correlate condition for in subquery or scalar subquery. So Calcite should support them as well.
e.g.

SELECT * FROM EMPLOYEE WHERE location in (select location from DEPART where EMPLOYEE.dept = DEPART.name) limit 1

If there is correlate condition for in or scalar subsearch, shall we do similar operation like above?

[LantaoJin]

Added the same logic for correlated in-subquery. For correlated scalar-subquery, since there is always an aggregation will be perform in subquery, sysLimit is not necessary.

[opensearch-trigger-bot]

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repositorycd$(git rev-parse --show-toplevel)# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working treepushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4501-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fddbb705a6aeae138915e2174d5d7ea3ccbd3e9e
# Push it to GitHub
git push --set-upstream origin backport/backport-4501-to-2.19-dev
# Go back to the original working treepopd# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4501-to-2.19-dev.

[LantaoJin]

Description

Add two configurable limitations for PPL.

1. plugins.ppl.subsearch.maxout (default value 10000, similar to maxout in [subsearch], ref)
2. plugins.ppl.join.subsearch_maxout (default value 50000, similar to subsearch_maxout in [join], ref)

Related Issues

Resolves#3731 and #4430

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[penghuo]

nit, is it possible to avoid access private method?

2cents, Add a frame in CalcitePlanContext, frame is boundary of subsearch, and define limit on frame. When visit subsearch, append LogicalSystemLimit to subsearch on each frame.

[LantaoJin]

nit, is it possible to avoid access private method?

I don't think so.

When visit the subsearch side (right in join for example), the right plan was pushed to stack.

public RelNode analyze(UnresolvedPlan unresolved, CalcitePlanContext context) {
return unresolved.accept(this, context);
}

RelBuilder.pop() is private either. So we don't have a way to replace it.

Here was my previous try code for join

 public RelNode visitJoin(Join node, CalcitePlanContext context) {
// visit the main side
analyze(node.getLeft(), context);
if (context.sysLimit.joinSubsearchLimit() >= 0) {
// add join.subsearch_maxout limit to subsearch side
RelNode withLimit = context.relBuilder.with(
analyze(node.getRight(), context),
r -> LogicalSystemLimit.create(
SystemLimitType.JOIN_SUBSEARCH_MAXOUT,
r.peek(),
r.literal(context.sysLimit.joinSubsearchLimit())));
context.relBuilder.push(withLimit); // push the new subsearch plan
} else {
// visit the subsearch side
analyze(node.getRight(), context);
}

The code use relBuilder.with(), but the first parameter analyze(node.getRight(), context) will push the subsearch to stack, and the with() method push it twice.

 /** Evaluates an expression with a relational expression temporarily on the
* stack. */
public <E> E with(RelNode r, Function<RelBuilder, E> fn) {
try {
push(r);
return fn.apply(this);
} finally {
stack.pop();
}
}

1. push left plan by analyze(node.getLeft(), context), stack size is 1
2. push right plan by the first parameter of with(analyze(node.getRight(), context)), stack size is 2
3. push duplicated right plan by push in with, stack size is 3
4. pop duplicated right plan by pop in with, stack size is 2
5. push new right plan by context.relBuilder.push(withLimit), stack size is 3 (incorrect)

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[LantaoJin]

Sync offline. We still cannot use relbuilder.build() + relbuilder.push(newTop) since it will empty the fields of Frame.

 private void replaceTop(RelNode node) {
final Frame frame = stack.pop();
stack.push(new Frame(node, frame.fields)); // <--- frame.fields will be kept all the time
}

[penghuo]

How does SQL Join translate to RelNode? It use the private method?

[penghuo]

What if set to 0? Join/Subquery will be optimzied by Calcite?

[LantaoJin]

For now, I followed the standard behaviour in database:

select * from t_outer where exists (select 1 from t_inner where t_outer.id = t_inner.id limit 0);
select * from t_outer where id in (select id from t_inner limit 0);
select * from t_outer where id = (select id from t_inner limit 0);
select * from t_outer where id = (select count(*) from t_inner limit 0);

All above queries return empty in SQL (postgresql).

The implementation is here https://github.com/opensearch-project/sql/pull/4501/files#diff-e5198d773af75bf3173ef25676a2803a0091cb51e32d6ae30241273519d30261R601-R605

[LantaoJin]

What's your thoughts, set both 0 and negative value to unlimited? @penghuo

[penghuo]

discussed offline, 0 and -1 means unlimited.

[LantaoJin]

discussed offline, 0 and -1 means unlimited.

sure, let me update the code and doc to
0 means unlimited, and minValue=0 in Settings

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[qianheng-aws]

[question]Will there be case that there is join or union in subsearch? In those case there will be more than 1 input for the specific operators? If so, the current code will construct incorrect plan.

[LantaoJin]

Fixed in latest commit. For BiRel or SetOp, just return.

[qianheng-aws]

SQL support correlate condition for in subquery or scalar subquery. So Calcite should support them as well.
e.g.

SELECT * FROM EMPLOYEE WHERE location in (select location from DEPART where EMPLOYEE.dept = DEPART.name) limit 1

If there is correlate condition for in or scalar subsearch, shall we do similar operation like above?

[LantaoJin]

Added the same logic for correlated in-subquery. For correlated scalar-subquery, since there is always an aggregation will be perform in subquery, sysLimit is not necessary.

[opensearch-trigger-bot]

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repositorycd$(git rev-parse --show-toplevel)# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working treepushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4501-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fddbb705a6aeae138915e2174d5d7ea3ccbd3e9e
# Push it to GitHub
git push --set-upstream origin backport/backport-4501-to-2.19-dev
# Go back to the original working treepopd# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4501-to-2.19-dev.

[LantaoJin]

Description

Add two configurable limitations for PPL.

1. plugins.ppl.subsearch.maxout (default value 10000, similar to maxout in [subsearch], ref)
2. plugins.ppl.join.subsearch_maxout (default value 50000, similar to subsearch_maxout in [join], ref)

Related Issues

Resolves#3731 and #4430

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[penghuo]

nit, is it possible to avoid access private method?

2cents, Add a frame in CalcitePlanContext, frame is boundary of subsearch, and define limit on frame. When visit subsearch, append LogicalSystemLimit to subsearch on each frame.

[LantaoJin]

nit, is it possible to avoid access private method?

I don't think so.

When visit the subsearch side (right in join for example), the right plan was pushed to stack.

public RelNode analyze(UnresolvedPlan unresolved, CalcitePlanContext context) {
return unresolved.accept(this, context);
}

RelBuilder.pop() is private either. So we don't have a way to replace it.

Here was my previous try code for join

 public RelNode visitJoin(Join node, CalcitePlanContext context) {
// visit the main side
analyze(node.getLeft(), context);
if (context.sysLimit.joinSubsearchLimit() >= 0) {
// add join.subsearch_maxout limit to subsearch side
RelNode withLimit = context.relBuilder.with(
analyze(node.getRight(), context),
r -> LogicalSystemLimit.create(
SystemLimitType.JOIN_SUBSEARCH_MAXOUT,
r.peek(),
r.literal(context.sysLimit.joinSubsearchLimit())));
context.relBuilder.push(withLimit); // push the new subsearch plan
} else {
// visit the subsearch side
analyze(node.getRight(), context);
}

The code use relBuilder.with(), but the first parameter analyze(node.getRight(), context) will push the subsearch to stack, and the with() method push it twice.

 /** Evaluates an expression with a relational expression temporarily on the
* stack. */
public <E> E with(RelNode r, Function<RelBuilder, E> fn) {
try {
push(r);
return fn.apply(this);
} finally {
stack.pop();
}
}

1. push left plan by analyze(node.getLeft(), context), stack size is 1
2. push right plan by the first parameter of with(analyze(node.getRight(), context)), stack size is 2
3. push duplicated right plan by push in with, stack size is 3
4. pop duplicated right plan by pop in with, stack size is 2
5. push new right plan by context.relBuilder.push(withLimit), stack size is 3 (incorrect)

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[LantaoJin]

Sync offline. We still cannot use relbuilder.build() + relbuilder.push(newTop) since it will empty the fields of Frame.

 private void replaceTop(RelNode node) {
final Frame frame = stack.pop();
stack.push(new Frame(node, frame.fields)); // <--- frame.fields will be kept all the time
}

[penghuo]

How does SQL Join translate to RelNode? It use the private method?

[penghuo]

What if set to 0? Join/Subquery will be optimzied by Calcite?

[LantaoJin]

For now, I followed the standard behaviour in database:

select * from t_outer where exists (select 1 from t_inner where t_outer.id = t_inner.id limit 0);
select * from t_outer where id in (select id from t_inner limit 0);
select * from t_outer where id = (select id from t_inner limit 0);
select * from t_outer where id = (select count(*) from t_inner limit 0);

All above queries return empty in SQL (postgresql).

The implementation is here https://github.com/opensearch-project/sql/pull/4501/files#diff-e5198d773af75bf3173ef25676a2803a0091cb51e32d6ae30241273519d30261R601-R605

[LantaoJin]

What's your thoughts, set both 0 and negative value to unlimited? @penghuo

[penghuo]

discussed offline, 0 and -1 means unlimited.

[LantaoJin]

discussed offline, 0 and -1 means unlimited.

sure, let me update the code and doc to
0 means unlimited, and minValue=0 in Settings

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[qianheng-aws]

[question]Will there be case that there is join or union in subsearch? In those case there will be more than 1 input for the specific operators? If so, the current code will construct incorrect plan.

[LantaoJin]

Fixed in latest commit. For BiRel or SetOp, just return.

[qianheng-aws]

SQL support correlate condition for in subquery or scalar subquery. So Calcite should support them as well.
e.g.

SELECT * FROM EMPLOYEE WHERE location in (select location from DEPART where EMPLOYEE.dept = DEPART.name) limit 1

If there is correlate condition for in or scalar subsearch, shall we do similar operation like above?

[LantaoJin]

Added the same logic for correlated in-subquery. For correlated scalar-subquery, since there is always an aggregation will be perform in subquery, sysLimit is not necessary.

[opensearch-trigger-bot]

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repositorycd$(git rev-parse --show-toplevel)# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working treepushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4501-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fddbb705a6aeae138915e2174d5d7ea3ccbd3e9e
# Push it to GitHub
git push --set-upstream origin backport/backport-4501-to-2.19-dev
# Go back to the original working treepopd# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4501-to-2.19-dev.

[LantaoJin]

Description

Add two configurable limitations for PPL.

1. plugins.ppl.subsearch.maxout (default value 10000, similar to maxout in [subsearch], ref)
2. plugins.ppl.join.subsearch_maxout (default value 50000, similar to subsearch_maxout in [join], ref)

Related Issues

Resolves#3731 and #4430

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[penghuo]

nit, is it possible to avoid access private method?

2cents, Add a frame in CalcitePlanContext, frame is boundary of subsearch, and define limit on frame. When visit subsearch, append LogicalSystemLimit to subsearch on each frame.

[LantaoJin]

nit, is it possible to avoid access private method?

I don't think so.

When visit the subsearch side (right in join for example), the right plan was pushed to stack.

public RelNode analyze(UnresolvedPlan unresolved, CalcitePlanContext context) {
return unresolved.accept(this, context);
}

RelBuilder.pop() is private either. So we don't have a way to replace it.

Here was my previous try code for join

 public RelNode visitJoin(Join node, CalcitePlanContext context) {
// visit the main side
analyze(node.getLeft(), context);
if (context.sysLimit.joinSubsearchLimit() >= 0) {
// add join.subsearch_maxout limit to subsearch side
RelNode withLimit = context.relBuilder.with(
analyze(node.getRight(), context),
r -> LogicalSystemLimit.create(
SystemLimitType.JOIN_SUBSEARCH_MAXOUT,
r.peek(),
r.literal(context.sysLimit.joinSubsearchLimit())));
context.relBuilder.push(withLimit); // push the new subsearch plan
} else {
// visit the subsearch side
analyze(node.getRight(), context);
}

The code use relBuilder.with(), but the first parameter analyze(node.getRight(), context) will push the subsearch to stack, and the with() method push it twice.

 /** Evaluates an expression with a relational expression temporarily on the
* stack. */
public <E> E with(RelNode r, Function<RelBuilder, E> fn) {
try {
push(r);
return fn.apply(this);
} finally {
stack.pop();
}
}

1. push left plan by analyze(node.getLeft(), context), stack size is 1
2. push right plan by the first parameter of with(analyze(node.getRight(), context)), stack size is 2
3. push duplicated right plan by push in with, stack size is 3
4. pop duplicated right plan by pop in with, stack size is 2
5. push new right plan by context.relBuilder.push(withLimit), stack size is 3 (incorrect)

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[LantaoJin]

Sync offline. We still cannot use relbuilder.build() + relbuilder.push(newTop) since it will empty the fields of Frame.

 private void replaceTop(RelNode node) {
final Frame frame = stack.pop();
stack.push(new Frame(node, frame.fields)); // <--- frame.fields will be kept all the time
}

[penghuo]

How does SQL Join translate to RelNode? It use the private method?

[penghuo]

What if set to 0? Join/Subquery will be optimzied by Calcite?

[LantaoJin]

For now, I followed the standard behaviour in database:

select * from t_outer where exists (select 1 from t_inner where t_outer.id = t_inner.id limit 0);
select * from t_outer where id in (select id from t_inner limit 0);
select * from t_outer where id = (select id from t_inner limit 0);
select * from t_outer where id = (select count(*) from t_inner limit 0);

All above queries return empty in SQL (postgresql).

The implementation is here https://github.com/opensearch-project/sql/pull/4501/files#diff-e5198d773af75bf3173ef25676a2803a0091cb51e32d6ae30241273519d30261R601-R605

[LantaoJin]

What's your thoughts, set both 0 and negative value to unlimited? @penghuo

[penghuo]

discussed offline, 0 and -1 means unlimited.

[LantaoJin]

discussed offline, 0 and -1 means unlimited.

sure, let me update the code and doc to
0 means unlimited, and minValue=0 in Settings

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[qianheng-aws]

[question]Will there be case that there is join or union in subsearch? In those case there will be more than 1 input for the specific operators? If so, the current code will construct incorrect plan.

[LantaoJin]

Fixed in latest commit. For BiRel or SetOp, just return.

[qianheng-aws]

SQL support correlate condition for in subquery or scalar subquery. So Calcite should support them as well.
e.g.

SELECT * FROM EMPLOYEE WHERE location in (select location from DEPART where EMPLOYEE.dept = DEPART.name) limit 1

If there is correlate condition for in or scalar subsearch, shall we do similar operation like above?

[LantaoJin]

Added the same logic for correlated in-subquery. For correlated scalar-subquery, since there is always an aggregation will be perform in subquery, sysLimit is not necessary.

[opensearch-trigger-bot]

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repositorycd$(git rev-parse --show-toplevel)# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working treepushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4501-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fddbb705a6aeae138915e2174d5d7ea3ccbd3e9e
# Push it to GitHub
git push --set-upstream origin backport/backport-4501-to-2.19-dev
# Go back to the original working treepopd# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4501-to-2.19-dev.

[LantaoJin]

Description

Add two configurable limitations for PPL.

1. plugins.ppl.subsearch.maxout (default value 10000, similar to maxout in [subsearch], ref)
2. plugins.ppl.join.subsearch_maxout (default value 50000, similar to subsearch_maxout in [join], ref)

Related Issues

Resolves#3731 and #4430

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[penghuo]

nit, is it possible to avoid access private method?

2cents, Add a frame in CalcitePlanContext, frame is boundary of subsearch, and define limit on frame. When visit subsearch, append LogicalSystemLimit to subsearch on each frame.

[LantaoJin]

nit, is it possible to avoid access private method?

I don't think so.

When visit the subsearch side (right in join for example), the right plan was pushed to stack.

public RelNode analyze(UnresolvedPlan unresolved, CalcitePlanContext context) {
return unresolved.accept(this, context);
}

RelBuilder.pop() is private either. So we don't have a way to replace it.

Here was my previous try code for join

 public RelNode visitJoin(Join node, CalcitePlanContext context) {
// visit the main side
analyze(node.getLeft(), context);
if (context.sysLimit.joinSubsearchLimit() >= 0) {
// add join.subsearch_maxout limit to subsearch side
RelNode withLimit = context.relBuilder.with(
analyze(node.getRight(), context),
r -> LogicalSystemLimit.create(
SystemLimitType.JOIN_SUBSEARCH_MAXOUT,
r.peek(),
r.literal(context.sysLimit.joinSubsearchLimit())));
context.relBuilder.push(withLimit); // push the new subsearch plan
} else {
// visit the subsearch side
analyze(node.getRight(), context);
}

The code use relBuilder.with(), but the first parameter analyze(node.getRight(), context) will push the subsearch to stack, and the with() method push it twice.

 /** Evaluates an expression with a relational expression temporarily on the
* stack. */
public <E> E with(RelNode r, Function<RelBuilder, E> fn) {
try {
push(r);
return fn.apply(this);
} finally {
stack.pop();
}
}

1. push left plan by analyze(node.getLeft(), context), stack size is 1
2. push right plan by the first parameter of with(analyze(node.getRight(), context)), stack size is 2
3. push duplicated right plan by push in with, stack size is 3
4. pop duplicated right plan by pop in with, stack size is 2
5. push new right plan by context.relBuilder.push(withLimit), stack size is 3 (incorrect)

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[LantaoJin]

Sync offline. We still cannot use relbuilder.build() + relbuilder.push(newTop) since it will empty the fields of Frame.

 private void replaceTop(RelNode node) {
final Frame frame = stack.pop();
stack.push(new Frame(node, frame.fields)); // <--- frame.fields will be kept all the time
}

[penghuo]

How does SQL Join translate to RelNode? It use the private method?

[penghuo]

What if set to 0? Join/Subquery will be optimzied by Calcite?

[LantaoJin]

For now, I followed the standard behaviour in database:

select * from t_outer where exists (select 1 from t_inner where t_outer.id = t_inner.id limit 0);
select * from t_outer where id in (select id from t_inner limit 0);
select * from t_outer where id = (select id from t_inner limit 0);
select * from t_outer where id = (select count(*) from t_inner limit 0);

All above queries return empty in SQL (postgresql).

The implementation is here https://github.com/opensearch-project/sql/pull/4501/files#diff-e5198d773af75bf3173ef25676a2803a0091cb51e32d6ae30241273519d30261R601-R605

[LantaoJin]

What's your thoughts, set both 0 and negative value to unlimited? @penghuo

[penghuo]

discussed offline, 0 and -1 means unlimited.

[LantaoJin]

discussed offline, 0 and -1 means unlimited.

sure, let me update the code and doc to
0 means unlimited, and minValue=0 in Settings

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[qianheng-aws]

[question]Will there be case that there is join or union in subsearch? In those case there will be more than 1 input for the specific operators? If so, the current code will construct incorrect plan.

[LantaoJin]

Fixed in latest commit. For BiRel or SetOp, just return.

[qianheng-aws]

SQL support correlate condition for in subquery or scalar subquery. So Calcite should support them as well.
e.g.

SELECT * FROM EMPLOYEE WHERE location in (select location from DEPART where EMPLOYEE.dept = DEPART.name) limit 1

If there is correlate condition for in or scalar subsearch, shall we do similar operation like above?

[LantaoJin]

Added the same logic for correlated in-subquery. For correlated scalar-subquery, since there is always an aggregation will be perform in subquery, sysLimit is not necessary.

[opensearch-trigger-bot]

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repositorycd$(git rev-parse --show-toplevel)# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working treepushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4501-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fddbb705a6aeae138915e2174d5d7ea3ccbd3e9e
# Push it to GitHub
git push --set-upstream origin backport/backport-4501-to-2.19-dev
# Go back to the original working treepopd# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4501-to-2.19-dev.

[LantaoJin]

Description

Add two configurable limitations for PPL.

1. plugins.ppl.subsearch.maxout (default value 10000, similar to maxout in [subsearch], ref)
2. plugins.ppl.join.subsearch_maxout (default value 50000, similar to subsearch_maxout in [join], ref)

Related Issues

Resolves#3731 and #4430

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[penghuo]

nit, is it possible to avoid access private method?

2cents, Add a frame in CalcitePlanContext, frame is boundary of subsearch, and define limit on frame. When visit subsearch, append LogicalSystemLimit to subsearch on each frame.

[LantaoJin]

nit, is it possible to avoid access private method?

I don't think so.

When visit the subsearch side (right in join for example), the right plan was pushed to stack.

public RelNode analyze(UnresolvedPlan unresolved, CalcitePlanContext context) {
return unresolved.accept(this, context);
}

RelBuilder.pop() is private either. So we don't have a way to replace it.

Here was my previous try code for join

 public RelNode visitJoin(Join node, CalcitePlanContext context) {
// visit the main side
analyze(node.getLeft(), context);
if (context.sysLimit.joinSubsearchLimit() >= 0) {
// add join.subsearch_maxout limit to subsearch side
RelNode withLimit = context.relBuilder.with(
analyze(node.getRight(), context),
r -> LogicalSystemLimit.create(
SystemLimitType.JOIN_SUBSEARCH_MAXOUT,
r.peek(),
r.literal(context.sysLimit.joinSubsearchLimit())));
context.relBuilder.push(withLimit); // push the new subsearch plan
} else {
// visit the subsearch side
analyze(node.getRight(), context);
}

The code use relBuilder.with(), but the first parameter analyze(node.getRight(), context) will push the subsearch to stack, and the with() method push it twice.

 /** Evaluates an expression with a relational expression temporarily on the
* stack. */
public <E> E with(RelNode r, Function<RelBuilder, E> fn) {
try {
push(r);
return fn.apply(this);
} finally {
stack.pop();
}
}

1. push left plan by analyze(node.getLeft(), context), stack size is 1
2. push right plan by the first parameter of with(analyze(node.getRight(), context)), stack size is 2
3. push duplicated right plan by push in with, stack size is 3
4. pop duplicated right plan by pop in with, stack size is 2
5. push new right plan by context.relBuilder.push(withLimit), stack size is 3 (incorrect)

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[LantaoJin]

Sync offline. We still cannot use relbuilder.build() + relbuilder.push(newTop) since it will empty the fields of Frame.

 private void replaceTop(RelNode node) {
final Frame frame = stack.pop();
stack.push(new Frame(node, frame.fields)); // <--- frame.fields will be kept all the time
}

[penghuo]

How does SQL Join translate to RelNode? It use the private method?

[penghuo]

What if set to 0? Join/Subquery will be optimzied by Calcite?

[LantaoJin]

For now, I followed the standard behaviour in database:

select * from t_outer where exists (select 1 from t_inner where t_outer.id = t_inner.id limit 0);
select * from t_outer where id in (select id from t_inner limit 0);
select * from t_outer where id = (select id from t_inner limit 0);
select * from t_outer where id = (select count(*) from t_inner limit 0);

All above queries return empty in SQL (postgresql).

The implementation is here https://github.com/opensearch-project/sql/pull/4501/files#diff-e5198d773af75bf3173ef25676a2803a0091cb51e32d6ae30241273519d30261R601-R605

[LantaoJin]

What's your thoughts, set both 0 and negative value to unlimited? @penghuo

[penghuo]

discussed offline, 0 and -1 means unlimited.

[LantaoJin]

discussed offline, 0 and -1 means unlimited.

sure, let me update the code and doc to
0 means unlimited, and minValue=0 in Settings

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[qianheng-aws]

[question]Will there be case that there is join or union in subsearch? In those case there will be more than 1 input for the specific operators? If so, the current code will construct incorrect plan.

[LantaoJin]

Fixed in latest commit. For BiRel or SetOp, just return.

[qianheng-aws]

SQL support correlate condition for in subquery or scalar subquery. So Calcite should support them as well.
e.g.

SELECT * FROM EMPLOYEE WHERE location in (select location from DEPART where EMPLOYEE.dept = DEPART.name) limit 1

If there is correlate condition for in or scalar subsearch, shall we do similar operation like above?

[LantaoJin]

Added the same logic for correlated in-subquery. For correlated scalar-subquery, since there is always an aggregation will be perform in subquery, sysLimit is not necessary.

[opensearch-trigger-bot]

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repositorycd$(git rev-parse --show-toplevel)# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working treepushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4501-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fddbb705a6aeae138915e2174d5d7ea3ccbd3e9e
# Push it to GitHub
git push --set-upstream origin backport/backport-4501-to-2.19-dev
# Go back to the original working treepopd# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4501-to-2.19-dev.

[LantaoJin]

Description

Add two configurable limitations for PPL.

1. plugins.ppl.subsearch.maxout (default value 10000, similar to maxout in [subsearch], ref)
2. plugins.ppl.join.subsearch_maxout (default value 50000, similar to subsearch_maxout in [join], ref)

Related Issues

Resolves#3731 and #4430

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[penghuo]

nit, is it possible to avoid access private method?

2cents, Add a frame in CalcitePlanContext, frame is boundary of subsearch, and define limit on frame. When visit subsearch, append LogicalSystemLimit to subsearch on each frame.

[LantaoJin]

nit, is it possible to avoid access private method?

I don't think so.

When visit the subsearch side (right in join for example), the right plan was pushed to stack.

public RelNode analyze(UnresolvedPlan unresolved, CalcitePlanContext context) {
return unresolved.accept(this, context);
}

RelBuilder.pop() is private either. So we don't have a way to replace it.

Here was my previous try code for join

 public RelNode visitJoin(Join node, CalcitePlanContext context) {
// visit the main side
analyze(node.getLeft(), context);
if (context.sysLimit.joinSubsearchLimit() >= 0) {
// add join.subsearch_maxout limit to subsearch side
RelNode withLimit = context.relBuilder.with(
analyze(node.getRight(), context),
r -> LogicalSystemLimit.create(
SystemLimitType.JOIN_SUBSEARCH_MAXOUT,
r.peek(),
r.literal(context.sysLimit.joinSubsearchLimit())));
context.relBuilder.push(withLimit); // push the new subsearch plan
} else {
// visit the subsearch side
analyze(node.getRight(), context);
}

The code use relBuilder.with(), but the first parameter analyze(node.getRight(), context) will push the subsearch to stack, and the with() method push it twice.

 /** Evaluates an expression with a relational expression temporarily on the
* stack. */
public <E> E with(RelNode r, Function<RelBuilder, E> fn) {
try {
push(r);
return fn.apply(this);
} finally {
stack.pop();
}
}

1. push left plan by analyze(node.getLeft(), context), stack size is 1
2. push right plan by the first parameter of with(analyze(node.getRight(), context)), stack size is 2
3. push duplicated right plan by push in with, stack size is 3
4. pop duplicated right plan by pop in with, stack size is 2
5. push new right plan by context.relBuilder.push(withLimit), stack size is 3 (incorrect)

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[LantaoJin]

Sync offline. We still cannot use relbuilder.build() + relbuilder.push(newTop) since it will empty the fields of Frame.

 private void replaceTop(RelNode node) {
final Frame frame = stack.pop();
stack.push(new Frame(node, frame.fields)); // <--- frame.fields will be kept all the time
}

[penghuo]

How does SQL Join translate to RelNode? It use the private method?

[penghuo]

What if set to 0? Join/Subquery will be optimzied by Calcite?

[LantaoJin]

For now, I followed the standard behaviour in database:

select * from t_outer where exists (select 1 from t_inner where t_outer.id = t_inner.id limit 0);
select * from t_outer where id in (select id from t_inner limit 0);
select * from t_outer where id = (select id from t_inner limit 0);
select * from t_outer where id = (select count(*) from t_inner limit 0);

All above queries return empty in SQL (postgresql).

The implementation is here https://github.com/opensearch-project/sql/pull/4501/files#diff-e5198d773af75bf3173ef25676a2803a0091cb51e32d6ae30241273519d30261R601-R605

[LantaoJin]

What's your thoughts, set both 0 and negative value to unlimited? @penghuo

[penghuo]

discussed offline, 0 and -1 means unlimited.

[LantaoJin]

discussed offline, 0 and -1 means unlimited.

sure, let me update the code and doc to
0 means unlimited, and minValue=0 in Settings

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[qianheng-aws]

[question]Will there be case that there is join or union in subsearch? In those case there will be more than 1 input for the specific operators? If so, the current code will construct incorrect plan.

[LantaoJin]

Fixed in latest commit. For BiRel or SetOp, just return.

[qianheng-aws]

SQL support correlate condition for in subquery or scalar subquery. So Calcite should support them as well.
e.g.

SELECT * FROM EMPLOYEE WHERE location in (select location from DEPART where EMPLOYEE.dept = DEPART.name) limit 1

If there is correlate condition for in or scalar subsearch, shall we do similar operation like above?

[LantaoJin]

Added the same logic for correlated in-subquery. For correlated scalar-subquery, since there is always an aggregation will be perform in subquery, sysLimit is not necessary.

[opensearch-trigger-bot]

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repositorycd$(git rev-parse --show-toplevel)# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working treepushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4501-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fddbb705a6aeae138915e2174d5d7ea3ccbd3e9e
# Push it to GitHub
git push --set-upstream origin backport/backport-4501-to-2.19-dev
# Go back to the original working treepopd# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4501-to-2.19-dev.

[LantaoJin]

Description

Add two configurable limitations for PPL.

1. plugins.ppl.subsearch.maxout (default value 10000, similar to maxout in [subsearch], ref)
2. plugins.ppl.join.subsearch_maxout (default value 50000, similar to subsearch_maxout in [join], ref)

Related Issues

Resolves#3731 and #4430

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[penghuo]

nit, is it possible to avoid access private method?

2cents, Add a frame in CalcitePlanContext, frame is boundary of subsearch, and define limit on frame. When visit subsearch, append LogicalSystemLimit to subsearch on each frame.

[LantaoJin]

nit, is it possible to avoid access private method?

I don't think so.

When visit the subsearch side (right in join for example), the right plan was pushed to stack.

public RelNode analyze(UnresolvedPlan unresolved, CalcitePlanContext context) {
return unresolved.accept(this, context);
}

RelBuilder.pop() is private either. So we don't have a way to replace it.

Here was my previous try code for join

 public RelNode visitJoin(Join node, CalcitePlanContext context) {
// visit the main side
analyze(node.getLeft(), context);
if (context.sysLimit.joinSubsearchLimit() >= 0) {
// add join.subsearch_maxout limit to subsearch side
RelNode withLimit = context.relBuilder.with(
analyze(node.getRight(), context),
r -> LogicalSystemLimit.create(
SystemLimitType.JOIN_SUBSEARCH_MAXOUT,
r.peek(),
r.literal(context.sysLimit.joinSubsearchLimit())));
context.relBuilder.push(withLimit); // push the new subsearch plan
} else {
// visit the subsearch side
analyze(node.getRight(), context);
}

The code use relBuilder.with(), but the first parameter analyze(node.getRight(), context) will push the subsearch to stack, and the with() method push it twice.

 /** Evaluates an expression with a relational expression temporarily on the
* stack. */
public <E> E with(RelNode r, Function<RelBuilder, E> fn) {
try {
push(r);
return fn.apply(this);
} finally {
stack.pop();
}
}

1. push left plan by analyze(node.getLeft(), context), stack size is 1
2. push right plan by the first parameter of with(analyze(node.getRight(), context)), stack size is 2
3. push duplicated right plan by push in with, stack size is 3
4. pop duplicated right plan by pop in with, stack size is 2
5. push new right plan by context.relBuilder.push(withLimit), stack size is 3 (incorrect)

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[LantaoJin]

Sync offline. We still cannot use relbuilder.build() + relbuilder.push(newTop) since it will empty the fields of Frame.

 private void replaceTop(RelNode node) {
final Frame frame = stack.pop();
stack.push(new Frame(node, frame.fields)); // <--- frame.fields will be kept all the time
}

[penghuo]

How does SQL Join translate to RelNode? It use the private method?

[penghuo]

What if set to 0? Join/Subquery will be optimzied by Calcite?

[LantaoJin]

For now, I followed the standard behaviour in database:

select * from t_outer where exists (select 1 from t_inner where t_outer.id = t_inner.id limit 0);
select * from t_outer where id in (select id from t_inner limit 0);
select * from t_outer where id = (select id from t_inner limit 0);
select * from t_outer where id = (select count(*) from t_inner limit 0);

All above queries return empty in SQL (postgresql).

The implementation is here https://github.com/opensearch-project/sql/pull/4501/files#diff-e5198d773af75bf3173ef25676a2803a0091cb51e32d6ae30241273519d30261R601-R605

[LantaoJin]

What's your thoughts, set both 0 and negative value to unlimited? @penghuo

[penghuo]

discussed offline, 0 and -1 means unlimited.

[LantaoJin]

discussed offline, 0 and -1 means unlimited.

sure, let me update the code and doc to
0 means unlimited, and minValue=0 in Settings

[qianheng-aws]

Does it work by using relbuilder.build() + relbuilder.push(newTop)? relbuilder.build() will do pop while public.

[qianheng-aws]

[question]Will there be case that there is join or union in subsearch? In those case there will be more than 1 input for the specific operators? If so, the current code will construct incorrect plan.

[LantaoJin]

Fixed in latest commit. For BiRel or SetOp, just return.

[qianheng-aws]

SQL support correlate condition for in subquery or scalar subquery. So Calcite should support them as well.
e.g.

SELECT * FROM EMPLOYEE WHERE location in (select location from DEPART where EMPLOYEE.dept = DEPART.name) limit 1

If there is correlate condition for in or scalar subsearch, shall we do similar operation like above?

[LantaoJin]

Added the same logic for correlated in-subquery. For correlated scalar-subquery, since there is always an aggregation will be perform in subquery, sysLimit is not necessary.

[opensearch-trigger-bot]

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repositorycd$(git rev-parse --show-toplevel)# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working treepushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4501-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fddbb705a6aeae138915e2174d5d7ea3ccbd3e9e
# Push it to GitHub
git push --set-upstream origin backport/backport-4501-to-2.19-dev
# Go back to the original working treepopd# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4501-to-2.19-dev.