Skip to content

PHOENIX-5072 Cursor Query Loops Eternally with Local Index, Returns F… - #1287

Merged
richardantal merged 1 commit into
apache:masterfrom
richardantal:PHOENIX-5072
Sep 14, 2021
Merged

PHOENIX-5072 Cursor Query Loops Eternally with Local Index, Returns F…#1287
richardantal merged 1 commit into
apache:masterfrom
richardantal:PHOENIX-5072

Conversation

@richardantal

Copy link
Copy Markdown
Contributor

…ine Without It

I found out that when we had (local) index then we didn't have the CursorFetchPlan but it was optimized to a ScanPlan on the index.

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec6m 50sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall24m 37smaster passed
+0hbaserecompile33m 42sHBase recompiled.
+1 💚compile1m 7smaster passed
+1 💚checkstyle1m 5smaster passed
+1 💚javadoc0m 52smaster passed
+0 🆗spotbugs3m 26sphoenix-core in master has 973 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall16m 51sthe patch passed
+0hbaserecompile27m 19sHBase recompiled.
+1 💚compile1m 6sthe patch passed
+1 💚javac1m 6sthe patch passed
-1 ❌checkstyle1m 7sphoenix-core: The patch generated 6 new + 212 unchanged - 3 fixed = 218 total (was 215)
-1 ❌whitespace0m 0sThe patch 3 line(s) with tabs.
+1 💚javadoc0m 51sthe patch passed
+1 💚spotbugs3m 29sthe patch passed
_ Other Tests _
+1 💚unit120m 59sphoenix-core in the patch passed.
+1 💚asflicense0m 13sThe patch does not generate ASF License warnings.
199m 53s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#1287
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
unameLinux 1040f3c493cf 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revisionmaster / c4d297d
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
whitespacehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/1/artifact/yetus-general-check/output/whitespace-tabs.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/1/testReport/
Max. process+thread count9697 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/1/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

throw new RowValueConstructorOffsetNotCoercibleException("No table or index could be coerced to the PK as the offset. Or an uncovered index was attempted");
}

if (applicablePlans.get(0) instanceof CursorFetchPlan) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather put in a generic query method if possible. This code seems to imply that the cursor fetch plan is always first followed by several normal execution plans. This special case handling seems awkward at best is. it possible to generalize this somehow either in initial plan creation ie create them as separate cursor plans or maybe as part of a general plan method? I probably am missing a bunch of details.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the review @dbwong !

@richardantal

Copy link
Copy Markdown
ContributorAuthor

The ScanPlan in CursorFetchPlan was not using the index when we called the optimize on the CursorFetchPlan it added a simple ScanPlan (using the index) to the applicablePlans and at the and it got selected as the best one.

I've updated the constructor to optimize the ScanPlan at the first place.

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec6m 32sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall23m 47smaster passed
+0hbaserecompile30m 52sHBase recompiled.
+1 💚compile1m 6smaster passed
+1 💚checkstyle1m 2smaster passed
+1 💚javadoc0m 53smaster passed
+0 🆗spotbugs3m 21sphoenix-core in master has 972 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall17m 10sthe patch passed
+0hbaserecompile27m 34sHBase recompiled.
+1 💚compile1m 5sthe patch passed
+1 💚javac1m 5sthe patch passed
-1 ❌checkstyle1m 2sphoenix-core: The patch generated 2 new + 21 unchanged - 1 fixed = 23 total (was 22)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 52sthe patch passed
+1 💚spotbugs3m 31sthe patch passed
_ Other Tests _
-1 ❌unit120m 27sphoenix-core in the patch failed.
+1 💚asflicense0m 34sThe patch does not generate ASF License warnings.
198m 29s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#1287
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
unameLinux 99d9f524882d 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revisionmaster / 9dfb423
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/2/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/2/testReport/
Max. process+thread count9240 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/2/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

try {
compilePlan = statement.getConnection().getQueryServices().getOptimizer().optimize(statement, queryPlan);
} catch (SQLException e) {
e.printStackTrace();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use logger here.

QueryPlan compilePlan = queryPlan;
try {
compilePlan = statement.getConnection().getQueryServices().getOptimizer().optimize(statement, queryPlan);
} catch (SQLException e) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we want to handle the exception, shouldn't we just propogate?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to handle the exception to minimise the effect of this change in case the optimize() throws an Exception.
Even if we handle the exception here (in the DeclareCursorCompiler) later optimize() will be called on the CursorFetchPlan and the Exception would be thrown there so it doesn't really makes sense to handle it here probably.

@richardantal

Copy link
Copy Markdown
ContributorAuthor

Thank you @ankitsinghal for the review.
I've updated the PR.

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec7m 2sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall27m 13smaster passed
+0hbaserecompile35m 56sHBase recompiled.
+1 💚compile1m 20smaster passed
+1 💚checkstyle1m 11smaster passed
+1 💚javadoc1m 3smaster passed
+0 🆗spotbugs4m 31sphoenix-core in master has 972 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall19m 43sthe patch passed
+0hbaserecompile32m 24sHBase recompiled.
+1 💚compile1m 25sthe patch passed
+1 💚javac1m 25sthe patch passed
-1 ❌checkstyle1m 14sphoenix-core: The patch generated 1 new + 21 unchanged - 1 fixed = 22 total (was 22)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc1m 0sthe patch passed
+1 💚spotbugs4m 31sthe patch passed
_ Other Tests _
-1 ❌unit117m 12sphoenix-core in the patch failed.
+1 💚asflicense0m 42sThe patch does not generate ASF License warnings.
208m 39s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#1287
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
unameLinux b0ce7ffc49e8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revisionmaster / 9dfb423
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/3/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/3/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/3/testReport/
Max. process+thread count14450 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/3/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@stotystoty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some nits on the test.

this.statement = statement;
this.operation = operation;
this.queryPlan = queryPlan;
this.queryPlan = statement.getConnection().getQueryServices().getOptimizer()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this change help?
By pre-optimizing the query, we force the next optimization plan to be a No-Op ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this change we optimize the plan inside the CursorFetchPlan.
Later when the next optimize is called, the original CursorFetchPlan will be selected as there won't be any better plans.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a code comment with this information and a reference to this ticket.

@stoty

stoty commented Sep 8, 2021

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec8m 50sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall27m 29smaster passed
+0hbaserecompile35m 49sHBase recompiled.
+1 💚compile1m 24smaster passed
+1 💚checkstyle1m 20smaster passed
+1 💚javadoc1m 1smaster passed
+0 🆗spotbugs3m 54sphoenix-core in master has 972 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall19m 43sthe patch passed
+0hbaserecompile32m 40sHBase recompiled.
+1 💚compile1m 20sthe patch passed
+1 💚javac1m 20sthe patch passed
-1 ❌checkstyle1m 15sphoenix-core: The patch generated 1 new + 21 unchanged - 1 fixed = 22 total (was 22)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc1m 1sthe patch passed
+1 💚spotbugs4m 22sthe patch passed
_ Other Tests _
-1 ❌unit139m 42sphoenix-core in the patch failed.
+1 💚asflicense0m 37sThe patch does not generate ASF License warnings.
232m 45s
ReasonTests
Failed junit testsphoenix.end2end.AuditLoggingIT
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#1287
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
unameLinux 48a473be5f8c 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revisionmaster / 19b2260
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/4/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/4/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/4/testReport/
Max. process+thread count7881 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/4/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@stotystoty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM (but better check with @ankitsinghal too)

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec4m 36sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall20m 54smaster passed
+0hbaserecompile27m 25sHBase recompiled.
+1 💚compile1m 5smaster passed
+1 💚checkstyle0m 56smaster passed
+1 💚javadoc0m 53smaster passed
+0 🆗spotbugs3m 8sphoenix-core in master has 972 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall13m 35sthe patch passed
+0hbaserecompile23m 24sHBase recompiled.
+1 💚compile1m 4sthe patch passed
+1 💚javac1m 4sthe patch passed
-1 ❌checkstyle0m 56sphoenix-core: The patch generated 3 new + 20 unchanged - 2 fixed = 23 total (was 22)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 50sthe patch passed
+1 💚spotbugs3m 19sthe patch passed
_ Other Tests _
-1 ❌unit112m 29sphoenix-core in the patch failed.
+1 💚asflicense0m 45sThe patch does not generate ASF License warnings.
180m 53s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#1287
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
unameLinux 13afcb19e82d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revisionmaster / 19b2260
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/5/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/5/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/5/testReport/
Max. process+thread count15556 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1287/5/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@richardantal
richardantal merged commit c55b472 into apache:masterSep 14, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@richardantal@stoty@ankitsinghal@dbwong