PHOENIX-6408 LIMIT on local index query with uncovered columns in the WHERE returns wrong result. by lhofhansl · Pull Request #1161 · apache/phoenix

lhofhansl · 2021-03-10T03:37:32Z

See description in the jira.
The fix is relatively straight forward, if not particularly pretty:

don't enforce the limit via a filter
instead count the rows after the extraWhere filter has run for uncovered local index WHERE clauses

I added extra tests for this.

stoty · 2021-03-10T06:28:10Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	6m 42s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	17m 3s	master passed
+0	hbaserecompile	23m 59s	HBase recompiled.
+1 💚	compile	1m 4s	master passed
+1 💚	checkstyle	0m 51s	master passed
+1 💚	javadoc	0m 53s	master passed
+0 🆗	spotbugs	3m 18s	phoenix-core in master has 959 extant spotbugs warnings.
		_ Patch Compile Tests _
+1 💚	mvninstall	9m 51s	the patch passed
+0	hbaserecompile	20m 2s	HBase recompiled.
+1 💚	compile	1m 4s	the patch passed
+1 💚	javac	1m 4s	the patch passed
-1 ❌	checkstyle	0m 50s	phoenix-core: The patch generated 11 new + 1205 unchanged - 1 fixed = 1216 total (was 1206)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 52s	the patch passed
+1 💚	spotbugs	3m 27s	the patch passed
		_ Other Tests _
-1 ❌	unit	106m 49s	phoenix-core in the patch failed.
+1 💚	asflicense	0m 33s	The patch does not generate ASF License warnings.
		170m 1s

Reason	Tests
Failed junit tests	phoenix.end2end.PointInTimeScanQueryIT

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1161/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1161
Optional Tests	dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
uname	Linux dfc0ef6c9cde 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / `5d78494`
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstyle	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1161/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unit	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1161/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1161/1/testReport/
Max. process+thread count	10407 (vs. ulimit of 30000)
modules	C: phoenix-core U: phoenix-core
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1161/1/console
versions	git=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

kadirozde · 2021-03-10T06:30:14Z

 return next;
 }
 }
+ if (extraLimit >= 0 && --extraLimit == 0) {


Should not this check be at the end of this method just before it returns?

It does not matter as long as this is after the local index extra where filtering (and there are no other exit points after this). I wanted to isolate that to local indexing only, to not affect the other code paths.

Oh. You are right. We're not building the right tuple in this case if we have a tuple projector.
In fact this caused PointInTimeScanQueryIT to fail.

lhofhansl · 2021-03-10T17:41:31Z

I'm looking into the PointInTimeLimitedIT failures. They look pertinent.
In the worst case we can move this issue to 5.1.2 or even 5.2.0.

lhofhansl · 2021-03-10T18:05:53Z

Updated as per @kadirozde's suggestion, which also fixes the test failures.

… WHERE returns wrong result.

kadirozde

+1, Thanks

lhofhansl · 2021-03-10T18:17:20Z

Thanks! I'll await the next test run and then merge.

stoty · 2021-03-10T21:08:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 11s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	17m 24s	master passed
+0	hbaserecompile	24m 15s	HBase recompiled.
+1 💚	compile	1m 4s	master passed
+1 💚	checkstyle	0m 50s	master passed
+1 💚	javadoc	0m 50s	master passed
+0 🆗	spotbugs	3m 17s	phoenix-core in master has 959 extant spotbugs warnings.
		_ Patch Compile Tests _
+1 💚	mvninstall	9m 55s	the patch passed
+0	hbaserecompile	21m 8s	HBase recompiled.
+1 💚	compile	1m 4s	the patch passed
+1 💚	javac	1m 4s	the patch passed
-1 ❌	checkstyle	0m 51s	phoenix-core: The patch generated 11 new + 1205 unchanged - 1 fixed = 1216 total (was 1206)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 51s	the patch passed
+1 💚	spotbugs	3m 29s	the patch passed
		_ Other Tests _
+1 💚	unit	125m 33s	phoenix-core in the patch passed.
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		182m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1161/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1161
Optional Tests	dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
uname	Linux 723631e213cb 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / `b6b41ce`
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstyle	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1161/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1161/2/testReport/
Max. process+thread count	10426 (vs. ulimit of 30000)
modules	C: phoenix-core U: phoenix-core
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1161/2/console
versions	git=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

lhofhansl requested a review from kadirozde March 10, 2021 03:37

kadirozde reviewed Mar 10, 2021
View reviewed changes

lhofhansl force-pushed the 6408 branch from 2c05694 to 6f4f3d4Compare March 10, 2021 18:05

PHOENIX-6408 LIMIT on local index query with uncovered columns in the…
6f4f3d4
… WHERE returns wrong result.

kadirozde approved these changes Mar 10, 2021
View reviewed changes

lhofhansl merged commit dcf3bf9 into apache:masterMar 10, 2021

Navigation Menu

Uh oh!

PHOENIX-6408 LIMIT on local index query with uncovered columns in the WHERE returns wrong result. - #1161

PHOENIX-6408 LIMIT on local index query with uncovered columns in the WHERE returns wrong result.#1161
lhofhansl merged 1 commit into
apache:masterfrom
lhofhansl:6408

lhofhansl commented Mar 10, 2021•
edited
Loading

Uh oh!

stoty commented Mar 10, 2021

Uh oh!

kadirozdeMar 10, 2021

Uh oh!

lhofhanslMar 10, 2021•
edited
Loading

Uh oh!

lhofhanslMar 10, 2021

Uh oh!

lhofhansl commented Mar 10, 2021

Uh oh!

lhofhansl commented Mar 10, 2021

Uh oh!

kadirozde left a comment

Uh oh!

lhofhansl commented Mar 10, 2021

Uh oh!

stoty commented Mar 10, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

lhofhansl commented Mar 10, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stoty commented Mar 10, 2021

Uh oh!

kadirozdeMar 10, 2021

Choose a reason for hiding this comment

Uh oh!

lhofhanslMar 10, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lhofhanslMar 10, 2021

Choose a reason for hiding this comment

Uh oh!

lhofhansl commented Mar 10, 2021

Uh oh!

lhofhansl commented Mar 10, 2021

Uh oh!

kadirozde left a comment

Choose a reason for hiding this comment

Uh oh!

lhofhansl commented Mar 10, 2021

Uh oh!

stoty commented Mar 10, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lhofhansl commented Mar 10, 2021•
edited
Loading

lhofhanslMar 10, 2021•
edited
Loading