Skip to content

PHOENIX-6476 Index tool when verifying from index to data doesn't correctly split page into tasks - #1240

Merged
swaroopak merged 3 commits into
apache:4.xfrom
tkhurana:PHOENIX-6476-4.x
Jun 2, 2021
Merged

PHOENIX-6476 Index tool when verifying from index to data doesn't correctly split page into tasks#1240
swaroopak merged 3 commits into
apache:4.xfrom
tkhurana:PHOENIX-6476-4.x

Conversation

@tkhurana

Copy link
Copy Markdown
Contributor

When verifying from index table to data table, there were 2 issues:

  1. Data table region boundary keys were not being respected so the splits were happening only on the basis of per task max size.
  2. The actual index mutation map was was not being split for every task but the data row keys were being split. This caused the tool to report extra index rows which were actually false positives.

@tkhurana

tkhurana commented May 26, 2021

Copy link
Copy Markdown
ContributorAuthor

@kadirozde@gokceni

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec7m 2sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ 4.x Compile Tests _
+1 💚mvninstall15m 20s4.x passed
+1 💚compile1m 8s4.x passed
+1 💚checkstyle0m 31s4.x passed
+1 💚javadoc0m 49s4.x passed
+0 🆗spotbugs3m 18sphoenix-core in 4.x has 951 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall6m 47sthe patch passed
+1 💚compile1m 6sthe patch passed
+1 💚javac1m 6sthe patch passed
-1 ❌checkstyle0m 30sphoenix-core: The patch generated 10 new + 123 unchanged - 1 fixed = 133 total (was 124)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 47sthe patch passed
+1 💚spotbugs3m 29sthe patch passed
_ Other Tests _
-1 ❌unit205m 6sphoenix-core in the patch failed.
+1 💚asflicense0m 39sThe patch does not generate ASF License warnings.
249m 12s
ReasonTests
Failed junit testsphoenix.end2end.AuditLoggingIT
phoenix.end2end.IndexRepairRegionScannerIT
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1240/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#1240
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux 59fc83cccf50 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 69a9ec3
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1240/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1240/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1240/1/testReport/
Max. process+thread count5143 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1240/1/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 6sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ 4.x Compile Tests _
+1 💚mvninstall15m 14s4.x passed
+1 💚compile1m 5s4.x passed
+1 💚checkstyle0m 30s4.x passed
+1 💚javadoc0m 48s4.x passed
+0 🆗spotbugs3m 17sphoenix-core in 4.x has 951 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall6m 44sthe patch passed
+1 💚compile1m 5sthe patch passed
+1 💚javac1m 5sthe patch passed
-1 ❌checkstyle0m 30sphoenix-core: The patch generated 11 new + 123 unchanged - 1 fixed = 134 total (was 124)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 49sthe patch passed
+1 💚spotbugs3m 29sthe patch passed
_ Other Tests _
+1 💚unit200m 53sphoenix-core in the patch passed.
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
236m 35s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1240/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#1240
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux 31b2d5fcc797 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 69a9ec3
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1240/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1240/2/testReport/
Max. process+thread count5056 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1240/2/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

byte[] dataRowKey = indexMaintainer.buildDataRowKey(new ImmutableBytesWritable(indexRowKey), viewConstants);
for (int i = 0; i < dataRowKeysSetList.size(); ++i) {
if (dataRowKeysSetList.get(i).contains(dataRowKey)) {
mapList.get(i).put(indexRowKey, actualMutationList);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't you need to break?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that would avoid unnecessary lookups.

@swaroopakswaroopak left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, Thanks for the patch @tkhurana

@swaroopak
swaroopak merged commit 5d6b3f3 into apache:4.xJun 2, 2021
tkhurana added a commit to tkhurana/phoenix that referenced this pull request Jun 16, 2021
…rectly split page into tasks (apache#1240)
* PHOENIX-6476 Index tool when verifying from index to data doesn't correctly split page into tasks
* Fix failing tests, add comments
* Break early when the matching set is found
swaroopak pushed a commit that referenced this pull request Jun 16, 2021
…rectly split page into tasks (#1240) (#1248)
* PHOENIX-6476 Index tool when verifying from index to data doesn't correctly split page into tasks
* Fix failing tests, add comments
* Break early when the matching set is found
virajjasani pushed a commit that referenced this pull request Jul 23, 2021
…rectly split page into tasks (#1240) (#1248)
* PHOENIX-6476 Index tool when verifying from index to data doesn't correctly split page into tasks
* Fix failing tests, add comments
* Break early when the matching set is found
jpisaac pushed a commit to jpisaac/phoenix that referenced this pull request Jun 10, 2022
* PHOENIX-6476 Index tool when verifying from index to data doesn't correctly split page into tasks (apache#1240)
* PHOENIX-6476 Index tool when verifying from index to data doesn't correctly split page into tasks
* Fix failing tests, add comments
* Break early when the matching set is found
* Fix import usage
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@tkhurana@stoty@swaroopak@gokceni