Skip to content

PHOENIX-6181 IndexRepairRegionScanner to verify and repair every glob… - #915

Merged
kadirozde merged 4 commits into
apache:4.xfrom
kadirozde:6181
Oct 16, 2020
Merged

PHOENIX-6181 IndexRepairRegionScanner to verify and repair every glob…#915
kadirozde merged 4 commits into
apache:4.xfrom
kadirozde:6181

Conversation

@kadirozde

Copy link
Copy Markdown
Contributor

…al index row

@stoty

stoty commented Oct 8, 2020

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec6m 24sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 2 new or modified test files.
_ 4.x Compile Tests _
+1 💚mvninstall8m 39s4.x passed
+1 💚compile0m 53s4.x passed
+1 💚checkstyle1m 13s4.x passed
+1 💚javadoc0m 45s4.x passed
+0 🆗spotbugs2m 54sphoenix-core in 4.x has 957 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall4m 54sthe patch passed
+1 💚compile0m 56sthe patch passed
+1 💚javac0m 56sthe patch passed
-1 ❌checkstyle1m 21sphoenix-core: The patch generated 542 new + 1433 unchanged - 372 fixed = 1975 total (was 1805)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 41sthe patch passed
-1 ❌spotbugs3m 3sphoenix-core generated 1 new + 956 unchanged - 1 fixed = 957 total (was 957)
_ Other Tests _
-1 ❌unit1m 12sphoenix-core in the patch failed.
+1 💚asflicense0m 10sThe patch does not generate ASF License warnings.
34m 35s
ReasonTests
FindBugsmodule:phoenix-core
org.apache.phoenix.coprocessor.GlobalIndexRegionScanner.shouldVerify(IndexTool$IndexVerifyType, byte[], Scan, Region, IndexMaintainer, IndexVerificationResultRepository, boolean) may expose internal representation by storing an externally mutable object into GlobalIndexRegionScanner.indexRowKeyforReadRepair At GlobalIndexRegionScanner.java:IndexVerificationResultRepository, boolean) may expose internal representation by storing an externally mutable object into GlobalIndexRegionScanner.indexRowKeyforReadRepair At GlobalIndexRegionScanner.java:[line 328]
Failed junit testsphoenix.index.ShouldVerifyTest
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#915
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux b294a7ca6906 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 3008ca9
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
spotbugshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/1/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/1/testReport/
Max. process+thread count487 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/1/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Comment on lines +127 to +157
batchSize += mutationList.size();
if (batchSize >= maxBatchSize) {
ungroupedAggregateRegionObserver.checkForRegionClosing();
region.batchMutate(indexUpdates.toArray(new Mutation[indexUpdates.size()]),
HConstants.NO_NONCE, HConstants.NO_NONCE);
batchSize = 0;
indexUpdates = new ArrayList<Mutation>(maxBatchSize);
}
}
if (batchSize > 0) {
ungroupedAggregateRegionObserver.checkForRegionClosing();
region.batchMutate(indexUpdates.toArray(new Mutation[indexUpdates.size()]),
HConstants.NO_NONCE, HConstants.NO_NONCE);
}
batchSize = 0;
indexUpdates = new ArrayList<Mutation>(maxBatchSize);
for (Mutation mutation : indexRowsToBeDeleted) {
indexUpdates.add(mutation);
batchSize ++;
if (batchSize >= maxBatchSize) {
ungroupedAggregateRegionObserver.checkForRegionClosing();
region.batchMutate(indexUpdates.toArray(new Mutation[indexUpdates.size()]),
HConstants.NO_NONCE, HConstants.NO_NONCE);
batchSize = 0;
indexUpdates = new ArrayList<Mutation>(maxBatchSize);
}
}
if (batchSize > 0) {
ungroupedAggregateRegionObserver.checkForRegionClosing();
region.batchMutate(indexUpdates.toArray(new Mutation[indexUpdates.size()]),
HConstants.NO_NONCE, HConstants.NO_NONCE);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel the same logic of checking if the region is closed and then sending the mutations is duplicated in multiple places and we can move it to a function to reduce the code duplication.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: Here we are calling region.batchMutate to update the index table but in IndexRebuildRegionScanner we call indexHTable.batch(). Why the difference ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will reduce the code duplication here. When the index table is local (in the case of repair) we use the region API and when the index table is remote (in the case of rebuild), we use the table API.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

// All other types of rebuilds/verification should be incrementally performed if appropriate param is passed
byte[] lastVerifyTimeValue = scan.getAttribute(UngroupedAggregateRegionObserver.INDEX_RETRY_VERIFY);
Long lastVerifyTime = lastVerifyTimeValue == null ? 0 : Bytes.toLong(lastVerifyTimeValue);
if(indexRowKeyforReadRepair != null || lastVerifyTime == 0 || shouldVerifyCheckDone) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once this function is called, shouldVerifyCheckDone is set to true and from that point on, it will always be true.
Even if we want incremental, we will always do verify then why do we check lastVerifyTime? If lastVerifyTime is not 0, then shouldVerify check will not be set to true.
Can we have just 1 global variable to decide if we should verify than multiple to make this easier?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lastVerifyTime is not a global variable. We can improve this method by checking shouldVerifyCheckDone at the entry of this method.

@VisibleForTesting
public int setIndexTableTTL(int ttl) {
indexTableTTL = ttl;
return 0;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the purpose of always returning 0?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is used only for the unit test. @swaroopak, can you answer the question?

return;
}
if (!CellUtil.matchingValue(actualCell, expectedCell)) {
String errorMsg = "Not matching value (in iteration " + iteration + ") for " + Bytes.toString(family) + ":" + Bytes.toString(qualifier);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Family qualifier will not be super useful if we do 5928

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you mean 5923? Single cell format for indexes will be optional. Also, please note that there will be a separate cell for each family.

Comment on lines +127 to +157
batchSize += mutationList.size();
if (batchSize >= maxBatchSize) {
ungroupedAggregateRegionObserver.checkForRegionClosing();
region.batchMutate(indexUpdates.toArray(new Mutation[indexUpdates.size()]),
HConstants.NO_NONCE, HConstants.NO_NONCE);
batchSize = 0;
indexUpdates = new ArrayList<Mutation>(maxBatchSize);
}
}
if (batchSize > 0) {
ungroupedAggregateRegionObserver.checkForRegionClosing();
region.batchMutate(indexUpdates.toArray(new Mutation[indexUpdates.size()]),
HConstants.NO_NONCE, HConstants.NO_NONCE);
}
batchSize = 0;
indexUpdates = new ArrayList<Mutation>(maxBatchSize);
for (Mutation mutation : indexRowsToBeDeleted) {
indexUpdates.add(mutation);
batchSize ++;
if (batchSize >= maxBatchSize) {
ungroupedAggregateRegionObserver.checkForRegionClosing();
region.batchMutate(indexUpdates.toArray(new Mutation[indexUpdates.size()]),
HConstants.NO_NONCE, HConstants.NO_NONCE);
batchSize = 0;
indexUpdates = new ArrayList<Mutation>(maxBatchSize);
}
}
if (batchSize > 0) {
ungroupedAggregateRegionObserver.checkForRegionClosing();
region.batchMutate(indexUpdates.toArray(new Mutation[indexUpdates.size()]),
HConstants.NO_NONCE, HConstants.NO_NONCE);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

import org.slf4j.LoggerFactory;

import com.google.common.collect.Maps;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of the other classes have small descriptions of what they do. Let's add one here as well.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

return new IndexerRegionScanner(scanner, region, scan, env, this);
} else {
return new IndexRebuildRegionScanner(scanner, region, scan, env, this);
if (region.getTableDesc().hasCoprocessor(IndexRegionObserver.class.getCanonicalName())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we have IndexRegionObserver for new design all the time? What is the case when the index doesn't have both Indexer and IndexRegionObserver?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gokceni IndexRegionObserver is only on the data table. Index table has neither Indexer nor IndexRegionObserver

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes you are right @tkhurana. I meant data table. Don't we remove Indexer and add IndexRegionObserver during upgrade?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this an optimization for checking if this table is Index table @kadirozde rather than querying the PTable and looking at its type?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have not removed Indexer yet. Regarding the PTable comment, this is a server side code and in general we do not want to access the syscat on the server side for performance reasons mainly.

}
return;
}
if (verifyType == IndexTool.IndexVerifyType.AFTER) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The AFTER option will not remove the extra verified rows in the index table. Same with the NONE option. Does it make sense to have these options when using the index table as the source ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will leave this decision to be made within IndexTool. I think we can still allow these options and state in the help text for IndexTook that they do not remove the stale index rows.

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 8sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 2 new or modified test files.
_ 4.x Compile Tests _
+1 💚mvninstall32m 47s4.x passed
+1 💚compile0m 58s4.x passed
+1 💚checkstyle1m 0s4.x passed
+1 💚javadoc0m 49s4.x passed
+0 🆗spotbugs3m 8sphoenix-core in 4.x has 956 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall29m 2sthe patch passed
+1 💚compile0m 58sthe patch passed
+1 💚javac0m 58sthe patch passed
-1 ❌checkstyle1m 3sphoenix-core: The patch generated 575 new + 1421 unchanged - 385 fixed = 1996 total (was 1806)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 48sthe patch passed
-1 ❌spotbugs3m 21sphoenix-core generated 1 new + 955 unchanged - 1 fixed = 956 total (was 956)
_ Other Tests _
-1 ❌unit1m 26sphoenix-core in the patch failed.
+1 💚asflicense0m 9sThe patch does not generate ASF License warnings.
77m 59s
ReasonTests
FindBugsmodule:phoenix-core
org.apache.phoenix.coprocessor.GlobalIndexRegionScanner.shouldVerify(IndexTool$IndexVerifyType, byte[], Scan, Region, IndexMaintainer, IndexVerificationResultRepository, boolean) may expose internal representation by storing an externally mutable object into GlobalIndexRegionScanner.indexRowKeyforReadRepair At GlobalIndexRegionScanner.java:IndexVerificationResultRepository, boolean) may expose internal representation by storing an externally mutable object into GlobalIndexRegionScanner.indexRowKeyforReadRepair At GlobalIndexRegionScanner.java:[line 328]
Failed junit testsphoenix.index.ShouldVerifyTest
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#915
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux 23e85e95bce0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 264310b
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
spotbugshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/2/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/2/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/2/testReport/
Max. process+thread count442 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/2/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec5m 28sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 2 new or modified test files.
_ 4.x Compile Tests _
+1 💚mvninstall32m 29s4.x passed
+1 💚compile1m 1s4.x passed
+1 💚checkstyle0m 59s4.x passed
+1 💚javadoc0m 47s4.x passed
+0 🆗spotbugs3m 9sphoenix-core in 4.x has 956 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall28m 50sthe patch passed
+1 💚compile0m 58sthe patch passed
+1 💚javac0m 58sthe patch passed
-1 ❌checkstyle1m 3sphoenix-core: The patch generated 582 new + 1420 unchanged - 386 fixed = 2002 total (was 1806)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 46sthe patch passed
-1 ❌spotbugs3m 23sphoenix-core generated 1 new + 955 unchanged - 1 fixed = 956 total (was 956)
_ Other Tests _
-1 ❌unit1m 25sphoenix-core in the patch failed.
+1 💚asflicense0m 9sThe patch does not generate ASF License warnings.
81m 51s
ReasonTests
FindBugsmodule:phoenix-core
org.apache.phoenix.coprocessor.GlobalIndexRegionScanner.shouldVerify(IndexTool$IndexVerifyType, byte[], Scan, Region, IndexMaintainer, IndexVerificationResultRepository, boolean) may expose internal representation by storing an externally mutable object into GlobalIndexRegionScanner.indexRowKeyforReadRepair At GlobalIndexRegionScanner.java:IndexVerificationResultRepository, boolean) may expose internal representation by storing an externally mutable object into GlobalIndexRegionScanner.indexRowKeyforReadRepair At GlobalIndexRegionScanner.java:[line 333]
Failed junit testsphoenix.index.ShouldVerifyTest
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#915
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux 1e8a21fbdd5a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 2359f54
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/3/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
spotbugshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/3/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/3/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/3/testReport/
Max. process+thread count447 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-915/3/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@gokcenigokceni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@kadirozde@stoty@tkhurana@gokceni