Merge PHOENIX-6182: IndexTool to verify and repair every index row (#…

[tkhurana]

…1022)

• PHOENIX-6198 Add option to IndexTool to specify the source table for scan (PHOENIX-6198 Add option to IndexTool to specify the source table for scan #937)

• PHOENIX-6198 Add option to IndexTool to specify the source table for scan

• Addressed feedback for PHOENIX-6198

Extended the -from-index option to support -vBOTH, -vAFTER and -vNONE.
Added the disclaimer for -vAFTER. Also, using the source table enum from
IndexScrutinyTool.

• PHOENIX-6199 Generate different query plan depending upon if the source (PHOENIX-6199 Generate different query plan depending upon if the source is index table or data table #958)

is index table or data table

• PHOENIX-6200 Add counters for extra index rows, log results to PIT and PIT_RESULT table (PHOENIX-6200 Add counters for extra index rows, log results to PIT and PIT_RESULT table #995)

• PHOENIX-6200 Add counters for extra index rows, log results to PIT and PIT_RESULT table

• Address feedback

• PHOENIX-6200 (addendum) Fix test case because invalid rows now are
  reported as beyond max lookback when max lookback is set to 0
  Also add ASF license to one file

[stoty]

I see you have started working on it.
I try to do the same at #1126.

It looks like your test had hung.
I have fixed that particular problem in my version, HBase 2 cannot create tables while the time stands still.

However, I have a test failure in org.apache.phoenix.end2end.IndexRepairRegionScannerIT.testFromIndexToolForIncrementalVerify() that I'm note really making progress on.

[tkhurana]

@stoty what failure are you seeing ?

I have not been able to run tests locally on master branch. My test setup fails everytime Caused by: java.lang.RuntimeException: Master not active after 30000ms at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:232) at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:188) at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:430) at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:259) ... 30 more

[stoty]

@tkhurana
Local branch test failure:

I haven't encountered that one locally.
If you are running the full suite, and have less than 16GB RAM, you coul try halving numForkedIT.

My problem with org.apache.phoenix.end2end.IndexRepairRegionScannerIT.testFromIndexToolForIncrementalVerify():
(this is after fixing the EnvireonmentEdgeManager not to be used while creating the table and index)

In the second set of asserts
assertEquals(NROWS,
mrJobCounters.findCounter(BEFORE_REPAIR_EXTRA_VERIFIED_INDEX_ROW_COUNT.name()).getValue());
returns 3 instead of 4.

Which is very scary.

[stoty]

@tkhurana
Are you rebuilding HBase locally as described in BUILDING.md ?
IIRC missing that step can also cause this error.

[stoty]

#1126 now seems to be working.