Skip to content

PHOENIX-6528 Fix view index read repair for the pks with variable length - #1286

Closed
gokceni wants to merge 1 commit into
apache:4.xfrom
gokceni:PHOENIX-6528
Closed

PHOENIX-6528 Fix view index read repair for the pks with variable length#1286
gokceni wants to merge 1 commit into
apache:4.xfrom
gokceni:PHOENIX-6528

Conversation

@gokceni

Copy link
Copy Markdown
Contributor

No description provided.

@gokceni

Copy link
Copy Markdown
ContributorAuthor

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec6m 28sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ 4.x Compile Tests _
+1 💚mvninstall20m 1s4.x passed
+1 💚compile1m 6s4.x passed
+1 💚checkstyle1m 4s4.x passed
+1 💚javadoc0m 48s4.x passed
+0 🆗spotbugs3m 21sphoenix-core in 4.x has 958 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall11m 51sthe patch passed
+1 💚compile1m 5sthe patch passed
+1 💚javac1m 5sthe patch passed
-1 ❌checkstyle1m 4sphoenix-core: The patch generated 10 new + 456 unchanged - 10 fixed = 466 total (was 466)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 47sthe patch passed
+1 💚spotbugs3m 29sthe patch passed
_ Other Tests _
-1 ❌unit1m 45sphoenix-core in the patch failed.
+1 💚asflicense0m 9sThe patch does not generate ASF License warnings.
54m 21s
ReasonTests
Failed junit testsphoenix.index.IndexMaintainerTest
phoenix.query.PhoenixStatsCacheLoaderTest
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#1286
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux 51d4accc6888 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 6e8fe10
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/1/testReport/
Max. process+thread count476 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/1/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 3sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ 4.x Compile Tests _
+1 💚mvninstall19m 46s4.x passed
+1 💚compile1m 7s4.x passed
+1 💚checkstyle1m 5s4.x passed
+1 💚javadoc0m 47s4.x passed
+0 🆗spotbugs3m 20sphoenix-core in 4.x has 958 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall11m 55sthe patch passed
+1 💚compile1m 8sthe patch passed
+1 💚javac1m 8sthe patch passed
-1 ❌checkstyle1m 6sphoenix-core: The patch generated 8 new + 456 unchanged - 10 fixed = 464 total (was 466)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 46sthe patch passed
+1 💚spotbugs3m 29sthe patch passed
_ Other Tests _
-1 ❌unit206m 46sphoenix-core in the patch failed.
+1 💚asflicense0m 41sThe patch does not generate ASF License warnings.
255m 35s
ReasonTests
Failed junit testsphoenix.end2end.AuditLoggingIT
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#1286
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux ab253a6aec7b 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 6e8fe10
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/2/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/2/testReport/
Max. process+thread count5293 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/2/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

String actualExplainPlan = QueryUtil.getExplainPlan(rs1);
assertTrue(actualExplainPlan.contains("_IDX_" + fullTableName));

SingleCellIndexIT.dumpTable("_IDX_" + fullTableName);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove dumpTable statements before merging

// (but we still need to write it if it's DESC to ensure sort order is correct).
byte sepByte = SchemaUtil.getSeparatorByte(rowKeyOrderOptimizable, ptr.getLength() == 0, dataRowKeySchema.getField(i));
if (!dataRowKeySchema.getField(i).getDataType().isFixedWidth() && (((i+1) != dataRowKeySchema.getFieldCount()) || sepByte == QueryConstants.DESC_SEPARATOR_BYTE)) {
if (!dataRowKeySchema.getField(i).getDataType().isFixedWidth()){

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to fix comment on line 848 which refers to the logic you're changing

// Remove trailing nulls
int index = dataRowKeySchema.getFieldCount() - 1;
while (index >= 0 && !dataRowKeySchema.getField(index).getDataType().isFixedWidth() && length > minLength && dataRowKey[length-1] == QueryConstants.SEPARATOR_BYTE) {
while (trailingVariableWidthColumnNum > 0 && dataRowKey[length-1] == QueryConstants.SEPARATOR_BYTE) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if it's the DESC_SEPARATOR_BYTE? wouldn't we need to continue the loop and decrement trailingVariableWidthColumnNum? The old logic only needed to check the asc SEPARATOR_BYTE but that's because it only wrote a SEPARATOR_BYTE in the asc case, but that's changed now above.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternately, do we need to leave the sepByte == QueryConstants.DESC_SEPARATOR_BYTE part of the check back above on 849?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gjacoby126 In line 850, if the field is variable length with this new change we write the desc or asc sep byte always regardless of it is last one or not.
In the previous case, if it is a variable length and last field but not desc, we didn't write it but if it is variable len, last but asc, we didn't.
In line 860, if it is desc_sep_byte, we don't continue the loop in before and now. In the old case, it wrote desc_byte all the time even if it is the last field.
For example: Think about this case: col1 VARCHAR, col2 VARCHAR DESC
In the old version it wrote: sepbyteforasc,sepbytefordesc
In the new version it writes: sepbyteforasc,sepbytefordesc
And line 860 doesn't remove these

For this case: col1 VARCHAR, col2 VARCHAR
In old version: sepbyteforasc
In new version: sepbyteforasc,sepbyteforasc
In line 860,
In old version last one is removed, in new version last 2 is removed.

For this case: col1 VARCHAR DESC, col2 VARCHAR
In old version: sepbytefordesc
In new version: sepbytefordesc,sepbyteforasc
In line 860,
In old version last one is not removed, in new version last 1 is removed and they had sepbytefordesc as the only sep byte.

Am I missing something?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, got it. Thanks @gokceni

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 1sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ 4.x Compile Tests _
+1 💚mvninstall20m 21s4.x passed
+1 💚compile1m 6s4.x passed
+1 💚checkstyle1m 4s4.x passed
+1 💚javadoc0m 47s4.x passed
+0 🆗spotbugs3m 19sphoenix-core in 4.x has 958 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall11m 59sthe patch passed
+1 💚compile1m 8sthe patch passed
+1 💚javac1m 8sthe patch passed
-1 ❌checkstyle1m 5sphoenix-core: The patch generated 8 new + 456 unchanged - 10 fixed = 464 total (was 466)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 48sthe patch passed
+1 💚spotbugs3m 31sthe patch passed
_ Other Tests _
+1 💚unit205m 29sphoenix-core in the patch passed.
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
252m 43s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#1286
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux 378ec2154115 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 6e8fe10
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/3/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/3/testReport/
Max. process+thread count4995 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1286/3/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@gjacoby126gjacoby126 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@gokceni

Copy link
Copy Markdown
ContributorAuthor

merged

@gokcenigokceni closed this Feb 4, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@gokceni@stoty@gjacoby126