Skip to content

PHOENIX-6179 : Relax maxLookBackAge check during an upgrade - #942

Closed
virajjasani wants to merge 1 commit into
apache:masterfrom
virajjasani:PHOENIX-6179-master
Closed

PHOENIX-6179 : Relax maxLookBackAge check during an upgrade#942
virajjasani wants to merge 1 commit into
apache:masterfrom
virajjasani:PHOENIX-6179-master

Conversation

@virajjasani

Copy link
Copy Markdown
Contributor

No description provided.

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec7m 8sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall12m 31smaster passed
+1 💚compile0m 55smaster passed
+1 💚checkstyle0m 39smaster passed
+1 💚javadoc0m 44smaster passed
+0 🆗spotbugs2m 51sphoenix-core in master has 969 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall7m 31sthe patch passed
+1 💚compile0m 52sthe patch passed
+1 💚javac0m 52sthe patch passed
+1 💚checkstyle0m 40sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 44sthe patch passed
+1 💚spotbugs3m 5sthe patch passed
_ Other Tests _
-1 ❌unit94m 38sphoenix-core in the patch failed.
+1 💚asflicense0m 27sThe patch does not generate ASF License warnings.
135m 22s
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#942
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux bbff880e688e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revisionmaster / 6d84d0f
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/1/testReport/
Max. process+thread count6977 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/1/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@virajjasani

Copy link
Copy Markdown
ContributorAuthor

@gjacoby126@jpisaac@ChinmaySKulkarni Could you please take a look? 4.x patch is also attached to Jira.

@virajjasani

Copy link
Copy Markdown
ContributorAuthor

FYI @yanxinyi

Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
if (setMaxLookBackAge) {
props.put(
QueryServices.GLOBAL_INDEX_ROW_AGE_THRESHOLD_TO_DELETE_MS_ATTRIB,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need to set this property?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, this is not needed, it came as part of copy from existing test. Removed it.
Thanks

QueryServices.GLOBAL_INDEX_ROW_AGE_THRESHOLD_TO_DELETE_MS_ATTRIB,
Long.toString(0));
props.put(CompatBaseScannerRegionObserver.PHOENIX_MAX_LOOKBACK_AGE_CONF_KEY,
Integer.toString(15));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why 15 here?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With 15, we are able to reproduce the exception without source code change on this patch. Is it fine? I can update this if there is any better suggestion that I might be unaware of.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any value less than 31 should work for testing purpose. Because during upgrade, SCN is set to be Phoenix version timestamp:

 public static final long MIN_SYSTEM_TABLE_TIMESTAMP_4_16_0 = MIN_TABLE_TIMESTAMP + 31;
public static final long MIN_SYSTEM_TABLE_TIMESTAMP_5_1_0 = MIN_SYSTEM_TABLE_TIMESTAMP_4_16_0;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add explanation comments here so that in the future ppl have a better understanding, thanks

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done @yanxinyi
Thanks

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec6m 30sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
-1 ❌mvninstall15m 24sroot in master failed.
+1 💚compile1m 4smaster passed
+1 💚checkstyle0m 41smaster passed
+1 💚javadoc0m 49smaster passed
+0 🆗spotbugs3m 18sphoenix-core in master has 969 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall10m 4sthe patch passed
+1 💚compile0m 57sthe patch passed
+1 💚javac0m 57sthe patch passed
+1 💚checkstyle0m 37sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 50sthe patch passed
+1 💚spotbugs3m 35sthe patch passed
_ Other Tests _
-1 ❌unit156m 11sphoenix-core in the patch failed.
+1 💚asflicense0m 24sThe patch does not generate ASF License warnings.
202m 49s
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#942
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux 77063e7d854a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revisionmaster / ff46e35
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
mvninstallhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/2/artifact/yetus-general-check/output/branch-mvninstall-root.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/2/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/2/testReport/
Max. process+thread count6196 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/2/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec4m 41sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall13m 5smaster passed
+1 💚compile0m 59smaster passed
+1 💚checkstyle0m 38smaster passed
+1 💚javadoc0m 44smaster passed
+0 🆗spotbugs2m 59sphoenix-core in master has 969 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall7m 31sthe patch passed
+1 💚compile0m 54sthe patch passed
+1 💚javac0m 54sthe patch passed
+1 💚checkstyle0m 39sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 46sthe patch passed
+1 💚spotbugs3m 17sthe patch passed
_ Other Tests _
-1 ❌unit95m 35sphoenix-core in the patch failed.
+1 💚asflicense0m 27sThe patch does not generate ASF License warnings.
134m 51s
ReasonTests
Failed junit testsphoenix.end2end.OrphanViewToolIT
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#942
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux 3f63c001b4c4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revisionmaster / ff46e35
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/3/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/3/testReport/
Max. process+thread count6909 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/3/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@jpisaacjpisaac left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, added a suggestion, Thanks for the patch

// Executes the SQL commands listed in the given operation file from the sql_files directory
private void executeQueriesWithCurrentVersion(String operation) throws Exception {
private void executeQueriesWithCurrentVersion(String operation,
boolean setMaxLookBackAge) throws Exception {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@virajjasani Was wondering whether you would entertain the idea of using an EnumSet instead of boolean (which is specific to maxLookBaackAge). The thinking was we could then reuse this method more easily for other property checks and combinations of them.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done @jpisaac
Thanks

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 30sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall12m 35smaster passed
+1 💚compile0m 56smaster passed
+1 💚checkstyle0m 40smaster passed
+1 💚javadoc0m 45smaster passed
+0 🆗spotbugs2m 54sphoenix-core in master has 969 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall7m 21sthe patch passed
+1 💚compile0m 51sthe patch passed
+1 💚javac0m 51sthe patch passed
-1 ❌checkstyle0m 40sphoenix-core: The patch generated 2 new + 380 unchanged - 2 fixed = 382 total (was 382)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 44sthe patch passed
+1 💚spotbugs3m 2sthe patch passed
_ Other Tests _
-1 ❌unit95m 2sphoenix-core in the patch failed.
+1 💚asflicense0m 27sThe patch does not generate ASF License warnings.
129m 2s
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#942
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux 34d20ecdccb4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revisionmaster / ff46e35
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/4/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/4/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/4/testReport/
Max. process+thread count6935 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-942/4/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@virajjasani
virajjasani deleted the PHOENIX-6179-master branch November 6, 2020 06:12
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@virajjasani@stoty@jpisaac@yanxinyi