Skip to content

PHOENIX-5940 Pre-4.15 client cannot connect to 4.15+ server after SYS… - #950

Closed
yanxinyi wants to merge 1 commit into
apache:4.xfrom
yanxinyi:PHOENIX-5940-4.x
Closed

PHOENIX-5940 Pre-4.15 client cannot connect to 4.15+ server after SYS…#950
yanxinyi wants to merge 1 commit into
apache:4.xfrom
yanxinyi:PHOENIX-5940-4.x

Conversation

@yanxinyi

Copy link
Copy Markdown
Contributor

…TEM.CATALOG region has split

@virajjasanivirajjasani left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small request, looks good otherwise.

@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec4m 18sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ 4.x Compile Tests _
+1 💚mvninstall10m 45s4.x passed
+1 💚compile0m 55s4.x passed
+1 💚checkstyle1m 13s4.x passed
+1 💚javadoc0m 42s4.x passed
+0 🆗spotbugs2m 53sphoenix-core in 4.x has 953 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall5m 18sthe patch passed
+1 💚compile0m 55sthe patch passed
+1 💚javac0m 55sthe patch passed
-1 ❌checkstyle1m 12sphoenix-core: The patch generated 2 new + 1723 unchanged - 2 fixed = 1725 total (was 1725)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 42sthe patch passed
+1 💚spotbugs3m 1sthe patch passed
_ Other Tests _
-1 ❌unit142m 25sphoenix-core in the patch failed.
+1 💚asflicense0m 37sThe patch does not generate ASF License warnings.
177m 40s
ReasonTests
Failed junit testsphoenix.end2end.index.SaltedIndexIT
phoenix.end2end.join.WhereOptimizerForJoinFiltersIT
phoenix.end2end.index.AsyncIndexDisabledIT
phoenix.end2end.salted.SaltedTableIT
phoenix.end2end.index.txn.MutableRollbackIT
phoenix.end2end.index.AlterIndexIT
phoenix.end2end.index.LocalImmutableNonTxIndexIT
phoenix.end2end.OrphanViewToolIT
phoenix.end2end.index.LocalImmutableTxIndexIT
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#950
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux ccb215169317 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 2aaf2e2
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/1/testReport/
Max. process+thread count6859 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/1/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ServerUtil.createIOException(
SchemaUtil.getPhysicalTableName(PhoenixDatabaseMetaData.SYSTEM_CATALOG_NAME_BYTES,
isTablesMappingEnabled).toString(), t));
if (request.getClientVersion() >= MIN_SPLITTABLE_SYSTEM_CATALOG) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not so sure about this fix. If there is a legit problem loading SYSCAT region (can happen even if there is just 1 region), we won't be setting a syscat timestamp in this case. This in turn will allow weird cases like 4.14 client + 4.16 server + 4.13 metadata without throwing an UpgradeRequiredException.

How about we add an else block like this:

elseif (<syscatheaderrownotinregion>) {
// ignore
} else {
// this is a legit exception so set a controller exception with the same body as the if clause
}

We can refactor the if and else since the if and last else have the same body

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ChinmaySKulkarni
One question out of curiosity: when I checked CQSI invoking coprocessor hook for this call, version is set from client side:

GetVersionRequest.Builder builder =
GetVersionRequest.newBuilder();
builder.setClientVersion(
VersionUtil.encodeVersion(PHOENIX_MAJOR_VERSION,
PHOENIX_MINOR_VERSION, PHOENIX_PATCH_NUMBER));
instance.getVersion(controller, builder.build(), rpcCallback);

Hence, I thought it was always guaranteed to get clientVersion() from old client if old client is checking compatibility with new server.
However, looks like this check might not be reliable in all cases? In the above example, we might get 4.13 version by any chance (while checking for client 4.14 <-> server 4.16 compatibility?)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is the client jar version. The getVersion call does 3 things:

  1. Returns the phoenix server jar version + HBase version
  2. Embeds the namespace mapping prop in the response for 1.
  3. Returns the system catalog timestamp i.e. metadata version if syscat exists.

On the client-side after getting the response from the server, we check client-server compatibility based on these 3 things

@yanxinyi
yanxinyiforce-pushed the PHOENIX-5940-4.x branch 2 times, most recently from 15a2dad to 65ba7fbCompareOctober 31, 2020 06:14
@stoty

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 32sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ 4.x Compile Tests _
+1 💚mvninstall10m 48s4.x passed
+1 💚compile0m 55s4.x passed
+1 💚checkstyle1m 11s4.x passed
+1 💚javadoc0m 44s4.x passed
+0 🆗spotbugs2m 57sphoenix-core in 4.x has 953 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall5m 41sthe patch passed
+1 💚compile1m 2sthe patch passed
+1 💚javac1m 2sthe patch passed
-1 ❌checkstyle1m 22sphoenix-core: The patch generated 8 new + 1737 unchanged - 3 fixed = 1745 total (was 1740)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 56sthe patch passed
+1 💚spotbugs3m 43sthe patch passed
_ Other Tests _
-1 ❌unit140m 48sphoenix-core in the patch failed.
-1 ❌asflicense0m 37sThe patch generated 1 ASF License warnings.
174m 2s
ReasonTests
Failed junit testsphoenix.end2end.index.SaltedIndexIT
phoenix.replication.SystemCatalogWALEntryFilterIT
phoenix.end2end.UpsertSelectIT
TEST-[RangeScanIT_0]
phoenix.end2end.BackwardCompatibilityForSplitableSyscatIT
phoenix.end2end.index.DropColumnIT
phoenix.end2end.index.IndexVerificationResultRepositoryIT
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#950
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux 9109f0338931 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 7cbebb4
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/2/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/2/testReport/
asflicensehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/2/artifact/yetus-general-check/output/patch-asflicense-problems.txt
Max. process+thread count7201 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/2/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

}
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yanxinyi my assumption is there is no real diff in this class and is just a bunch of stuff copied from BackwardCompatibilityIT right?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, no logic changes. The only diff is adding extra param for zkQuorm or URL for the execute script.

@yanxinyi

Copy link
Copy Markdown
ContributorAuthor

For additional testing on my local machine, I created more than 2k views, and the syscat split into 2 regions(confirmed by HBase UI page). Ran the select query from 4.14 and current 4.x clients without any issue.

@ChinmaySKulkarni

@ChinmaySKulkarniChinmaySKulkarni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Please make sure the test failures aren't related to your changes and squash all commits before pushing. Thanks for fixing this @yanxinyi

@stoty

stoty commented Nov 7, 2020

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec4m 23sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 1sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ 4.x Compile Tests _
+1 💚mvninstall11m 14s4.x passed
+1 💚compile0m 55s4.x passed
+1 💚checkstyle1m 14s4.x passed
+1 💚javadoc0m 43s4.x passed
+0 🆗spotbugs2m 50sphoenix-core in 4.x has 946 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall5m 17sthe patch passed
+1 💚compile0m 55sthe patch passed
+1 💚javac0m 55sthe patch passed
-1 ❌checkstyle1m 15sphoenix-core: The patch generated 8 new + 1738 unchanged - 2 fixed = 1746 total (was 1740)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 44sthe patch passed
+1 💚spotbugs3m 4sthe patch passed
_ Other Tests _
-1 ❌unit122m 33sphoenix-core in the patch failed.
+1 💚asflicense0m 40sThe patch does not generate ASF License warnings.
158m 31s
ReasonTests
Failed junit testsphoenix.end2end.index.GlobalImmutableNonTxIndexIT
phoenix.end2end.BackwardCompatibilityForSplitableSyscatIT
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#950
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux 6fc899b65a12 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 31c9912
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/6/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/6/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/6/testReport/
Max. process+thread count6887 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/6/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@stoty

stoty commented Nov 7, 2020

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 32sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 1sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ 4.x Compile Tests _
+1 💚mvninstall10m 38s4.x passed
+1 💚compile0m 54s4.x passed
+1 💚checkstyle1m 12s4.x passed
+1 💚javadoc0m 45s4.x passed
+0 🆗spotbugs2m 52sphoenix-core in 4.x has 946 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall5m 13sthe patch passed
+1 💚compile0m 56sthe patch passed
+1 💚javac0m 56sthe patch passed
-1 ❌checkstyle1m 13sphoenix-core: The patch generated 9 new + 1737 unchanged - 3 fixed = 1746 total (was 1740)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 41sthe patch passed
+1 💚spotbugs3m 4sthe patch passed
_ Other Tests _
-1 ❌unit223m 48sphoenix-core in the patch failed.
+1 💚asflicense0m 41sThe patch does not generate ASF License warnings.
255m 14s
ReasonTests
Failed junit testsphoenix.util.IndexScrutinyIT
phoenix.rpc.PhoenixServerRpcIT
phoenix.end2end.PointInTimeQueryIT
phoenix.end2end.OrphanViewToolIT
phoenix.end2end.index.GlobalMutableNonTxIndexIT
phoenix.end2end.index.IndexWithTableSchemaChangeIT
phoenix.end2end.BackwardCompatibilityForSplitableSyscatIT
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/7/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#950
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux 0aebbfcebc0a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 50edcc9
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/7/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/7/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/7/testReport/
Max. process+thread count6907 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/7/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@stoty

stoty commented Nov 9, 2020

Copy link
Copy Markdown
Contributor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 32sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-1 ❌test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ 4.x Compile Tests _
+1 💚mvninstall10m 41s4.x passed
+1 💚compile0m 56s4.x passed
+1 💚checkstyle1m 12s4.x passed
+1 💚javadoc0m 45s4.x passed
+0 🆗spotbugs2m 52sphoenix-core in 4.x has 946 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚mvninstall5m 21sthe patch passed
+1 💚compile0m 56sthe patch passed
+1 💚javac0m 56sthe patch passed
-1 ❌checkstyle1m 15sphoenix-core: The patch generated 8 new + 1738 unchanged - 2 fixed = 1746 total (was 1740)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚javadoc0m 42sthe patch passed
+1 💚spotbugs2m 59sthe patch passed
_ Other Tests _
-1 ❌unit124m 54sphoenix-core in the patch failed.
+1 💚asflicense0m 37sThe patch does not generate ASF License warnings.
156m 28s
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/9/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#950
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
unameLinux 411fda1912db 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 50edcc9
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstylehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/9/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/9/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/9/testReport/
Max. process+thread count6920 (vs. ulimit of 30000)
modulesC: phoenix-core U: phoenix-core
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-950/9/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@yanxinyi@stoty@ChinmaySKulkarni@virajjasani