Uh oh!
There was an error while loading. Please reload this page.
PHOENIX-7964: Wait for Phoenix system tables to initialize before sta… - #14
Merged
Conversation
…rting REST server Before starting the REST server in integration tests, ensure Phoenix system tables are fully initialized by calling TestUtils.awaitPhoenixReady(). This eliminates the race condition that caused intermittent test failures with: org.apache.phoenix.exception.PhoenixIOException: SYSTEM.CATALOG Caused by: org.apache.hadoop.hbase.TableNotFoundException: SYSTEM.CATALOG The fix adds TestUtils.awaitPhoenixReady(url) calls to all 82 integration test classes, either directly in their @BeforeClass methods or through inheritance from base classes like BaseSegmentScanIT, KclTestBase, UpdateItemBaseTests, and SegmentScanIT. The awaitPhoenixReady() method polls for up to 30 seconds (vs RESTServer's 5 seconds) for Phoenix to become ready, providing sufficient time for SYSTEM.CATALOG initialization even on slow CI workers. Changes: - Added TestUtils.awaitPhoenixReady() calls to 58 integration test classes - Fixed DeleteTableIT to construct Phoenix JDBC URL before REST server initialization - 24 additional test classes inherit the fix through base class inheritance - Total coverage: 82/82 integration test classes
palashc
requested changes
Jul 23, 2026
| restServer = new RESTServer(utility.getConfiguration()); | ||
| restServer.run(); | ||
| String url = "http://" + restServer.getServerAddress(); |
Contributor
There was a problem hiding this comment.
We need to build a JDBC URL from the ZK quorum here, not the rest server url.
Comment on lines
+65
to
+66
| TestUtils.awaitPhoenixReady(url); | ||
Contributor
There was a problem hiding this comment.
We have 2 options:
- remove
awaitPhoenixReady()from this test - remove this test entirely
I prefer option 1, which means this single IT test will not be protected by awaitPhoenixReady() and can lead to flaky test failure, which is okay.
| utility = new HBaseTestingUtility(conf); | ||
| setUpConfigForMiniCluster(conf); | ||
| utility.startMiniCluster(); | ||
| String zkQuorum = "127.0.0.1:" + utility.getZkCluster().getClientPort(); |
Contributor
There was a problem hiding this comment.
Why we have 127.0.0.1 here instead of localhost (everywhere else)?
virajjasani
left a comment
Contributor
There was a problem hiding this comment.
One thing to consider before we are good to go
| setUpConfigForMiniCluster(conf); | ||
| utility.startMiniCluster(); | ||
| String zkQuorum = "127.0.0.1:" + utility.getZkCluster().getClientPort(); | ||
| String zkQuorum = "localhost:" + utility.getZkCluster().getClientPort(); |
Contributor
There was a problem hiding this comment.
Here we should let it be 127.0.0.1 otherwise we have test failure in the latest build?
Contributor
There was a problem hiding this comment.
Basically no change needed in this IT test :)
virajjasani
approved these changes
Aug 1, 2026
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…rting REST server
Before starting the REST server in integration tests, ensure Phoenix system tables
are fully initialized by calling TestUtils.awaitPhoenixReady().
This eliminates the race condition that caused intermittent test failures with:
org.apache.phoenix.exception.PhoenixIOException: SYSTEM.CATALOG
Caused by: org.apache.hadoop.hbase.TableNotFoundException: SYSTEM.CATALOG
The fix adds TestUtils.awaitPhoenixReady(url) calls to all 82 integration test classes, either directly in their @BeforeClass methods or through inheritance from base classes like BaseSegmentScanIT, KclTestBase, UpdateItemBaseTests, and SegmentScanIT.
The awaitPhoenixReady() method polls for up to 30 seconds (vs RESTServer's 5 seconds) for Phoenix to become ready, providing sufficient time for SYSTEM.CATALOG initialization even on slow CI workers.
Changes: