Skip to content

HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. - #7021

Merged
jojochuang merged 1 commit into
apache:masterfrom
jojochuang:HDDS-11270
Aug 6, 2024
Merged

HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test.#7021
jojochuang merged 1 commit into
apache:masterfrom
jojochuang:HDDS-11270

Conversation

@jojochuang

@jojochuangjojochuang commented Aug 2, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test.

Please describe your PR in detail:
This is a test-only PR built on top of HDDS-11259.

Add a test class TestDatanodeUpgradeToHBaseSupport which has two test cases

  • PutBlock request with incremental chunk list
  • Block finalization request

These two requests are rejected before upgrade finalizes and accepted after upgrade.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11270

How was this patch tested?

Test only code

errose28
errose28 reviewed Aug 2, 2024
View reviewed changes

@errose28errose28 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this @jojochuang. The new test class looks good overall. Once #7012 is merged and the diff is reduced I can do a full review.

Comment thread...c/test/java/org/apache/hadoop/ozone/container/upgrade/TestDatanodeUpgradeToHBaseSupport.java Outdated
Comment thread...c/test/java/org/apache/hadoop/ozone/container/upgrade/TestDatanodeUpgradeToHBaseSupport.java Outdated
Comment thread...c/test/java/org/apache/hadoop/ozone/container/upgrade/TestDatanodeUpgradeToHBaseSupport.java
@smengclsmengcl added the hbase HBase on Ozone support label Aug 3, 2024
@jojochuang

jojochuang commented Aug 5, 2024

Copy link
Copy Markdown
ContributorAuthor

will rebase this PR against master after the HDDS-7593 feature branch gets merged.

errose28
errose28 approved these changes Aug 5, 2024
View reviewed changes

@errose28errose28 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this @jojochuang. Just one minor comment otherwise LGTM. If you want to wait until HBase branch is merged into master that's ok.

Comment threadhadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/container/ContainerTestHelper.java Outdated
@errose28

errose28 commented Aug 5, 2024

Copy link
Copy Markdown
Contributor

BTW I filed HDDS-11282 to extract the common setup code for these types of upgrade tests.

@jojochuang
HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) u…
3b66b1b
…pgrade test.
Change-Id: I9e3296ddbf5922d81d9754d80aac7e3618720565
Add tests covering incremental chunk list PutBlock requests during upgrade.
Change-Id: If473d1e5b5cc82adfac72fbe88a9ed71f7eaa169
Address review comments.
add additional check to make sure containers don't get marked unhealthy after DataNode rejects an unsupported operation.
Change-Id: I73dd83987d617bf5e17ff8e46d1d183b20c46b10
Move all reference of INCREMENTAL_CHUNK_LIST to OzoneConstants.
Change-Id: I00e7f2362b6a83d03c0eb3f615c1415c06cd668e
@jojochuang
jojochuang changed the base branch from HDDS-7593 to masterAugust 6, 2024 17:44
@jojochuang
jojochuang merged commit 91b2b91 into apache:masterAug 6, 2024
@jojochuang

jojochuang commented Aug 6, 2024

Copy link
Copy Markdown
ContributorAuthor

Merged. Thanks @errose28 !

errose28 added a commit to errose28/ozone that referenced this pull request Aug 7, 2024
@errose28
Merge branch 'master' into HDDS-10239-container-reconciliation
98369a8
* master: (181 commits)
HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024)
HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043)
HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040)
HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036)
HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003)
HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724)
HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021)
HDDS-11272. Statistics some node status information (apache#7025)
HDDS-11278. Move code out of Hadoop util package (apache#7028)
HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones
HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026)
HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032)
HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990)
HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023)
HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007)
HDDS-11224. Increase hdds.datanode.handler.count (apache#7011)
HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012)
HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014)
HDDS-11226. Make ExponentialBackoffPolicy maxRetries configurable (apache#6985)
HDDS-11260. [hsync] Add Ozone Manager protocol version (apache#7015)
...
Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/audit/DNAction.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java
hadoop-hdds/interface-client/src/main/proto/DatanodeClientProtocol.proto
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestContainerReportHandler.java
errose28 added a commit to errose28/ozone that referenced this pull request Aug 13, 2024
@errose28
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-11254-re…
6f68df6
…p-supervisor
Merge conflicts are resolved but the change does not yet build.
* HDDS-10239-container-reconciliation: (183 commits)
HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945)
HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024)
HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043)
HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040)
HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036)
HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003)
HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724)
HDDS-10926. Block deletion should update container merkle tree. (apache#6875)
HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021)
HDDS-11272. Statistics some node status information (apache#7025)
HDDS-11278. Move code out of Hadoop util package (apache#7028)
HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones
HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026)
HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032)
HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990)
HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023)
HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007)
HDDS-11224. Increase hdds.datanode.handler.count (apache#7011)
HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012)
HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014)
...
Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
errose28 added a commit to errose28/ozone that referenced this pull request Aug 16, 2024
@errose28
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-11253-co…
8601118
…rrupt-files
* HDDS-10239-container-reconciliation: (183 commits)
HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945)
HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024)
HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043)
HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040)
HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036)
HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003)
HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724)
HDDS-10926. Block deletion should update container merkle tree. (apache#6875)
HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021)
HDDS-11272. Statistics some node status information (apache#7025)
HDDS-11278. Move code out of Hadoop util package (apache#7028)
HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones
HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026)
HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032)
HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990)
HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023)
HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007)
HDDS-11224. Increase hdds.datanode.handler.count (apache#7011)
HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012)
HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014)
...
Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@errose28errose28errose28 approved these changes
@ChenSammiChenSammiAwaiting requested review from ChenSammi
@ashishkumar50ashishkumar50Awaiting requested review from ashishkumar50

Assignees

No one assigned

Labels

hbaseHBase on Ozone support

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jojochuang@errose28@smengcl