Skip to content

HDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to naming conventions - #1401

Merged
elek merged 2 commits into
apache:masterfrom
smengcl:HDDS-3927
Sep 13, 2020
Merged

HDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to naming conventions#1401
elek merged 2 commits into
apache:masterfrom
smengcl:HDDS-3927

Conversation

@smengcl

@smengclsmengcl commented Sep 5, 2020•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Rename the environment variables to be in accordance with the convention, and keep the compatibility by deprecating the old variable names.

  • HDFS_OM_OPTS -> OZONE_OM_OPTS
  • HDDS_DN_OPTS -> OZONE_DATANODE_OPTS
  • HDFS_STORAGECONTAINERMANAGER_OPTS -> OZONE_SCM_OPTS

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-3927

How was this patch tested?

No new tests added. Tested manually:

  1. HADOOP_SHELL_SCRIPT_DEBUG=1 OZONE_OM_OPTS=-Dtest123 ./ozone om
DEBUG: Appending OZONE_OM_OPTS onto HADOOP_OPTS
...
DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dtest123 -Dlog4j.configurationFile=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/etc/hadoop/om-audit-log4j2.properties -Dhadoop.log.dir=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT -Dhadoop.id.str=smeng -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
  1. HADOOP_SHELL_SCRIPT_DEBUG=1 OZONE_DATANODE_OPTS=-Dtest123 ./ozone datanode
DEBUG: Appending OZONE_DATANODE_OPTS onto HADOOP_OPTS
...
DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dorg.apache.ratis.thirdparty.io.netty.allocator.useCacheForAllThreads=false -Dorg.apache.ratis.thirdparty.io.netty.leakDetection.level=disabled -Dlog4j.configurationFile=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/etc/hadoop/dn-audit-log4j2.properties -Dtest123 -Dhadoop.log.dir=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT -Dhadoop.id.str=smeng -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
  1. HADOOP_SHELL_SCRIPT_DEBUG=1 OZONE_SCM_OPTS=-Dtest123 ./ozone scm
DEBUG: Appending OZONE_SCM_OPTS onto HADOOP_OPTS
...
DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dtest123 -Dlog4j.configurationFile=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/etc/hadoop/scm-audit-log4j2.properties -Dhadoop.log.dir=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT -Dhadoop.id.str=smeng -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
  1. Deprecation warning: HADOOP_SHELL_SCRIPT_DEBUG=1 HDFS_OM_OPTS=-Dtest123 ./ozone om
WARNING: HDFS_OM_OPTS has been replaced by OZONE_OM_OPTS. Using value of HDFS_OM_OPTS.

Caveat: when both deprecated and new variables names are present, the new variable will be ignored. So admins should refrain from using both at the same time. e.g. for HDFS_OM_OPTS=-Dtest123 OZONE_OM_OPTS=-Dtest456 ./ozone om, OZONE_OM_OPTS=-Dtest456 will be completely ignored, only HDFS_OM_OPTS=-Dtest123 will be respected.

Note

hadoop_deprecate_envvar takes care of the copying the values from the old variable name to the new one, and prints the deprecation warning.

@smengcl
HDDS-3927. Rename HDFS_OM_OPTS/HDDS_DN_OPTS to OZONE_MANAGER_OPTS/OZO…
8e5bfc2
…NE_DATANODE_OPTS
Change-Id: I629f87486b62b6a3b7f108addf89ce1ec2bc9e18
@smengclsmengcl self-assigned this Sep 5, 2020
elek
elek requested changes Sep 7, 2020
View reviewed changes

@elekelek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks to @smengcl the patch. I like this change.

Recently I learned that some OPTS are added automatically by hadoop_subcommand_opts, I think we can use that functionality (and naming convention).

Also: I would consider renaming HDFS_STORAGECONTAINERMANAGER_OPTS to OZONE_SCM_OPTS, too. Just for consistency.

Comment threadhadoop-hdds/common/src/main/conf/hadoop-env.sh Outdated
Comment threadhadoop-ozone/dist/src/shell/ozone/ozone Outdated
Comment threadhadoop-ozone/dist/src/shell/ozone/ozone Outdated
@smengcl
Conform to hadoop_subcommand_opts's OPTS naming conventions.
350b98b
Change-Id: I67259db97eabf04a55b9b5bbc582a6e5732cdb58
@smengclsmengcl changed the title HDDS-3927. Rename HDFS_OM_OPTS/HDDS_DN_OPTS to OZONE_MANAGER_OPTS/OZONE_DATANODE_OPTSHDDS-3927. Rename HDFS_OM_OPTS,HDDS_DN_OPTS,HDFS_STORAGECONTAINERMANAGER_OPTS to OZONE_OM_OPTS,OZONE_DATANODE_OPTS,OZONE_SCM_OPTSSep 7, 2020
@smengclsmengcl changed the title HDDS-3927. Rename HDFS_OM_OPTS,HDDS_DN_OPTS,HDFS_STORAGECONTAINERMANAGER_OPTS to OZONE_OM_OPTS,OZONE_DATANODE_OPTS,OZONE_SCM_OPTSHDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to naming conventionsSep 7, 2020
@smengcl

smengcl commented Sep 9, 2020

Copy link
Copy Markdown
ContributorAuthor

Thanks @elek for the review. I have made changes accordingly.

Do the changes look good to you?

elek
elek approved these changes Sep 13, 2020
View reviewed changes

@elekelek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the changes look good to you?

Sure, I am +1. Thank you very much for the update.

@elek
elek merged commit 48e8e50 into apache:masterSep 13, 2020
llemec pushed a commit to llemec/hadoop-ozone that referenced this pull request Sep 15, 2020
@smengcl@llemec
HDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to namin…
a530246
…g conventions (apache#1401)
errose28 added a commit to errose28/ozone that referenced this pull request Sep 18, 2020
@errose28
Merge branch 'master' into HDDS-4122-quota-attempt2
90ab92d
* master: (47 commits)
HDDS-4104. Provide a way to get the default value and key of java-based-configuration easily (apache#1369)
HDDS-4250. Fix wrong logger name (apache#1429)
HDDS-4244. Container deleted wrong replica cause mis-replicated. (apache#1423)
HDDS-4053. Volume space: add quotaUsageInBytes and update it when write and delete key. (apache#1296)
HDDS-4210. ResolveBucket during checkAcls fails. (apache#1398)
HDDS-4075. Retry request on different OM on AccessControlException (apache#1303)
HDDS-4166. Documentation index page redirects to the wrong address (apache#1372)
HDDS-4039. Reduce the number of fields in hdds.proto to improve performance (apache#1289)
HDDS-4155. Directory and filename can end up with same name in a path. (apache#1361)
HDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to naming conventions (apache#1401)
HDDS-4119. Improve performance of the BufferPool management of Ozone client (apache#1336)
HDDS-4217.Remove test TestOzoneContainerRatis (apache#1408)
HDDS-4218.Remove test TestRatisManager (apache#1409)
HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE. (apache#1337)
HDDS-4228: add field 'num' to ALLOCATE_BLOCK of scm audit log. (apache#1413)
HDDS-4196. Add an endpoint in Recon to query Prometheus (apache#1390)
HDDS-4211. [OFS] Better owner and group display for listing Ozone volumes and buckets (apache#1397)
HDDS-4150. recon.api.TestEndpoints test is flaky (apache#1396)
HDDS-4170 - Fix typo in method description. (apache#1406)
HDDS-4064. Show container verbose info with verbose option (apache#1290)
...
errose28 added a commit to errose28/ozone that referenced this pull request Sep 18, 2020
@errose28
Incorporate volume size tracking into open key commit request and res…
5fccc60
…ponse
* HDDS-4122-quota-attempt2: (51 commits)
Remove redundant check status calls in children of AbstractOMOpenKeyDeleteRequest
Remove unused inports and fix super constructor calls
Move common volume byte usage update code to AbstractOMKeyDeleteResponse
Add volume quota update to open key delete response, and group duplicate code
HDDS-4104. Provide a way to get the default value and key of java-based-configuration easily (apache#1369)
HDDS-4250. Fix wrong logger name (apache#1429)
HDDS-4244. Container deleted wrong replica cause mis-replicated. (apache#1423)
HDDS-4053. Volume space: add quotaUsageInBytes and update it when write and delete key. (apache#1296)
HDDS-4210. ResolveBucket during checkAcls fails. (apache#1398)
HDDS-4075. Retry request on different OM on AccessControlException (apache#1303)
HDDS-4166. Documentation index page redirects to the wrong address (apache#1372)
HDDS-4039. Reduce the number of fields in hdds.proto to improve performance (apache#1289)
HDDS-4155. Directory and filename can end up with same name in a path. (apache#1361)
HDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to naming conventions (apache#1401)
HDDS-4119. Improve performance of the BufferPool management of Ozone client (apache#1336)
HDDS-4217.Remove test TestOzoneContainerRatis (apache#1408)
HDDS-4218.Remove test TestRatisManager (apache#1409)
HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE. (apache#1337)
HDDS-4228: add field 'num' to ALLOCATE_BLOCK of scm audit log. (apache#1413)
HDDS-4196. Add an endpoint in Recon to query Prometheus (apache#1390)
...
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@elekelekelek approved these changes
@bharatviswa504bharatviswa504Awaiting requested review from bharatviswa504

Assignees

@smengclsmengcl

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@smengcl@elek