Skip to content

HDDS-1651. Create a http.policy config for Ozone - #930

Closed
shwetayakkali wants to merge 3 commits into
apache:trunkfrom
shwetayakkali:HDDS-1651
Closed

HDDS-1651. Create a http.policy config for Ozone#930
shwetayakkali wants to merge 3 commits into
apache:trunkfrom
shwetayakkali:HDDS-1651

Conversation

@shwetayakkali

Copy link
Copy Markdown

Change-Id: Ia284f685f6d39a512124e6055537615d325ae96b

Change-Id: Ia284f685f6d39a512124e6055537615d325ae96b
@hadoop-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec35Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1@author0The patch does not contain any @author tags.
-1test4tests0The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1mvninstall527trunk passed
+1compile294trunk passed
+1checkstyle90trunk passed
+1mvnsite0trunk passed
+1shadedclient890branch has no errors when building and testing our client artifacts.
+1javadoc188trunk passed
0spotbugs333Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs524trunk passed
_ Patch Compile Tests _
+1mvninstall458the patch passed
+1compile292the patch passed
+1javac292the patch passed
+1checkstyle97the patch passed
+1mvnsite0the patch passed
+1whitespace0The patch has no whitespace issues.
+1xml1The patch has no ill-formed XML file.
+1shadedclient694patch has no errors when building and testing our client artifacts.
+1javadoc181the patch passed
+1findbugs538the patch passed
_ Other Tests _
-1unit148hadoop-hdds in the patch failed.
-1unit1337hadoop-ozone in the patch failed.
+1asflicense57The patch does not generate ASF License warnings.
6546
ReasonTests
Failed junit testshadoop.ozone.container.common.impl.TestHddsDispatcher
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestWatchForCommit
hadoop.ozone.client.rpc.TestCommitWatcher
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
SubsystemReport/Notes
DockerClient=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-930/1/artifact/out/Dockerfile
GITHUB PR#930
Optional Testsdupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
unameLinux 5a57ea44076b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/hadoop.sh
git revisiontrunk / 76b94c2
Default Java1.8.0_212
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/1/artifact/out/patch-unit-hadoop-hdds.txt
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Resultshttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/1/testReport/
Max. process+thread count5361 (vs. ulimit of 5500)
modulesC: hadoop-hdds/common U: hadoop-hdds/common
Console outputhttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/1/console
versionsgit=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

public static final int DFS_CONTAINER_RATIS_IPC_PORT_DEFAULT = 9858;

public static final String OZONE_HTTP_POLICY = "ozone.http.policy";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This newly added property should be used during HTTP server start.
That part is missing.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor NIT: ozone.http.policy -> Use OzoneConfigKeys.OZONE_HTTP_POLICY.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line get() second param value should be a default value for the property.

Change-Id: I047aafc733c936fb82f926bcde489595cf51d928

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to set it back?

@bharatviswa504bharatviswa504 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank You @shwetayakkali for the contribution.
I have a few comments.

String policyStr = conf.get("ozone.http.policy", OzoneConfigKeys.OZONE_HTTP_POLICY);
if(policyStr == null || policyStr.length() == 0) {
policyStr = conf.get("dfs.http.policy", DFSConfigKeys.DFS_HTTP_POLICY_DEFAULT);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If HTTP_ONLY is used as the default,
we can conf.get("OzoneConfigKeys.OZONE_HTTP_POLICY", DFSConfigKeys.DFS_HTTP_POLICY_DEFAULT);
Then we don't need checks of null, and we can make this logic simple.

@hadoop-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec36Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0mvndep63Maven dependency ordering for branch
+1mvninstall552trunk passed
+1compile311trunk passed
+1checkstyle80trunk passed
+1mvnsite0trunk passed
+1shadedclient857branch has no errors when building and testing our client artifacts.
+1javadoc181trunk passed
0spotbugs350Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs536trunk passed
_ Patch Compile Tests _
0mvndep30Maven dependency ordering for patch
+1mvninstall473the patch passed
+1compile302the patch passed
+1javac302the patch passed
-0checkstyle38hadoop-hdds: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1mvnsite0the patch passed
+1whitespace0The patch has no whitespace issues.
+1xml1The patch has no ill-formed XML file.
+1shadedclient641patch has no errors when building and testing our client artifacts.
+1javadoc180the patch passed
+1findbugs545the patch passed
_ Other Tests _
-1unit114hadoop-hdds in the patch failed.
-1unit1267hadoop-ozone in the patch failed.
+1asflicense58The patch does not generate ASF License warnings.
6506
ReasonTests
Failed junit testshadoop.hdds.server.TestBaseHttpServer
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.TestMiniChaosOzoneCluster
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
SubsystemReport/Notes
DockerClient=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-930/2/artifact/out/Dockerfile
GITHUB PR#930
Optional Testsdupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
unameLinux 31c208148507 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/hadoop.sh
git revisiontrunk / 4f45529
Default Java1.8.0_212
checkstylehttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/2/artifact/out/diff-checkstyle-hadoop-hdds.txt
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/2/artifact/out/patch-unit-hadoop-hdds.txt
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Resultshttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/2/testReport/
Max. process+thread count4866 (vs. ulimit of 5500)
modulesC: hadoop-hdds/common hadoop-hdds/framework hadoop-ozone/ozone-manager U: .
Console outputhttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/2/console
versionsgit=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec92Docker mode activated.
_ Prechecks _
+1dupname1No case conflicting files found.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0mvndep73Maven dependency ordering for branch
+1mvninstall579trunk passed
+1compile353trunk passed
+1checkstyle98trunk passed
+1mvnsite0trunk passed
+1shadedclient1084branch has no errors when building and testing our client artifacts.
+1javadoc192trunk passed
0spotbugs395Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs629trunk passed
_ Patch Compile Tests _
0mvndep31Maven dependency ordering for patch
+1mvninstall574the patch passed
+1compile348the patch passed
+1javac348the patch passed
-0checkstyle50hadoop-hdds: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1mvnsite0the patch passed
+1whitespace0The patch has no whitespace issues.
+1xml1The patch has no ill-formed XML file.
+1shadedclient843patch has no errors when building and testing our client artifacts.
+1javadoc174the patch passed
+1findbugs573the patch passed
_ Other Tests _
-1unit137hadoop-hdds in the patch failed.
-1unit1315hadoop-ozone in the patch failed.
+1asflicense50The patch does not generate ASF License warnings.
7455
ReasonTests
Failed junit testshadoop.hdds.server.TestBaseHttpServer
hadoop.ozone.client.rpc.TestBCSID
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
SubsystemReport/Notes
DockerClient=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-930/3/artifact/out/Dockerfile
GITHUB PR#930
Optional Testsdupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
unameLinux 6a08fa8fe25d 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/hadoop.sh
git revisiontrunk / 4f45529
Default Java1.8.0_212
checkstylehttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/3/artifact/out/diff-checkstyle-hadoop-hdds.txt
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/3/artifact/out/patch-unit-hadoop-hdds.txt
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Resultshttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/3/testReport/
Max. process+thread count4424 (vs. ulimit of 5500)
modulesC: hadoop-hdds/common hadoop-hdds/framework hadoop-ozone/ozone-manager U: .
Console outputhttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/3/console
versionsgit=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@bharatviswa504

Copy link
Copy Markdown
Contributor

One more comment, we need to change the code in OzoneManagerSnapShotProvider class, to use this newly added method.

Line 89: this.httpPolicy = DFSUtil.getHttpPolicy(conf);

Change-Id: I23b1ddc619488bb290be9136a18ec5e2aab7dd77
@hadoop-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec519Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0mvndep67Maven dependency ordering for branch
+1mvninstall498trunk passed
+1compile263trunk passed
+1checkstyle73trunk passed
+1mvnsite0trunk passed
+1shadedclient863branch has no errors when building and testing our client artifacts.
+1javadoc164trunk passed
0spotbugs313Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs505trunk passed
_ Patch Compile Tests _
0mvndep34Maven dependency ordering for patch
+1mvninstall439the patch passed
+1compile273the patch passed
+1javac273the patch passed
-0checkstyle39hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1mvnsite0the patch passed
+1whitespace0The patch has no whitespace issues.
+1xml1The patch has no ill-formed XML file.
+1shadedclient682patch has no errors when building and testing our client artifacts.
+1javadoc161the patch passed
+1findbugs524the patch passed
_ Other Tests _
-1unit148hadoop-hdds in the patch failed.
-1unit1092hadoop-ozone in the patch failed.
+1asflicense50The patch does not generate ASF License warnings.
6616
ReasonTests
Failed junit testshadoop.ozone.container.common.impl.TestHddsDispatcher
hadoop.ozone.client.rpc.TestFailureHandlingByClient
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestWatchForCommit
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
SubsystemReport/Notes
DockerClient=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-930/4/artifact/out/Dockerfile
GITHUB PR#930
Optional Testsdupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
unameLinux a041c46a08f4 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/hadoop.sh
git revisiontrunk / 37bd5bb
Default Java1.8.0_212
checkstylehttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/4/artifact/out/diff-checkstyle-hadoop-hdds.txt
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/4/artifact/out/patch-unit-hadoop-hdds.txt
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/4/artifact/out/patch-unit-hadoop-ozone.txt
Test Resultshttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/4/testReport/
Max. process+thread count5006 (vs. ulimit of 5500)
modulesC: hadoop-hdds/common hadoop-hdds/framework hadoop-ozone/ozone-manager U: .
Console outputhttps://builds.apache.org/job/hadoop-multibranch/job/PR-930/4/console
versionsgit=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@bharatviswa504

bharatviswa504 commented Jun 25, 2019

Copy link
Copy Markdown
Contributor

Hi,
I have a question:
What is the reason to create a new http policy for ozone? Because when https is enabled, some of the additional config like key store location we still use dfs.https.server.keystore.resource.

I feel ozone can also re-use the hdfs config. In this way, we don't miss the code changes where HTTP policy is being used, like OzoneManagerSnapShotProvider which uses still DFSUtils.getHttpPolicy.

And also we use to create HttpServer2.Builder builder with below code. That again uses dfs.http.policy. So, this is the reason we need to set ozone.http.policy to dfs.http.policy to make it work. So, I feel to avoid all these, we can use dfs.http.policy as before. Let me know your thoughts on this.

 builder = DFSUtil.httpServerTemplateForNNAndJN(conf, this.httpAddress,
this.httpsAddress, name, getSpnegoPrincipal(), getKeytabFile());

import java.util.Optional;
import java.util.TimeZone;

import org.apache.hadoop.HadoopIllegalArgumentException;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: Unused import.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for using dfs.http..policy to reduce redundant config knobs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank You @eyanghwx for review.
I also feel the same. Can we close this as won't Fix?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank You @eyanghwx for the confirmation.
I will close this as Won't fix.

@bharatviswa504

Copy link
Copy Markdown
Contributor

Thank You @shwetayakkali for the contribution.
Closing this based on the above discussion.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@shwetayakkali@hadoop-yetus@bharatviswa504@eyanghwx