Skip to content

HDDS-1382. Create customized CSI server for Ozone. - #693

Closed
elek wants to merge 9 commits into
apache:trunkfrom
elek:HDDS-1382
Closed

HDDS-1382. Create customized CSI server for Ozone.#693
elek wants to merge 9 commits into
apache:trunkfrom
elek:HDDS-1382

Conversation

@elek

@elekelek commented Apr 4, 2019

Copy link
Copy Markdown
Member

@elekelek added the ozone label Apr 4, 2019
Comment threadhadoop-ozone/csi/src/main/java/org/apache/hadoop/ozone/csi/CsiServer.java Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make 1000_000_000 configurable?

@xiaoyuyaoxiaoyuyaoApr 17, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: LOG.info("Executing {}", mountCommand);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: LOG.info("Executing {}", umountCommand);

@xiaoyuyao

Copy link
Copy Markdown
Contributor

Thanks @elek for working on this, the change LGTM overall, just few minor comments.
Please also fix the the nightly test run. +1 after that.

@hadoop-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec44Docker mode activated.
_ Prechecks _
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0mvndep72Maven dependency ordering for branch
+1mvninstall1137trunk passed
+1compile1359trunk passed
+1checkstyle152trunk passed
-1mvnsite117hadoop-ozone in trunk failed.
-1mvnsite46integration-test in trunk failed.
+1shadedclient781branch has no errors when building and testing our client artifacts.
0findbugs0Skipped patched modules with no Java source: hadoop-ozone hadoop-ozone/dist hadoop-ozone/integration-test
+1findbugs163trunk passed
+1javadoc267trunk passed
_ Patch Compile Tests _
0mvndep28Maven dependency ordering for patch
-1mvninstall174hadoop-ozone in the patch failed.
-1mvninstall28integration-test in the patch failed.
+1compile1002the patch passed
+1cc1002the patch passed
+1javac1002the patch passed
+1checkstyle152the patch passed
-1hadolint0The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2)
-1mvnsite123hadoop-ozone in the patch failed.
-1mvnsite47integration-test in the patch failed.
+1shellcheck26There were no new shellcheck issues.
+1shelldocs33There were no new shelldocs issues.
+1whitespace1The patch has no whitespace issues.
+1xml7The patch has no ill-formed XML file.
+1shadedclient750patch has no errors when building and testing our client artifacts.
0findbugs0Skipped patched modules with no Java source: hadoop-ozone hadoop-ozone/dist hadoop-ozone/integration-test
+1findbugs252the patch passed
+1javadoc315the patch passed
_ Other Tests _
+1unit95common in the patch passed.
-1unit173hadoop-ozone in the patch failed.
+1unit53common in the patch passed.
+1unit45csi in the patch passed.
+1unit39dist in the patch passed.
-1unit47integration-test in the patch failed.
+1asflicense51The patch does not generate ASF License warnings.
8439
SubsystemReport/Notes
DockerClient=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/Dockerfile
GITHUB PR#693
Optional Testsdupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs cc hadolint
unameLinux 7ba080e16418 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/hadoop.sh
git revisiontrunk / c35abcd
mavenversion: Apache Maven 3.3.9
Default Java1.8.0_191
mvnsitehttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/branch-mvnsite-hadoop-ozone.txt
mvnsitehttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/branch-mvnsite-hadoop-ozone_integration-test.txt
shellcheckv0.4.6
findbugsv3.1.0-RC1
mvninstallhttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
mvninstallhttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/patch-mvninstall-hadoop-ozone_integration-test.txt
hadolinthttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/diff-patch-hadolint.txt
mvnsitehttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/patch-mvnsite-hadoop-ozone.txt
mvnsitehttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/patch-mvnsite-hadoop-ozone_integration-test.txt
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/patch-unit-hadoop-ozone.txt
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Resultshttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/testReport/
Max. process+thread count338 (vs. ulimit of 5500)
modulesC: hadoop-hdds/common hadoop-ozone hadoop-ozone/common hadoop-ozone/csi hadoop-ozone/dist hadoop-ozone/integration-test U: .
Console outputhttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/console
Powered byApache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo in end?

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Removed the modification of ozone-default.xml as I switched to use the annotation based configs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tmp directory might be emptied by admins and others without knowledge of this file.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK it shouldn't be a problem. This is a socket which is created by the process to listen on and even if it's deleted the socket will work well (until the close).

But I modified it to /var/lib/csi.sock. It can be more meaningful (and maybe more safe If my previous statement is wrong ;-) )

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we use the new config annotations instead?

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely. I uploaded a new version with config annotations.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update this with @config annotations?

@ajayydv

Copy link
Copy Markdown
Contributor

@elek thanks for the patch. LGTM, added few comments. Seems this needs a rebase as well.

@elek
elek marked this pull request as ready for review May 29, 2019 12:39
@hadoop-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec126Docker mode activated.
_ Prechecks _
+1dupname1No case conflicting files found.
0shelldocs0Shelldocs was not available.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0mvndep125Maven dependency ordering for branch
+1mvninstall683trunk passed
+1compile295trunk passed
+1checkstyle105trunk passed
+1mvnsite0trunk passed
+1shadedclient845branch has no errors when building and testing our client artifacts.
+1javadoc168trunk passed
0spotbugs304Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs517trunk passed
_ Patch Compile Tests _
0mvndep39Maven dependency ordering for patch
+1mvninstall563the patch passed
+1compile322the patch passed
+1cc322the patch passed
-1javac214hadoop-ozone generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
-0checkstyle46hadoop-ozone: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
-1hadolint3The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1)
+1mvnsite0the patch passed
+1shellcheck25There were no new shellcheck issues.
+1whitespace0The patch has no whitespace issues.
+1xml10The patch has no ill-formed XML file.
+1shadedclient700patch has no errors when building and testing our client artifacts.
+1javadoc187the patch passed
+1findbugs554the patch passed
_ Other Tests _
-1unit370hadoop-hdds in the patch failed.
-1unit1442hadoop-ozone in the patch failed.
+1asflicense53The patch does not generate ASF License warnings.
7551
ReasonTests
Failed junit testshadoop.hdds.scm.container.placement.algorithms.TestSCMContainerPlacementRackAware
hadoop.hdds.scm.pipeline.TestNodeFailure
hadoop.ozone.TestMiniChaosOzoneCluster
hadoop.hdds.scm.pipeline.TestSCMRestart
hadoop.ozone.om.TestOzoneManagerHA
hadoop.hdds.scm.pipeline.TestNode2PipelineMap
hadoop.ozone.om.TestOzoneManager
hadoop.hdds.scm.safemode.TestSCMSafeModeWithPipelineRules
SubsystemReport/Notes
DockerClient=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/artifact/out/Dockerfile
GITHUB PR#693
Optional Testsdupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs cc hadolint
unameLinux 55952ec91b3e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/hadoop.sh
git revisiontrunk / 544876f
Default Java1.8.0_212
javachttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/artifact/out/diff-compile-javac-hadoop-ozone.txt
checkstylehttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/artifact/out/diff-checkstyle-hadoop-ozone.txt
hadolinthttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/artifact/out/diff-patch-hadolint.txt
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/artifact/out/patch-unit-hadoop-hdds.txt
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/artifact/out/patch-unit-hadoop-ozone.txt
Test Resultshttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/testReport/
Max. process+thread count2112 (vs. ulimit of 5500)
modulesC: hadoop-hdds/common hadoop-ozone hadoop-ozone/common hadoop-ozone/csi hadoop-ozone/dist hadoop-ozone/integration-test U: .
Console outputhttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/console
versionsgit=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1 hadolint=1.11.1-0-g0e692dd
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@anuengineer

Copy link
Copy Markdown
Contributor

+1, @elek Can you please fix the checkstyle issues while committing ? Thanks

@elek

elek commented May 31, 2019

Copy link
Copy Markdown
MemberAuthor

Thanks the review @anuengineer I am merging it right now (checkstyle issues are fixed).

And we can continue the work:

  • HDDS-1508: Example k8s files to deploy it
  • Documentation is required in the docs
  • We can make it more configurable (support other fuse drivers not just goofys?)

@elekelek closed this in 1ae062cMay 31, 2019
@hadoop-yetus

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec35Docker mode activated.
_ Prechecks _
+1dupname1No case conflicting files found.
0shelldocs0Shelldocs was not available.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0mvndep71Maven dependency ordering for branch
+1mvninstall528trunk passed
+1compile305trunk passed
+1checkstyle101trunk passed
+1mvnsite0trunk passed
+1shadedclient804branch has no errors when building and testing our client artifacts.
+1javadoc179trunk passed
0spotbugs309Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs503trunk passed
_ Patch Compile Tests _
0mvndep37Maven dependency ordering for patch
+1mvninstall540the patch passed
+1compile315the patch passed
+1cc315the patch passed
-1javac209hadoop-ozone generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1checkstyle101the patch passed
-1hadolint1The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1)
+1mvnsite0the patch passed
+1shellcheck25There were no new shellcheck issues.
+1whitespace0The patch has no whitespace issues.
+1xml7The patch has no ill-formed XML file.
+1shadedclient694patch has no errors when building and testing our client artifacts.
+1javadoc183the patch passed
+1findbugs618the patch passed
_ Other Tests _
+1unit224hadoop-hdds in the patch passed.
-1unit1467hadoop-ozone in the patch failed.
+1asflicense57The patch does not generate ASF License warnings.
7058
ReasonTests
Failed junit testshadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
SubsystemReport/Notes
DockerClient=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-693/5/artifact/out/Dockerfile
GITHUB PR#693
Optional Testsdupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs cc hadolint
unameLinux 6442ceeebd8b 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitypersonality/hadoop.sh
git revisiontrunk / 52128e3
Default Java1.8.0_212
javachttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/5/artifact/out/diff-compile-javac-hadoop-ozone.txt
hadolinthttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/5/artifact/out/diff-patch-hadolint.txt
unithttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/5/artifact/out/patch-unit-hadoop-ozone.txt
Test Resultshttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/5/testReport/
Max. process+thread count5294 (vs. ulimit of 5500)
modulesC: hadoop-hdds/common hadoop-ozone hadoop-ozone/common hadoop-ozone/csi hadoop-ozone/dist hadoop-ozone/integration-test U: .
Console outputhttps://builds.apache.org/job/hadoop-multibranch/job/PR-693/5/console
versionsgit=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1 hadolint=1.11.1-0-g0e692dd
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
…vel API
Author: Prateek Maheshwari <pmaheshwari@apache.org>
Reviewers: Yi Pan <nickpan47@gmail.com>, Jagadish Venkataraman <vjagadish1989@gmail.com>
Closesapache#693 from prateekm/intermediate-stream-serde
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@elek@xiaoyuyao@hadoop-yetus@ajayydv@anuengineer