[SPARK-41943][CORE] Use java api to create files and grant permissions

[smallzhongfeng]

What changes were proposed in this pull request?

For method createDirWithPermission770, using java api to create files and grant permissions instead of calling shell commands.

Why are the changes needed?

Safer and more efficient.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Origin uts.

[smallzhongfeng]

For TODO, maybe we can see https://bugs.openjdk.org/browse/JDK-8220013

[HyukjinKwon]

Mind summarizing why we can use this approach while the JDK issue isn't resolved?

[smallzhongfeng]

In unix like systems every process has a property called umask which is masked onto the permissions of any file created and inherited by child processes. the default is 0002 or "turn off write for others". So it's most likely that Java is setting the permission you seek and then it's being masked out. At present, this issue has not been resolved, so my current approach is to create a directory first, and then explicitly grant permissions.

[mridulm]

+CC @zhouyejoe

[smallzhongfeng]

cc @cloud-fan@HyukjinKwon@LuciferYang @@zhouyejoe Hope to get your opinion. :)

[zhouyejoe]

There was a long discussion thread regarding this implementation in this PR. There will be some issue with setgid.
Will add @otterc@akpatnam25@Kimahriman for additional comments and review.

[smallzhongfeng]

There was a long discussion thread regarding this implementation in this PR. There will be some issue with setgid. Will add @otterc@akpatnam25@Kimahriman for additional comments and review.

Thank you for reminding me @zhouyejoe . I looked at the results of the discussion and it seems to be consistent with my handling method. You can see #35085 (comment)

[Kimahriman]

Yeah I think this is fine. mkdir -p essentially does a mkdir syscall followed by a chmod sys call, which is effectively what this is doing as well. Both approaches get around the umask but lose the setgid bit which is why all the make shuffle files world readable was necessary in #35085

[smallzhongfeng]

Could you take another look for this? @mridulm@zhouyejoe@HyukjinKwon@Kimahriman@cloud-fan THX!

[AmplabJenkins]

Can one of the admins verify this patch?

[cloud-fan]

thanks, merging to master!

[smallzhongfeng]

Thanks all. @mridulm@zhouyejoe@Kimahriman@HyukjinKwon@cloud-fan