[BLAZE-1149] Fix shuffle file permission issue when using BlazeShuffleManager

[turboFei]

Which issue does this PR close?

Closes#1149

Similar issue with gluten we met before, apache/gluten#9156

Rationale for this change

The purpose of this change is to ensure that the shuffle data files allowing read access for the others to fix shuffle fetch fail.

The error message on NodeManager:

Caused by: java.io.FileNotFoundException: /hadoop/2/yarn/local/usercache/b_stf/appcache/application_1754277161261_3068012/blockmgr-62f53ceb-e9f1-4ac8-bbc7-7442b47d6990/2c/shuffle_0_114_0.data (Permission denied)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at org.sparkproject.io.netty.channel.DefaultFileRegion.open(DefaultFileRegion.java:88)
at org.sparkproject.io.netty.channel.DefaultFileRegion.transferTo(DefaultFileRegion.java:128)

Set the shuffle file permissions to 0644 to keep it consistent with the permissions of the built-in shuffler manager in Spark.

1. https://github.com/apache/spark/blob/549c30acfebdc0f72916448a4cbd1dab6ff9b760/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala#L87

2. https://github.com/apache/spark/blob/549c30acfebdc0f72916448a4cbd1dab6ff9b760/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala#L231

Related spark PR: apache/spark#35085

What changes are included in this PR?

Are there any user-facing changes?

No.

How was this patch tested?

Have tested in our cluster, the file permission looks good now.

It can fetch the shuffle data after this patch.

ls -l /hadoop/2/yarn/local/usercache/b_stf/appcache/application_1754277161261_3329570/blockmgr-ddbec8d5-06d4-41e9-91dd-e07c2d79800d/3c/
total 259048
-rw-r--r-- 1 b_stf b_dpms 32008 Aug 11 11:59 shuffle_0_17718_0.index
-rw-r--r-- 1 b_stf b_dpms 32008 Aug 11 11:50 shuffle_0_20_0.index
-rw-r--r-- 1 b_stf b_dpms 265166734 Aug 11 12:02 shuffle_0_28109_0.data
-rw-r--r-- 1 b_stf b_dpms 32008 Aug 11 12:02 shuffle_0_28109_0.index.tmp

[turboFei]

below code would be impacted by umask.

OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.mode(0o644)
.open(path_ref)?;

So, we have to use std::fs::set_permissions

[turboFei]

I have tested in our cluster, it can fetch the shuffle data after this patch.

cc @richox

[turboFei]

gentle ping @richox

[richox]

i didnt see this setPermission logics in spark. so why do we need a different implementation here?

[turboFei]

i didnt see this setPermission logics in spark. so why do we need a different implementation here?

cc @wangyum

[turboFei]

1. https://github.com/apache/spark/blob/549c30acfebdc0f72916448a4cbd1dab6ff9b760/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala#L87

2. https://github.com/apache/spark/blob/549c30acfebdc0f72916448a4cbd1dab6ff9b760/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala#L231

Here is the logic if external shuffle service enabled.

[turboFei]

Have updated the PR description.

[wangyum]

i didnt see this setPermission logics in spark. so why do we need a different implementation here?

The directory permission created by the c++/rust method in a specific file system or operating system may be inconsistent with the directory permissions created by Spark through Java.