ParquetWriter is incompatible with JDK25

[hugoncosta]

Describe the bug, including details regarding any error messages, version, and platform.

Hello, I'm trying to upgrade my code base to JDK25 and I'm having this issue on my tests

     java.lang.UnsupportedOperationException: getSubject is not supported
        at java.base/javax.security.auth.Subject.getSubject(Subject.java:277)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:577)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3852)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3842)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3630)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:541)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
        at org.apache.parquet.hadoop.util.HadoopOutputFile.fromPath(HadoopOutputFile.java:58)
        at com.amazon.networkvalidator.parquet.ParquetWriter.write(ParquetWriter.kt:75)
        at com.amazon.networkvalidator.parquet.ParquetWriterTest$test writing to parquet$1.invokeSuspend(ParquetWriterTest.kt:88)
        at com.amazon.networkvalidator.parquet.ParquetWriterTest$test writing to parquet$1.invoke(ParquetWriterTest.kt)
        at com.amazon.networkvalidator.parquet.ParquetWriterTest$test writing to parquet$1.invoke(ParquetWriterTest.kt)
        at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt$runTest$2$1$1.invokeSuspend(TestBuilders.kt:318)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:101)
        at kotlinx.coroutines.test.TestDispatcher.processEvent$kotlinx_coroutines_test(TestDispatcher.kt:24)
        at kotlinx.coroutines.test.TestCoroutineScheduler.tryRunNextTaskUnless$kotlinx_coroutines_test(TestCoroutineScheduler.kt:99)
        at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt$runTest$2$1$workRunner$1.invokeSuspend(TestBuilders.kt:327)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:101)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:263)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:47)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at kotlinx.coroutines.test.TestBuildersJvmKt.createTestResult(TestBuildersJvm.kt:10)
        at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0(TestBuilders.kt:310)
        at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0(Unknown Source)
        at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0(TestBuilders.kt:168)
        at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0(Unknown Source)
        at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0$default(TestBuilders.kt:160)
        at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0$default(Unknown Source)
        at com.amazon.networkvalidator.parquet.ParquetWriterTest.test writing to parquet(ParquetWriterTest.kt:76) 

It appears this is an issue within the Hadoop libraries upon which Parquet Java depends on

Component(s)

No response

[hugoncosta]

Raised a Jira in Apache itself - [​HADOOP-19703] UserGroupInformation.java is using a non-support operation in JDK25 - ASF JIRA

[hugoncosta]

Work is ongoing in apache/hadoop#7892

[ChristopherWirt]

I managed to get around this by - disabling the filesystem cache HADOOP_CONF.set("fs.file.impl.disable.cache", "true");

[steveloughran]

FYI Hadoop 3.5.0 (RC1 out this week!) is compatible with later java versions as it deals with the getSubject removal which oracle included when they took out all the sandbox. Needless piece of pain there.