java.lang.NoClassDefFoundError: Could not initialize class org.apache.comet.package$

[andygrove]

Describe the bug

I am building a Docker image using the Dockerfile from this repo and then trying to execute queries in k8s. I get the following error.

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.comet.package$
  at org.apache.comet.vector.NativeUtil.<init>(NativeUtil.scala:48)
  at org.apache.comet.CometExecIterator.<init>(CometExecIterator.scala:52)                         

As far as I can tell, the root cause is an exception being thrown in the following code:

package object comet {

  val CometArrowAllocator = new RootAllocator(Long.MaxValue)

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

[tbar4]

Getting a similar error:

scala> (0 until 10).toDF("a").write.mode("overwrite").parquet("/tmp/test")
24/10/11 19:32:10 WARN CometSparkSessionExtensions: Comet extension is disabled because of error when loading native lib. Falling back to Spark
java.lang.NoClassDefFoundError: Could not initialize class org.apache.comet.NativeBase

[andygrove]

Getting a similar error:

scala> (0 until 10).toDF("a").write.mode("overwrite").parquet("/tmp/test")
24/10/11 19:32:10 WARN CometSparkSessionExtensions: Comet extension is disabled because of error when loading native lib. Falling back to Spark
java.lang.NoClassDefFoundError: Could not initialize class org.apache.comet.NativeBase

There is a specific issue for the NativeBase error: #999