Skip to content

Support gzip compression in native Parquet writes #4929

Description

@andygrove

What is the problem the feature request solves?

Comet's native Parquet writer supports none, snappy, lz4, and zstd. A write configured with spark.sql.parquet.compression.codec=gzip (or .option("compression", "gzip")) falls back to Spark's writer, since CometDataWritingCommand.getSupportLevel reports Unsupported("Unsupported compression codec: gzip").

gzip is a valid Spark Parquet codec and is widely used, so this is an avoidable loss of native write coverage.

While looking at this, two related gaps in the same code path:

  • parseCompressionCodec does not honor Spark's codec precedence. Spark's ParquetOptions resolves compression > parquet.compression (ParquetOutputFormat.COMPRESSION) > spark.sql.parquet.compression.codec, but Comet checks only the first and third. A write with .option("parquet.compression", "gzip") and the default SQLConf is therefore natively written as snappy, silently substituting the user's requested codec.
  • Spark accepts uncompressed as a spelling of none, but Comet's supported-codec set only lists none, so uncompressed falls back unnecessarily.

Describe the potential solution

Add a Gzip value to the protobuf CompressionCodec enum and map it in the native Parquet writer to Compression::GZIP(GzipLevel::default()) (level 6, matching parquet-mr's zlib default). Fix the codec precedence and accept the uncompressed alias.

Part of #1625.

Additional context

Compression levels remain non-configurable (gzip and zstd use fixed defaults), so parquet-mr's parquet.compression.codec.{gzip,zstd}.level configs are still ignored. That is out of scope here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions