Add Configurable Thrift Max Message Size for Parquet Metadata Reading

[cravani]

Describe the enhancement requested

When reading Parquet files with large metadata (e.g., files with thousands of columns), the default Thrift message size limit can be insufficient, causing TTransportException: Message size exceeds limit errors. Currently, the Thrift protocol configuration uses default max message size (100MB), preventing users from reading files with exceptionally large metadata footers.

There is similar fix for PyArrow: https://issues.apache.org/jira/browse/ARROW-16546

Exception while reading large footer message

Caused by: java.io.IOException: can not read FileMetaData: MaxMessageSize reached
	at org.apache.parquet.format.Util.readFileMetaData(Util.java:396)
	at org.apache.parquet.format.Util.readFileMetaData(Util.java:208)
	at org.apache.parquet.format.converter.ParquetMetadataConverter$3.visit(ParquetMetadataConverter.java:1581)
	at org.apache.parquet.format.converter.ParquetMetadataConverter$3.visit(ParquetMetadataConverter.java:1567)
	at org.apache.parquet.format.converter.ParquetMetadataConverter$SkipMetadataFilter.accept(ParquetMetadataConverter.java:1285)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:1567)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:629)
	at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:934)
	at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:925)
	at org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:710)
	at org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(ParquetFooterReader.java:85)
	at org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(ParquetFooterReader.java:76)
	at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.$anonfun$readParquetFootersInParallel$1(ParquetFileFormat.scala:451)
	at org.apache.spark.util.ThreadUtils$.$anonfun$parmap$2(ThreadUtils.scala:416)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:687)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
Caused by: shaded.parquet.org.apache.thrift.transport.TTransportException: MaxMessageSize reached
	at shaded.parquet.org.apache.thrift.transport.TEndpointTransport.checkReadBytesAvailable(TEndpointTransport.java:94)
	at shaded.parquet.org.apache.thrift.protocol.TCompactProtocol.checkStringReadLength(TCompactProtocol.java:726)
	at shaded.parquet.org.apache.thrift.protocol.TCompactProtocol.readString(TCompactProtocol.java:675)
	at org.apache.parquet.format.InterningProtocol.readString(InterningProtocol.java:269)
	at org.apache.parquet.format.KeyValue$KeyValueStandardScheme.read(KeyValue.java:415)
	at org.apache.parquet.format.KeyValue$KeyValueStandardScheme.read(KeyValue.java:392)
	at org.apache.parquet.format.KeyValue.read(KeyValue.java:327)
	at org.apache.parquet.format.event.TBaseStructConsumer.consumeStruct(Consumers.java:183)
	at org.apache.parquet.format.event.TypedConsumer$StructConsumer.read(TypedConsumer.java:141)
	at org.apache.parquet.format.event.TypedConsumer.read(TypedConsumer.java:247)
	at org.apache.parquet.format.event.DelegatingListElementsConsumer.consumeElement(Consumers.java:166)
	at org.apache.parquet.format.event.EventBasedThriftReader.readListContent(EventBasedThriftReader.java:125)
	at org.apache.parquet.format.event.TypedConsumer$ListConsumer.consumeList(TypedConsumer.java:167)
	at org.apache.parquet.format.event.TypedConsumer$ListConsumer.read(TypedConsumer.java:162)
	at org.apache.parquet.format.event.TypedConsumer.read(TypedConsumer.java:247)
	at org.apache.parquet.format.event.Consumers$DelegatingFieldConsumer.consumeField(Consumers.java:88)
	at org.apache.parquet.format.event.EventBasedThriftReader.readStructContent(EventBasedThriftReader.java:71)
	at org.apache.parquet.format.event.EventBasedThriftReader.readStruct(EventBasedThriftReader.java:54)
	at org.apache.parquet.format.Util.readFileMetaData(Util.java:394)
	... 21 more

Component(s)

Thrift