TestParquetReader fails with hadoop 3.4.2

[annimesh2809]

Describe the bug, including details regarding any error messages, version, and platform.

A couple of test cases of TestParquetReader suite started failing with errors like:

- testRangeFiltering[0] *** FAILED ***
 org.apache.parquet.bytes.TrackingByteBufferAllocator$LeakedByteBufferException: 24 ByteBuffer object(s) is/are remained unreleased after closing this allocator.
 at org.apache.parquet.bytes.TrackingByteBufferAllocator.close(TrackingByteBufferAllocator.java:161)
 at org.apache.parquet.hadoop.TestParquetReader.closeAllocator(TestParquetReader.java:175)
 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:569)

with hadoop 3.4.2 and parquet-mr 1.16.0
The leaks are happening when reading using vectored IO because we never pass the buffers to the releaser.

Component(s)

No response

[steveloughran]

problem with the vector IO buffer merging and the checksum blocks of local fs.

See HADOOP-18296: setting fs.file.checksum.verify to false will stop this by handing down to the raw local code. (bonus benefit: vector reads handoff to java nio scatter/gather and you get maximum perfomance).

1. parquet needs a PR for this
2. someone who can understand the relevant bits of ChecksumFileSystem could see if it could be clever here and return the original buffers

This doesn't actually surface in production because the ranges of parquet/ORC files is too big, though I think we've also been considering removing range merging from the s3a code. The issue there is correctly handling partial failures.

[steveloughran]

Let me fix this and remind me if I don't submit a PR: there's an internal PR we have which addresses this.

[Fokko]

@steveloughran I noticed this issue after creating the PR, WDYT of #3559?

[steveloughran]

I actually do want to sit down with claude and get the fundamental issue fixed, which is ChecksumFileSystem.readVectored

for 3.4.2+ you can actually turn off checksum validation here

fs.file.checksum.verify = false

see https://issues.apache.org/jira/browse/HADOOP-18296

This is tangibly faster for vector iO, as all requests should be mapped to jni scatter gather. There's no checksumming for any file:// reads, but if you trust your SSD you are safe, at least for unit tests.

we do this for our tests