Correct fixed_length_byte_array.parquet

[ccleva]

Follow-up of the discussion in #31.

The initial fixed_length_byte_array.parquet file contains null values in a required field. The issue was identified in apache/arrow#47662.

This PR replaces the initial file with a new one, generated with the same parquet-java version, where the field is optional. Page min/max values and null count are kept the same than in the initial file to facilitate compatibility with existing tests.

Tests using this file:
https://github.com/apache/arrow/blob/27513af4eaeb5fe22cfcb13206ff23a7ca6b52b1/cpp/src/parquet/reader_test.cc#L1203
https://github.com/apache/arrow/blob/27513af4eaeb5fe22cfcb13206ff23a7ca6b52b1/cpp/src/parquet/page_index_test.cc#L242
https://github.com/apache/arrow-go/blob/ccb6818822e7a86d11c9ec4a9f42505546838d2a/parquet/metadata/page_index_test.go#L213

The initial file is moved into the bad_data folder and renamed.

Modified script used to generate the new file:

packageexamples;
importorg.apache.hadoop.conf.Configuration;
importorg.apache.hadoop.fs.Path;
importorg.apache.parquet.column.ParquetProperties;
importorg.apache.parquet.example.data.Group;
importorg.apache.parquet.example.data.simple.SimpleGroupFactory;
importorg.apache.parquet.hadoop.ParquetWriter;
importorg.apache.parquet.hadoop.example.GroupWriteSupport;
importorg.apache.parquet.hadoop.metadata.CompressionCodecName;
importorg.apache.parquet.io.api.Binary;
importorg.apache.parquet.schema.MessageType;
importorg.apache.parquet.schema.PrimitiveType;
importorg.apache.parquet.schema.Types;
importjava.io.IOException;
importjava.nio.ByteBuffer;
publicclassGenerateTestFile {
publicstaticvoidmain(String[] args) {
Pathpath = newPath("/tmp/fixed_length_byte_array.parquet");
Configurationconf = newConfiguration();
MessageTypeschema = Types.buildMessage()
.optional(PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY).length(4)
.named("flba_field")
.named("schema");
SimpleGroupFactoryfact = newSimpleGroupFactory(schema);
GroupWriteSupport.setSchema(schema, conf);
int[] nulls_per_page = {9, 9, 19, 10, 13, 11, 11, 8, 9, 6};
try (
@SuppressWarnings("deprecation")
ParquetWriter<Group> writer = newParquetWriter<>(
path,
newGroupWriteSupport(),
CompressionCodecName.UNCOMPRESSED,
/*blockSize=*/1024 * 1024,
/*pageSize=*/128,
/*dictionaryPageSize=*/128,
/*enableDictionary=*/false,
/*validating=*/true,
ParquetProperties.WriterVersion.PARQUET_1_0,
conf)) {
for (inti = 1000; i > 0; --i) {
if (i % 100 == 99) {
// Add the required number of null values after the first page elementintnb_null = nulls_per_page[9 - i / 100];
for (intj = 0; j < nb_null; j++) {
writer.write(fact.newGroup());
}
i = i - nb_null + 1;
} else {
ByteBufferbuffer = ByteBuffer.allocate(Integer.BYTES);
writer.write(fact.newGroup()
.append("flba_field",
Binary.fromConstantByteArray(buffer.putInt(0, i).array())));
}
}
} catch (IOExceptione) {
thrownewRuntimeException(e);
}
}
}

[ccleva]

@pitrou@wgtmac if you could have a look
I am not setup to run the C++ or go tests, if you could give it a try that would be great, thanks!

[wgtmac]

Just curious why the new file (102 B) is way smaller than the old one (4.23 KB)?

[wgtmac]

Do we want to mention that this file was supposed to serve the purpose of data/fixed_length_byte_array.parquet before this bug has been spotted?

[ccleva]

Thank you for the review!

The new file is 4.33 KB, 102 B larger than the old file. It seems github shows the delta, not the actual size.

Do you want me to add something like "Incorrect version of data/fixed_length_byte_array.parquet" to the description?

[wgtmac]

Yes, that would be helpful! Thanks for the explanation.

[wgtmac]

Thanks @ccleva for fixing this!