ARROW-18420: [Parquet] Add fixed_length_byte_array.parquet for page index test

[wgtmac]

This patch adds a parquet file generated by parquet-mr with following attributes:

• a single column with fixed_length_byte_array type of size 4.
• values are ordered in the descending order.
• in total 1000 values written into 10 pages, with some random null values.
• page index is generated.

After this file has been committed, I can go ahead to finish test cases required for page index in the apache/arrow#14803

Below is the complete java code to generate the file:

packageorg.apache.parquet.cli.commands;
importorg.apache.hadoop.conf.Configuration;
importorg.apache.hadoop.fs.Path;
importorg.apache.parquet.column.ParquetProperties;
importorg.apache.parquet.example.data.Group;
importorg.apache.parquet.example.data.simple.SimpleGroupFactory;
importorg.apache.parquet.hadoop.ParquetWriter;
importorg.apache.parquet.hadoop.example.GroupWriteSupport;
importorg.apache.parquet.hadoop.metadata.CompressionCodecName;
importorg.apache.parquet.io.api.Binary;
importorg.apache.parquet.schema.MessageType;
importorg.apache.parquet.schema.PrimitiveType;
importorg.apache.parquet.schema.Types;
importjava.io.IOException;
importjava.nio.ByteBuffer;
importjava.util.Random;
publicclassGenerateTestFile {
publicstaticvoidmain(String[] args) {
Pathpath = newPath("/tmp/fixed_length_byte_array.parquet");
Configurationconf = newConfiguration();
MessageTypeschema = Types.buildMessage()
.required(PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY).length(4)
.named("flba_field")
.named("schema");
SimpleGroupFactoryfact = newSimpleGroupFactory(schema);
GroupWriteSupport.setSchema(schema, conf);
try (
ParquetWriter<Group> writer = newParquetWriter<>(
path,
newGroupWriteSupport(),
CompressionCodecName.UNCOMPRESSED,
/*blockSize=*/1024 * 1024,
/*pageSize=*/128,
/*dictionaryPageSize=*/128,
/*enableDictionary=*/false,
/*validating=*/false,
ParquetProperties.WriterVersion.PARQUET_1_0,
conf)) {
Randomrnd = newRandom();
for (inti = 1000; i > 0; --i) {
if (rnd.nextInt(10) == 5) {
writer.write(fact.newGroup());
} else {
ByteBufferbuffer = ByteBuffer.allocate(Integer.BYTES);
writer.write(fact.newGroup()
.append("flba_field",
Binary.fromConstantByteArray(buffer.putInt(0, i).array())));
}
}
} catch (IOExceptione) {
thrownewRuntimeException(e);
}
}
}

[wgtmac]

@pitrou@emkornfield Can you please take a look? Thanks!

[pitrou]

Nice, thank you!

[ccleva]

@wgtmac@pitrou sorry for coming back on this. The generated file has null values in a required field. This causes an EOFException when one tries to read it using the hadoop parquet reader and the equivalent error with pyArrow.
apache/parquet-java#3336
apache/arrow#47662
Is it something that should be fixed? Making the field optional solves the issue with both readers.

[pitrou]

Is it something that should be fixed? Making the field optional solves the issue with both readers.

Oh, yes, definitely, thanks for spotting this. Do you want to submit a PR fixing the file?

[ccleva]

Sure, I will do that