I ran parquet-cli's convert-csv with an input file which name starts with a numeric character without --schema option and got the following error:
$java -cp'target/*:target/dependency/*'org.apache.parquet.cli.Mainconvert-csv0sample.csv -osample.parquetUnknownerrorshaded.parquet.org.apache.avro.SchemaParseException: Illegalinitialcharacter: 0sampleatshaded.parquet.org.apache.avro.Schema.validateName(Schema.java:1498)
atshaded.parquet.org.apache.avro.Schema.access$200(Schema.java:86)
atshaded.parquet.org.apache.avro.Schema$Name.<init>(Schema.java:645)
atshaded.parquet.org.apache.avro.Schema.createRecord(Schema.java:182)
atshaded.parquet.org.apache.avro.SchemaBuilder$RecordBuilder.fields(SchemaBuilder.java:1805)
atorg.apache.parquet.cli.csv.AvroCSV.inferSchemaInternal(AvroCSV.java:158)
atorg.apache.parquet.cli.csv.AvroCSV.inferNullableSchema(AvroCSV.java:78)
atorg.apache.parquet.cli.commands.ConvertCSVCommand.run(ConvertCSVCommand.java:160)
atorg.apache.parquet.cli.Main.run(Main.java:147)
atorg.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
atorg.apache.parquet.cli.Main.main(Main.java:177)
This is because that convert-csv uses the input file name as the name for the output schema, while Avro requires its schema name to match the regex pattern [A-Za-z_][A-Za-z0-9_]*.
So users have to change the input file name or use the --schema option explicitly, but it's not so obvious from the error message.
It'd be nice if the message were improved, or the schema name were automatically replaced with valid characters to avoid this problem.
Reporter: Kengo Seki / @sekikn
Assignee: Kengo Seki / @sekikn
PRs and other links:
Note: This issue was originally created as PARQUET-1598. Please see the migration documentation for further details.
I ran parquet-cli's
convert-csvwith an input file which name starts with a numeric character without--schemaoption and got the following error:This is because that
convert-csvuses the input file name as the name for the output schema, while Avro requires its schema name to match the regex pattern[A-Za-z_][A-Za-z0-9_]*.So users have to change the input file name or use the
--schemaoption explicitly, but it's not so obvious from the error message.It'd be nice if the message were improved, or the schema name were automatically replaced with valid characters to avoid this problem.
Reporter: Kengo Seki / @sekikn
Assignee: Kengo Seki / @sekikn
PRs and other links:
Note: This issue was originally created as PARQUET-1598. Please see the migration documentation for further details.