Uh oh!
There was an error while loading. Please reload this page.
Parquet: Support constant map for partition values - #909
Conversation
| protected abstract void set(S struct, int pos, Object value); | ||
| protected Object prepareConstant(Type type, Object value) { |
There was a problem hiding this comment.
I'm moving this out of Avro and adding a callback to convert the constants to PartitionUtil.constantsMap. That way, Spark can supply a conversion function and use it in both places, instead of duplicating the conversion in Avro and Parquet readers.
6965f40 to
6c5db2dCompare| } | ||
| @Override | ||
| protected Object prepareConstant(Type type, Object value) { |
3e102e6 to
db7a4c7Compare
rdsr
left a comment
There was a problem hiding this comment.
Minor comment. Otherwise LGTM!
| appender.addAll(expected); | ||
| } | ||
| // add the Avro data file to the source table |
There was a problem hiding this comment.
Why not write the data for the parameterized format for which the test is running?
There was a problem hiding this comment.
This is just source data for the write from Spark with the target format. Since it isn't part of the test, we don't want it to change at all in ways that might affect the test.
rdblue
commented
Apr 11, 2020
Thanks for reviewing @rdsr! |
This is a follow-up to apache#896, which added the same constant map support for Avro. Fixes#575 for Parquet and replaces #585. Andrei did a lot of the work for this in #585. Co-authored-by: Andrei Ionescu <webdev.andrei@gmail.com>
This is a follow-up to #896, which added the same constant map support for Avro.
Fixes#575 for Parquet and replaces #585. Andrei did a lot of the work for this in #585.
Co-authored-by: Andrei Ionescu webdev.andrei@gmail.com