Uh oh!
There was an error while loading. Please reload this page.
Core, Kafka Connect: Support additional Avro temporal logical types - #16681
Core, Kafka Connect: Support additional Avro temporal logical types#16681JustMaris wants to merge 4 commits into
Conversation
f297f72 to
4bbb68dCompareMap time-millis and local-timestamp-{millis,micros,nanos} when reading Avro into Iceberg. These previously threw "Unknown logical type" in the readers, and local-timestamp types were unrecognized in schema conversion. time-millis is read as an int and scaled to microseconds; local-timestamp-* always map to without-zone timestamp types. Write path is unchanged.
Signed-off-by: Maris Popens <maris@popens.lv>Recognize the Connect schema names the AvroConverter assigns to sub-millisecond and zone-less Avro temporal types (timestamp-micros, timestamp-nanos, local-timestamp-{millis,micros,nanos}, time-micros) and map them to the correct Iceberg types. Thread the Connect source schema through value conversion so numeric temporal values are scaled by their actual unit instead of assuming milliseconds, and add TIMESTAMP_NANO support. Requires the AvroConverter to emit these as named int64.
Signed-off-by: Maris Popens <maris@popens.lv>4bbb68d to
84d522cCompare…mporal-logical-types Signed-off-by: Maris Popens <maris@popens.lv> # Conflicts: # kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
JustMaris
commented
Jul 23, 2026
Still relevant and ready for review. This adds read support for the Avro time-millis and local-timestamp / micros / nanos logical types in core and the Kafka Connect sink; CI is green, and the branch is up to date with main. Could a committer take a look? Happy to address any feedback. |
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
JustMaris
commented
Aug 25, 2026
Still relevant and ready for review. This adds read support for the Avro time-millis and local-timestamp / micros / nanos logical types in core and the Kafka Connect sink; CI is green, and the branch is up to date with main. Could a committer take a look? Happy to address any feedback. |
…mporal-logical-types Resolves conflicts from upstream's removal of the deprecated DataReader in favor of PlannedDataReader (apache#17699): - core/.../data/avro/DataReader.java and its test were deleted upstream; accepted the deletion since PlannedDataReader (which already carries this PR's temporal logical type support) supersedes it. - kafka-connect SchemaUtils.java: kept both the new AVRO_* schema-name constants added here and the unrelated MAX_DECIMAL_PRECISION constant added upstream.
What & why
Adds support for Avro temporal logical types that Iceberg does not handle today, across two
independent code paths. External producers (e.g. Kafka/Avro Java services via the Confluent
AvroConverter) commonly emittime-millis,time-micros, and thelocal-timestamp-*/timestamp-{micros,nanos}logical types. Currently the Avro readers throwIllegalArgumentException: Unknown logical typefortime-millisand alllocal-timestamp-*,and the Kafka Connect sink only recognizes Connect's millis-based
Timestamp/Time, soeverything else lands as a plain
long, silently losing temporal semantics.Core: Avro read path
SchemaToTypeand the four readers (GenericAvroReader,InternalReader,DataReader,PlannedDataReader) now handle:time-millis(int)time(scaled ms → µs)local-timestamp-millistimestamp(without zone)local-timestamp-microstimestamp(without zone)local-timestamp-nanostimestamp_ns(without zone)local-timestamp-*are zone-less by definition, so they always map to without-zone types. Thewrite path is unchanged — Iceberg still emits
timestamp-micros/timestamp-nanoswithadjust-to-utc; this only adds the ability to read externally-produced Avro.Kafka Connect: sink
The
AvroConverterpasses sub-millisecond and zone-less Avro types through as rawint64whoseunit and zone are encoded only in the Connect schema name.
SchemaUtilsnow maps those names andRecordConverterrecovers the unit so values are scaled correctly:timestamp-microstimestamptztimestamp-nanostimestamptz_nslocal-timestamp-millis/local-timestamp-microstimestamplocal-timestamp-nanostimestamp_nstime-microstimenumeric temporals are scaled by their actual unit instead of the previous hard-coded milliseconds.
TIMESTAMP_NANOconversion case.interpreted as milliseconds, exactly as before; existing
protectedconverter methods are keptas millis-defaulting wrappers.
Timestamp→ with-zone): Avro instant types(
timestamp-*) → with-zone;local-timestamp-*→ without-zone.Note: the
*-nanostypes map totimestamp_ns/timestamptz_ns, which require table format v3+.Testing
(
DataReader/PlannedDataReader, incl. pre/post-epoch) and the raw-long readers(
GenericAvroReader/InternalReader). Since Iceberg's write path never emits these types, thetests build Avro schemas with the logical types directly and decode encoded values.
SchemaUtilsmapping tests and end-to-endRecordConvertertests (structfields and nested list/map), covering with/without zone and the millis/micros/nanos units.