Filtering records with timestamp doesn't work in IcebergGenerics

[vanliu-tx]

Filtering records with timestamp field by IcebergGenerics leading to IllegalStateException.

In this test I got a iceberg table schema and partition below:

Schema schema = new Schema(
        required(1, "ts", Types.TimestampType.withZone()),
        required(2, "bank_balance", Types.LongType.get()),
        required(3, "salary", Types.IntegerType.get()),
        optional(4, "info", Types.StringType.get()));

PartitionSpec partitionSpec = PartitionSpec.builderFor(schema)
        .day("ts")
        .bucket("bank_balance", 2)
        .build();

Some generated records are committed to this table, then use IcebergGenerics to read this table with filter condition on "ts" and "bank_balance" field. The strangest thing is that, greaterThan("ts", "2020-01-01T23:59:59.999999-00:00") works while greaterThan("ts", "2020-01-01T11:59:59.999999-00:00") fails. Please run the attached demo code to take a look.

I think the first filter condition just filter the data files by partition info, and not filtering each records as records in the data file all matching this condition. But the second one need to filtering each records in data file which partition info is "ts_day=2020-01-01".

Here is the exception stack:

Exception in thread "main" java.lang.IllegalStateException: Not an instance of java.lang.Long: 2020-01-01T06:07:57.849863Z
at org.apache.iceberg.data.GenericRecord.get(GenericRecord.java:123)
at org.apache.iceberg.Accessors$PositionAccessor.get(Accessors.java:57)
at org.apache.iceberg.Accessors$PositionAccessor.get(Accessors.java:44)
at org.apache.iceberg.expressions.BoundReference.get(BoundReference.java:49)
at org.apache.iceberg.expressions.Evaluator$EvalVisitor.gt(Evaluator.java:116)
at org.apache.iceberg.expressions.Evaluator$EvalVisitor.gt(Evaluator.java:58)
at org.apache.iceberg.expressions.ExpressionVisitors$BoundExpressionVisitor.predicate(ExpressionVisitors.java:116)
at org.apache.iceberg.expressions.ExpressionVisitors.visitEvaluator(ExpressionVisitors.java:195)
at org.apache.iceberg.expressions.ExpressionVisitors.visitEvaluator(ExpressionVisitors.java:210)
at org.apache.iceberg.expressions.Evaluator$EvalVisitor.eval(Evaluator.java:63)
at org.apache.iceberg.expressions.Evaluator$EvalVisitor.access$000(Evaluator.java:58)
at org.apache.iceberg.expressions.Evaluator.eval(Evaluator.java:55)
at com.google.common.collect.Iterators$5.computeNext(Iterators.java:637)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
at org.apache.iceberg.data.TableScanIterable$ScanIterator.hasNext(TableScanIterable.java:129)
at java.lang.Iterable.forEach(Iterable.java:74)
at RecordFilterIssue.main(RecordFilterIssue.java:182)

Here is the test code:
RecordFilterIssue.java.txt

[rdblue]

Thanks for reporting this, @vanliu-tx! Looks like the accessors used in generics should be converting date and timestamp values before returning them to the evaluator.

[chenjunjiedada]

I will fix this.

[chenjunjiedada]

@vanliu-tx , I reproduced your issue and fixed locally. This PR: #983 should fix it, could you please have a try with it?

[vanliu-tx]

@chenjunjiedada the fix works, thank you!