Uh oh!
There was an error while loading. Please reload this page.
[BEAM-7554] Add MillisInstant logical type to replace DATETIME - #11456
[BEAM-7554] Add MillisInstant logical type to replace DATETIME #11456TheNeuralBit wants to merge 22 commits into
Conversation
Ohterwise the switch to datetime as a logical type fails to restore the timezone and fails assertions.
TheNeuralBit
commented
Apr 18, 2020
Run SQL PostCommit |
9e589b8 to
2f59493Compare6f6fa60 to
633710aCompare| FLOAT, | ||
| DOUBLE, | ||
| STRING, // String. | ||
| DATETIME, // Date and time. |
There was a problem hiding this comment.
I'm a little worried about this. Empirically many users are using schemas. Maybe we should start off by leaving DATETIME around and remove it later in another PR?
There was a problem hiding this comment.
Yeah we could deprecate it for a release or two before removing. Would we also need to keep support for the primitive DATETIME in IOs and SQL? I'd need to think about how that transition would work.
| : combined.aggregateFieldBaseValue( | ||
| inputs.get(0), combineFn, fieldAggregation.outputField); | ||
| ? byFields.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField) | ||
| : combined.aggregateField(inputs.get(0), combineFn, fieldAggregation.outputField); |
There was a problem hiding this comment.
I'm not sure I agree with this change. I think it's important that SQL work over user logical types by interpreting it as the base value. The user writing the SQL statement usually understands the base type of their logical type, and can write the SQL statement appropriately. This will break that.
There was a problem hiding this comment.
What about adding the option for SqlTransform to convert unknown logical types to their base type at the input? I think that behavior would be effectively the same
| Expressions.constant(index), | ||
| Expressions.constant(convertTo)), | ||
| convertTo); | ||
| Expressions.call(expression, "getValue", Expressions.constant(index)), convertTo); |
There was a problem hiding this comment.
I have the same concern with this change.
| /** A timestamp represented as milliseconds since the epoch. */ | ||
| public class MillisInstant extends MillisType<ReadableInstant> { | ||
| public static final String IDENTIFIER = "beam:logical_type:millis_instant:v1"; |
There was a problem hiding this comment.
@reuvenlax what do you think about making this type (and maybe NanosInstant as well) parameterized by timezone?
The arrow approach seems useful: un-specified timezone indicates time-zone naive (e.g. joda time Instant), otherwise time zone parameter should reference a value in tzdata (and would map to joda time DateTime).
cc: @alexvanboxel
There was a problem hiding this comment.
A timestamp type seems like it's by definition time-zone agnostic. If we want a datetime type, that should be a different type.
robinyqiu
commented
Jul 7, 2020
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions. |
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
This PR adds a
MillisInstantlogical type, and replaces all usages of the primitiveDATETIMEtype with it.MillisInstantmirrors theNanosInstanttype. It usesorg.joda.time.Instantrather thanjava.time.Instantfor consistency withDATETIME, and it is backed by anINT64representing the number of milliseconds since the epoch.The majority of the changes in this PR are relatively trivial, but there are a few significant (and some potentially controversial) ones:
ByteBuddyUtilsto produce and consume instances ofLongrather thanInstant. This is necessary because these functions are expected to produce/consume the base type for logical types.getBaseValuewithgetValue. Previously the only logical types used in SQL werePassThroughLogicalTypeinstances, where the input type and base type were the same.Changes that are not currently included in this PR but we might consider:
DateTime(addDateTimeField,addDateTimeValue, ..) withInstant, and deprecate or remove theDateTimereferences.Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.