Skip to content

Support datetime interval extended type and arithmetic - #230

Merged
DifferentialOrange merged 2 commits into
masterfrom
DifferentialOrange/gh-229-interval
Oct 5, 2022
Merged

Support datetime interval extended type and arithmetic#230
DifferentialOrange merged 2 commits into
masterfrom
DifferentialOrange/gh-229-interval

Conversation

@DifferentialOrange

@DifferentialOrangeDifferentialOrange commented Sep 9, 2022

Copy link
Copy Markdown
Member

msgpack: support datetime interval extended type

Tarantool supports datetime interval type since version 2.10.0 [1]. This patch introduced the support of Tarantool interval type in msgpack decoders and encoders.

Tarantool datetime interval objects are decoded to tarantool.Interval type. tarantool.Interval may be encoded to Tarantool interval objects.

You can create tarantool.Interval objects either from msgpack data or by using the same API as in Tarantool:

di=tarantool.Interval(year=-1, month=2, day=3,
hour=4, minute=-5, sec=6,
nsec=308543321,
adjust=tarantool.IntervalAdjust.NONE)

Its attributes (same as in init API) are exposed, so you can use them if needed.

datetime, numpy and pandas tools doesn't seem to be sufficient to cover all adjust cases supported by Tarantool.

This patch does not yet introduce the support of datetime interval arithmetic.

  1. Introduce TIMESTAMP type to Tarantool tarantool#5941

Part of #229

msgpack: support datetime interval arithmetic

Support datetime and interval arithmetic with the same rules as in Tarantool [1].

Valid operations:

  • tarantool.Datetime + tarantool.Interval = tarantool.Datetime
  • tarantool.Datetime - tarantool.Interval = tarantool.Datetime
  • tarantool.Datetime - tarantool.Datetime = tarantool.Interval
  • tarantool.Interval + tarantool.Interval = tarantool.Interval
  • tarantool.Interval - tarantool.Interval = tarantool.Interval

Since tarantool.Interval could contain month and year fields and such operations could be ambiguous, you can use adjust field to tune the logic.

  • tarantool.IntervalAdjust.NONE -- only truncation toward the end of month performed (default mode).

    >>> dt = tarantool.Datetime(year=2022, month=3, day=31)
    datetime: Timestamp('2022-03-31 00:00:00'), tz: ""
    >>> di = tarantool.Interval(month=1, adjust=tarantool.IntervalAdjust.NONE)
    >>> dt + di
    datetime: Timestamp('2022-04-30 00:00:00'), tz: ""
    
  • tarantool.IntervalAdjust.EXCESS -- overflow mode, without any snap or truncation to the end of month, straight addition of days in month, stopping over month boundaries if there is less number of days.

    >>> dt = tarantool.Datetime(year=2022, month=1, day=31)
    datetime: Timestamp('2022-01-31 00:00:00'), tz: ""
    >>> di = tarantool.Interval(month=1, adjust=tarantool.IntervalAdjust.EXCESS)
    >>> dt + di
    datetime: Timestamp('2022-03-02 00:00:00'), tz: ""
    
  • tarantool.IntervalAdjust.LAST -- mode when day snaps to the end of month, if happens.

    >>> dt = tarantool.Datetime(year=2022, month=2, day=28)
    datetime: Timestamp('2022-02-28 00:00:00'), tz: ""
    >>> di = tarantool.Interval(month=1, adjust=tarantool.IntervalAdjust.LAST)
    >>> dt + di
    datetime: Timestamp('2022-03-31 00:00:00'), tz: ""
    

Tarantool does not yet correctly support subtraction of datetime objects with different timezones [2] and addition of intervals to datetimes with non-fixed offset timezones [3]. tarantool-python implementation support them, but it could be reworked later if core team choose another solution.

  1. https://github.com/tarantool/tarantool/wiki/Datetime-Internals#interval-arithmetic
  2. Datetime with timezone subtraction tarantool#7698
  3. Interval ariphmetic with offset change tarantool#7700

Closes#229

@DifferentialOrange
DifferentialOrangeforce-pushed the DifferentialOrange/gh-229-interval branch 2 times, most recently from 5e331d7 to 406b2efCompareSeptember 9, 2022 11:48
@DifferentialOrangeDifferentialOrange changed the title msgpack: support datetime interval extended type and arithmeticSupport datetime interval extended type and arithmeticSep 9, 2022
@DifferentialOrange
DifferentialOrange changed the base branch from master to DifferentialOrange/gh-204-datetimeSeptember 9, 2022 11:53
@DifferentialOrange
DifferentialOrangeforce-pushed the DifferentialOrange/gh-229-interval branch from 406b2ef to f5deabeCompareSeptember 9, 2022 15:59
@DifferentialOrange
DifferentialOrangeforce-pushed the DifferentialOrange/gh-204-datetime branch 9 times, most recently from d4e0aed to 47d9864CompareSeptember 19, 2022 15:32
@DifferentialOrange
DifferentialOrangeforce-pushed the DifferentialOrange/gh-204-datetime branch from 47d9864 to e71a4acCompareSeptember 20, 2022 11:25
@DifferentialOrange
DifferentialOrangeforce-pushed the DifferentialOrange/gh-229-interval branch 3 times, most recently from e8b1be0 to bd9aebaCompareSeptember 21, 2022 13:46
@DifferentialOrange
DifferentialOrangeforce-pushed the DifferentialOrange/gh-204-datetime branch from e71a4ac to 8c92b86CompareSeptember 22, 2022 07:48
@DifferentialOrange
DifferentialOrangeforce-pushed the DifferentialOrange/gh-229-interval branch from bd9aeba to c629676CompareSeptember 22, 2022 15:31
@DifferentialOrange
DifferentialOrangeforce-pushed the DifferentialOrange/gh-204-datetime branch from 8c92b86 to aa7302aCompareSeptember 26, 2022 06:52
Base automatically changed from DifferentialOrange/gh-204-datetime to masterSeptember 26, 2022 07:01
@DifferentialOrange
DifferentialOrangeforce-pushed the DifferentialOrange/gh-229-interval branch from c629676 to afa8107CompareSeptember 26, 2022 07:04
@DifferentialOrange
DifferentialOrange marked this pull request as ready for review September 26, 2022 07:07
@DifferentialOrange
DifferentialOrangeforce-pushed the DifferentialOrange/gh-229-interval branch from afa8107 to d4c577cCompareSeptember 26, 2022 07:10
@oleg-jukovec
oleg-jukovec self-requested a review September 29, 2022 08:54

@oleg-jukovecoleg-jukovec left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the patch! I don't see a critical problems, just few issues.

Comment threadCHANGELOG.md
Comment threadtarantool/msgpack_ext/types/datetime.py
Comment threadtarantool/msgpack_ext/types/interval.py
Comment on lines +209 to +215
@skip_or_run_datetime_test
@unittest.expectedFailure # See https://github.com/tarantool/tarantool/issues/7698
def test_tarantool_datetime_sub_different_timezones(self):
case = self.datetime_sub_different_timezones_case

self.assertSequenceEqual(self.con.call('sub', case['arg_1'], case['arg_2']),
[case['res']])

@oleg-jukovecoleg-jukovecSep 29, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wil the test fail on a new versions of Tarantool with fixes? It seems better to skip the test.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it will. I think it's better to wait until the bug would be fixed and then replace xfail with conditional xfail based on version rather than set a skip and forget to ever run the test.

@oleg-jukovecoleg-jukovecSep 29, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should run the test only on CI? (by a environment variable or an another way that unittest support)

On the one hand, we'll see when our tests break. On the other hand, a user will not be confused with a failed test after updating Tarantool.

It looks a bit tricky, but for me it seems better than failed by default tests in the future.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the other hand, a user will not be confused with a failed test after updating Tarantool.

I think that when one should get this fail, it would be the moment we need to replace xfail with conditional xfail based on version.

@oleg-jukovecoleg-jukovecOct 3, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need to add a task to a sprint, fix it, and then release a new version... It may take a long time for users to get a working version.

It's not a good idea to make CI easier for us in the moment, but make life harder for users in the future.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get how choosing between xfail and skip affects user with release versions.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are running module tests, you are a developer. As a developer, you may either ignore a couple of failing tests after you find out what's the reason (it shouldn't be complicated since relevant issue link is provided), or report the issue to us (to be honest, I'm sure that developer would already be one of us), or fix it (it's an hour long work together with all required stuff like commit messages and PR).

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would be glad to provide skip tests with conditions, but these conditions not yet exist. Make a test that always will skip until one remembers to "uncomment it" is also doesn't seems like a good approach to me.

Comment threadtest/suites/test_datetime_arithmetic.py Outdated
Comment threadtest/suites/test_datetime_arithmetic.py Outdated
Comment threadtest/suites/test_datetime_arithmetic.py Outdated
Comment threadtest/suites/test_datetime_arithmetic.py Outdated
Comment threadtest/suites/test_interval.py
Comment threadtest/suites/test_interval.py
@DifferentialOrange
DifferentialOrangeforce-pushed the DifferentialOrange/gh-229-interval branch 2 times, most recently from e3525ef to fb9449bCompareSeptember 30, 2022 08:26
@DifferentialOrange
DifferentialOrangeforce-pushed the DifferentialOrange/gh-229-interval branch 5 times, most recently from 9efe2f2 to ac7a1e3CompareOctober 3, 2022 08:57

@oleg-jukovecoleg-jukovec left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Tarantool supports datetime interval type since version 2.10.0 [1].
This patch introduced the support of Tarantool interval type in
msgpack decoders and encoders.
Tarantool datetime interval objects are decoded to `tarantool.Interval`
type. `tarantool.Interval` may be encoded to Tarantool interval
objects.
You can create `tarantool.Interval` objects either from msgpack
data or by using the same API as in Tarantool:
```
di = tarantool.Interval(year=-1, month=2, day=3,
hour=4, minute=-5, sec=6,
nsec=308543321,
adjust=tarantool.IntervalAdjust.NONE)
```
Its attributes (same as in init API) are exposed, so you can
use them if needed.
datetime, numpy and pandas tools doesn't seem to be sufficient to
cover all adjust cases supported by Tarantool.
This patch does not yet introduce the support of datetime interval
arithmetic.
1. tarantool/tarantool#5941
Part of #229
Support datetime and interval arithmetic with the same rules as in
Tarantool [1].
Valid operations:
- `tarantool.Datetime` + `tarantool.Interval` = `tarantool.Datetime`
- `tarantool.Datetime` - `tarantool.Interval` = `tarantool.Datetime`
- `tarantool.Datetime` - `tarantool.Datetime` = `tarantool.Interval`
- `tarantool.Interval` + `tarantool.Interval` = `tarantool.Interval`
- `tarantool.Interval` - `tarantool.Interval` = `tarantool.Interval`
Since `tarantool.Interval` could contain `month` and `year` fields
and such operations could be ambiguous, you can use `adjust` field
to tune the logic. The behavior is the same as in Tarantool, see
Interval arithmetic RFC [1].
- `tarantool.IntervalAdjust.NONE` -- only truncation toward the end of
month performed (default mode).
```
>>> dt = tarantool.Datetime(year=2022, month=3, day=31)
datetime: Timestamp('2022-03-31 00:00:00'), tz: ""
>>> di = tarantool.Interval(month=1, adjust=tarantool.IntervalAdjust.NONE)
>>> dt + di
datetime: Timestamp('2022-04-30 00:00:00'), tz: ""
```
- `tarantool.IntervalAdjust.EXCESS` -- overflow mode, without any snap
or truncation to the end of month, straight addition of days in month,
stopping over month boundaries if there is less number of days.
```
>>> dt = tarantool.Datetime(year=2022, month=1, day=31)
datetime: Timestamp('2022-01-31 00:00:00'), tz: ""
>>> di = tarantool.Interval(month=1, adjust=tarantool.IntervalAdjust.EXCESS)
>>> dt + di
datetime: Timestamp('2022-03-02 00:00:00'), tz: ""
```
- `tarantool.IntervalAdjust.LAST` -- mode when day snaps to the end of
month, if happens.
```
>>> dt = tarantool.Datetime(year=2022, month=2, day=28)
datetime: Timestamp('2022-02-28 00:00:00'), tz: ""
>>> di = tarantool.Interval(month=1, adjust=tarantool.IntervalAdjust.LAST)
>>> dt + di
datetime: Timestamp('2022-03-31 00:00:00'), tz: ""
```
Tarantool does not yet correctly support subtraction of datetime objects
with different timezones [2] and addition of intervals to datetimes with
non-fixed offset timezones [3]. tarantool-python implementation support
them, but it could be reworked later if core team choose another
solution.
1. https://github.com/tarantool/tarantool/wiki/Datetime-Internals#interval-arithmetic
2. tarantool/tarantool#7698
3. tarantool/tarantool#7700Closes#229
@DifferentialOrange
DifferentialOrangeforce-pushed the DifferentialOrange/gh-229-interval branch from ac7a1e3 to b121ac8CompareOctober 4, 2022 09:36
@DifferentialOrange
DifferentialOrange merged commit 14f1c77 into masterOct 5, 2022
@DifferentialOrange
DifferentialOrange deleted the DifferentialOrange/gh-229-interval branch October 5, 2022 09:16
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support interval type

2 participants

@DifferentialOrange@oleg-jukovec