PHOENIX-6486 Phoenix uses inconsistent chronologies internally, break…

[richardantal]

…ing pre-Gregorian date handling

[stoty]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	20m 56s	master passed
+0	hbaserecompile	27m 26s	HBase recompiled.
+1 💚	compile	1m 1s	master passed
+1 💚	checkstyle	0m 59s	master passed
+1 💚	javadoc	0m 50s	master passed
+0 🆗	spotbugs	3m 7s	phoenix-core in master has 972 extant spotbugs warnings.
		_ Patch Compile Tests _
+1 💚	mvninstall	13m 39s	the patch passed
+0	hbaserecompile	23m 9s	HBase recompiled.
+1 💚	compile	1m 3s	the patch passed
+1 💚	javac	1m 3s	the patch passed
-1 ❌	checkstyle	1m 0s	phoenix-core: The patch generated 6 new + 144 unchanged - 19 fixed = 150 total (was 163)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 51s	the patch passed
+1 💚	spotbugs	3m 22s	the patch passed
		_ Other Tests _
-1 ❌	unit	112m 57s	phoenix-core in the patch failed.
+1 💚	asflicense	0m 45s	The patch does not generate ASF License warnings.
		177m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1289
Optional Tests	dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
uname	Linux 07c5ec37973d 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / bfc4621
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstyle	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unit	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/1/testReport/
Max. process+thread count	14094 (vs. ulimit of 30000)
modules	C: phoenix-core U: phoenix-core
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/1/console
versions	git=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[dbwong]

Question, likely we should follow what HBase/Hadoop does for storage/manipulation of temporal data. Does hbase use joda time internally or are they on only jdk, or another library?

[joshelser]

Does hbase use joda time internally or are they on only jdk, or another library?

HBase doesn't have any notion of time (in this sense). The timestamp on an HBase cell is just a number.

[dbwong]

If we don't have any wire dependencies then any reason we are preferring jodatime? I think long term we likely would want to consider something more robust for SQL temporal standards for interval queries (like oracle BETWEEN) or perhaps more common like jdk8/jsr-310. @joshelser@richardantal

[stoty]

If we don't have any wire dependencies then any reason we are preferring jodatime? I think long term we likely would want to consider something more robust for SQL temporal standards for interval queries (like oracle BETWEEN) or perhaps more common like jdk8/jsr-310. @joshelser@richardantal

Unfortunately we cannot use JSR310, because it's only available from JDK8, and we still have to support JDK7 for 4.x.

IMO the robustnes of Joda Time is not a problem, it's conceptually very similar to JSR310 (or rather the other way around), and actually has the same author.

[stoty]

@joshelser@dbwong
Also see the analysis in the ticket on choosing which Chronology to use internally,
and the cons and pros of the solution in this patch (using GJChronology internally)
i.e. we will have to do some extra work when we finally convert to JSR310.

[stoty]

@richardantal Can you add some tests to see how this works with new java.time APIs and the conversions methods from/to java.sql.* ?
(Obviously for master only)

[dbwong]

Mmy comment on robustness was not pointing toward jodatime or jsr being more robust than another but rather in consideration of other datetime libraries from a future feature point of view as I know many SQL users have wanted in the past from my experience implementing many date time functions in other large databases. Apologies if I missed any library discussion on the JIRA but I mostly saw a great job covering the underlying currently used libraries and the time representation we were using. Maybe these were library discussions, thus class names, when I was assuming you were only discussing the calendar system.
Note that joda time also wants people to migrate to js-310 and the implementer supports a maven backwards compatibility package for jdk7 though I haven't done full investigation on the performance.
One last question, can customer data could be "corrupt" based on how the data get entered into the table if from SQL constant vs function based path? Would this change break customer data if one path was used and not another?

[stoty]

@dbwong
I absolutely support migrating to JSR-310, as soon 4.x is out of the picture.
I got the impression that Java 7 JSR310 library is not really a production quality thing, but I may be wrong.
The JSR310 vs Java7 discussion happened in another JIRA earlier, not this one.

Yes, existing time data will be impacted.

AFACT we only use JodaTime (with ISO chronology) in the write path, and we always read with the java.util.Date classes, and the current approach uses the java.util.Date compatible chronology for both read and write.

However, some date/time functions do use Joda time internally, so the function output may be affected by the change, even if reading the values directly is not.

I wouldn't worry about it too mach, as only pre-gregorian dates are affected, and the current mess makes using those impractical anyway.

One issue with the approach in this patch is that we will need to re-implement GJChronology for the JSR310 migration, as it doesn't have that chronology by default.

[stoty]

To recap:

In this patch we use GJChronology internally, so

• upserting java.sql.Timestamp(10,10,10,10,10,10) will be read back as rs.getString() == "10-10-10 10:10:10" as a, but
• upserting java.sql.Timestamp.valueOf(java.time.LocalDateTme.of(10,10,10,10,10,10)) will be read back as rs.getString() == (10-10-12 10:10:10"

If we used ISOChronology internally instead , then

• upserting java.sql.Timestamp(10,10,10,10,10,10) would be read back as rs.getString() == "10-10-08 10:10:10", but
• upserting java.sql.Timestamp.valueOf(java.time.LocalDateTime.of(10,10,10,10,10,10)) will be read back as rs.getString() == "10-10-10 10:10:10"

(The above examples assume that the local TZ is UTC)

So we will will behave unexpectedly in some cases either way, no matter what we do, there will be surprises when using either the old java.util.Date , or the new java.time.API .

The current patch favors the old API.

[richardantal]

We had a private discussion about it with Istvan.
Thank you @stoty for the recap.

Added a test with java.time API to visualise some scenario.

Small correction about the current behaviour:

• upserting java.sql.Timestamp.valueOf(java.time.LocalDateTme.of(10,10,10,10,10,10)) will be read back as rs.getString() == "10-10-10 10:10:10"
• dayofmonth will return 10

but

• upserting new java.sql.Timestamp(java.time.Instant.parse("0010-10-10T10:10:10.111Z").toEpochMilli()) will be read back as rs.getString() == "10-10-12 10:10:10"
• dayofmonth will return 12

[stoty]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	6m 42s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	21m 54s	master passed
+0	hbaserecompile	28m 53s	HBase recompiled.
+1 💚	compile	1m 11s	master passed
+1 💚	checkstyle	1m 2s	master passed
+1 💚	javadoc	0m 54s	master passed
+0 🆗	spotbugs	3m 17s	phoenix-core in master has 972 extant spotbugs warnings.
		_ Patch Compile Tests _
+1 💚	mvninstall	14m 15s	the patch passed
+0	hbaserecompile	24m 11s	HBase recompiled.
+1 💚	compile	1m 5s	the patch passed
+1 💚	javac	1m 5s	the patch passed
-1 ❌	checkstyle	1m 1s	phoenix-core: The patch generated 6 new + 144 unchanged - 19 fixed = 150 total (was 163)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 52s	the patch passed
+1 💚	spotbugs	3m 24s	the patch passed
		_ Other Tests _
+1 💚	unit	124m 38s	phoenix-core in the patch passed.
+1 💚	asflicense	0m 23s	The patch does not generate ASF License warnings.
		195m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1289
Optional Tests	dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
uname	Linux 426233ea480c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / 9dfb423
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstyle	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/2/testReport/
Max. process+thread count	9132 (vs. ulimit of 30000)
modules	C: phoenix-core U: phoenix-core
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/2/console
versions	git=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[richardantal]

Thanks @dbwong and @stoty for the reviews. I squashed the commits to 1.
Will create a PR for 4.x to see test run and merge them together.

[stoty]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 19s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	20m 50s	master passed
+0	hbaserecompile	27m 28s	HBase recompiled.
+1 💚	compile	1m 2s	master passed
+1 💚	checkstyle	1m 0s	master passed
+1 💚	javadoc	0m 50s	master passed
+0 🆗	spotbugs	3m 4s	phoenix-core in master has 972 extant spotbugs warnings.
		_ Patch Compile Tests _
+1 💚	mvninstall	17m 28s	the patch passed
+0	hbaserecompile	27m 52s	HBase recompiled.
+1 💚	compile	1m 9s	the patch passed
+1 💚	javac	1m 9s	the patch passed
-1 ❌	checkstyle	1m 4s	phoenix-core: The patch generated 8 new + 142 unchanged - 21 fixed = 150 total (was 163)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 53s	the patch passed
+1 💚	spotbugs	3m 22s	the patch passed
		_ Other Tests _
-1 ❌	unit	116m 35s	phoenix-core in the patch failed.
+1 💚	asflicense	0m 43s	The patch does not generate ASF License warnings.
		186m 44s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1289
Optional Tests	dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
uname	Linux 4cc0ae40448c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / 9dfb423
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstyle	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/3/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unit	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/3/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/3/testReport/
Max. process+thread count	12045 (vs. ulimit of 30000)
modules	C: phoenix-core U: phoenix-core
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1289/3/console
versions	git=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.