Refactor DateLiteral class in FE by HangyuanLiu · Pull Request #1644 · apache/doris

HangyuanLiu · 2019-08-15T02:36:16Z

1、Add fe time zone function support
2、Refactor DateLiteral class in FE
#1583

…tion

…u/incubator-doris into fe-time-zone-function

…u/incubator-doris into fe-time-zone-function # Conflicts: # fe/src/main/java/org/apache/doris/rewrite/FEFunctions.java

…u/incubator-doris into fe-time-zone-function

…to fe-time-zone-function Conflicts: fe/src/main/java/org/apache/doris/rewrite/FEFunctions.java

morningman · 2019-08-20T10:46:41Z

 switch (type.getPrimitiveType()) {
 case DATE:
- return this.date.compareTo(TimeUtils.MIN_DATE) == 0;
+ return this.getStringValue().compareTo("1900-01-01") == 0;


I think min value and max value can be a static final member of DateLiteral？So that you can just use "=" to check this

We can create a static for min/max value. But we can't not use == to check if it is a minimal value. Because user can construct another object with minimal value string

ok
We can create a static for min/max value. But we can't not use == to check if it is a minimal value. Because user can construct another object with minimal value string
ok

morningman · 2019-08-20T10:51:36Z

+ .toString(FormatBuilder(pattern).toFormatter());
+ }
+
+ private static DateTimeFormatterBuilder FormatBuilder(String pattern) throws AnalysisException{


Suggested change
privatestaticDateTimeFormatterBuilderFormatBuilder(Stringpattern) throwsAnalysisException{
privatestaticDateTimeFormatterBuilderformatBuilder(Stringpattern) throwsAnalysisException{

morningman · 2019-08-20T10:52:34Z

+ return hour;
+ }
+
+ public void setHour(long hour) {


Do we really need these set or get functions? If not, it is better to remove them.

imay · 2019-08-20T12:13:36Z

- date = new Date(in.readLong());
+ if (Catalog.getCurrentCatalogJournalVersion() >= FeMetaVersion.VERSION_59) {
+ long packed_time = in.readLong();
+ microsecond = (packed_time % (1L << 24));


make it a function

imay · 2019-08-20T12:15:14Z

+ second = hms % (1 << 6);
+ minute = (hms >> 6) % (1 << 6);
+ hour = (hms >> 12);
+ this.type = Type.DATETIME;


why DATETIME? I think it may be DATE

The dates read from the meta are all datetime types, and Doris explicitly specifies type DATE/DATETIME by setType()

imay · 2019-08-20T12:25:04Z

+ return dateLiteral;
+ }
+
+ public String dateFormat(String pattern) throws AnalysisException{


You should add comments for this function. It's better to give some example that what function do

imay · 2019-08-20T12:28:03Z

+ DateLiteral dateLiteral = (DateLiteral) date;
+
+ DateLiteral result = new DateLiteral(dateLiteral);
+ result.setDay(dateLiteral.getDay() + day.getLongValue());


how about "2010-01-31" + 10

…tion # Conflicts: # fe/src/main/java/org/apache/doris/common/FeMetaVersion.java # fe/src/main/java/org/apache/doris/rewrite/FEFunctions.java

…u/incubator-doris into fe-time-zone-function

morningman · 2019-08-21T12:39:18Z

 public void readFields(DataInput in) throws IOException {
 super.readFields(in);
- date = new Date(in.readLong());
+ if (Catalog.getCurrentCatalogJournalVersion() >= FeMetaVersion.VERSION_59) {


VERSION_59 is used by support strict mode, change to VERSION_60

imay · 2019-08-22T06:21:00Z

+ return new DateLiteral(dateTime.getYear(), dateTime.getMonthOfYear(), dateTime.getDayOfMonth());
+ } else {
+ return new DateLiteral(dateTime.getYear(), dateTime.getMonthOfYear(), dateTime.getDayOfMonth(),
+ dateTime.getHourOfDay(), dateTime.getMinuteOfHour(), dateTime.getSecondOfMinute());


Why not move this logic to DateLiteral. And I think we can make formatter static which is thread-safe.

imay · 2019-08-22T06:24:44Z

- date = new Date(in.readLong());
+ if (Catalog.getCurrentCatalogJournalVersion() >= FeMetaVersion.VERSION_60) {
+ fromPackedDatetime(in.readLong());
+ } else {


I think we should persist type for later use. Or if we want to persist Date, we will have to change the format we use.

…to fe-time-zone-function

imay · 2019-08-26T15:59:49Z

 public static FloatLiteral timeDiff(LiteralExpr first, LiteralExpr second) throws AnalysisException {
- long timediff = (getTime(first) - getTime(second)) / 1000;
- return new FloatLiteral((double)timediff, Type.TIME);
+ SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");


why "yyyy-MM-dd" other than "yyyy-MM-dd hh:mm:ss"
I think we can write a function getTime() for DateLiteral.

imay · 2019-08-26T16:06:01Z

 import java.io.IOException;
 import java.nio.ByteBuffer;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;


And I see you use SimpleDateFormat and DateTimeFormatter.
Can we unify them to DateTimeFormatter?

imay · 2019-08-26T16:10:20Z

 }
- Date date = new Date(unixTime.getLongValue() * 1000);
- return new StringLiteral(dateFormat(date, "%Y-%m-%d %H:%i:%S"));
+ SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");


I think we can create a constructor of DateLiteral with unix timestamp and timezone.
Then this logic can be included in DateLiteral

imay

LGTM

morningman

LGTM

… missing out file (apache#1644)

HangyuanLiu added 7 commits August 13, 2019 20:24

add fe time zone
a60451c

add fe time zone
701d448

add
194f5e9

Merge remote-tracking branch 'upstream/master' into fe-time-zone-func…
37d7dfa
…tion

Merge branch 'fe-time-zone-function' ofhttps://github.com/HangyuanLi…
5d85ff4
…u/incubator-doris into fe-time-zone-function

add fe time zone
b575d4c

add fe time zone
98ab140

HangyuanLiu closed this Aug 19, 2019

HangyuanLiu added 11 commits August 19, 2019 14:26

add fe time zone
40c0dfe

Merge branch 'fe-time-zone-function' ofhttps://github.com/HangyuanLi…
e5ac2fe
…u/incubator-doris into fe-time-zone-function # Conflicts: # fe/src/main/java/org/apache/doris/rewrite/FEFunctions.java

Merge branch 'fe-time-zone-function' ofhttps://github.com/HangyuanLi…
01c8557
…u/incubator-doris into fe-time-zone-function # Conflicts: # fe/src/main/java/org/apache/doris/rewrite/FEFunctions.java

add
58e856b

add
00f523d

add
3a376c0

add
4c486b9

add
6f12379

add
e4917d0

Merge branch 'fe-time-zone-function' ofhttps://github.com/HangyuanLi…
740b0bc
…u/incubator-doris into fe-time-zone-function

refactor date literal
05335df

HangyuanLiu reopened this Aug 19, 2019

HangyuanLiu changed the title ~~Fe time zone function~~Refactor DateLiteral class in FEAug 19, 2019

Merge branch 'master' ofhttps://github.com/apache/incubator-dorisin…
98d6668
…to fe-time-zone-function Conflicts: fe/src/main/java/org/apache/doris/rewrite/FEFunctions.java

HangyuanLiu force-pushed the fe-time-zone-function branch from 3438f2f to 98d6668Compare August 19, 2019 13:48

morningman requested changes Aug 20, 2019
View reviewed changes

imay requested changes Aug 20, 2019
View reviewed changes

HangyuanLiu added 4 commits August 21, 2019 13:31

add
c587d55

Merge remote-tracking branch 'upstream/master' into fe-time-zone-func…
23e4fc1
…tion # Conflicts: # fe/src/main/java/org/apache/doris/common/FeMetaVersion.java # fe/src/main/java/org/apache/doris/rewrite/FEFunctions.java

add
6db4c68

Merge branch 'fe-time-zone-function' ofhttps://github.com/HangyuanLi…
a4427f7
…u/incubator-doris into fe-time-zone-function

HangyuanLiu force-pushed the fe-time-zone-function branch from 12e3678 to a4427f7Compare August 21, 2019 06:36

morningman requested changes Aug 21, 2019
View reviewed changes

HangyuanLiu force-pushed the fe-time-zone-function branch from 298dd35 to 58093b8Compare August 21, 2019 14:27

imay reviewed Aug 22, 2019
View reviewed changes

function name to camel
266e74b

HangyuanLiu force-pushed the fe-time-zone-function branch from 58093b8 to 266e74bCompare August 25, 2019 03:18

Merge branch 'master' ofhttps://github.com/apache/incubator-dorisin…
a3f5672
…to fe-time-zone-function

HangyuanLiu force-pushed the fe-time-zone-function branch 2 times, most recently from 9ac46ab to 63071d4Compare August 26, 2019 14:28

imay reviewed Aug 26, 2019
View reviewed changes

HangyuanLiu force-pushed the fe-time-zone-function branch from 63071d4 to 08bf2ecCompare August 27, 2019 04:14

add master forward time zone support
2412538

HangyuanLiu force-pushed the fe-time-zone-function branch from 08bf2ec to 2412538Compare August 27, 2019 04:59

imay approved these changes Aug 27, 2019
View reviewed changes

morningman approved these changes Aug 27, 2019
View reviewed changes

morningman merged commit 0c2e344 into apache:masterAug 27, 2019

imay mentioned this pull request Sep 26, 2019
Release Notes 0.11.0 #1891
Closed

HangyuanLiu deleted the fe-time-zone-function branch May 25, 2020 01:56

swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Jul 25, 2023
[feature](selectdb-cloud) Fix test_schema_change_with_readd_cluster…
737fad2
… missing out file (apache#1644)

	privatestaticDateTimeFormatterBuilderFormatBuilder(Stringpattern) throwsAnalysisException{
	privatestaticDateTimeFormatterBuilderformatBuilder(Stringpattern) throwsAnalysisException{

Uh oh!

Conversation

HangyuanLiu commented Aug 15, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

imay left a comment

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HangyuanLiu commented Aug 15, 2019•
edited
Loading