[feature](json-function) add json_insert, json_replace, json_set functions by xuefengze · Pull Request #24384 · apache/doris

xuefengze · 2023-09-14T10:10:58Z

Proposed changes

Issue Number: close#21268

add json_insert, json_replace, json_set functions

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-09-14T10:18:12Z

clang-tidy review says "All clean, LGTM! 👍"

xuefengze · 2023-09-14T16:07:02Z

run buildall

doris-robot · 2023-09-14T16:36:06Z

TeamCity be ut coverage result:
Function Coverage: 36.94% (7975/21587)
Line Coverage: 28.89% (63915/221218)
Region Coverage: 27.82% (33167/119236)
Branch Coverage: 24.39% (17023/69782)
Coverage Report: http://coverage.selectdb-in.cc/coverage/12db03f306c08d9fb4aaa3db5aacaea8c5a635f8_12db03f306c08d9fb4aaa3db5aacaea8c5a635f8/report/index.html

doris-robot · 2023-09-14T16:44:01Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.83 seconds
stream load tsv: 600 seconds loaded 74807831229 Bytes, about 118 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162433739 Bytes

xuefengze · 2023-09-15T02:21:44Z

run buildall

github-actions · 2023-09-15T02:24:58Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-15T02:36:32Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-15T02:41:37Z

TeamCity be ut coverage result:
Function Coverage: 36.93% (7973/21587)
Line Coverage: 28.88% (63889/221232)
Region Coverage: 27.80% (33151/119243)
Branch Coverage: 24.38% (17014/69788)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c5084c2b6a51e56c25a16c8dad6e5648e8f10c81_c5084c2b6a51e56c25a16c8dad6e5648e8f10c81/report/index.html

doris-robot · 2023-09-15T02:53:56Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.47 seconds
stream load tsv: 600 seconds loaded 74807831229 Bytes, about 118 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162468413 Bytes

github-actions · 2023-09-15T03:17:36Z

clang-tidy review says "All clean, LGTM! 👍"

xuefengze · 2023-09-15T03:22:42Z

run buildall

doris-robot · 2023-09-15T04:04:11Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.48 seconds
stream load tsv: 595 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162482677 Bytes

doris-robot · 2023-09-15T04:08:22Z

TeamCity be ut coverage result:
Function Coverage: 36.93% (7975/21593)
Line Coverage: 28.87% (63916/221401)
Region Coverage: 27.80% (33170/119313)
Branch Coverage: 24.38% (17023/69830)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0420ca994142dd0a6573448ae4e1bd5da0f796a9_0420ca994142dd0a6573448ae4e1bd5da0f796a9/report/index.html

xuefengze · 2023-09-15T05:35:49Z

run p0

eldenmoon · 2023-09-15T17:00:24Z


+struct FunctionJsonInsert {
+ static constexpr auto name = "json_insert";
+ static constexpr auto is_insert = true;


why not use enum? boolean value is not intuitive

I will do it.

eldenmoon · 2023-09-15T17:02:22Z

+ auto tok = get_json_token(path_string);
+#endif
+ std::vector<std::string> paths(tok.begin(), tok.end());
+ auto status = get_parsed_paths_with_status(paths, &parsed_paths);


why not use macro RETURN_IF_ERROR?

eldenmoon · 2023-09-15T17:09:10Z

+
+ std::vector<std::vector<std::vector<JsonPath>>> json_paths;
+ auto status = get_parsed_path_columns(json_paths, data_columns, input_rows_count);
+ if (UNLIKELY(status != Status::OK())) {


use macro RETURN_IF_ERROR?

github-actions · 2023-09-16T04:11:28Z

clang-tidy review says "All clean, LGTM! 👍"

xuefengze · 2023-09-16T04:12:28Z

run buildall

doris-robot · 2023-09-16T04:40:19Z

TeamCity be ut coverage result:
Function Coverage: 36.83% (7981/21667)
Line Coverage: 28.84% (64029/221999)
Region Coverage: 27.76% (33216/119635)
Branch Coverage: 24.36% (17047/69976)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c6e7e1f8d0cba04973fcad841408bb4edb74f931_c6e7e1f8d0cba04973fcad841408bb4edb74f931/report/index.html

doris-robot · 2023-09-16T04:54:27Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.72 seconds
stream load tsv: 593 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162182638 Bytes

xuefengze · 2023-09-16T05:19:47Z

run p0

eldenmoon

LGTM

github-actions · 2023-09-16T15:55:26Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-16T15:55:29Z

PR approved by anyone and no changes requested.

xuefengze · 2023-09-18T06:34:50Z

run buildall

doris-robot · 2023-09-18T07:25:46Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.06 seconds
stream load tsv: 594 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162380267 Bytes

doris-robot · 2023-09-18T07:32:24Z

TeamCity be ut coverage result:
Function Coverage: 36.80% (7980/21682)
Line Coverage: 28.82% (64021/222153)
Region Coverage: 27.71% (33226/119912)
Branch Coverage: 24.33% (17028/69990)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4cc611a0f374695295bf379236d13094de59b279_4cc611a0f374695295bf379236d13094de59b279/report/index.html

xuefengze · 2023-09-18T10:08:02Z

run p0

xuefengze · 2023-09-18T11:05:53Z

run p0

eldenmoon

LGTM

github-actions · 2023-09-19T03:43:56Z

PR approved by at least one committer and no changes requested.

yiguolei · 2023-09-22T01:10:52Z

@xuefengze Hi, the code is conflict. Could you please resolve conflicts?

yiguolei · 2023-09-22T01:13:14Z

run buildall

github-actions · 2023-09-22T01:19:58Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei · 2023-09-22T01:24:21Z

I have resolve the conflict and rebuild all

doris-robot · 2023-09-22T01:59:46Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.25 seconds
stream load tsv: 601 seconds loaded 74807831229 Bytes, about 118 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17162495202 Bytes

doris-robot · 2023-09-22T02:29:02Z

TeamCity be ut coverage result:
Function Coverage: 36.38% (8022/22050)
Line Coverage: 28.66% (64425/224753)
Region Coverage: 27.57% (33460/121347)
Branch Coverage: 24.24% (17116/70622)
Coverage Report: http://coverage.selectdb-in.cc/coverage/95f4cb0adb2b4b0fcbaf43f51f1d4eb5a9c58366_95f4cb0adb2b4b0fcbaf43f51f1d4eb5a9c58366/report/index.html

yiguolei

lgtm

github-actions · 2023-09-23T11:45:06Z

PR approved by at least one committer and no changes requested.

eldenmoon

LGTM

…tions (apache#24384) [feature](json-function) add three json funcitons

…tions (#24384) [feature](json-function) add three json funcitons

[feature](json-function) add three json funcitons
12db03f

refactor: remove unnecessary include headers
c5084c2

Merge branch 'master' into add_func_modifyjsonvalue
8e339b4

doc: fix typo
0420ca9

eldenmoon reviewed Sep 15, 2023
View reviewed changes

refactor: update code
c6e7e1f

eldenmoon previously approved these changes Sep 16, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 16, 2023

github-actionsBot added the reviewed label Sep 16, 2023

eldenmoon previously approved these changes Sep 19, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2023

Merge branch 'master' into add_func_modifyjsonvalue
95f4cb0

yiguolei dismissed eldenmoon’s stale review via 95f4cb0September 22, 2023 01:12

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 22, 2023

yiguolei approved these changes Sep 23, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 23, 2023

eldenmoon approved these changes Sep 25, 2023
View reviewed changes

eldenmoon merged commit 1b95ce1 into apache:masterSep 25, 2023

xiaokang added dev/2.0.3 merge_conflict dev/2.0.3-merged and removed dev/2.0.3 labels Sep 26, 2023

xiaokang pushed a commit to xiaokang/doris that referenced this pull request Sep 27, 2023
[feature](json-function) add json_insert, json_replace, json_set func…
89434df
…tions (apache#24384) [feature](json-function) add three json funcitons

xiaokang pushed a commit that referenced this pull request Oct 3, 2023
[feature](json-function) add json_insert, json_replace, json_set func…
e032324
…tions (#24384) [feature](json-function) add three json funcitons

xuefengze deleted the add_func_modifyjsonvalue branch November 14, 2023 05:32

xiaokang mentioned this pull request Dec 4, 2023
Release Note 2.0.3 #27909
Closed

Uh oh!

Conversation

xuefengze commented Sep 14, 2023

Proposed changes

Further comments

Uh oh!

github-actionsBot commented Sep 14, 2023

Uh oh!

xuefengze commented Sep 14, 2023

Uh oh!

doris-robot commented Sep 14, 2023

Uh oh!

doris-robot commented Sep 14, 2023

Uh oh!

xuefengze commented Sep 15, 2023

Uh oh!

github-actionsBot commented Sep 15, 2023

Uh oh!

github-actionsBot commented Sep 15, 2023

Uh oh!

doris-robot commented Sep 15, 2023

Uh oh!

doris-robot commented Sep 15, 2023

Uh oh!

github-actionsBot commented Sep 15, 2023

Uh oh!

xuefengze commented Sep 15, 2023

Uh oh!

doris-robot commented Sep 15, 2023

Uh oh!

doris-robot commented Sep 15, 2023

Uh oh!

xuefengze commented Sep 15, 2023

Uh oh!

eldenmoonSep 15, 2023

Choose a reason for hiding this comment

Uh oh!

xuefengzeSep 16, 2023

Choose a reason for hiding this comment

Uh oh!

eldenmoonSep 15, 2023

Choose a reason for hiding this comment

Uh oh!

xuefengzeSep 16, 2023

Choose a reason for hiding this comment

Uh oh!

eldenmoonSep 15, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 16, 2023

Uh oh!

xuefengze commented Sep 16, 2023

Uh oh!

doris-robot commented Sep 16, 2023

Uh oh!

doris-robot commented Sep 16, 2023

Uh oh!

xuefengze commented Sep 16, 2023

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 16, 2023

Uh oh!

github-actionsBot commented Sep 16, 2023

Uh oh!

xuefengze commented Sep 18, 2023

Uh oh!

doris-robot commented Sep 18, 2023

Uh oh!

doris-robot commented Sep 18, 2023

Uh oh!

xuefengze commented Sep 18, 2023

Uh oh!

xuefengze commented Sep 18, 2023

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 19, 2023