[feature](function) Support SHA family functions by zclllyybb · Pull Request #24342 · apache/doris

zclllyybb · 2023-09-13T23:57:28Z

Proposed changes

Issue Number: close #xxx

mysql>select sha("123");
+------------------------------------------+
| sha1('123') |
+------------------------------------------+
| 40bd001563085fc35165329ea1ff5c5ecbdbbeef |
+------------------------------------------+1 row inset (0.13 sec)
mysql>select sha2('abc', 224);
+----------------------------------------------------------+
| sha2('abc', 224) |
+----------------------------------------------------------+
| 23097d223405d8228642a477bda255b32aadbce4bda0b3f7e36c9da7 |
+----------------------------------------------------------+1 row inset (0.13 sec)
mysql>select sha2('abc', 384);
+--------------------------------------------------------------------------------------------------+
| sha2('abc', 384) |
+--------------------------------------------------------------------------------------------------+
| cb00753f45a35e8bb5a03d699ac65007272c32ab0eded1631a8b605a43ff5bed8086072ba1e7cc2358baeca134c825a7 |
+--------------------------------------------------------------------------------------------------+1 row inset (0.13 sec)
mysql>select sha2(NULL, 512);
+-----------------+
| sha2(NULL, 512) |
+-----------------+
| NULL |
+-----------------+1 row inset (0.09 sec)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

zclllyybb · 2023-09-13T23:57:35Z

run buildall

github-actions · 2023-09-14T00:04:02Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-14T00:34:29Z

TeamCity be ut coverage result:
Function Coverage: 36.92% (7968/21582)
Line Coverage: 28.92% (63850/220788)
Region Coverage: 27.87% (33159/118969)
Branch Coverage: 24.43% (17008/69616)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2789e6266674806902ee6b68e7ded3cc9a5792cf_2789e6266674806902ee6b68e7ded3cc9a5792cf/report/index.html

zclllyybb · 2023-09-14T02:49:43Z

run buildall

github-actions · 2023-09-14T02:56:05Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-14T03:15:52Z

TeamCity be ut coverage result:
Function Coverage: 36.92% (7968/21584)
Line Coverage: 28.91% (63837/220823)
Region Coverage: 27.86% (33153/118998)
Branch Coverage: 24.42% (17004/69630)
Coverage Report: http://coverage.selectdb-in.cc/coverage/34a587b05a1a29103d19daea66bedced79a27c67_34a587b05a1a29103d19daea66bedced79a27c67/report/index.html

doris-robot · 2023-09-14T04:35:34Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.75 seconds
stream load tsv: 604 seconds loaded 74807831229 Bytes, about 118 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162440258 Bytes

zclllyybb · 2023-09-15T07:21:15Z

run buildall

github-actions · 2023-09-15T07:28:29Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-15T07:48:08Z

TeamCity be ut coverage result:
Function Coverage: 36.93% (7981/21610)
Line Coverage: 28.90% (63986/221403)
Region Coverage: 27.83% (33194/119272)
Branch Coverage: 24.41% (17036/69804)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c8d5278032a58f8232d5fe5511d5d539a6427dff_c8d5278032a58f8232d5fe5511d5d539a6427dff/report/index.html

zclllyybb · 2023-09-15T08:24:46Z

run buildall

github-actions · 2023-09-15T08:31:48Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-15T08:45:03Z

TeamCity be ut coverage result:
Function Coverage: 36.80% (7979/21682)
Line Coverage: 28.84% (63970/221839)
Region Coverage: 27.77% (33198/119527)
Branch Coverage: 24.38% (17039/69896)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2fd00f42364b065a222ca3165ad237a46778dc7c_2fd00f42364b065a222ca3165ad237a46778dc7c/report/index.html

doris-robot · 2023-09-15T08:59:32Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.81 seconds
stream load tsv: 592 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162462279 Bytes

HappenLee

LGTM

github-actions · 2023-09-15T09:25:07Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-15T09:25:09Z

PR approved by anyone and no changes requested.

zclllyybb · 2023-09-17T08:59:37Z

run buildall

doris-robot · 2023-09-17T09:33:15Z

TeamCity be ut coverage result:
Function Coverage: 36.79% (7980/21689)
Line Coverage: 28.83% (64017/222052)
Region Coverage: 27.72% (33220/119826)
Branch Coverage: 24.35% (17025/69926)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2fd00f42364b065a222ca3165ad237a46778dc7c_2fd00f42364b065a222ca3165ad237a46778dc7c/report/index.html

doris-robot · 2023-09-17T09:36:07Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.43 seconds
stream load tsv: 619 seconds loaded 74807831229 Bytes, about 115 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162539370 Bytes

zclllyybb · 2023-09-18T02:13:10Z

run buildall

zclllyybb · 2023-09-19T08:54:41Z

run buildall

github-actions · 2023-09-19T09:02:13Z

clang-tidy review says "All clean, LGTM! 👍"

BiteTheDDDDt

LGTM

github-actions · 2023-09-19T09:05:38Z

PR approved by at least one committer and no changes requested.

doris-robot · 2023-09-19T09:20:21Z

TeamCity be ut coverage result:
Function Coverage: 36.80% (7982/21689)
Line Coverage: 28.85% (64060/222081)
Region Coverage: 27.75% (33249/119835)
Branch Coverage: 24.37% (17044/69932)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cdeb36e35a2b579132e010ced522b942af8b9ef3_cdeb36e35a2b579132e010ced522b942af8b9ef3/report/index.html

doris-robot · 2023-09-19T09:29:46Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 51.02 seconds
stream load tsv: 597 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162450562 Bytes

zclllyybb · 2023-09-20T03:14:46Z

run buildall

github-actions · 2023-09-20T03:21:22Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-20T03:53:30Z

TeamCity be ut coverage result:
Function Coverage: 36.64% (7989/21807)
Line Coverage: 28.68% (64069/223373)
Region Coverage: 27.60% (33260/120514)
Branch Coverage: 24.23% (17037/70326)
Coverage Report: http://coverage.selectdb-in.cc/coverage/970dea6fe6ef9679175a60496445d40cda617e5d_970dea6fe6ef9679175a60496445d40cda617e5d/report/index.html

doris-robot · 2023-09-20T03:57:18Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.84 seconds
stream load tsv: 594 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162492405 Bytes

Gabriel39

LGTM

github-actions · 2023-09-20T06:26:19Z

PR approved by at least one committer and no changes requested.

HappenLee

LGTM

Related PR: #24342 before: ```sql select sha2(k0, k1) from str; ``` it works **but the result may doesn't match with the input!** now: ```sql ERROR 1105 (HY000): errCode = 2, detailMessage = sha2 needs two params, and the second is must be a integer constant: sha2(k0, k1) ```

) Related PR: apache#24342 before: ```sql select sha2(k0, k1) from str; ``` it works **but the result may doesn't match with the input!** now: ```sql ERROR 1105 (HY000): errCode = 2, detailMessage = sha2 needs two params, and the second is must be a integer constant: sha2(k0, k1) ```

HappenLee reviewed Sep 14, 2023
View reviewed changes

Comment threadgensrc/script/doris_builtins_functions.py

HappenLee reviewed Sep 14, 2023
View reviewed changes

Comment threadbe/src/vec/functions/function_string.h

HappenLee reviewed Sep 14, 2023
View reviewed changes

Comment threadbe/src/vec/functions/function_string.h Outdated

HappenLee reviewed Sep 14, 2023
View reviewed changes

Comment threadbe/src/util/sha.cpp Outdated

zclllyybb force-pushed the sha2 branch from 34a587b to c8d5278Compare September 15, 2023 07:20

zclllyybb changed the title ~~[feature](function) Support SHA2 function~~[feature](function) Support SHA family functionsSep 15, 2023

HappenLee previously approved these changes Sep 15, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 15, 2023

github-actionsBot added the reviewed label Sep 15, 2023

BiteTheDDDDt reviewed Sep 18, 2023
View reviewed changes

Comment threadbe/src/util/sha.cpp

zclllyybb force-pushed the sha2 branch from 2fd00f4 to cdeb36eCompare September 19, 2023 08:54

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 19, 2023

BiteTheDDDDt previously approved these changes Sep 19, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2023

Merge branch 'master' into sha2
970dea6

zclllyybb dismissed BiteTheDDDDt’s stale review via 970dea6September 20, 2023 03:14

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 20, 2023

Gabriel39 approved these changes Sep 20, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 20, 2023

HappenLee approved these changes Sep 20, 2023
View reviewed changes

HappenLee merged commit 81e65f4 into apache:masterSep 20, 2023

zclllyybb deleted the sha2 branch October 11, 2023 19:07

zhangstar333 added the dev/2.0.3 label Oct 20, 2023

xiaokang added dev/2.0.3-merged and removed dev/2.0.3 labels Oct 22, 2023

xiaokang pushed a commit that referenced this pull request Oct 22, 2023
[feature](function) Support SHA family functions (#24342)
bd3ee3e

xiaokang mentioned this pull request Dec 4, 2023
Release Note 2.0.3 #27909
Closed

zclllyybb mentioned this pull request May 22, 2025
[Fix](function) Add lost check of function SHA2 in nereids #51163
Merged
16 tasks

Uh oh!

Conversation

zclllyybb commented Sep 13, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

zclllyybb commented Sep 13, 2023

Uh oh!

github-actionsBot commented Sep 14, 2023

Uh oh!

doris-robot commented Sep 14, 2023

Uh oh!

zclllyybb commented Sep 14, 2023

Uh oh!

github-actionsBot commented Sep 14, 2023

Uh oh!

doris-robot commented Sep 14, 2023

Uh oh!

doris-robot commented Sep 14, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zclllyybb commented Sep 15, 2023

Uh oh!

github-actionsBot commented Sep 15, 2023

Uh oh!

doris-robot commented Sep 15, 2023

Uh oh!

zclllyybb commented Sep 15, 2023

Uh oh!

github-actionsBot commented Sep 15, 2023

Uh oh!

doris-robot commented Sep 15, 2023

Uh oh!

doris-robot commented Sep 15, 2023

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 15, 2023

Uh oh!

github-actionsBot commented Sep 15, 2023

Uh oh!

zclllyybb commented Sep 17, 2023

Uh oh!

doris-robot commented Sep 17, 2023

Uh oh!

doris-robot commented Sep 17, 2023

Uh oh!

zclllyybb commented Sep 18, 2023

Uh oh!

Uh oh!

zclllyybb commented Sep 19, 2023

Uh oh!

github-actionsBot commented Sep 19, 2023

Uh oh!

BiteTheDDDDt left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 19, 2023

Uh oh!

doris-robot commented Sep 19, 2023

Uh oh!

doris-robot commented Sep 19, 2023

Uh oh!

zclllyybb commented Sep 20, 2023

Uh oh!

github-actionsBot commented Sep 20, 2023

Uh oh!

doris-robot commented Sep 20, 2023

Uh oh!

doris-robot commented Sep 20, 2023

Uh oh!

Gabriel39 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 20, 2023

zclllyybb commented Sep 13, 2023•
edited
Loading