[Exec](join) Support column string64 to avoid join failed in string size overflow the uint32 by HappenLee · Pull Request #33511 · apache/doris

HappenLee · 2024-04-10T17:24:29Z

Proposed changes

Before:
join right table column string over uint32
string column Length is too Large

After:
get the right result, but may exec slowly

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-10T17:24:36Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

HappenLee · 2024-04-10T17:24:39Z

run buildall

github-actions · 2024-04-10T17:30:36Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-10T18:50:37Z

TeamCity be ut coverage result:
Function Coverage: 35.63% (8911/25012)
Line Coverage: 27.33% (73142/267624)
Region Coverage: 26.46% (37825/142958)
Branch Coverage: 23.21% (19274/83044)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2d0c934b491fad724a2b4e41272ad6f2a1290366_2d0c934b491fad724a2b4e41272ad6f2a1290366/report/index.html

HappenLee · 2024-04-16T16:04:20Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-04-16T16:10:10Z

+ void sort_column(const ColumnString64& column, EqualFlags& flags, IColumn::Permutation& perms,
+ EqualRange& range, bool last_column) const {


warning: method 'sort_column' can be made static [readability-convert-member-functions-to-static]
Suggested change
voidsort_column(constColumnString64&column, EqualFlags&flags, IColumn::Permutation&perms,
EqualRange&range, boollast_column) const{
staticvoidsort_column(constColumnString64&column, EqualFlags&flags, IColumn::Permutation&perms,
EqualRange&range, boollast_column) {

HappenLee · 2024-04-16T16:30:17Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-04-16T16:37:25Z

+ for (size_t i = 0; i < length; ++i)
+ cur_offsets[old_size + i] = src_offsets[start + i] - nested_offset + prev_max_offset;


warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
for (size_t i = 0; i < length; ++i)
cur_offsets[old_size + i] = src_offsets[start + i] - nested_offset + prev_max_offset;
for (size_t i = 0; i < length; ++i) {
cur_offsets[old_size + i] = src_offsets[start + i] - nested_offset + prev_max_offset;
}

github-actions · 2024-04-16T16:37:26Z

 }
 }

+void ColumnMap::insert_range_from_ignore_overflow(const IColumn& src, size_t start, size_t length) {


warning: method 'insert_range_from_ignore_overflow' can be made static [readability-convert-member-functions-to-static]
be/src/vec/columns/column_map.h:104:
- void insert_range_from_ignore_overflow(const IColumn& src, size_t start,+ static void insert_range_from_ignore_overflow(const IColumn& src, size_t start,

github-actions · 2024-04-16T16:37:26Z

 }
 }

+void ColumnStruct::insert_range_from_ignore_overflow(const IColumn& src, size_t start,


warning: method 'insert_range_from_ignore_overflow' can be made static [readability-convert-member-functions-to-static]
be/src/vec/columns/column_struct.h:147:
- void insert_range_from_ignore_overflow(const IColumn& src, size_t start,+ static void insert_range_from_ignore_overflow(const IColumn& src, size_t start,

HappenLee · 2024-04-16T17:23:43Z

run buildall

doris-robot · 2024-04-16T18:10:24Z

TeamCity be ut coverage result:
Function Coverage: 35.48% (8907/25107)
Line Coverage: 27.20% (73142/268889)
Region Coverage: 26.33% (37817/143637)
Branch Coverage: 23.11% (19271/83376)
Coverage Report: http://coverage.selectdb-in.cc/coverage/719a6d1be5a331bfd2794ef12f00cb805f97b86f_719a6d1be5a331bfd2794ef12f00cb805f97b86f/report/index.html

HappenLee · 2024-04-16T18:54:12Z

run buildall

HappenLee · 2024-04-16T18:56:23Z

run buildall

doris-robot · 2024-04-16T20:20:51Z

TeamCity be ut coverage result:
Function Coverage: 35.49% (8910/25109)
Line Coverage: 27.21% (73184/268926)
Region Coverage: 26.33% (37826/143642)
Branch Coverage: 23.12% (19276/83378)
Coverage Report: http://coverage.selectdb-in.cc/coverage/26833e0cd1968d5ad7d4efec47d689acfe55b94e_26833e0cd1968d5ad7d4efec47d689acfe55b94e/report/index.html

HappenLee · 2024-04-17T02:53:55Z

run buildall

doris-robot · 2024-04-17T03:15:26Z

TeamCity be ut coverage result:
Function Coverage: 35.47% (8908/25114)
Line Coverage: 27.20% (73160/268988)
Region Coverage: 26.32% (37811/143675)
Branch Coverage: 23.10% (19267/83400)
Coverage Report: http://coverage.selectdb-in.cc/coverage/eadfde7b23619fc03ea6aefcb14d5f41ed942087_eadfde7b23619fc03ea6aefcb14d5f41ed942087/report/index.html

HappenLee · 2024-04-17T05:34:40Z

run buildall

HappenLee · 2024-04-17T05:51:16Z

run buildall

HappenLee · 2024-04-17T06:04:57Z

run buildall

doris-robot · 2024-04-17T07:20:31Z

TeamCity be ut coverage result:
Function Coverage: 35.44% (8910/25143)
Line Coverage: 27.15% (73178/269566)
Region Coverage: 26.27% (37821/143989)
Branch Coverage: 23.07% (19270/83546)
Coverage Report: http://coverage.selectdb-in.cc/coverage/14a1b54cfa5dd0f9d1161f055a574791d4b359cf_14a1b54cfa5dd0f9d1161f055a574791d4b359cf/report/index.html

HappenLee · 2024-04-17T10:55:24Z

run buildall

github-actions · 2024-04-17T11:29:34Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-17T11:29:37Z

PR approved by anyone and no changes requested.

doris-robot · 2024-04-17T12:11:00Z

TeamCity be ut coverage result:
Function Coverage: 35.43% (8910/25149)
Line Coverage: 27.13% (73191/269828)
Region Coverage: 26.26% (37836/144062)
Branch Coverage: 23.06% (19277/83584)
Coverage Report: http://coverage.selectdb-in.cc/coverage/34f195c557c5e64b662cc597e112b50b18fec600_34f195c557c5e64b662cc597e112b50b18fec600/report/index.html

…ize overflow the uint32 (apache#33511)

…ize overflow the uint32 (#33511) (#33850)

This is imported by #33511. wrongly used ColumnStr<T> (); which violate C++20 standard(see https://wg21.cmeerw.net/cwg/issue2237) but still supported by clang up until now(see llvm/llvm-project#58112)

…ize overflow the uint32 (apache#33511)

This is imported by apache#33511. wrongly used ColumnStr<T> (); which violate C++20 standard(see https://wg21.cmeerw.net/cwg/issue2237) but still supported by clang up until now(see llvm/llvm-project#58112)

HappenLee added 2 commits April 17, 2024 00:03

save column append data
e5adcff

add col str 64
2bcca5f

HappenLee force-pushed the log branch from 2d0c934 to 2bcca5fCompare April 16, 2024 16:03

github-actionsBot reviewed Apr 16, 2024
View reviewed changes

save code except column string
fe89493

github-actionsBot reviewed Apr 16, 2024
View reviewed changes

support the columnstring overflow
719a6d1

HappenLee force-pushed the log branch from c36f87b to 719a6d1Compare April 16, 2024 16:41

fix copy build column core
26833e0

HappenLee force-pushed the log branch from dea461e to 26833e0Compare April 16, 2024 18:56

fix rf failed
eadfde7

change code
216b0af

HappenLee changed the title ~~save column append data~~[Exec](join) Support column string64 to avoid join failed in string size overflow the uint32Apr 17, 2024

HappenLee marked this pull request as ready for review April 17, 2024 05:34

BiteTheDDDDt reviewed Apr 17, 2024
View reviewed changes

Comment threadbe/src/vec/functions/in.h Outdated

change by cr
3c61a4b

BiteTheDDDDt reviewed Apr 17, 2024
View reviewed changes

Comment threadbe/src/vec/columns/column_string.cpp

change comment
14a1b54

impl the column string insert_range_from_ignore_overflow
34f195c

BiteTheDDDDt approved these changes Apr 17, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 17, 2024

github-actionsBot added the reviewed label Apr 17, 2024

Gabriel39 approved these changes Apr 18, 2024
View reviewed changes

HappenLee merged commit 7258692 into apache:masterApr 18, 2024

yiguolei added the 2.1.0-conflict label Apr 18, 2024

HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Apr 18, 2024
[Exec](join) Support column string64 to avoid join failed in string s…
b4f2d85
…ize overflow the uint32 (apache#33511)

yiguolei pushed a commit that referenced this pull request Apr 18, 2024
[Exec](join) Support column string64 to avoid join failed in string s…
1300317
…ize overflow the uint32 (#33511) (#33850)

zclllyybb mentioned this pull request Apr 18, 2024
[Fix](compile) Fix gcc compile on master #33864
Merged

yiguolei removed the 2.1.0-conflict label May 18, 2024

HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[Exec](join) Support column string64 to avoid join failed in string s…
02853de
…ize overflow the uint32 (apache#33511)

		void sort_column(const ColumnString64& column, EqualFlags& flags, IColumn::Permutation& perms,
		EqualRange& range, bool last_column) const {

-voidsort_column(constColumnString64&column, EqualFlags&flags, IColumn::Permutation&perms,
-EqualRange&range, boollast_column) const{
+staticvoidsort_column(constColumnString64&column, EqualFlags&flags, IColumn::Permutation&perms,
+EqualRange&range, boollast_column) {

		for (size_t i = 0; i < length; ++i)
		cur_offsets[old_size + i] = src_offsets[start + i] - nested_offset + prev_max_offset;

Uh oh!

Conversation

HappenLee commented Apr 10, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 10, 2024

Uh oh!

HappenLee commented Apr 10, 2024

Uh oh!

github-actionsBot commented Apr 10, 2024

Uh oh!

doris-robot commented Apr 10, 2024

Uh oh!

HappenLee commented Apr 16, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotApr 16, 2024

Choose a reason for hiding this comment

Uh oh!

HappenLee commented Apr 16, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotApr 16, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotApr 16, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotApr 16, 2024

Choose a reason for hiding this comment

Uh oh!

HappenLee commented Apr 16, 2024

Uh oh!

doris-robot commented Apr 16, 2024

Uh oh!

HappenLee commented Apr 16, 2024

Uh oh!

HappenLee commented Apr 16, 2024

Uh oh!

doris-robot commented Apr 16, 2024

Uh oh!

HappenLee commented Apr 17, 2024

Uh oh!

doris-robot commented Apr 17, 2024

Uh oh!

HappenLee commented Apr 17, 2024

Uh oh!

Uh oh!

HappenLee commented Apr 17, 2024

Uh oh!

Uh oh!

HappenLee commented Apr 17, 2024

Uh oh!

doris-robot commented Apr 17, 2024

Uh oh!

HappenLee commented Apr 17, 2024

Uh oh!

github-actionsBot commented Apr 17, 2024

Uh oh!

github-actionsBot commented Apr 17, 2024

Uh oh!

doris-robot commented Apr 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

HappenLee commented Apr 10, 2024•
edited
Loading