[FIX](resize) fix array and map offsets resize with default value by amorynan · Pull Request #25669 · apache/doris

amorynan · 2023-10-20T06:06:25Z

Proposed changes

Issue Number: close #xxx
fix array and map offsets resize with default value
if we create_column() then resize() , offsets in array\map may has wrong default value to push

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions

clang-tidy made some suggestions

github-actions · 2023-10-20T06:16:01Z

+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest-message.h>


warning: 'gtest/gtest-message.h' file not found [clang-diagnostic-error]
#include<gtest/gtest-message.h> ^

github-actions · 2023-10-20T06:16:01Z

+ return dataTypes;
+}
+
+TEST(ResizeTest, ScalaTypeTest) {


warning: all parameters should be named in a function [readability-named-parameter]
Suggested change
TEST(ResizeTest, ScalaTypeTest) {
TEST(ResizeTest/*unused*/, ScalaTypeTest/*unused*/) {

github-actions · 2023-10-20T06:16:01Z

+ }
+}
+
+TEST(ResizeTest, ArrayTypeTest) {


warning: all parameters should be named in a function [readability-named-parameter]
Suggested change
TEST(ResizeTest, ArrayTypeTest) {
TEST(ResizeTest/*unused*/, ArrayTypeTest/*unused*/) {

github-actions · 2023-10-20T06:16:01Z

+ }
+}
+
+TEST(ResizeTest, ArrayTypeWithValuesTest) {


warning: all parameters should be named in a function [readability-named-parameter]
Suggested change
TEST(ResizeTest, ArrayTypeWithValuesTest) {
TEST(ResizeTest/*unused*/, ArrayTypeWithValuesTest/*unused*/) {

github-actions · 2023-10-20T06:16:01Z

+ col_a->insert(af);
+ col_a->insert(af);
+
+ col_a->resize(10);


warning: 10 is a magic number; consider replacing it with a named constant [readability-magic-numbers]
col_a->resize(10); ^

github-actions · 2023-10-20T06:16:02Z

+ MutableColumnPtr b = a->create_column();
+ b->insert_range_from(*col_a, 0, 10);
+ EXPECT_EQ(b->size(), 10);
+ ColumnMap* col_map = reinterpret_cast<ColumnMap*>(b.get());


warning: use auto when initializing with a cast to avoid duplicating the type name [modernize-use-auto]
Suggested change
ColumnMap* col_map = reinterpret_cast<ColumnMap*>(b.get());
auto* col_map = reinterpret_cast<ColumnMap*>(b.get());

github-actions · 2023-10-20T06:16:02Z

+ }
+}
+
+TEST(ResizeTest, StructTypeTest) {


warning: all parameters should be named in a function [readability-named-parameter]
Suggested change
TEST(ResizeTest, StructTypeTest) {
TEST(ResizeTest/*unused*/, StructTypeTest/*unused*/) {

github-actions · 2023-10-20T06:16:02Z

+
+ DataTypePtr a = std::make_shared<DataTypeStruct>(dataTypes);
+ auto col_a = a->create_column();
+ col_a->resize(10);


warning: 10 is a magic number; consider replacing it with a named constant [readability-magic-numbers]
col_a->resize(10); ^

github-actions · 2023-10-20T06:16:02Z

+ auto col_a = a->create_column();
+ col_a->resize(10);
+ MutableColumnPtr b = a->create_column();
+ b->insert_range_from(*col_a, 0, 10);


warning: 10 is a magic number; consider replacing it with a named constant [readability-magic-numbers]
b->insert_range_from(*col_a, 0, 10); ^

github-actions · 2023-10-20T06:16:03Z

+ col_a->resize(10);
+ MutableColumnPtr b = a->create_column();
+ b->insert_range_from(*col_a, 0, 10);
+ EXPECT_EQ(b->size(), 10);


warning: 10 is a magic number; consider replacing it with a named constant [readability-magic-numbers]
EXPECT_EQ(b->size(), 10); ^

amorynan · 2023-10-20T06:31:31Z

run buildall

eldenmoon · 2023-10-20T06:38:09Z

- get_offsets().resize(n);
+ auto last_off = get_offsets().back();
+ get_offsets().resize_fill(n, last_off);
 get_data().resize(n);


Is this get_data().resize(n) correct? for example if this array contains 3 element s[1, 2][1, 2][1, 2], so the inner data size is 6 instead of 3. so after doing ::resize(3), the inner data size became 3 and lost 3 nested element.Is this semantic right?

I think it should not call get_data().resize(n). Just append offsets with the same value.

doris-robot · 2023-10-20T07:23:13Z

TeamCity be ut coverage result:
Function Coverage: 37.08% (8309/22409)
Line Coverage: 29.20% (66744/228598)
Region Coverage: 27.84% (34639/124407)
Branch Coverage: 24.43% (17591/72018)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e4709fd4bff8637b4422595677e2b236a8ae9d05_e4709fd4bff8637b4422595677e2b236a8ae9d05/report/index.html

xiaokang · 2023-10-20T07:30:36Z

- get_offsets().resize(n);
+ auto last_off = get_offsets().back();
+ get_offsets().resize_fill(n, last_off);
 get_data().resize(n);


I think it should not call get_data().resize(n). Just append offsets with the same value.

xiaokang · 2023-10-20T07:31:05Z

 void ColumnArray::resize(size_t n) {
- get_offsets().resize(n);
+ auto last_off = get_offsets().back();
+ get_offsets().resize_fill(n, last_off);


We need to check if resize_fill will modify old offsets value.

if (n > old_size) { this->reserve(n); std::fill(t_end(), t_end() + n - old_size, value); } this->c_end = this->c_start + this->byte_size(n);

xiaokang · 2023-10-20T07:31:59Z

- get_offsets().resize(n);
+ auto last_off = get_offsets().back();
+ get_offsets().resize_fill(n, last_off);
 keys_column->resize(n);


should not call resize on keys and values column

xiaokang · 2023-10-20T07:32:43Z

@@ -422,7 +422,8 @@ void ColumnArray::reserve(size_t n) {

 //please check you real need size in data column, because it's maybe need greater size when data is string column
 void ColumnArray::resize(size_t n) {


Is there the same problem for ColumnStruct?

struct has no offsets array so do not has this problem

doris-robot · 2023-10-20T07:44:05Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.54 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17162350704 Bytes

hello-stephen · 2023-10-20T13:47:09Z

run p1

amorynan · 2023-10-23T07:19:30Z

run buildall

amorynan · 2023-10-23T08:18:02Z

run compile

amorynan · 2023-10-23T08:18:11Z

run beut

amorynan · 2023-10-23T08:18:31Z

run clickbench

doris-robot · 2023-10-23T08:53:15Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.88 seconds
stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162131278 Bytes

doris-robot · 2023-10-23T09:51:17Z

TeamCity be ut coverage result:
Function Coverage: 37.05% (8304/22410)
Line Coverage: 29.18% (66661/228417)
Region Coverage: 27.77% (34633/124696)
Branch Coverage: 24.42% (17585/72010)
Coverage Report: http://coverage.selectdb-in.cc/coverage/741457aa1ebb3122d447f07b2fa2551a89347aad_741457aa1ebb3122d447f07b2fa2551a89347aad/report/index.html

amorynan · 2023-10-24T02:02:52Z

run p0

amorynan · 2023-10-24T02:02:56Z

run p1

amorynan · 2023-10-24T02:03:59Z

run external

airborne12

LGTM

github-actions · 2023-10-24T03:26:26Z

PR approved by anyone and no changes requested.

eldenmoon

LGTM

github-actions · 2023-10-24T03:39:06Z

PR approved by at least one committer and no changes requested.

xiaokang

LGTM

…5669)

…ache#25669)

fix array and map offsets resize with default value
e4709fd

wm1581066 requested review from xiaokang and yiguolei October 20, 2023 06:15

github-actionsBot reviewed Oct 20, 2023
View reviewed changes

eldenmoon reviewed Oct 20, 2023
View reviewed changes

xiaokang added usercase Important user case type label p0_c dev/2.0.3 labels Oct 20, 2023

xiaokang reviewed Oct 20, 2023
View reviewed changes

delete resize nested column
741457a

airborne12 approved these changes Oct 24, 2023
View reviewed changes

github-actionsBot added the reviewed label Oct 24, 2023

eldenmoon approved these changes Oct 24, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 24, 2023

xiaokang approved these changes Oct 24, 2023
View reviewed changes

xiaokang merged commit 9160834 into apache:masterOct 24, 2023

xiaokang added dev/2.0.3-merged and removed dev/2.0.3 labels Oct 24, 2023

xiaokang pushed a commit that referenced this pull request Oct 24, 2023
[FIX](resize) fix array and map offsets resize with default value (#2…
5f93c91
…5669)

dutyu pushed a commit to dutyu/doris that referenced this pull request Oct 28, 2023
[FIX](resize) fix array and map offsets resize with default value (ap…
e1c159d
…ache#25669)

eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Oct 30, 2023
[FIX](resize) fix array and map offsets resize with default value (ap…
8f16ce5
…ache#25669)

gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023
[FIX](resize) fix array and map offsets resize with default value (ap…
52f52da
…ache#25669)

xiaokang mentioned this pull request Dec 4, 2023
Release Note 2.0.3 #27909
Closed

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
[FIX](resize) fix array and map offsets resize with default value (ap…
2def945
…ache#25669)

xiaokang added dev/2.0.x dev/2.1.x labels Oct 14, 2024

eldenmoon added dev/2.1.7-merged and removed dev/2.0.x dev/2.1.x labels Oct 17, 2024

	TEST(ResizeTest, ScalaTypeTest) {
	TEST(ResizeTest/unused/, ScalaTypeTest/unused/) {

	TEST(ResizeTest, ArrayTypeTest) {
	TEST(ResizeTest/unused/, ArrayTypeTest/unused/) {

	TEST(ResizeTest, ArrayTypeWithValuesTest) {
	TEST(ResizeTest/unused/, ArrayTypeWithValuesTest/unused/) {

	ColumnMap* col_map = reinterpret_cast<ColumnMap*>(b.get());
	auto* col_map = reinterpret_cast<ColumnMap*>(b.get());

	TEST(ResizeTest, StructTypeTest) {
	TEST(ResizeTest/unused/, StructTypeTest/unused/) {

		@@ -422,7 +422,8 @@ void ColumnArray::reserve(size_t n) {

		//please check you real need size in data column, because it's maybe need greater size when data is string column
		void ColumnArray::resize(size_t n) {

Uh oh!

Conversation

amorynan commented Oct 20, 2023

Proposed changes

Further comments

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 20, 2023

Choose a reason for hiding this comment

Uh oh!

amorynan commented Oct 20, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Oct 20, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amorynanOct 23, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Oct 20, 2023

Uh oh!

hello-stephen commented Oct 20, 2023

Uh oh!

amorynan commented Oct 23, 2023

Uh oh!

amorynan commented Oct 23, 2023

Uh oh!

amorynan commented Oct 23, 2023

Uh oh!

amorynan commented Oct 23, 2023

Uh oh!

doris-robot commented Oct 23, 2023

Uh oh!

doris-robot commented Oct 23, 2023

Uh oh!

amorynan commented Oct 24, 2023

amorynanOct 23, 2023•
edited
Loading