v2 segment support string encode(#1766) by wangbo · Pull Request #1816 · apache/doris

wangbo · 2019-09-17T05:49:39Z

#1766
major change

change data format of binary dict page, appending (dict page data) and (dict page offset) to binary dict page;
add new decoding method for new binary dict page format
add ut for segment test
set the elements of initial array to 0 ,when calling arena.AllocateNewBlock
hard code way to choose dict coding for string

0919 commit major change

change dict file format:when saving binary dict page, separate dict page from dict page,one dict page may have multi data pages;when reading a binary dict page,one ColumnReader keeps one dict page
loading dict when calling column_reader._read_page
3.rollback BinaryDictPage
no longer using memset(0) to inital column_zonemap.max_value

0926 17 commit major change

init column_zone_map min value column_zone_map slice's data array;
set char/varchar column_zone_map'max value size to 0
add ut for char column zone map query hit/miss

0929 10 commit major change

allocate mem for column_zone_map 's max and min value
direct copy content to column_zone_map's max and min value

keep dict page handle in FileColumnIterator;

gaodayue · 2019-09-25T08:45:42Z

+ for (int j = 0; j < 4; ++j) {
+ auto cell = row.cell(j);
+ cell.set_not_null();
+ set_column_value_by_type(tablet_schema->_cols[j]._type, i * 10 + j, (char*)cell.mutable_cell_ptr(), tablet_schema->_cols[j]._length);


I'm confused, how can you access private field _cols and _type?

me too;not FRIEND_TEST,not "#define private public"

kangpinghuang · 2019-09-29T01:58:10Z

+ } else if (_type_info->type() == OLAP_FIELD_TYPE_CHAR) {
+ Slice *min_value = (Slice *)_zone_map.min_value;
+ min_value->data = _max_char_value;
+ min_value->size = OLAP_CHAR_MAX_LENGTH;


why set char type's max length to 255? I think it should be equal to OLAP_STRING_MAX_LENGTH.

Fe defines max char and varchar length,see ScalarType;this is a temporary define,later I will use columns real length

yes, for char type, it is a bit complex

gaodayue

LGTM generally. One thing I'm not sure is the change to varchar's set_to_max since it's been used by segment v1's zonemap. Would it break backward compatibility? @kangpinghuang Could you verify it?

gaodayue · 2019-09-29T09:56:54Z

 auto slice = reinterpret_cast<Slice*>(buf);
- slice->size = 1;
- memset(slice->data, 0xFF, 1);
+ memset(slice->data, 0xFF, slice->size);


should first reset slice->size to OLAP_STRING_MAX_LENGTH

2. interpret not cast 3.remove useless var 4. set_to_min/max consist with master

kangpinghuang · 2019-09-30T02:11:20Z

I have check set_to_max in olap, I think original code has bug and this pr's code has fix it.
I check set_to_max used in two place of v1:

in zone map building. But for char/varchar type, there are no zone map for them, so there is no influence.
in split_range for query, I think the new max is larger than the old one, will not cause problems.
LGTM

kangpinghuang

LGTM

gaodayue

LGTM

wangbo · 2019-09-30T02:31:08Z

current commit is old version:in varchar's set_to_max,slice size set to 1
because set to max_string_value will cause delete_handler_test core dump,I will see this case today

wangbo · 2019-09-30T04:38:04Z

current commit is old version:in varchar's set_to_max,slice size set to 1
because set to max_string_value will cause delete_handler_test core dump,I will see this case today

Core dump stack

#0 SLL_Next (t=0xffffffffffffffff) at src/linked_list.h:45
#1 SLL_TryPop (rv=, list=0x436dce0) at src/linked_list.h:69
#2 TryPop (rv=, this=0x436dce0) at src/thread_cache.h:220
#3 Allocate (oom_handler=0x25e5700 <tcmalloc::cpp_throw_oom(unsigned long)>, cl=5, size=64, this=) at src/thread_cache.h:379
#4 malloc_fast_pathtcmalloc::cpp_throw_oom (size=64) at src/tcmalloc.cc:1855
#5 tc_new (size=64) at src/tcmalloc.cc:1976
#6 0x0000000000c21672 in doris::WrapperField::WrapperField (this=0x4e07d40, rep=, variable_len=,
is_string_type=) at /home/wangbo/incubator-doris/be/src/olap/wrapper_field.cpp:84
#7 0x0000000000c21b9e in doris::WrapperField::create (column=..., len=, len@entry=0)
at /home/wangbo/incubator-doris/be/src/olap/wrapper_field.cpp:51
#8 0x0000000000cb88e1 in doris::ColumnDataWriter::init (this=0x4bc4780) at /home/wangbo/incubator-doris/be/src/olap/rowset/column_data_writer.cpp:72
#9 0x0000000000ca273c in doris::AlphaRowsetWriter::_init (this=this@entry=0x4bf83c0)
at /home/wangbo/incubator-doris/be/src/olap/rowset/alpha_rowset_writer.cpp:275
#10 0x0000000000ca3694 in doris::AlphaRowsetWriter::init (this=0x4bf83c0, rowset_writer_context=...)
at /home/wangbo/incubator-doris/be/src/olap/rowset/alpha_rowset_writer.cpp:73
#11 0x0000000000c91b03 in doris::RowsetFactory::create_rowset_writer (context=..., output=output@entry=0x7fffcdf75318)
at /home/wangbo/incubator-doris/be/src/olap/rowset/rowset_factory.cpp:50
#12 0x0000000000bc9b3f in doris::TabletManager::_create_inital_rowset (this=this@entry=0x4d5a180, tablet=..., request=...)
at /home/wangbo/incubator-doris/be/src/olap/tablet_manager.cpp:1250
#13 0x0000000000bcc919 in doris::TabletManager::_internal_create_tablet (this=this@entry=0x4d5a180, alter_type=alter_type@entry=doris::SCHEMA_CHANGE,
request=..., is_schema_change_tablet=is_schema_change_tablet@entry=false, ref_tablet=..., data_dirs=...)
at /home/wangbo/incubator-doris/be/src/olap/tablet_manager.cpp:371
#14 0x0000000000bcdd72 in doris::TabletManager::create_tablet (this=this@entry=0x4d5a180, request=..., stores=...)
at /home/wangbo/incubator-doris/be/src/olap/tablet_manager.cpp:318
#15 0x0000000000b82642 in doris::StorageEngine::create_tablet (this=, request=...)
at /home/wangbo/incubator-doris/be/src/olap/storage_engine.cpp:760
#16 0x0000000000b4fac3 in doris::TestDeleteConditionHandler::SetUp (this=0x4e08460)
at /home/wangbo/incubator-doris/be/test/olap/delete_handler_test.cpp:173
#17 0x0000000002530f37 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) ()
#18 0x000000000252ba9c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) ()
#19 0x0000000002510df7 in testing::Test::Run() ()
#20 0x00000000025116ce in testing::TestInfo::Run() ()
#21 0x0000000002511d25 in testing::TestCase::Run() ()
#22 0x0000000002518899 in testing::internal::UnitTestImpl::RunAllTests() ()
#23 0x0000000002531eb3 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) ()
#24 0x000000000252c7a2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) ()
#25 0x0000000002517570 in testing::UnitTest::Run() ()
#26 0x00000000009fb316 in RUN_ALL_TESTS () at /home/wangbo/incubator-doris/thirdparty/installed/include/gtest/gtest.h:2233
#27 main (argc=, argv=0x7fffcdf75f88) at /home/wangbo/incubator-doris/be/test/olap/delete_handler_test.cpp:951

core dump happends in wrapper_field.cpp

line:80 _string_content.reset(new char[slice->size]);
the slice->size is 16,it seems reasonable
the code logic here is that we want to create a varchar wrapper_field

before that in column_data_writer.cpp

line:68 WrapperField::create // create varchar wrapper_field first
line:70 _zone_maps[i].first->set_to_max();
...
line:72 WrapperField::create # core dump here,when we want to create a another varchar wrapper_field

in line:68,we get a char[64]
in line:70, varchar-> set_to_max, memset(char[64],0xff,65536)
it seems that unreasonable,correct is:"memset(char[64],0xff,strlen(char[64])])"
So I think that we show set_to_max with the correct column length.

I prepare to solve it with a new pr which allocate ColumnZoneMapBuilder max/min value with real column length

wangbo · 2019-09-30T07:56:34Z

@imay please review again

add comment for dict lazy init
2.merge master branch to resolve conflict

imay

LGTM

wangbo added 3 commits September 17, 2019 13:45

v2 segment support string encode(apache#1766)
0835bc1

Merge remote-tracking branch 'upstream/master'
b7e5bb7

segment test using uint64
2380732

imay requested changes Sep 17, 2019
View reviewed changes

Comment threadbe/src/olap/rowset/segment_v2/binary_dict_page.cpp Outdated
Comment threadbe/src/olap/rowset/segment_v2/binary_dict_page.cpp Outdated
Comment threadbe/src/util/arena.cpp Outdated

wangbo added 2 commits September 19, 2019 19:21

update commit
0a40fa8

update zone map
61f2868

imay reviewed Sep 21, 2019
View reviewed changes

Comment threadbe/src/olap/rowset/segment_v2/column_reader.cpp Outdated
Comment threadbe/src/olap/rowset/segment_v2/column_reader.h Outdated

gaodayue requested changes Sep 22, 2019
View reviewed changes

wangbo added 2 commits September 24, 2019 12:07

update commit 0924:
12e31ef
keep dict page handle in FileColumnIterator;

Merge remote-tracking branch 'upstream/master' into master_test
c191220

gaodayue requested changes Sep 24, 2019
View reviewed changes

update commit 0925 10
52195ad

gaodayue requested changes Sep 25, 2019
View reviewed changes

wangbo added 2 commits September 25, 2019 20:31

update commit 0925 8
2d519fc

Merge remote-tracking branch 'upstream/master'
694b1be

kangpinghuang requested changes Sep 25, 2019
View reviewed changes

Comment threadbe/src/olap/rowset/segment_v2/binary_dict_page.h Outdated
Comment threadbe/src/olap/rowset/segment_v2/binary_dict_page.h Outdated
Comment threadbe/src/olap/rowset/segment_v2/column_zone_map.cpp Outdated
Comment threadbe/src/olap/rowset/segment_v2/encoding_info.cpp

kangpinghuang reviewed Sep 26, 2019
View reviewed changes

Comment threadbe/src/olap/rowset/segment_v2/binary_dict_page.h Outdated

kangpinghuang reviewed Sep 26, 2019
View reviewed changes

Comment threadbe/src/olap/rowset/segment_v2/column_reader.cpp

update commit 0926:17
349b251

wangbo force-pushed the master branch from 56d951a to 349b251Compare September 27, 2019 07:19

re fresh commit
ec186d8

wangbo force-pushed the master branch from d740714 to ec186d8Compare September 27, 2019 07:28

wangbo added 4 commits September 27, 2019 17:58

reuse column zone map char array
118bce2

reuse column zone map char array
c09aabc

add get_type_value_with_arena
2489045

Merge branch 'upstream_master' into master_test
839bbd8

kangpinghuang requested changes Sep 29, 2019
View reviewed changes

Merge branch 'master_test'
40a498a

gaodayue requested changes Sep 29, 2019
View reviewed changes

1. use allocate from
429d372
2. interpret not cast 3.remove useless var 4. set_to_min/max consist with master

wangbo force-pushed the master branch from 9754aeb to 429d372Compare September 29, 2019 14:48

refresh commit
98fdf6f

kangpinghuang previously approved these changes Sep 30, 2019
View reviewed changes

gaodayue previously approved these changes Sep 30, 2019
View reviewed changes

wangbo closed this Sep 30, 2019

wangbo reopened this Sep 30, 2019

imay reviewed Sep 30, 2019
View reviewed changes

Comment threadbe/src/olap/rowset/segment_v2/column_reader.cpp
Comment threadbe/src/olap/types.h

add commit for lazy init dict
e0d6ca0

wangbo dismissed stale reviews from gaodayue and kangpinghuang via e0d6ca0September 30, 2019 05:03

Merge remote-tracking branch 'remotes/upstream/master' into master_test
abcf356

imay approved these changes Sep 30, 2019
View reviewed changes

imay merged commit 8aa8e08 into apache:masterSep 30, 2019

Uh oh!

Conversation

wangbo commented Sep 17, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gaodayueSep 25, 2019

Choose a reason for hiding this comment

Uh oh!

wangboSep 25, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kangpinghuangSep 29, 2019

Choose a reason for hiding this comment

Uh oh!

wangboSep 29, 2019

Choose a reason for hiding this comment

Uh oh!

kangpinghuangSep 30, 2019

Choose a reason for hiding this comment

Uh oh!

gaodayue left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gaodayueSep 29, 2019

Choose a reason for hiding this comment

Uh oh!

wangboSep 29, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kangpinghuang commented Sep 30, 2019

Uh oh!

kangpinghuang left a comment

Choose a reason for hiding this comment

Uh oh!

gaodayue left a comment

Choose a reason for hiding this comment

Uh oh!

wangbo commented Sep 30, 2019

Uh oh!

Uh oh!

Uh oh!

wangbo commented Sep 30, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wangbo commented Sep 30, 2019

Uh oh!

wangbo commented Sep 17, 2019•
edited
Loading

wangbo commented Sep 30, 2019•
edited
Loading