HLL support default value by HangyuanLiu · Pull Request #1825 · apache/doris

HangyuanLiu · 2019-09-18T10:47:50Z

When load date don't have HLL value , HLL value is a empty HyperLogLog

eg:
CREATE TABLE test_uv_9 (
pin_id bigint(20) NOT NULL COMMENT "",
id bigint(20) NULL COMMENT "",
uv1 hll HLL_UNION NOT NULL COMMENT "",
uv2 hll HLL_UNION NOT NULL COMMENT ""
) ENGINE=OLAP
AGGREGATE KEY(pin_id, id)
DISTRIBUTED BY HASH(pin_id) BUCKETS 16
PROPERTIES (
"storage_type" = "COLUMN"
)
1、curl --location-trusted -u root:123456 -H column_separator:, -H label:test_uv_14 -H "columns:pin_id,idx,u1,u2,id=12 ,uv1=hll_hash(u1)" -T uv_test http://11.40.166.162:8030/api/test/test_uv_9/_stream_load
Result ：hll_union_agg(uv2) should be 0，And hll_union_agg(uv1) has values > 0

2、curl --location-trusted -u root:123456 -H column_separator:, -H label:test_uv_15 -H "columns:pin_id,idx,u1,u2,id=12" -T uv_test http://11.40.166.162:8030/api/test/test_uv_10/_stream_load
Result ：hll_union_agg(uv1) and hll_union_agg(uv2) should be 0

imay · 2019-09-18T14:38:11Z

@HangyuanLiu
Can you add some document or usage example for this change?

kangkaisen · 2019-09-19T03:07:32Z

 char* mem = arena->Allocate(dst_slice->size);
- dst_slice->data = (char*) new (mem) HyperLogLog(src_slice->data);
+ if (src_slice->empty()) {
+ dst_slice->data = (char*) new (mem) HyperLogLog();


When the slice size is empty?

When HLL columns are NULL in the LOAD data

HangyuanLiu · 2019-09-19T04:02:55Z

Update example @imay

kangkaisen · 2019-09-19T06:16:24Z

curl --location-trusted -u root:123456 -H column_separator:, -H label:test_uv_15 -H "columns:pin_id,idx,u1,u2,id=12" -T uv_test http://11.40.166.162:8030/api/test/test_uv_10/_stream_load
Result ：hll_union_agg(uv1) and hll_union_agg(uv2) should be 0

I think we don't support this usage.

When load data with HLL column, we must use hll_hash function, hll_hash function will handle null value.

HangyuanLiu · 2019-09-19T06:29:47Z

This PR will support this capability.
There is a scenario where a Doris table may have multiple HLL columns that may come from different streams. The A stream may not contain the required HLL columns in the B stream and may not have NULL fields. The number of fields in this stream may be less than the number of fields in the Doris table
@kangkaisen

kangkaisen · 2019-09-19T07:02:10Z

@HangyuanLiu Hi, I See.
Could we add a empty_hll function support this capability? add then, we could still ensure the HLL input in aggregate_func.h will not null.

HangyuanLiu · 2019-09-19T07:20:44Z

@kangkaisen
If we add a empty_hll function , so we should write load command like .

LOAD LABEL test.uv
DATA INFILE ("hdfs://ns1017/streamA/*") INTO TABLE test_uv
(pin, id, u1)
set (
uv1=hll_hash(u1),
uv2= empty_hll(),
);

LOAD LABEL test.uv
DATA INFILE ("hdfs://ns1017/streamB/*") INTO TABLE test_uv
(pin, id, u2)
set (
uv1= empty_hll(),
uv2= hll_hash(u2),
);

uv2 may be another business stream
It is unreasonable to let the import data for A business care about the other B business HLL type.

morningman · 2019-09-21T04:35:25Z

@HangyuanLiu I think using empty_hll() is more reasonable. If business A does not care not column uv2, It should leave uv2 unchanged by explicitly assigning a function means unchanged, which is what empty_hll() will do.

If user assign nothing to HLL column, HLL column's content will be undefined. Because HLL is not like other types of column, which can be filled by default value or null.

empty_hll() to HLL is like 0 to SUM.

And also, should will add a function empty_bitmap() for BITMAP column? @kangkaisen

kangkaisen · 2019-09-21T04:43:12Z

And also, should will and a function empty_bitmap() for BITMAP column

@morningman I agree with you.

imay · 2019-09-21T05:44:37Z

And also, should will and a function empty_bitmap() for BITMAP column
@morningman I agree with you.

I agree too

@HangyuanLiu

Will you add a hll_empty() function? And it will be better if you can create an issue to record this improvement.

HangyuanLiu · 2019-09-21T11:53:36Z

OK,I can add a function

…to hll_default Conflicts: be/src/olap/aggregate_func.h

imay · 2019-09-23T10:57:08Z

+ const int HLL_EMPTY_SIZE = 1;
+ std::string buf;
+ std::unique_ptr<HyperLogLog> hll;
+ hll.reset(new HyperLogLog());


A stack variable is enough

# Conflicts: # be/src/exprs/hll_function.cpp

imay · 2019-09-24T13:46:11Z


 int64_t estimate_cardinality();

+ std::string empty() {


why not make this function static?

imay · 2019-09-24T13:46:22Z

 HLL_HASH(column_name)
 生成HLL列类型，用于insert或导入的时候，导入的使用见相关说明
+
+ EMPTY_HLL()


english doc

HangyuanLiu · 2019-09-27T06:39:41Z

#1895

hll default value
3eb2d82

kangkaisen reviewed Sep 19, 2019
View reviewed changes

HangyuanLiu added 3 commits September 23, 2019 18:33

add empty_hll
9996ebe

Merge branch 'master' ofhttps://github.com/apache/incubator-dorisin…
839b31f
…to hll_default Conflicts: be/src/olap/aggregate_func.h

fix
6ed3dfb

imay reviewed Sep 23, 2019
View reviewed changes

HangyuanLiu force-pushed the hll_default branch from 6ed3dfb to c8de048Compare September 24, 2019 13:35

fix
7105674

HangyuanLiu force-pushed the hll_default branch from c8de048 to 7105674Compare September 24, 2019 13:36

HangyuanLiu added 2 commits September 24, 2019 21:37

Add empty_hll doc
9bf922f

Merge remote-tracking branch 'origin/hll_default' into hll_default
fe966b0
# Conflicts: # be/src/exprs/hll_function.cpp

imay reviewed Sep 24, 2019
View reviewed changes

fix
25f3a6a

HangyuanLiu force-pushed the hll_default branch from fe966b0 to 25f3a6aCompare September 24, 2019 14:17

HangyuanLiu added 3 commits September 24, 2019 22:27

Merge remote-tracking branch 'origin/hll_default' into hll_default
6e2978a

Add empty_hll doc
c403e5d

Add empty_hll doc
cbe3153

imay approved these changes Sep 25, 2019
View reviewed changes

imay merged commit 40b9c35 into apache:masterSep 25, 2019

imay mentioned this pull request Sep 26, 2019
Release Notes 0.11.0 #1891
Closed

swjtu-zhanglei pushed a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Jul 25, 2023
(selectdb-cloud) Update recycler regression tests (apache#1825)
00b4f7a

Uh oh!

Conversation

HangyuanLiu commented Sep 18, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

imay commented Sep 18, 2019

Uh oh!

kangkaisenSep 19, 2019

Choose a reason for hiding this comment

Uh oh!

HangyuanLiuSep 19, 2019

Choose a reason for hiding this comment

Uh oh!

HangyuanLiu commented Sep 19, 2019

Uh oh!

kangkaisen commented Sep 19, 2019

Uh oh!

HangyuanLiu commented Sep 19, 2019

Uh oh!

kangkaisen commented Sep 19, 2019

Uh oh!

HangyuanLiu commented Sep 19, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

morningman commented Sep 21, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kangkaisen commented Sep 21, 2019

Uh oh!

imay commented Sep 21, 2019

Uh oh!

HangyuanLiu commented Sep 21, 2019

Uh oh!

imaySep 23, 2019

Choose a reason for hiding this comment

Uh oh!

imaySep 24, 2019

Choose a reason for hiding this comment

Uh oh!

imaySep 24, 2019

Choose a reason for hiding this comment

Uh oh!

HangyuanLiu commented Sep 27, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HangyuanLiu commented Sep 18, 2019•
edited
Loading

HangyuanLiu commented Sep 19, 2019•
edited
Loading

morningman commented Sep 21, 2019•
edited
Loading