Skip to content

[testcase](scalar_types_p0) make scalar_types_p0 testcases faster - #16104

Merged
dataroaring merged 1 commit into
apache:masterfrom
xiaokang:move_scalar_case
Jan 19, 2023
Merged

[testcase](scalar_types_p0) make scalar_types_p0 testcases faster#16104
dataroaring merged 1 commit into
apache:masterfrom
xiaokang:move_scalar_case

Conversation

@xiaokang

@xiaokangxiaokang commented Jan 19, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx

Problem summary

make scalar_types_p0 testcases faster by reducing dataset to 100k lines and move dataset with 1000k lines to scalar_types_p2. Runtime of scalar_types_p0 will be about 1m and scalar_types_p2 will be about 10m.

Checklist(Required)

  1. Does it affect the original behavior:
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:
    • Yes
    • No
    • No Need
  3. Has document been added or modified:
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies:
    • Yes
    • No
  5. Are there any changes that cannot be rolled back:
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@xiaokangxiaokang changed the title (testcase)[dataset] make scalar_types_p0 testcases faster[testcase](scalar_types_p0) make scalar_types_p0 testcases fasterJan 19, 2023
dataroaring
dataroaring approved these changes Jan 19, 2023
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 19, 2023
@github-actions

github-actionsBot commented Jan 19, 2023

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Jan 19, 2023

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@dataroaring
dataroaring merged commit 2018b49 into apache:masterJan 19, 2023
xiaokang added a commit to xiaokang/doris that referenced this pull request Jan 29, 2023
@xiaokang
[opt](test) scalar_types_p0 use 100k lines dataset and scalar_types_p…
1e047e0
…2 use 1000k (apache#16104)
dutyu pushed a commit to dutyu/doris that referenced this pull request Feb 1, 2023
@xiaokangwangxiangyu@360shuke.com
[opt](test) scalar_types_p0 use 100k lines dataset and scalar_types_p…
a897baf
…2 use 1000k (apache#16104)
luwei16 pushed a commit to luwei16/Doris that referenced this pull request Apr 7, 2023
@xiaokang
[pick](topn) Sync and pick topn from doris master (apache#1431)
652ccc7
Issue Number: close http://jira.selectdb-in.cc/browse/CORE-1462
Describe the overview of changes.
commit e1697741a82f875ca42b0d18caa7972eaa225bee
Author: Kang <kxiao.tiger@gmail.com>
Date: Thu Jan 19 22:59:29 2023 +0800
[opt](test) scalar_types_p0 use 100k lines dataset and scalar_types_p2 use 1000k (apache#16104)
commit 33a47e8d02644123ffd8c5c4353653c1c175e96a
Author: Kang <kxiao.tiger@gmail.com>
Date: Wed Jan 18 14:17:24 2023 +0800
[testcase](bitmap index)bitmap index testcase (apache#15975)
* add bitmap index testcases for all scalar types
commit 260a631441834ca7e23da4b77c922eb818eddca7
Author: Kang <kxiao.tiger@gmail.com>
Date: Mon Jan 16 16:49:59 2023 +0800
[regression-test](topn)add test cases for nonkey topn query for each scalar type (apache#15790)
related to apache#15558apache#15693
1. dup key table with 17 scalar datatypes
2. unique key table with mow enabled
3. unique key table with mow disabled
commit 81cea5219ae86df950f10aa123072df78c7cdf23
Author: Kang <kxiao.tiger@gmail.com>
Date: Sun Feb 19 23:28:33 2023 +0800
[bugfix](topn) fix topn read_orderby_key_columns nullptr (apache#16896)
The SQL `SELECT nationkey FROM regression_test_query_p0_limit.tpch_tiny_nation ORDER BY nationkey DESC LIMIT 5`
make be core dump since dereference a nullptr `read_orderby_key_columns in VCollectIterator::_topn_next`,
triggered by skipping _colname_to_value_range init in apache#16818 .
This PR makes two changes:
1. avoid read_orderby_key_columns nullptr in TabletReader::_init_orderby_keys_param
2. return error if read_orderby_key_columns is nullptr unexpected in VCollectIterator::_topn_next to avoid core dump
commit 2fee1d1d79942e49eddaafdc2b49e49b0651b109
Author: Kang <kxiao.tiger@gmail.com>
Date: Fri Feb 10 12:56:33 2023 +0800
[Improvement](topn) add limit threashold session variable and fuzzy for topn optimizations (apache#16514)
1. add limit threshold for topn runtime pushdown and key topn optimization
2. use unified session variable topn_opt_limit_threshold for all topn optimizations
3. add fuzzy support for topn_opt_limit_threshold
commit 1696bed39129fcc891f32f64ff1fb43f9531fcd4
Author: Kang <kxiao.tiger@gmail.com>
Date: Thu Feb 2 09:13:32 2023 +0800
[bugfix](topn) fix topn runtime predicate getting value bug for decimal type (apache#16331)
* fix topn runtime predicate getting value bug for decimal type
* fix cast_to_string bug for TYPE_DECIMALV2
commit d70cdf61521a23417c9bc734a3cdb668265a15b0
Author: Kang <kxiao.tiger@gmail.com>
Date: Wed Feb 22 16:18:46 2023 +0800
topn sync doris order by key topn query optimization apache#15663
commit 1df514c8f0b66ae9a8438617163a31848e519949
Author: Kang <kxiao.tiger@gmail.com>
Date: Wed Feb 22 15:14:43 2023 +0800
sync with doris runtime prune for topn query apache#15558
swjtu-zhanglei pushed a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Jul 25, 2023
@xiaokang@gavinchou
[pick](topn) Sync and pick topn from doris master (apache#1431)
99fbafb
Issue Number: close http://jira.selectdb-in.cc/browse/CORE-1462
Describe the overview of changes.
commit e1697741a82f875ca42b0d18caa7972eaa225bee
Author: Kang <kxiao.tiger@gmail.com>
Date: Thu Jan 19 22:59:29 2023 +0800
[opt](test) scalar_types_p0 use 100k lines dataset and scalar_types_p2 use 1000k (apache#16104)
commit 33a47e8d02644123ffd8c5c4353653c1c175e96a
Author: Kang <kxiao.tiger@gmail.com>
Date: Wed Jan 18 14:17:24 2023 +0800
[testcase](bitmap index)bitmap index testcase (apache#15975)
* add bitmap index testcases for all scalar types
commit 260a631441834ca7e23da4b77c922eb818eddca7
Author: Kang <kxiao.tiger@gmail.com>
Date: Mon Jan 16 16:49:59 2023 +0800
[regression-test](topn)add test cases for nonkey topn query for each scalar type (apache#15790)
related to apache#15558apache#15693
1. dup key table with 17 scalar datatypes
2. unique key table with mow enabled
3. unique key table with mow disabled
commit 81cea5219ae86df950f10aa123072df78c7cdf23
Author: Kang <kxiao.tiger@gmail.com>
Date: Sun Feb 19 23:28:33 2023 +0800
[bugfix](topn) fix topn read_orderby_key_columns nullptr (apache#16896)
The SQL `SELECT nationkey FROM regression_test_query_p0_limit.tpch_tiny_nation ORDER BY nationkey DESC LIMIT 5`
make be core dump since dereference a nullptr `read_orderby_key_columns in VCollectIterator::_topn_next`,
triggered by skipping _colname_to_value_range init in apache#16818 .
This PR makes two changes:
1. avoid read_orderby_key_columns nullptr in TabletReader::_init_orderby_keys_param
2. return error if read_orderby_key_columns is nullptr unexpected in VCollectIterator::_topn_next to avoid core dump
commit 2fee1d1d79942e49eddaafdc2b49e49b0651b109
Author: Kang <kxiao.tiger@gmail.com>
Date: Fri Feb 10 12:56:33 2023 +0800
[Improvement](topn) add limit threashold session variable and fuzzy for topn optimizations (apache#16514)
1. add limit threshold for topn runtime pushdown and key topn optimization
2. use unified session variable topn_opt_limit_threshold for all topn optimizations
3. add fuzzy support for topn_opt_limit_threshold
commit 1696bed39129fcc891f32f64ff1fb43f9531fcd4
Author: Kang <kxiao.tiger@gmail.com>
Date: Thu Feb 2 09:13:32 2023 +0800
[bugfix](topn) fix topn runtime predicate getting value bug for decimal type (apache#16331)
* fix topn runtime predicate getting value bug for decimal type
* fix cast_to_string bug for TYPE_DECIMALV2
commit d70cdf61521a23417c9bc734a3cdb668265a15b0
Author: Kang <kxiao.tiger@gmail.com>
Date: Wed Feb 22 16:18:46 2023 +0800
topn sync doris order by key topn query optimization apache#15663
commit 1df514c8f0b66ae9a8438617163a31848e519949
Author: Kang <kxiao.tiger@gmail.com>
Date: Wed Feb 22 15:14:43 2023 +0800
sync with doris runtime prune for topn query apache#15558
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.kind/testreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@xiaokang@dataroaring