[enhance](orc) Optimize ORC Predicate Pushdown for OR-connected Predicate by suxiaogang223 · Pull Request #43255 · apache/doris

suxiaogang223 · 2024-11-05T06:56:03Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
This issue addresses a limitation in Apache Doris where only predicates joined by AND are pushed down to the ORC reader, leaving OR-connected predicates unoptimized. By extending pushdown functionality to handle these OR conditions, the aim is to better leverage ORC’s predicate pushdown capabilities, reducing data reads and improving query performance.

Check List (For Committer)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No colde files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.
Release note
None

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-05T06:56:08Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

suxiaogang223 · 2024-11-06T08:14:07Z

run buildall

doris-robot · 2024-11-06T08:37:07Z

TeamCity be ut coverage result:
Function Coverage: 37.88% (9849/26001)
Line Coverage: 29.03% (81909/282154)
Region Coverage: 28.24% (42185/149374)
Branch Coverage: 24.83% (21408/86206)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5ebe1d5c1da127bdc70c7a871ced06d390ccce62_5ebe1d5c1da127bdc70c7a871ced06d390ccce62/report/index.html

github-actions

clang-tidy made some suggestions

suxiaogang223 · 2024-11-06T17:54:09Z

run buildall

github-actions

clang-tidy made some suggestions

suxiaogang223 · 2024-11-07T01:55:21Z

run buildall

suxiaogang223 · 2024-11-07T14:32:44Z

run buildall

doris-robot · 2024-11-07T16:08:09Z

TeamCity be ut coverage result:
Function Coverage: 38.11% (9910/26007)
Line Coverage: 29.23% (82500/282281)
Region Coverage: 28.45% (42525/149455)
Branch Coverage: 25.07% (21622/86260)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5056b99e1268c1ab03ba3f30355c2f082a8ae5f1_5056b99e1268c1ab03ba3f30355c2f082a8ae5f1/report/index.html

suxiaogang223 · 2024-11-08T06:31:26Z

run external

suxiaogang223 · 2024-11-08T08:04:01Z

run external

github-actions

clang-tidy made some suggestions

suxiaogang223 · 2024-11-08T13:09:34Z

run external

suxiaogang223 · 2024-11-10T12:53:30Z

run buildall

doris-robot · 2024-11-10T14:27:22Z

TeamCity be ut coverage result:
Function Coverage: 38.15% (9931/26030)
Line Coverage: 29.30% (82887/282911)
Region Coverage: 28.45% (42696/150068)
Branch Coverage: 25.03% (21667/86562)
Coverage Report: http://coverage.selectdb-in.cc/coverage/22388b7bbe19673f8616d7cb9be19701415fc9b8_22388b7bbe19673f8616d7cb9be19701415fc9b8/report/index.html

morningman · 2024-11-10T14:48:52Z

-bool OrcReader::_init_search_argument(
- std::unordered_map<std::string, ColumnValueRangeType>* colname_to_value_range) {
- if ((!_enable_filter_by_min_max) || colname_to_value_range->empty()) {
+bool OrcReader::_init_search_argument(const VExprContextSPtrs& conjuncts) {


For a certain query, the conjuncts is identical in all orc readers.
So I think we can just init search argument once and use the result for every orc reader?

it is possible to cache searchArgument into member variables of OrcReader

github-actions

clang-tidy made some suggestions

suxiaogang223 · 2024-11-11T04:14:49Z

run external

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

### What problem does this PR solve? Related PR: apache#43255 Problem Summary: Example: ```sql CREATE TABLE table_a ( id INT, age INT ) STORED AS ORC; INSERT INTO table_a VALUES (1, null), (2, 18), (3, null), (4, 25); CREATE TABLE table_b ( id INT, age INT ) STORED AS ORC; INSERT INTO table_b VALUES (1, null), (2, null), (3, 1000000), (4, 100); ``` run sql ``` select * from table_a inner join table_b on table_a.age <=> table_b.age and table_b.id in (1,3); ``` When executing this SQL, the backend generates a runtime filter on the table_a side during the join operation, resulting in a condition like WHERE table_a.age IN (NULL, 1000000). It’s important to note that since <=> is a null-aware comparison operator, the IN predicate must also be null-aware. However, the ORC predicate pushdown API does not support null-aware IN predicates. As a result, our current approach ignores null values, leading to an empty result set for this query. To fix this bug, we’ve adjusted the logic so that predicates with null-aware comparisons are not pushed down, ensuring the correct result as follows: ```text +------+------+------+------+ | id | age | id | age | +------+------+------+------+ | 1 | NULL | 1 | NULL | | 3 | NULL | 1 | NULL | +------+------+------+------+ ```

### What problem does this PR solve? ### Release note relate pr: apache#43255 Improved ACID table column handling - Added support for ACID column prefix in ORC column initialization - Fixed column name handling for ACID tables - Improved type mapping for ACID table columns

…cate (apache#43255) Problem Summary: This issue addresses a limitation in Apache Doris where only predicates joined by AND are pushed down to the ORC reader, leaving OR-connected predicates unoptimized. By extending pushdown functionality to handle these OR conditions, the aim is to better leverage ORC’s predicate pushdown capabilities, reducing data reads and improving query performance.

…4615) In the old logic, the `check_expr_can_push_down` function does not check whether the `orc::Literal` are constructed successfully, but only checks during `build_search_argument`. However, if it is found that the `orc::Literal` fails to be constructed after `builder->startNot`, it will fail because the builder cannot end `startNot`. Therefore, we advance the behavior of constructing `orc::Literal` to the `check_expr_can_push_down` function and save the result to the map, so that it will never fail in the `build_search_argument` phase. Related PR: apache#43255

Related PR: apache#43255 Problem Summary: Example: ```sql CREATE TABLE table_a ( id INT, age INT ) STORED AS ORC; INSERT INTO table_a VALUES (1, null), (2, 18), (3, null), (4, 25); CREATE TABLE table_b ( id INT, age INT ) STORED AS ORC; INSERT INTO table_b VALUES (1, null), (2, null), (3, 1000000), (4, 100); ``` run sql ``` select * from table_a inner join table_b on table_a.age <=> table_b.age and table_b.id in (1,3); ``` When executing this SQL, the backend generates a runtime filter on the table_a side during the join operation, resulting in a condition like WHERE table_a.age IN (NULL, 1000000). It’s important to note that since <=> is a null-aware comparison operator, the IN predicate must also be null-aware. However, the ORC predicate pushdown API does not support null-aware IN predicates. As a result, our current approach ignores null values, leading to an empty result set for this query. To fix this bug, we’ve adjusted the logic so that predicates with null-aware comparisons are not pushed down, ensuring the correct result as follows: ```text +------+------+------+------+ | id | age | id | age | +------+------+------+------+ | 1 | NULL | 1 | NULL | | 3 | NULL | 1 | NULL | +------+------+------+------+ ```

…ins (apache#45104) Related PR: apache#43255 Problem Summary: Should ignore null values when the literals of in_predicate contains null value, like `in (1, null)` For example, init table in hive: ```sql CREATE TABLE sample_orc_table ( id INT, name STRING, age INT ) STORED AS ORC; INSERT INTO TABLE sample_orc_table VALUES (1, 'Alice', 25), (2, NULL, NULL); ``` select result in Doris should be: ```sql mysql> select * from sample_orc_table where age in (null,25); +------+-------+------+ | id | name | age | +------+-------+------+ | 1 | Alice | 25 | +------+-------+------+ 1 row in set (0.30 sec) mysql> select * from sample_orc_table where age in (25); +------+-------+------+ | id | name | age | +------+-------+------+ | 1 | Alice | 25 | +------+-------+------+ 1 row in set (0.27 sec) mysql> select * from sample_orc_table where age in (null); Empty set (0.01 sec) mysql> select * from sample_orc_table where age is null; +------+------+------+ | id | name | age | +------+------+------+ | 2 | NULL | NULL | +------+------+------+ 1 row in set (0.11 sec) ```

relate pr: apache#43255 Improved ACID table column handling - Added support for ACID column prefix in ORC column initialization - Fixed column name handling for ACID tables - Improved type mapping for ACID table columns