Uh oh!
There was an error while loading. Please reload this page.
[improvement](mtmv) Support to use current_date() when create async mv - #36111
Merged
morrySnow merged 7 commits intoJun 21, 2024
Merged
Conversation
doris-robot
commented
Jun 11, 2024
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
seawinde
commented
Jun 11, 2024
MemberAuthor
run buildall |
doris-robot
commented
Jun 11, 2024
TPC-H: Total hot run time: 39959 ms |
doris-robot
commented
Jun 11, 2024
TPC-DS: Total hot run time: 169159 ms |
doris-robot
commented
Jun 11, 2024
ClickBench: Total hot run time: 30.71 s |
seawinde
commented
Jun 11, 2024
MemberAuthor
run buildall |
doris-robot
commented
Jun 11, 2024
TPC-H: Total hot run time: 40702 ms |
doris-robot
commented
Jun 11, 2024
TPC-DS: Total hot run time: 173055 ms |
doris-robot
commented
Jun 11, 2024
ClickBench: Total hot run time: 30.72 s |
morrySnow
commented
Jun 12, 2024
Contributor
support all nondeterministic function ? |
seawinde
commented
Jun 12, 2024
MemberAuthor
run buildall |
doris-robot
commented
Jun 12, 2024
TPC-H: Total hot run time: 39864 ms |
doris-robot
commented
Jun 12, 2024
TPC-DS: Total hot run time: 170309 ms |
doris-robot
commented
Jun 12, 2024
ClickBench: Total hot run time: 30.39 s |
| * Identify the function is deterministic or not, such as UnixTimestamp, when it's children is not empty | ||
| * it's deterministic | ||
| */ | ||
| boolean isDeterministic(); |
Contributor
There was a problem hiding this comment.
add this interface to expression with default implement check all check isDeterministic. Nondeterministic default return false. unixTimestamp do not implement Nondeterministic anymore. all check isntance of Nondeterministic change to check expression.isDeterministic().
| * the expression in whiteFunctionSet would not be collected | ||
| */ | ||
| public static class FunctionCollectContext { | ||
| private final List<Expression> collectedExpressions = new LinkedList<>(); |
Contributor
There was a problem hiding this comment.
do not use java's linkedList in any case. it has poor perf with no adv with ArrayList
| return collectedExpressions; | ||
| } | ||
| expressions.forEach(expression -> { | ||
| collectedExpressions.addAll(expression.collect(Nondeterministic.class::isInstance)); |
Contributor
There was a problem hiding this comment.
i think u could collected isDeterministic == true after move this interface into expression
seawinde
commented
Jun 12, 2024
MemberAuthor
run buildall |
doris-robot
commented
Jun 12, 2024
TPC-H: Total hot run time: 39478 ms |
doris-robot
commented
Jun 12, 2024
TPC-DS: Total hot run time: 174267 ms |
doris-robot
commented
Jun 12, 2024
ClickBench: Total hot run time: 30.69 s |
| * Identify the expression is deterministic or not | ||
| */ | ||
| default boolean isDeterministic() { | ||
| return true; |
Contributor
There was a problem hiding this comment.
recursive call children's isDeterministic
seawinde
commented
Jun 17, 2024
MemberAuthor
run buildall |
doris-robot
commented
Jun 17, 2024
TPC-H: Total hot run time: 40459 ms |
seawinde
commented
Jun 17, 2024
MemberAuthor
run buildall |
1 similar comment
seawinde
commented
Jun 18, 2024
MemberAuthor
run buildall |
seawinde
commented
Jun 18, 2024
MemberAuthor
run feut |
seawinde
commented
Jun 20, 2024
MemberAuthor
run buildall |
doris-robot
commented
Jun 20, 2024
TPC-H: Total hot run time: 39522 ms |
doris-robot
commented
Jun 20, 2024
TPC-DS: Total hot run time: 173457 ms |
doris-robot
commented
Jun 20, 2024
ClickBench: Total hot run time: 30.8 s |
zddr
approved these changes
Jun 20, 2024
zddr
left a comment
Contributor
There was a problem hiding this comment.
It may be strange to set enable_dondeterministic_function to false through the alter statement after creating a materialized view, but it does not affect the business logic
Contributor
PR approved by anyone and no changes requested. |
morrySnow
approved these changes
Jun 21, 2024
Contributor
PR approved by at least one committer and no changes requested. |
dataroaring pushed a commit
that referenced
this pull request
Jun 21, 2024
…async mv (#36111) Support to use current_date() when create async materialized view by adding 'enable_nondeterministic_function' = 'true' in properties when create materialized view. `enable_nondeterministic_function` is default false. Here is a example, it will success > CREATE MATERIALIZED VIEW mv_name > BUILD DEFERRED REFRESH AUTO ON MANUAL > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ( > 'replication_num' = '1', > 'enable_nondeterministic_function' = 'true' > ) > AS > SELECT *, unix_timestamp(k3, '%Y-%m-%d %H:%i-%s') from ${tableName} where current_date() > k3; Note: unix_timestamp is nondeterministic when has no params. it is deterministic when has params which means format column k3 date another example, it will success > CREATE MATERIALIZED VIEW mv_name > BUILD DEFERRED REFRESH AUTO ON MANUAL > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ( > 'replication_num' = '1', > 'enable_nondeterministic_function' = 'true' > ) > AS > SELECT *, unix_timestamp() from ${tableName} where current_date() > k3; though unix_timestamp() is nondeterministic, we add 'enable_date_nondeterministic_function' = 'true' in properties
seawinde added a commit
to seawinde/doris
that referenced
this pull request
Jul 11, 2024
…async mv (apache#36111) Support to use current_date() when create async materialized view by adding 'enable_nondeterministic_function' = 'true' in properties when create materialized view. `enable_nondeterministic_function` is default false. Here is a example, it will success > CREATE MATERIALIZED VIEW mv_name > BUILD DEFERRED REFRESH AUTO ON MANUAL > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ( > 'replication_num' = '1', > 'enable_nondeterministic_function' = 'true' > ) > AS > SELECT *, unix_timestamp(k3, '%Y-%m-%d %H:%i-%s') from ${tableName} where current_date() > k3; Note: unix_timestamp is nondeterministic when has no params. it is deterministic when has params which means format column k3 date another example, it will success > CREATE MATERIALIZED VIEW mv_name > BUILD DEFERRED REFRESH AUTO ON MANUAL > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ( > 'replication_num' = '1', > 'enable_nondeterministic_function' = 'true' > ) > AS > SELECT *, unix_timestamp() from ${tableName} where current_date() > k3; though unix_timestamp() is nondeterministic, we add 'enable_date_nondeterministic_function' = 'true' in properties
morrySnow pushed a commit
that referenced
this pull request
Jul 12, 2024
starocean999 pushed a commit
that referenced
this pull request
Aug 26, 2024
…a expression is nondeterministic or not (#39801) ## Proposed changes In #36111, we add `isDeterministic` method in class `ExpressionTrait` to identify the expression is deterministic or not. But `unix_timestamp` doesn't extend Nondeterministic, but it is not deterministic when it's children is empty. and is not deterministic when children is not empty. If we use` instanceOf Nondeterministic `to indentify if expression is is not deterministic, that is confused. So we do something as fllowing: 1. Remove Nondeterministic class, and use `isDeterministic` to indentify it's deterministic. 2. Add `containsNondeterministic` method in `ExpressionTrait` to identify it contains nondeterministic expression or not. 3. `isDeterministic` only identify current expression is deterministic or not. would identify if contains nondeterministic or not
seawinde added a commit
to seawinde/doris
that referenced
this pull request
Aug 29, 2024
…a expression is nondeterministic or not (apache#39801) ## Proposed changes In apache#36111, we add `isDeterministic` method in class `ExpressionTrait` to identify the expression is deterministic or not. But `unix_timestamp` doesn't extend Nondeterministic, but it is not deterministic when it's children is empty. and is not deterministic when children is not empty. If we use` instanceOf Nondeterministic `to indentify if expression is is not deterministic, that is confused. So we do something as fllowing: 1. Remove Nondeterministic class, and use `isDeterministic` to indentify it's deterministic. 2. Add `containsNondeterministic` method in `ExpressionTrait` to identify it contains nondeterministic expression or not. 3. `isDeterministic` only identify current expression is deterministic or not. would identify if contains nondeterministic or not
yiguolei pushed a commit
to seawinde/doris
that referenced
this pull request
Aug 29, 2024
…a expression is nondeterministic or not (apache#39801) ## Proposed changes In apache#36111, we add `isDeterministic` method in class `ExpressionTrait` to identify the expression is deterministic or not. But `unix_timestamp` doesn't extend Nondeterministic, but it is not deterministic when it's children is empty. and is not deterministic when children is not empty. If we use` instanceOf Nondeterministic `to indentify if expression is is not deterministic, that is confused. So we do something as fllowing: 1. Remove Nondeterministic class, and use `isDeterministic` to indentify it's deterministic. 2. Add `containsNondeterministic` method in `ExpressionTrait` to identify it contains nondeterministic expression or not. 3. `isDeterministic` only identify current expression is deterministic or not. would identify if contains nondeterministic or not
dataroaring pushed a commit
that referenced
this pull request
Sep 26, 2024
…a expression is nondeterministic or not (#39801) ## Proposed changes In #36111, we add `isDeterministic` method in class `ExpressionTrait` to identify the expression is deterministic or not. But `unix_timestamp` doesn't extend Nondeterministic, but it is not deterministic when it's children is empty. and is not deterministic when children is not empty. If we use` instanceOf Nondeterministic `to indentify if expression is is not deterministic, that is confused. So we do something as fllowing: 1. Remove Nondeterministic class, and use `isDeterministic` to indentify it's deterministic. 2. Add `containsNondeterministic` method in `ExpressionTrait` to identify it contains nondeterministic expression or not. 3. `isDeterministic` only identify current expression is deterministic or not. would identify if contains nondeterministic or not
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
…async mv (apache#36111) Support to use current_date() when create async materialized view by adding 'enable_nondeterministic_function' = 'true' in properties when create materialized view. `enable_nondeterministic_function` is default false. Here is a example, it will success > CREATE MATERIALIZED VIEW mv_name > BUILD DEFERRED REFRESH AUTO ON MANUAL > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ( > 'replication_num' = '1', > 'enable_nondeterministic_function' = 'true' > ) > AS > SELECT *, unix_timestamp(k3, '%Y-%m-%d %H:%i-%s') from ${tableName} where current_date() > k3; Note: unix_timestamp is nondeterministic when has no params. it is deterministic when has params which means format column k3 date another example, it will success > CREATE MATERIALIZED VIEW mv_name > BUILD DEFERRED REFRESH AUTO ON MANUAL > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ( > 'replication_num' = '1', > 'enable_nondeterministic_function' = 'true' > ) > AS > SELECT *, unix_timestamp() from ${tableName} where current_date() > k3; though unix_timestamp() is nondeterministic, we add 'enable_date_nondeterministic_function' = 'true' in properties
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
…a expression is nondeterministic or not (apache#39801) ## Proposed changes In apache#36111, we add `isDeterministic` method in class `ExpressionTrait` to identify the expression is deterministic or not. But `unix_timestamp` doesn't extend Nondeterministic, but it is not deterministic when it's children is empty. and is not deterministic when children is not empty. If we use` instanceOf Nondeterministic `to indentify if expression is is not deterministic, that is confused. So we do something as fllowing: 1. Remove Nondeterministic class, and use `isDeterministic` to indentify it's deterministic. 2. Add `containsNondeterministic` method in `ExpressionTrait` to identify it contains nondeterministic expression or not. 3. `isDeterministic` only identify current expression is deterministic or not. would identify if contains nondeterministic or not
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Support to use current_date() when create async materialized view by adding
'enable_nondeterministic_function' = 'true'in properties when create materialized view.enable_nondeterministic_functionis default false.Here is a example, it will success
Note:
unix_timestampis nondeterministic when has no params. it is deterministic when has params which means format column k3 dateanother example, it will success
though
unix_timestamp()is nondeterministic, we add'enable_date_nondeterministic_function' = 'true'in properties