[feat](mtmv)mtmv support paimon partition refresh by zddr · Pull Request #43959 · apache/doris

zddr · 2024-11-14T08:16:12Z

What problem does this PR solve?

Previously, when using Paimon to create MTMV, it was not possible to perceive changes in partition lists and data, so only refresh materialized view mv1 complete could be used to force full refresh.

This PR obtains the partition list of Paimon, the last update time of the partition, and the latest snapshotId of the table.

Therefore, MTMV can be partitioned based on Paimon tables and perceive changes in data, automatically refreshing partitions

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
mtmv support paimon partition refresh

Release note

mtmv support paimon partition refresh

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-14T08:16:17Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

…on_refresh1

zddr · 2024-11-14T08:26:16Z

run buildall

zddr · 2024-11-15T03:02:47Z

run buildall

zddr · 2024-11-15T10:58:53Z

run buildall

zddr · 2024-11-18T02:05:19Z

run external

zddr · 2024-11-18T06:38:11Z

run buildall

zddr · 2024-11-18T12:14:28Z

run buildall

zddr · 2024-11-19T02:22:27Z

run buildall

zddr · 2024-11-20T02:59:11Z

run buildall

doris-robot · 2024-11-20T03:54:56Z

TPC-H: Total hot run time: 45480 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 84cb3364c70fa46d6117a43136de8295be0a02b4, data reload: false
------ Round 1 ----------------------------------
q1	17731	7407	7331	7331
q2	2223	1164	1166	1164
q3	9972	1209	1186	1186
q4	10219	739	786	739
q5	7599	2751	2732	2732
q6	238	154	156	154
q7	987	620	599	599
q8	9364	2368	2360	2360
q9	6703	6519	6469	6469
q10	7018	2318	2347	2318
q11	470	267	264	264
q12	417	227	215	215
q13	17773	3046	3070	3046
q14	240	217	215	215
q15	583	536	530	530
q16	663	582	595	582
q17	1010	542	549	542
q18	7307	6725	6829	6725
q19	1334	1051	1017	1017
q20	2899	2691	2701	2691
q21	4026	3296	3264	3264
q22	1407	1337	1339	1337
Total cold run time: 110183 ms
Total hot run time: 45480 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7463	7294	7639	7294
q2	342	235	245	235
q3	3073	3063	2961	2961
q4	2084	1807	1828	1807
q5	5672	5720	5773	5720
q6	219	141	149	141
q7	2221	1831	1761	1761
q8	3321	3625	3620	3620
q9	8997	8948	8968	8948
q10	3616	3574	3596	3574
q11	590	496	492	492
q12	816	599	597	597
q13	11877	3244	3303	3244
q14	315	293	264	264
q15	573	527	539	527
q16	678	641	654	641
q17	1866	1641	1609	1609
q18	8261	7648	7639	7639
q19	1722	1547	1583	1547
q20	2129	1855	1908	1855
q21	5467	5457	5438	5438
q22	642	555	590	555
Total cold run time: 71944 ms
Total hot run time: 60469 ms

doris-robot · 2024-11-20T04:00:27Z

ClickBench: Total hot run time: 33.76 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 84cb3364c70fa46d6117a43136de8295be0a02b4, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.06	0.07
query4	1.62	0.11	0.10
query5	0.41	0.43	0.41
query6	1.17	0.66	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.57	0.49	0.50
query10	0.55	0.54	0.58
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.61
query14	2.72	2.72	2.85
query15	0.91	0.84	0.84
query16	0.38	0.38	0.37
query17	1.01	1.03	1.05
query18	0.20	0.20	0.21
query19	1.89	1.86	1.97
query20	0.01	0.01	0.01
query21	15.37	0.59	0.60
query22	2.81	2.66	2.47
query23	16.94	1.09	0.92
query24	2.76	1.75	1.24
query25	0.30	0.10	0.15
query26	0.38	0.14	0.14
query27	0.04	0.04	0.05
query28	10.15	1.13	1.10
query29	12.52	3.26	3.22
query30	0.25	0.06	0.06
query31	2.84	0.39	0.38
query32	3.27	0.46	0.47
query33	3.00	2.96	3.08
query34	16.89	4.52	4.60
query35	4.59	4.53	4.59
query36	0.68	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.15	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.99 s
Total hot run time: 33.76 s

github-actions · 2024-11-20T07:55:23Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-20T07:55:25Z

PR approved by anyone and no changes requested.

### What problem does this PR solve? Previously, when using Paimon to create MTMV, it was not possible to perceive changes in partition lists and data, so only `refresh materialized view mv1 complete` could be used to force full refresh. This PR obtains the partition list of Paimon, the last update time of the partition, and the latest snapshotId of the table. Therefore, MTMV can be partitioned based on Paimon tables and perceive changes in data, automatically refreshing partitions ### Release note mtmv support paimon partition refresh

…44911 (#45660) pick: #44911 #43959 only pick code about paimon, not pick some code about MTMV REFRESH

…43959 apache#44911 (apache#45660) pick: apache#44911 apache#43959 only pick code about paimon, not pick some code about MTMV REFRESH

) pick: #44911 #43959 only pick code about paimon, not pick some code about MTMV REFRESH

### What problem does this PR solve? `catalog.getPaimonTable(dbName, name);` will try forward request to master FE. When in replay logic, forwarding will fail, cause fe failed to start. Introduced from #43959

Previously, when using Paimon to create MTMV, it was not possible to perceive changes in partition lists and data, so only `refresh materialized view mv1 complete` could be used to force full refresh. This PR obtains the partition list of Paimon, the last update time of the partition, and the latest snapshotId of the table. Therefore, MTMV can be partitioned based on Paimon tables and perceive changes in data, automatically refreshing partitions mtmv support paimon partition refresh

PaimonUtil PaimonPartitionInfo PaimonSchemaCacheValue PaimonExternalTable use latest Previously, when using Paimon to create MTMV, it was not possible to perceive changes in partition lists and data, so only `refresh materialized view mv1 complete` could be used to force full refresh. This PR obtains the partition list of Paimon, the last update time of the partition, and the latest snapshotId of the table. Therefore, MTMV can be partitioned based on Paimon tables and perceive changes in data, automatically refreshing partitions mtmv support paimon partition refresh

Cherry-pick from #43959 #44419 #44415 #44567 #44673 #44998 #45273 #44911 #44726 #45652 #45659 #46257 #46641 #47026 #47166 #48172 #49956 #50979 --------- Co-authored-by: James <lijibing@selectdb.com> Co-authored-by: Tiewei Fang <fangtiewei@selectdb.com>

### What problem does this PR solve? Previously, when using Paimon to create MTMV, it was not possible to perceive changes in partition lists and data, so only `refresh materialized view mv1 complete` could be used to force full refresh. This PR obtains the partition list of Paimon, the last update time of the partition, and the latest snapshotId of the table. Therefore, MTMV can be partitioned based on Paimon tables and perceive changes in data, automatically refreshing partitions ### Release note mtmv support paimon partition refresh

zddr added 15 commits November 13, 2024 19:39

1
8485678

1
d1a6780

1
edbe24a

1
5af3e68

1
4ddad64

1
2be1729

1
0d58c44

1
e360a24

1
63313e1

1
6375815

1
f66b585

1
74f5e6d

1
426ffcb

1
3f48b29

1
9496c30

zddr added 2 commits November 14, 2024 16:24

1
aba5fb4

Merge remote-tracking branch 'origin/mv_paimon_refresh1' into mv_paim…
8eedc2b
…on_refresh1

zddr added 2 commits November 15, 2024 18:57

add ut
6d0d00b

Merge branch 'master' into mv_paimon_refresh1
b01e981

Merge branch 'master' into mv_paimon_refresh1
8224130

drop case test pipeline
2206584

drop case test pipeline
90b6ae9

Merge branch 'master' into mv_paimon_refresh1
84cb336

morrySnow approved these changes Nov 20, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 20, 2024

github-actionsBot added the reviewed label Nov 20, 2024

seawinde approved these changes Nov 20, 2024
View reviewed changes

morrySnow merged commit 95e9765 into apache:masterNov 21, 2024

zddr mentioned this pull request Dec 19, 2024
[feat](mtmv)mtmv support paimon partition refresh(#43959,#44911) #45660
Merged

morrySnow pushed a commit that referenced this pull request Dec 24, 2024
branch-3.0: [feat](mtmv)mtmv support paimon partition refresh#43959#…
f4e5727
…44911 (#45660) pick: #44911 #43959 only pick code about paimon, not pick some code about MTMV REFRESH

morrySnow added the dev/3.0.4-merged label Dec 24, 2024

zddr mentioned this pull request Dec 24, 2024
[feat](mtmv)mtmv support paimon partition refresh(#43959,#44911) #45878
Merged

yiguolei pushed a commit that referenced this pull request Dec 25, 2024
[feat](mtmv)mtmv support paimon partition refresh(#43959,#44911) (#45878
fdb6519
) pick: #44911 #43959 only pick code about paimon, not pick some code about MTMV REFRESH

yiguolei added the dev/2.1.8-merged label Dec 25, 2024

This was referenced Feb 20, 2025
branch-2.1: [fix](paimon) avoid get paimon table when replay #48151
Merged
branch-3.0: [fix](paimon) avoid get paimon table when replay #48152
Merged

zddr mentioned this pull request Jun 20, 2025
branch-3.1:[enhance](mtmv)MTMV support iceberg/hudi/paimon #52040
Merged

morrySnow added dev/3.1.0-merged and removed dev/3.1.0-merged labels Jun 23, 2025

Navigation Menu

Uh oh!

[feat](mtmv)mtmv support paimon partition refresh - #43959

[feat](mtmv)mtmv support paimon partition refresh#43959
morrySnow merged 29 commits into
apache:masterfrom
zddr:mv_paimon_refresh1

zddr commented Nov 14, 2024•
edited
Loading

Uh oh!

doris-robot commented Nov 14, 2024

Uh oh!

zddr commented Nov 14, 2024

Uh oh!

zddr commented Nov 15, 2024

Uh oh!

zddr commented Nov 15, 2024

Uh oh!

zddr commented Nov 18, 2024

Uh oh!

zddr commented Nov 18, 2024

Uh oh!

zddr commented Nov 18, 2024

Uh oh!

zddr commented Nov 19, 2024

Uh oh!

zddr commented Nov 20, 2024

Uh oh!

doris-robot commented Nov 20, 2024

Uh oh!

doris-robot commented Nov 20, 2024

Uh oh!

github-actionsBot commented Nov 20, 2024

Uh oh!

github-actionsBot commented Nov 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

zddr commented Nov 14, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 14, 2024

Uh oh!

zddr commented Nov 14, 2024

Uh oh!

zddr commented Nov 15, 2024

Uh oh!

zddr commented Nov 15, 2024

Uh oh!

zddr commented Nov 18, 2024

Uh oh!

zddr commented Nov 18, 2024

Uh oh!

zddr commented Nov 18, 2024

Uh oh!

zddr commented Nov 19, 2024

Uh oh!

zddr commented Nov 20, 2024

Uh oh!

doris-robot commented Nov 20, 2024

Uh oh!

doris-robot commented Nov 20, 2024

Uh oh!

github-actionsBot commented Nov 20, 2024

Uh oh!

github-actionsBot commented Nov 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zddr commented Nov 14, 2024•
edited
Loading