[chore](fe) Returns dropped tables in GetMeta request by w41ter · Pull Request #38019 · apache/doris

w41ter · 2024-07-17T12:08:24Z

The CCR syncer needs to know the distribution of tables, partitions, indexes, and replicas when synchronizing binlogs. If a table is deleted before the binlog synchronization is complete, the CCR syncer cannot continue synchronizing. This PR will record the deleted tables and include them in the get meta response, allowing the CCR syncer to filter out the binlogs that belong to these tables.

The CCR syncer part PR is selectdb/ccr-syncer#123.

The CCR syncer needs to know the distribution of tables, partitions, indexes, and replicas when synchronizing binlogs. If a table is deleted before the binlog synchronization is complete, the CCR syncer cannot continue synchronizing. This PR will record the deleted tables and include them in the get meta response, allowing the CCR syncer to filter out the binlogs that belong to these tables. The CCR syncer part PR is selectdb/ccr-syncer#123.

doris-robot · 2024-07-17T12:08:30Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

w41ter · 2024-07-17T12:08:35Z

run buildall

github-actions · 2024-07-17T12:14:08Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-17T13:49:56Z

TPC-H: Total hot run time: 41723 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ff89966af5a17bc56e150b9d203a2fed5ce7f137, data reload: false
------ Round 1 ----------------------------------
q1	18157	4619	4565	4565
q2	2026	198	202	198
q3	10517	1295	1240	1240
q4	10233	884	815	815
q5	7568	2842	2791	2791
q6	240	139	141	139
q7	965	623	631	623
q8	9225	2286	2314	2286
q9	9036	6856	6855	6855
q10	8887	3926	3892	3892
q11	461	238	241	238
q12	438	223	236	223
q13	17767	3015	3041	3015
q14	290	244	247	244
q15	546	471	501	471
q16	510	399	390	390
q17	1043	693	661	661
q18	8273	7598	7500	7500
q19	7295	1521	1481	1481
q20	700	324	327	324
q21	5170	3467	4138	3467
q22	364	316	305	305
Total cold run time: 119711 ms
Total hot run time: 41723 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4643	4458	4579	4458
q2	422	280	274	274
q3	3322	3020	3060	3020
q4	2138	1714	1785	1714
q5	5660	5630	5754	5630
q6	243	144	137	137
q7	2345	1858	1898	1858
q8	3553	3737	3751	3737
q9	8927	8906	8937	8906
q10	4197	4041	4053	4041
q11	625	534	496	496
q12	879	671	646	646
q13	17358	3228	3146	3146
q14	321	301	295	295
q15	552	501	513	501
q16	517	472	443	443
q17	2011	1641	1584	1584
q18	8212	8029	7906	7906
q19	1916	1794	1791	1791
q20	2227	1936	1902	1902
q21	5265	4960	5139	4960
q22	625	561	516	516
Total cold run time: 75958 ms
Total hot run time: 57961 ms

doris-robot · 2024-07-17T14:01:08Z

TPC-DS: Total hot run time: 175157 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ff89966af5a17bc56e150b9d203a2fed5ce7f137, data reload: false
query1	923	375	366	366
query2	6815	1831	1887	1831
query3	6634	218	221	218
query4	28355	17640	17601	17601
query5	3813	483	478	478
query6	271	178	158	158
query7	4582	294	295	294
query8	247	194	198	194
query9	8423	2390	2357	2357
query10	431	267	291	267
query11	10563	10077	10211	10077
query12	111	87	81	81
query13	1637	363	363	363
query14	10104	7895	8110	7895
query15	229	165	170	165
query16	7168	327	322	322
query17	1409	547	519	519
query18	1752	288	294	288
query19	192	148	147	147
query20	88	80	79	79
query21	205	136	125	125
query22	4366	4105	4102	4102
query23	34218	33752	33645	33645
query24	9316	2959	2936	2936
query25	606	393	407	393
query26	705	148	148	148
query27	2264	284	282	282
query28	5925	2072	2063	2063
query29	898	644	634	634
query30	266	162	153	153
query31	983	784	748	748
query32	97	56	58	56
query33	628	319	372	319
query34	894	500	500	500
query35	684	610	602	602
query36	1140	1008	968	968
query37	146	82	86	82
query38	2999	2905	2877	2877
query39	899	861	856	856
query40	206	122	121	121
query41	46	45	44	44
query42	114	102	96	96
query43	500	461	482	461
query44	1061	720	712	712
query45	195	165	158	158
query46	1087	716	721	716
query47	1851	1824	1798	1798
query48	372	285	284	284
query49	815	410	446	410
query50	777	393	390	390
query51	6942	6851	6672	6672
query52	107	107	94	94
query53	355	292	286	286
query54	748	445	445	445
query55	77	78	73	73
query56	281	259	275	259
query57	1126	1084	1076	1076
query58	255	253	245	245
query59	2863	2619	2573	2573
query60	302	269	278	269
query61	97	94	98	94
query62	784	636	656	636
query63	327	301	290	290
query64	9188	2296	7567	2296
query65	3144	3130	3106	3106
query66	765	332	321	321
query67	15631	14987	15066	14987
query68	4504	530	531	530
query69	473	317	335	317
query70	1152	1193	1158	1158
query71	368	270	281	270
query72	6990	5557	5692	5557
query73	723	332	324	324
query74	6135	5817	5715	5715
query75	3374	2691	2660	2660
query76	2134	952	940	940
query77	436	305	317	305
query78	9659	9082	8985	8985
query79	2688	532	517	517
query80	2702	480	475	475
query81	609	227	219	219
query82	832	132	138	132
query83	306	179	165	165
query84	283	91	84	84
query85	2025	318	304	304
query86	475	307	323	307
query87	3302	3135	3153	3135
query88	3847	2388	2363	2363
query89	482	393	391	391
query90	1729	197	192	192
query91	141	111	114	111
query92	64	52	54	52
query93	3570	516	511	511
query94	1155	225	226	225
query95	417	323	331	323
query96	615	280	272	272
query97	3189	3073	3081	3073
query98	240	203	196	196
query99	1674	1271	1248	1248
Total cold run time: 277295 ms
Total hot run time: 175157 ms

doris-robot · 2024-07-17T14:06:41Z

ClickBench: Total hot run time: 30.91 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ff89966af5a17bc56e150b9d203a2fed5ce7f137, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.69	0.08	0.08
query5	0.51	0.50	0.49
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.56	0.52	0.55
query11	0.15	0.11	0.12
query12	0.16	0.13	0.13
query13	0.60	0.58	0.58
query14	0.77	0.79	0.79
query15	0.86	0.81	0.82
query16	0.36	0.37	0.37
query17	1.04	1.05	1.02
query18	0.23	0.22	0.21
query19	1.92	1.85	1.84
query20	0.01	0.01	0.01
query21	15.45	0.76	0.66
query22	4.33	6.58	1.89
query23	18.28	1.38	1.25
query24	2.06	0.25	0.23
query25	0.15	0.09	0.08
query26	0.29	0.21	0.22
query27	0.45	0.23	0.23
query28	13.25	1.03	1.01
query29	12.66	3.31	3.29
query30	0.25	0.06	0.06
query31	2.86	0.40	0.39
query32	3.27	0.48	0.46
query33	2.87	2.94	2.89
query34	17.12	4.36	4.37
query35	4.41	4.49	4.51
query36	0.65	0.48	0.48
query37	0.18	0.15	0.16
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.11	0.12
query41	0.10	0.05	0.05
query42	0.06	0.06	0.06
query43	0.05	0.04	0.04
Total cold run time: 110.05 s
Total hot run time: 30.91 s

dataroaring

LGTM

github-actions · 2024-07-17T14:32:30Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-17T14:32:32Z

PR approved by anyone and no changes requested.

The CCR syncer needs to know the distribution of tables, partitions, indexes, and replicas when synchronizing binlogs. If a table is deleted before the binlog synchronization is complete, the CCR syncer cannot continue synchronizing. This PR will record the deleted tables and include them in the get meta response, allowing the CCR syncer to filter out the binlogs that belong to these tables. The CCR syncer part PR is selectdb/ccr-syncer#123.

Cherry-pick apache#38019

Cherry-pick #38019

Cherry-pick apache#38019

The CCR syncer needs to know the distribution of tables, partitions, indexes, and replicas when synchronizing binlogs. If a table is deleted before the binlog synchronization is complete, the CCR syncer cannot continue synchronizing. This PR will record the deleted tables and include them in the get meta response, allowing the CCR syncer to filter out the binlogs that belong to these tables. The CCR syncer part PR is selectdb/ccr-syncer#123.

w41ter requested review from CalvinKirs, dataroaring and morningman as code owners July 17, 2024 12:08

github-actionsBot added the meta-change label Jul 17, 2024

dataroaring approved these changes Jul 17, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 17, 2024

github-actionsBot added the reviewed label Jul 17, 2024

CalvinKirs approved these changes Jul 17, 2024
View reviewed changes

w41ter merged commit ddcbb1c into masterJul 18, 2024

w41ter deleted the filter_dropped_tables branch July 18, 2024 02:01

w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 30, 2024
[chore](fe) Returns dropped tables in GetMeta request
6048814
Cherry-pick apache#38019

w41ter mentioned this pull request Jul 30, 2024
[chore](fe) Returns dropped tables in GetMeta request #38541
Merged

w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 30, 2024
[chore](fe) Returns dropped tables in GetMeta request
a29afde
Cherry-pick apache#38019

w41ter mentioned this pull request Jul 30, 2024
[chore](fe) Returns dropped tables in GetMeta request #38542
Merged

w41ter added a commit that referenced this pull request Jul 31, 2024
[chore](fe) Returns dropped tables in GetMeta request (#38542)
ce7c979
Cherry-pick #38019

w41ter added the dev/2.0.14-merged label Jul 31, 2024

w41ter added a commit that referenced this pull request Jul 31, 2024
[chore](fe) Returns dropped tables in GetMeta request (#38541)
182bf4d
Cherry-pick #38019

w41ter added the dev/2.1.6-merged label Jul 31, 2024

dataroaring added the dev/3.0.1-merged label Aug 5, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
[chore](fe) Returns dropped tables in GetMeta request (apache#38542)
37d7d96
Cherry-pick apache#38019

Navigation Menu

Uh oh!

[chore](fe) Returns dropped tables in GetMeta request - #38019

[chore](fe) Returns dropped tables in GetMeta request#38019
w41ter merged 1 commit into
masterfrom
filter_dropped_tables

w41ter commented Jul 17, 2024

Uh oh!

doris-robot commented Jul 17, 2024

Uh oh!

w41ter commented Jul 17, 2024

Uh oh!

github-actionsBot commented Jul 17, 2024

Uh oh!

doris-robot commented Jul 17, 2024

Uh oh!

doris-robot commented Jul 17, 2024

Uh oh!

doris-robot commented Jul 17, 2024

Uh oh!

dataroaring left a comment

Uh oh!

github-actionsBot commented Jul 17, 2024

Uh oh!

github-actionsBot commented Jul 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

w41ter commented Jul 17, 2024

Uh oh!

doris-robot commented Jul 17, 2024

Uh oh!

w41ter commented Jul 17, 2024

Uh oh!

github-actionsBot commented Jul 17, 2024

Uh oh!

doris-robot commented Jul 17, 2024

Uh oh!

doris-robot commented Jul 17, 2024

Uh oh!

doris-robot commented Jul 17, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 17, 2024

Uh oh!

github-actionsBot commented Jul 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants