[chore][fe] Get meta returns dropped partitions in binlog manager by w41ter · Pull Request #37196 · apache/doris

w41ter · 2024-07-03T03:51:06Z

The CCR syncer needs to know the distribution of tables, partitions, indexes, and replicas when synchronizing binlogs. If a partition is deleted before the binlog synchronization is complete, the CCR syncer cannot continue synchronizing. This PR will record the deleted partitions and include them in the get meta response, allowing the CCR syncer to filter out the binlogs that belong to these partitions.

The CCR syncer part PR is selectdb/ccr-syncer#117.

doris-robot · 2024-07-03T03:51:10Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

w41ter · 2024-07-03T03:51:21Z

run buildall

w41ter · 2024-07-03T03:55:35Z

run buildall

github-actions · 2024-07-03T03:56:38Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-07-03T04:01:08Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-03T05:22:34Z

TPC-H: Total hot run time: 39966 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8419362b3bec9b489024001bad964a252b07e9a3, data reload: false
------ Round 1 ----------------------------------
q1	18067	5148	4283	4283
q2	2014	188	194	188
q3	10549	1171	1192	1171
q4	10217	779	792	779
q5	7478	2673	2612	2612
q6	219	137	138	137
q7	965	607	603	603
q8	9235	2064	2079	2064
q9	9052	6498	6502	6498
q10	8922	3717	3759	3717
q11	447	238	245	238
q12	476	235	233	233
q13	17760	2992	2963	2963
q14	278	226	225	225
q15	526	481	492	481
q16	531	400	378	378
q17	981	708	702	702
q18	8156	7446	7489	7446
q19	6012	1454	1538	1454
q20	725	317	316	316
q21	4988	3150	3255	3150
q22	391	328	338	328
Total cold run time: 117989 ms
Total hot run time: 39966 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4357	4190	4239	4190
q2	392	276	263	263
q3	2956	2935	2857	2857
q4	2078	1693	1812	1693
q5	5615	5519	5499	5499
q6	227	143	130	130
q7	2290	1870	1834	1834
q8	3230	3418	3420	3418
q9	8759	8771	8832	8771
q10	4125	3710	3830	3710
q11	606	512	500	500
q12	850	654	631	631
q13	15944	3144	3183	3144
q14	311	284	281	281
q15	544	483	499	483
q16	510	440	457	440
q17	1847	1522	1520	1520
q18	8080	7975	7670	7670
q19	2248	1665	1746	1665
q20	2164	1853	1844	1844
q21	5211	4918	4707	4707
q22	653	544	557	544
Total cold run time: 72997 ms
Total hot run time: 55794 ms

doris-robot · 2024-07-03T05:33:29Z

TPC-DS: Total hot run time: 175063 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8419362b3bec9b489024001bad964a252b07e9a3, data reload: false
query1	910	394	385	385
query2	6428	2506	2487	2487
query3	6641	217	205	205
query4	19190	17120	17333	17120
query5	3595	512	499	499
query6	280	169	163	163
query7	4595	296	293	293
query8	309	303	302	302
query9	8552	2378	2367	2367
query10	582	303	287	287
query11	10538	10103	10171	10103
query12	118	87	87	87
query13	1646	377	381	377
query14	8629	7887	7579	7579
query15	273	186	196	186
query16	7989	265	263	263
query17	1917	553	519	519
query18	2051	284	269	269
query19	206	149	151	149
query20	95	85	87	85
query21	212	135	127	127
query22	4207	4089	4003	4003
query23	33806	33828	33799	33799
query24	10331	2840	2902	2840
query25	589	387	383	383
query26	712	171	161	161
query27	2219	339	325	325
query28	5764	2155	2144	2144
query29	902	656	644	644
query30	259	169	157	157
query31	993	766	759	759
query32	95	53	56	53
query33	666	299	304	299
query34	873	505	502	502
query35	732	682	637	637
query36	1135	968	973	968
query37	146	81	87	81
query38	2964	2822	2856	2822
query39	894	831	825	825
query40	221	133	134	133
query41	53	53	52	52
query42	109	105	107	105
query43	603	598	577	577
query44	1087	779	745	745
query45	195	168	168	168
query46	1073	719	734	719
query47	1859	1755	1765	1755
query48	369	306	313	306
query49	845	417	417	417
query50	784	395	387	387
query51	6929	6801	6805	6801
query52	105	98	90	90
query53	376	305	294	294
query54	888	463	458	458
query55	75	75	74	74
query56	308	271	271	271
query57	1144	1027	1043	1027
query58	254	253	260	253
query59	3480	3163	3323	3163
query60	312	289	285	285
query61	107	94	92	92
query62	598	455	464	455
query63	330	300	294	294
query64	8511	2302	1835	1835
query65	3197	3116	3118	3116
query66	740	328	329	328
query67	15557	15003	14842	14842
query68	4509	538	540	538
query69	470	311	315	311
query70	1161	1079	1061	1061
query71	364	289	281	281
query72	6971	5355	5662	5355
query73	741	332	328	328
query74	5954	5576	5490	5490
query75	3372	2646	2628	2628
query76	2368	949	862	862
query77	441	305	318	305
query78	10392	9861	10199	9861
query79	2137	533	530	530
query80	2073	492	484	484
query81	595	230	224	224
query82	625	112	107	107
query83	286	172	168	168
query84	262	92	96	92
query85	1798	282	276	276
query86	471	337	340	337
query87	3231	3041	3068	3041
query88	3765	2450	2430	2430
query89	473	377	382	377
query90	1753	188	190	188
query91	130	103	100	100
query92	61	50	48	48
query93	1804	521	519	519
query94	1131	192	189	189
query95	407	320	319	319
query96	589	273	266	266
query97	3168	3029	3050	3029
query98	230	200	194	194
query99	1339	829	836	829
Total cold run time: 265223 ms
Total hot run time: 175063 ms

doris-robot · 2024-07-03T05:39:01Z

ClickBench: Total hot run time: 30.53 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8419362b3bec9b489024001bad964a252b07e9a3, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.06	0.06
query4	1.67	0.08	0.07
query5	0.50	0.48	0.49
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.54	0.53	0.55
query11	0.15	0.12	0.12
query12	0.16	0.12	0.12
query13	0.58	0.60	0.58
query14	0.75	0.77	0.78
query15	0.87	0.81	0.81
query16	0.35	0.37	0.35
query17	1.06	0.98	0.95
query18	0.22	0.25	0.24
query19	1.74	1.70	1.66
query20	0.01	0.01	0.01
query21	15.44	0.74	0.65
query22	5.00	6.49	2.00
query23	18.24	1.44	1.32
query24	2.08	0.23	0.23
query25	0.16	0.09	0.09
query26	0.26	0.17	0.17
query27	0.08	0.07	0.08
query28	13.27	1.01	1.00
query29	12.57	3.26	3.31
query30	0.25	0.06	0.05
query31	2.89	0.38	0.40
query32	3.24	0.47	0.46
query33	2.88	2.89	2.86
query34	16.96	4.43	4.44
query35	4.50	4.43	4.48
query36	0.65	0.46	0.47
query37	0.18	0.15	0.16
query38	0.15	0.14	0.15
query39	0.04	0.03	0.04
query40	0.17	0.15	0.14
query41	0.09	0.04	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.89 s
Total hot run time: 30.53 s

w41ter · 2024-07-03T06:21:52Z

run buildall

github-actions · 2024-07-03T06:27:02Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-03T06:57:57Z

TPC-H: Total hot run time: 40823 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 60785964359f524c451afa8f671cb285fb505505, data reload: false
------ Round 1 ----------------------------------
q1	17649	4691	4393	4393
q2	2615	207	202	202
q3	11131	1243	1157	1157
q4	10576	782	795	782
q5	8102	2728	2652	2652
q6	224	144	142	142
q7	976	628	630	628
q8	9527	2095	2057	2057
q9	8925	6489	6518	6489
q10	8961	3730	3734	3730
q11	479	269	246	246
q12	501	239	231	231
q13	17766	2988	3010	2988
q14	254	234	216	216
q15	537	487	499	487
q16	524	390	370	370
q17	972	630	667	630
q18	8084	7480	7573	7480
q19	6370	1553	1399	1399
q20	653	332	334	332
q21	4918	3957	3870	3870
q22	399	342	344	342
Total cold run time: 120143 ms
Total hot run time: 40823 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4413	4289	4265	4265
q2	368	274	273	273
q3	3026	2771	2749	2749
q4	1873	1625	1588	1588
q5	5257	5296	5281	5281
q6	218	130	131	130
q7	2053	1758	1734	1734
q8	3205	3368	3322	3322
q9	8316	8384	8385	8384
q10	3918	3691	3715	3691
q11	612	518	492	492
q12	784	577	589	577
q13	16279	3008	3015	3008
q14	298	282	268	268
q15	519	497	499	497
q16	462	422	440	422
q17	1793	1508	1494	1494
q18	7551	7529	7343	7343
q19	1704	1603	1549	1549
q20	1975	1762	1785	1762
q21	5007	4644	4661	4644
q22	652	536	560	536
Total cold run time: 70283 ms
Total hot run time: 54009 ms

doris-robot · 2024-07-03T07:09:03Z

TPC-DS: Total hot run time: 174373 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 60785964359f524c451afa8f671cb285fb505505, data reload: false
query1	915	397	374	374
query2	6469	2592	2392	2392
query3	6645	207	213	207
query4	18742	17651	17475	17475
query5	4198	486	499	486
query6	265	177	165	165
query7	4598	306	306	306
query8	308	311	290	290
query9	8853	2401	2373	2373
query10	604	315	288	288
query11	10533	10230	10165	10165
query12	138	89	91	89
query13	1641	380	374	374
query14	10238	7653	7916	7653
query15	267	187	198	187
query16	8172	273	266	266
query17	1896	553	538	538
query18	2127	275	288	275
query19	199	145	155	145
query20	95	84	86	84
query21	214	136	139	136
query22	4355	4320	4042	4042
query23	33811	33228	33121	33121
query24	11155	2874	2920	2874
query25	598	367	364	364
query26	1127	158	158	158
query27	2957	321	320	320
query28	7465	2086	2060	2060
query29	853	634	619	619
query30	278	154	158	154
query31	961	736	759	736
query32	98	53	53	53
query33	765	295	324	295
query34	892	484	479	479
query35	755	633	629	629
query36	1106	948	944	944
query37	152	76	75	75
query38	2849	2752	2743	2743
query39	854	816	837	816
query40	213	126	131	126
query41	53	53	56	53
query42	115	100	101	100
query43	605	550	565	550
query44	1194	738	757	738
query45	206	162	167	162
query46	1082	717	733	717
query47	1852	1754	1768	1754
query48	382	309	298	298
query49	1091	426	425	425
query50	765	391	402	391
query51	6938	6911	6754	6754
query52	109	95	95	95
query53	357	293	297	293
query54	912	456	460	456
query55	78	75	75	75
query56	331	259	273	259
query57	1119	1042	1038	1038
query58	262	254	275	254
query59	3564	3257	3496	3257
query60	334	286	288	286
query61	95	96	105	96
query62	641	448	452	448
query63	330	293	296	293
query64	8948	2291	1761	1761
query65	3177	3145	3129	3129
query66	797	343	326	326
query67	15484	15041	15083	15041
query68	4968	546	558	546
query69	602	463	337	337
query70	1199	1163	1084	1084
query71	468	286	288	286
query72	7887	5117	5226	5117
query73	768	331	328	328
query74	5948	5470	5562	5470
query75	3775	2631	2669	2631
query76	3661	998	895	895
query77	677	317	306	306
query78	10359	9864	9611	9611
query79	2433	528	526	526
query80	1504	502	484	484
query81	576	222	231	222
query82	760	113	112	112
query83	207	186	244	186
query84	270	91	89	89
query85	1396	288	276	276
query86	467	328	331	328
query87	3312	3092	3125	3092
query88	3924	2461	2434	2434
query89	481	388	380	380
query90	1832	194	194	194
query91	128	101	102	101
query92	59	51	49	49
query93	2836	524	516	516
query94	1148	200	195	195
query95	410	329	324	324
query96	604	275	267	267
query97	3201	3043	3058	3043
query98	229	206	199	199
query99	1257	846	820	820
Total cold run time: 276528 ms
Total hot run time: 174373 ms

doris-robot · 2024-07-03T07:14:33Z

ClickBench: Total hot run time: 30.88 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 60785964359f524c451afa8f671cb285fb505505, data reload: false
query1	0.04	0.04	0.02
query2	0.08	0.04	0.05
query3	0.23	0.06	0.06
query4	1.66	0.08	0.10
query5	0.50	0.49	0.48
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.55	0.50	0.49
query10	0.56	0.55	0.55
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	0.83	0.79	0.78
query15	0.85	0.82	0.82
query16	0.37	0.36	0.36
query17	1.02	0.98	1.00
query18	0.22	0.26	0.26
query19	1.80	1.69	1.68
query20	0.01	0.01	0.01
query21	15.44	0.77	0.64
query22	3.81	6.89	2.18
query23	18.29	1.37	1.31
query24	2.06	0.25	0.22
query25	0.14	0.10	0.08
query26	0.26	0.18	0.17
query27	0.08	0.08	0.07
query28	13.26	1.02	0.98
query29	12.67	3.28	3.25
query30	0.26	0.06	0.06
query31	2.94	0.39	0.39
query32	3.24	0.48	0.46
query33	2.89	2.92	2.94
query34	16.81	4.42	4.41
query35	4.47	4.50	4.46
query36	0.64	0.46	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.18	0.15	0.15
query41	0.08	0.04	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 108.86 s
Total hot run time: 30.88 s

w41ter · 2024-07-03T11:44:32Z

run buildall

github-actions · 2024-07-03T11:47:53Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-03T13:18:25Z

TPC-H: Total hot run time: 40051 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1cc6eca00d9df46c8aeb7eefa38d372155274267, data reload: false
------ Round 1 ----------------------------------
q1	18317	4864	4356	4356
q2	2019	200	187	187
q3	10541	1229	1216	1216
q4	10216	791	724	724
q5	7593	2924	2627	2627
q6	222	138	139	138
q7	961	603	606	603
q8	9231	2048	2082	2048
q9	9101	6467	6452	6452
q10	9084	3777	3789	3777
q11	440	238	247	238
q12	487	232	240	232
q13	17963	2972	2973	2972
q14	269	230	220	220
q15	528	478	473	473
q16	524	381	384	381
q17	970	674	733	674
q18	8157	7512	7355	7355
q19	5177	1533	1486	1486
q20	665	338	334	334
q21	4951	3228	3261	3228
q22	402	330	334	330
Total cold run time: 117818 ms
Total hot run time: 40051 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4478	4323	4298	4298
q2	365	265	287	265
q3	2971	2930	2933	2930
q4	2009	1719	1765	1719
q5	5593	5500	5472	5472
q6	231	132	131	131
q7	2215	1868	1811	1811
q8	3268	3415	3436	3415
q9	8771	8685	8826	8685
q10	4160	3899	3771	3771
q11	599	512	515	512
q12	851	644	649	644
q13	16304	3212	3186	3186
q14	310	273	277	273
q15	527	494	498	494
q16	486	427	453	427
q17	1829	1566	1503	1503
q18	8067	7865	7878	7865
q19	1738	1586	1600	1586
q20	2237	1854	1903	1854
q21	5333	4975	4949	4949
q22	638	536	547	536
Total cold run time: 72980 ms
Total hot run time: 56326 ms

doris-robot · 2024-07-03T13:29:32Z

TPC-DS: Total hot run time: 173604 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1cc6eca00d9df46c8aeb7eefa38d372155274267, data reload: false
query1	939	376	372	372
query2	6460	2499	2240	2240
query3	6638	215	221	215
query4	19060	17392	17557	17392
query5	3589	483	489	483
query6	286	177	157	157
query7	4591	311	307	307
query8	319	278	282	278
query9	8532	2374	2351	2351
query10	559	300	270	270
query11	10459	10079	10035	10035
query12	113	92	85	85
query13	1630	365	370	365
query14	10154	7779	6952	6952
query15	235	186	185	185
query16	7757	271	260	260
query17	1887	543	518	518
query18	1951	269	268	268
query19	190	150	154	150
query20	89	84	83	83
query21	205	133	134	133
query22	4249	4061	3994	3994
query23	33632	33904	33862	33862
query24	10880	2939	2856	2856
query25	595	397	381	381
query26	708	174	164	164
query27	2234	311	335	311
query28	6236	2120	2113	2113
query29	869	649	636	636
query30	268	161	159	159
query31	986	754	756	754
query32	95	57	58	57
query33	683	296	281	281
query34	881	470	481	470
query35	780	649	642	642
query36	1160	980	1006	980
query37	149	86	84	84
query38	2930	2867	2827	2827
query39	883	844	851	844
query40	209	131	128	128
query41	55	54	53	53
query42	117	107	102	102
query43	623	567	544	544
query44	1125	713	720	713
query45	194	161	163	161
query46	1076	744	737	737
query47	1849	1782	1757	1757
query48	365	289	289	289
query49	837	414	414	414
query50	774	391	377	377
query51	6893	6731	6697	6697
query52	102	97	101	97
query53	363	296	295	295
query54	874	449	440	440
query55	76	72	76	72
query56	308	285	296	285
query57	1148	1068	1055	1055
query58	275	250	261	250
query59	3297	3317	3090	3090
query60	323	297	307	297
query61	118	112	113	112
query62	610	448	460	448
query63	324	299	296	296
query64	8680	2374	1862	1862
query65	3178	3070	3118	3070
query66	758	341	348	341
query67	15305	14860	14955	14860
query68	4544	520	536	520
query69	594	405	336	336
query70	1125	1130	1042	1042
query71	452	286	286	286
query72	8547	5241	5468	5241
query73	744	328	321	321
query74	5875	5580	5481	5481
query75	4275	2664	2664	2664
query76	3188	982	884	884
query77	643	309	311	309
query78	10594	9851	9647	9647
query79	5060	515	514	514
query80	1709	473	466	466
query81	568	219	218	218
query82	1144	109	106	106
query83	353	176	172	172
query84	271	87	90	87
query85	2046	332	264	264
query86	478	319	325	319
query87	3306	3106	3066	3066
query88	4680	2352	2340	2340
query89	501	398	383	383
query90	2048	188	184	184
query91	128	101	98	98
query92	60	49	49	49
query93	5348	515	512	512
query94	1411	181	183	181
query95	415	305	302	302
query96	601	264	267	264
query97	3160	3010	2964	2964
query98	227	195	222	195
query99	1142	858	833	833
Total cold run time: 278954 ms
Total hot run time: 173604 ms

doris-robot · 2024-07-03T13:35:04Z

ClickBench: Total hot run time: 30.14 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1cc6eca00d9df46c8aeb7eefa38d372155274267, data reload: false
query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.22	0.06	0.05
query4	1.67	0.07	0.07
query5	0.51	0.50	0.49
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.56	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.58	0.58
query14	0.78	0.78	0.78
query15	0.86	0.82	0.82
query16	0.36	0.36	0.37
query17	1.03	0.97	0.97
query18	0.23	0.25	0.25
query19	1.89	1.67	1.76
query20	0.01	0.02	0.01
query21	15.41	0.75	0.66
query22	4.31	7.63	1.55
query23	18.44	1.40	1.24
query24	2.08	0.24	0.21
query25	0.16	0.08	0.08
query26	0.27	0.17	0.17
query27	0.08	0.07	0.08
query28	13.29	1.02	0.99
query29	12.63	3.31	3.28
query30	0.25	0.06	0.06
query31	2.87	0.39	0.38
query32	3.29	0.47	0.47
query33	2.85	2.91	2.90
query34	16.99	4.47	4.39
query35	4.48	4.58	4.47
query36	0.66	0.46	0.48
query37	0.18	0.16	0.16
query38	0.16	0.15	0.14
query39	0.05	0.03	0.03
query40	0.17	0.14	0.16
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.69 s
Total hot run time: 30.14 s

dataroaring

LGTM

github-actions · 2024-07-03T13:45:49Z

PR approved by at least one committer and no changes requested.

This branch was missed in the PR #37196

The CCR syncer needs to know the distribution of tables, partitions, indexes, and replicas when synchronizing binlogs. If a partition is deleted before the binlog synchronization is complete, the CCR syncer cannot continue synchronizing. This PR will record the deleted partitions and include them in the get meta response, allowing the CCR syncer to filter out the binlogs that belong to these partitions. The CCR syncer part PR is selectdb/ccr-syncer#117.

This branch was missed in the PR #37196

Cherry-pick apache#37196. The CCR syncer needs to know the distribution of tables, partitions, indexes, and replicas when synchronizing binlogs. If a partition is deleted before the binlog synchronization is complete, the CCR syncer cannot continue synchronizing. This PR will record the deleted partitions and include them in the get meta response, allowing the CCR syncer to filter out the binlogs that belong to these partitions. The CCR syncer part PR is selectdb/ccr-syncer#117.

Cherry-pick #37196, #37326. The CCR syncer needs to know the distribution of tables, partitions, indexes, and replicas when synchronizing binlogs. If a partition is deleted before the binlog synchronization is complete, the CCR syncer cannot continue synchronizing. This PR will record the deleted partitions and include them in the get meta response, allowing the CCR syncer to filter out the binlogs that belong to these partitions. The CCR syncer part PR is selectdb/ccr-syncer#117.

Cherry-pick apache#37196, apache#37326. The CCR syncer needs to know the distribution of tables, partitions, indexes, and replicas when synchronizing binlogs. If a partition is deleted before the binlog synchronization is complete, the CCR syncer cannot continue synchronizing. This PR will record the deleted partitions and include them in the get meta response, allowing the CCR syncer to filter out the binlogs that belong to these partitions. The CCR syncer part PR is selectdb/ccr-syncer#117.

The field partition ID of DropPartitionInfo was added in PR: apache#37196, the old version doesn't contain this field so it will be null.

The field partition ID of DropPartitionInfo was added in PR: #37196, the old version doesn't contain this field so it will be null.

Cherry-pick #39909 The field partition ID of DropPartitionInfo was added in PR: #37196, the old version doesn't contain this field so it will be null.

The field partition ID of DropPartitionInfo was added in PR: #37196, the old version doesn't contain this field so it will be null.

The CCR syncer needs to know the distribution of tables, partitions, indexes, and replicas when synchronizing binlogs. If a partition is deleted before the binlog synchronization is complete, the CCR syncer cannot continue synchronizing. This PR will record the deleted partitions and include them in the get meta response, allowing the CCR syncer to filter out the binlogs that belong to these partitions. The CCR syncer part PR is selectdb/ccr-syncer#117.

…he#37326) This branch was missed in the PR apache#37196

The field partition ID of DropPartitionInfo was added in PR: apache#37196, the old version doesn't contain this field so it will be null.

w41ter requested review from CalvinKirs, dataroaring and morningman as code owners July 3, 2024 03:51

github-actionsBot added the meta-change label Jul 3, 2024

w41ter force-pushed the filter_drop_partitions branch from 7ffe8b4 to 8419362Compare July 3, 2024 03:55

w41ter force-pushed the filter_drop_partitions branch from 8419362 to 6078596Compare July 3, 2024 06:21

w41ter force-pushed the filter_drop_partitions branch from 6078596 to 1cc6ecaCompare July 3, 2024 11:41

dataroaring previously approved these changes Jul 3, 2024
View reviewed changes

dataroaring added dev/2.1.x dev/2.0.x labels Jul 3, 2024

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 3, 2024

dataroaring pushed a commit that referenced this pull request Jul 5, 2024
[fix](binlog) gc dropped partition records in disable binlog gc (#37326)
f5c18ee
This branch was missed in the PR #37196

dataroaring pushed a commit that referenced this pull request Jul 15, 2024
[fix](binlog) gc dropped partition records in disable binlog gc (#37326)
a78e67e
This branch was missed in the PR #37196

dataroaring pushed a commit that referenced this pull request Jul 17, 2024
[fix](binlog) gc dropped partition records in disable binlog gc (#37326)
38ec8b3
This branch was missed in the PR #37196

w41ter mentioned this pull request Jul 18, 2024
[chore](fe) Returns dropped partitions in GetMeta request #38069
Merged

w41ter mentioned this pull request Jul 18, 2024
[chore](fe) Returns dropped partitions in GetMeta request #38071
Merged

dataroaring added dev/2.0.14-merged and removed dev/2.0.x labels Jul 18, 2024

dataroaring added dev/2.1.5-merged and removed dev/2.1.x labels Jul 18, 2024

dataroaring added the dev/3.0.1-merged label Aug 5, 2024

w41ter added a commit to w41ter/incubator-doris that referenced this pull request Aug 26, 2024
[fix](binlog) Fix NPE when recover binlogs
00033eb
The field partition ID of DropPartitionInfo was added in PR: apache#37196, the old version doesn't contain this field so it will be null.

w41ter mentioned this pull request Aug 26, 2024
[fix](binlog) Fix NPE when recover binlogs #39909
Merged

w41ter mentioned this pull request Aug 26, 2024
[fix](binlog) Fix NPE when recover binlogs (#39909) #39910
Merged

w41ter mentioned this pull request Aug 26, 2024
[fix](binlog) Fix NPE when recover binlogs (#39909) #39911
Merged

dataroaring pushed a commit that referenced this pull request Aug 26, 2024
[fix](binlog) Fix NPE when recover binlogs (#39909)
5eb4824
The field partition ID of DropPartitionInfo was added in PR: #37196, the old version doesn't contain this field so it will be null.

w41ter added a commit that referenced this pull request Aug 26, 2024
[fix](binlog) Fix NPE when recover binlogs (#39909) (#39911)
e7435bd
Cherry-pick #39909 The field partition ID of DropPartitionInfo was added in PR: #37196, the old version doesn't contain this field so it will be null.

w41ter added a commit that referenced this pull request Aug 26, 2024
[fix](binlog) Fix NPE when recover binlogs (#39909) (#39910)
fb5002d
Cherry-pick #39909 The field partition ID of DropPartitionInfo was added in PR: #37196, the old version doesn't contain this field so it will be null.

dataroaring pushed a commit that referenced this pull request Aug 26, 2024
[fix](binlog) Fix NPE when recover binlogs (#39909)
ccd6809
The field partition ID of DropPartitionInfo was added in PR: #37196, the old version doesn't contain this field so it will be null.

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[fix](binlog) gc dropped partition records in disable binlog gc (apac…
4ad9fc0
…he#37326) This branch was missed in the PR apache#37196

Uh oh!

Conversation

w41ter commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

w41ter commented Jul 3, 2024

Uh oh!

w41ter commented Jul 3, 2024

Uh oh!

github-actionsBot commented Jul 3, 2024

Uh oh!

github-actionsBot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

w41ter commented Jul 3, 2024

Uh oh!

github-actionsBot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

w41ter commented Jul 3, 2024

Uh oh!

github-actionsBot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

doris-robot commented Jul 3, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants