[fix](load) fix miss writer in concurrency incremental open by zclllyybb · Pull Request #38605 · apache/doris

zclllyybb · 2024-07-31T09:24:20Z

Proposed changes

Issue Number: close #xxx

fix bugs like:

W20240730 23:06:19.640558 108249 status.h:421] meet error status: [INTERNAL_ERROR]unknown tablet to append data, tablet=17285252124319
0# doris::BaseTabletsChannel::_write_block_data(doris::PTabletWriterAddBlockRequest const&, long, std::unordered_map<long, std::vector<unsigned int, std::allocator<unsigned int> >, std::hash<long>, std::equal_to<long>, std::allocator<std::pair<long const, std::vector<unsigned int, std::allocator<unsigned int> > > > >&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/common/status.h:0
1# doris::CloudTabletsChannel::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/cloud/cloud_tablets_channel.cpp:0
2# doris::LoadChannel::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/runtime/load_channel.cpp:179
3# doris::LoadChannelMgr::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/runtime/load_channel_mgr.cpp:0
4# std::_Function_handler<void (), doris::PInternalService::tablet_writer_add_block(google::protobuf::RpcController*, doris::PTabletWriterAddBlockRequest const*, doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /root/doris/be/src/common/status.h:488
5# doris::WorkThreadPool<false>::work_thread(int) at /root/doris/be/src/util/work_thread_pool.hpp:159
6# execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85
7# ?
8# clone

doris-robot · 2024-07-31T09:24:24Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zclllyybb · 2024-07-31T09:24:27Z

run buildall

github-actions · 2024-07-31T09:57:59Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-31T10:38:00Z

TPC-H: Total hot run time: 41449 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f1dfafcc257b44b513b0a1d009571916b40dcc7, data reload: false
------ Round 1 ----------------------------------
q1	17601	4088	4116	4088
q2	2024	220	200	200
q3	10483	1285	1356	1285
q4	10173	823	929	823
q5	7658	2969	2961	2961
q6	222	136	136	136
q7	1029	619	615	615
q8	9426	1781	1965	1781
q9	8553	6628	6591	6591
q10	8733	3812	3829	3812
q11	450	248	244	244
q12	431	231	227	227
q13	17760	2989	2933	2933
q14	277	242	237	237
q15	526	485	501	485
q16	527	401	387	387
q17	968	903	917	903
q18	8038	7231	7236	7231
q19	1432	1226	1216	1216
q20	561	322	346	322
q21	5339	4686	4787	4686
q22	354	300	286	286
Total cold run time: 112565 ms
Total hot run time: 41449 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4023	4020	3991	3991
q2	323	218	216	216
q3	2973	3018	3145	3018
q4	1993	2009	1976	1976
q5	5576	5511	5405	5405
q6	217	127	135	127
q7	2130	1737	1793	1737
q8	3319	3378	3356	3356
q9	8624	8652	8696	8652
q10	4011	4003	4055	4003
q11	544	447	453	447
q12	722	584	567	567
q13	14994	3124	3111	3111
q14	294	305	271	271
q15	523	491	499	491
q16	447	405	400	400
q17	1756	1713	1723	1713
q18	8264	7731	7792	7731
q19	1750	1722	1752	1722
q20	2055	1862	1850	1850
q21	5731	5451	5315	5315
q22	522	456	469	456
Total cold run time: 70791 ms
Total hot run time: 56555 ms

doris-robot · 2024-07-31T10:48:36Z

TPC-DS: Total hot run time: 170323 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f1dfafcc257b44b513b0a1d009571916b40dcc7, data reload: false
query1	928	362	373	362
query2	6488	1739	1629	1629
query3	6666	220	233	220
query4	19607	17674	17300	17300
query5	3661	519	508	508
query6	265	176	171	171
query7	4595	317	297	297
query8	257	206	185	185
query9	8504	2396	2407	2396
query10	451	292	268	268
query11	10443	9983	10027	9983
query12	128	94	89	89
query13	1620	376	386	376
query14	9812	7968	7774	7774
query15	203	161	166	161
query16	6761	453	447	447
query17	966	588	556	556
query18	1911	289	279	279
query19	195	152	152	152
query20	94	92	85	85
query21	205	101	105	101
query22	4129	4002	4138	4002
query23	33817	33692	33469	33469
query24	9191	3219	3123	3123
query25	606	427	438	427
query26	732	167	149	149
query27	2164	281	284	281
query28	6022	2062	2030	2030
query29	918	434	456	434
query30	234	151	157	151
query31	954	765	761	761
query32	101	59	56	56
query33	670	300	314	300
query34	916	487	500	487
query35	879	748	775	748
query36	1090	879	898	879
query37	147	86	88	86
query38	3081	2847	2839	2839
query39	910	816	815	815
query40	211	110	113	110
query41	44	40	44	40
query42	121	100	103	100
query43	463	414	420	414
query44	1202	728	742	728
query45	213	178	177	177
query46	1093	811	766	766
query47	1800	1702	1702	1702
query48	361	293	292	292
query49	861	443	420	420
query50	899	422	433	422
query51	6733	6660	6633	6633
query52	99	94	88	88
query53	259	181	186	181
query54	598	456	449	449
query55	76	75	76	75
query56	268	250	262	250
query57	1113	1057	1020	1020
query58	272	271	262	262
query59	2473	2358	2227	2227
query60	312	268	269	268
query61	93	92	93	92
query62	870	657	661	657
query63	213	187	189	187
query64	4218	1942	1876	1876
query65	3166	3107	3087	3087
query66	832	332	331	331
query67	15277	14948	14865	14865
query68	4334	549	563	549
query69	450	305	291	291
query70	1112	1030	1052	1030
query71	374	286	274	274
query72	7136	2669	2476	2476
query73	765	328	331	328
query74	6050	5626	5655	5626
query75	3387	2735	2732	2732
query76	2117	1372	1406	1372
query77	427	305	309	305
query78	9465	8939	8906	8906
query79	2728	597	533	533
query80	1179	523	497	497
query81	558	232	223	223
query82	960	133	137	133
query83	239	171	176	171
query84	272	78	81	78
query85	1176	322	333	322
query86	491	305	282	282
query87	3252	3070	3120	3070
query88	3918	2429	2430	2429
query89	381	291	294	291
query90	1705	184	187	184
query91	124	95	101	95
query92	57	49	49	49
query93	1988	625	614	614
query94	779	301	300	300
query95	379	271	279	271
query96	603	284	283	283
query97	3230	3040	3053	3040
query98	243	202	192	192
query99	1672	1303	1282	1282
Total cold run time: 256012 ms
Total hot run time: 170323 ms

doris-robot · 2024-07-31T10:54:05Z

ClickBench: Total hot run time: 30 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6f1dfafcc257b44b513b0a1d009571916b40dcc7, data reload: false
query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.48	0.48	0.48
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.59	0.51	0.52
query10	0.56	0.58	0.57
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.60
query14	0.78	0.80	0.80
query15	0.91	0.85	0.88
query16	0.35	0.35	0.36
query17	1.02	0.96	1.00
query18	0.22	0.21	0.20
query19	1.84	1.68	1.76
query20	0.02	0.01	0.01
query21	15.40	0.79	0.66
query22	3.95	7.30	1.35
query23	18.12	1.26	1.26
query24	2.27	0.21	0.22
query25	0.18	0.08	0.08
query26	0.32	0.22	0.22
query27	0.46	0.24	0.23
query28	13.17	0.99	0.98
query29	12.54	3.29	3.29
query30	0.25	0.06	0.05
query31	2.88	0.41	0.41
query32	3.23	0.49	0.48
query33	2.99	2.95	2.95
query34	15.45	4.27	4.26
query35	4.29	4.30	4.29
query36	0.67	0.48	0.48
query37	0.19	0.16	0.16
query38	0.16	0.15	0.16
query39	0.04	0.04	0.03
query40	0.16	0.13	0.14
query41	0.09	0.05	0.04
query42	0.04	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.79 s
Total hot run time: 30 s

HappenLee

LGTM

github-actions · 2024-08-01T02:50:33Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-01T02:50:36Z

PR approved by anyone and no changes requested.

…8605) Issue Number: close #xxx fix bugs like: ```log W20240730 23:06:19.640558 108249 status.h:421] meet error status: [INTERNAL_ERROR]unknown tablet to append data, tablet=17285252124319 0# doris::BaseTabletsChannel::_write_block_data(doris::PTabletWriterAddBlockRequest const&, long, std::unordered_map<long, std::vector<unsigned int, std::allocator<unsigned int> >, std::hash<long>, std::equal_to<long>, std::allocator<std::pair<long const, std::vector<unsigned int, std::allocator<unsigned int> > > > >&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/common/status.h:0 1# doris::CloudTabletsChannel::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/cloud/cloud_tablets_channel.cpp:0 2# doris::LoadChannel::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/runtime/load_channel.cpp:179 3# doris::LoadChannelMgr::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/runtime/load_channel_mgr.cpp:0 4# std::_Function_handler<void (), doris::PInternalService::tablet_writer_add_block(google::protobuf::RpcController*, doris::PTabletWriterAddBlockRequest const*, doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /root/doris/be/src/common/status.h:488 5# doris::WorkThreadPool<false>::work_thread(int) at /root/doris/be/src/util/work_thread_pool.hpp:159 6# execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85 7# ? 8# clone ```

…38605) (#38793) pick #38605

## Proposed changes Issue Number: close #xxx fix bugs like: ```log W20240730 23:06:19.640558 108249 status.h:421] meet error status: [INTERNAL_ERROR]unknown tablet to append data, tablet=17285252124319 0# doris::BaseTabletsChannel::_write_block_data(doris::PTabletWriterAddBlockRequest const&, long, std::unordered_map<long, std::vector<unsigned int, std::allocator<unsigned int> >, std::hash<long>, std::equal_to<long>, std::allocator<std::pair<long const, std::vector<unsigned int, std::allocator<unsigned int> > > > >&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/common/status.h:0 1# doris::CloudTabletsChannel::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/cloud/cloud_tablets_channel.cpp:0 2# doris::LoadChannel::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/runtime/load_channel.cpp:179 3# doris::LoadChannelMgr::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/runtime/load_channel_mgr.cpp:0 4# std::_Function_handler<void (), doris::PInternalService::tablet_writer_add_block(google::protobuf::RpcController*, doris::PTabletWriterAddBlockRequest const*, doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /root/doris/be/src/common/status.h:488 5# doris::WorkThreadPool<false>::work_thread(int) at /root/doris/be/src/util/work_thread_pool.hpp:159 6# execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85 7# ? 8# clone ```

…8605) ## Proposed changes Issue Number: close #xxx fix bugs like: ```log W20240730 23:06:19.640558 108249 status.h:421] meet error status: [INTERNAL_ERROR]unknown tablet to append data, tablet=17285252124319 0# doris::BaseTabletsChannel::_write_block_data(doris::PTabletWriterAddBlockRequest const&, long, std::unordered_map<long, std::vector<unsigned int, std::allocator<unsigned int> >, std::hash<long>, std::equal_to<long>, std::allocator<std::pair<long const, std::vector<unsigned int, std::allocator<unsigned int> > > > >&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/common/status.h:0 1# doris::CloudTabletsChannel::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/cloud/cloud_tablets_channel.cpp:0 2# doris::LoadChannel::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/runtime/load_channel.cpp:179 3# doris::LoadChannelMgr::add_batch(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) at /root/doris/be/src/runtime/load_channel_mgr.cpp:0 4# std::_Function_handler<void (), doris::PInternalService::tablet_writer_add_block(google::protobuf::RpcController*, doris::PTabletWriterAddBlockRequest const*, doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /root/doris/be/src/common/status.h:488 5# doris::WorkThreadPool<false>::work_thread(int) at /root/doris/be/src/util/work_thread_pool.hpp:159 6# execute_native_thread_routine at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:85 7# ? 8# clone ```

1
6f1dfaf

HappenLee approved these changes Aug 1, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 1, 2024

BiteTheDDDDt added dev/2.1.x dev/3.0.x labels Aug 2, 2024

BiteTheDDDDt approved these changes Aug 2, 2024
View reviewed changes

BiteTheDDDDt merged commit 159fc3f into apache:masterAug 2, 2024

zclllyybb deleted the fix_iot_lock branch August 2, 2024 07:50

zclllyybb mentioned this pull request Aug 2, 2024
[branch-2.1](load) fix miss writer in concurrency incremental open (#38605) #38793
Merged

yiguolei pushed a commit that referenced this pull request Aug 5, 2024
[branch-2.1](load) fix miss writer in concurrency incremental open (#…
8fa0710
…38605) (#38793) pick #38605

yiguolei added dev/2.1.6-merged and removed dev/2.1.x labels Aug 5, 2024

dataroaring added dev/3.0.2-merged and removed dev/3.0.x labels Aug 12, 2024

gavinchou mentioned this pull request Oct 13, 2024
Release Note 3.0.2 #41558
Closed

Navigation Menu

Uh oh!

[fix](load) fix miss writer in concurrency incremental open - #38605

[fix](load) fix miss writer in concurrency incremental open#38605
BiteTheDDDDt merged 1 commit into
apache:masterfrom
zclllyybb:fix_iot_lock

zclllyybb commented Jul 31, 2024

Uh oh!

doris-robot commented Jul 31, 2024

Uh oh!

zclllyybb commented Jul 31, 2024

Uh oh!

github-actionsBot commented Jul 31, 2024

Uh oh!

doris-robot commented Jul 31, 2024

Uh oh!

doris-robot commented Jul 31, 2024

Uh oh!

doris-robot commented Jul 31, 2024

Uh oh!

HappenLee left a comment

Uh oh!

github-actionsBot commented Aug 1, 2024

Uh oh!

github-actionsBot commented Aug 1, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

zclllyybb commented Jul 31, 2024

Proposed changes

Uh oh!

doris-robot commented Jul 31, 2024

Uh oh!

zclllyybb commented Jul 31, 2024

Uh oh!

github-actionsBot commented Jul 31, 2024

Uh oh!

doris-robot commented Jul 31, 2024

Uh oh!

doris-robot commented Jul 31, 2024

Uh oh!

doris-robot commented Jul 31, 2024

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 1, 2024

Uh oh!

github-actionsBot commented Aug 1, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants