[fix](audit-loader) fix invalid token check logic by morningman · Pull Request #32095 · apache/doris

morningman · 2024-03-12T05:38:33Z

Proposed changes

The check of the token should be forwarded to Master FE.
I add a new RPC method checkToken() in Frontend for this logic.
Otherwise, after enable the audit loader, the log from non-master FE can not be loaded to audit table
with Invalid token error.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-03-12T05:38:37Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

morningman · 2024-03-12T05:39:57Z

run buildall

doris-robot · 2024-03-12T06:09:53Z

TPC-H: Total hot run time: 37245 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0115e46ebdbf94fba100719f599b9016814d70fc, data reload: false
------ Round 1 ----------------------------------
q1	17651	4289	4235	4235
q2	2533	157	151	151
q3	11480	1018	943	943
q4	5014	734	721	721
q5	7855	2943	2885	2885
q6	190	132	129	129
q7	1244	852	872	852
q8	9845	2118	2049	2049
q9	7971	6520	6548	6520
q10	8133	2538	2550	2538
q11	428	224	221	221
q12	805	303	299	299
q13	17979	2965	2970	2965
q14	279	257	256	256
q15	500	467	450	450
q16	473	411	401	401
q17	952	557	542	542
q18	6577	5887	5837	5837
q19	1560	1439	1486	1439
q20	537	284	298	284
q21	7459	3267	3235	3235
q22	832	293	318	293
Total cold run time: 110297 ms
Total hot run time: 37245 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4085	4069	4106	4069
q2	321	238	224	224
q3	2988	2866	2841	2841
q4	1840	1585	1537	1537
q5	5211	5286	5227	5227
q6	208	114	119	114
q7	2258	1888	1872	1872
q8	3168	3333	3323	3323
q9	8590	8594	8622	8594
q10	6146	3701	3693	3693
q11	535	443	443	443
q12	726	542	543	542
q13	9933	2848	2824	2824
q14	294	251	250	250
q15	475	460	441	441
q16	455	406	414	406
q17	1733	1467	1475	1467
q18	7786	7271	6997	6997
q19	2977	1563	1519	1519
q20	1917	1700	1735	1700
q21	5003	4776	4958	4776
q22	528	454	466	454
Total cold run time: 67177 ms
Total hot run time: 53313 ms

doris-robot · 2024-03-12T06:22:10Z

TPC-DS: Total hot run time: 187397 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0115e46ebdbf94fba100719f599b9016814d70fc, data reload: false
query1	930	371	345	345
query2	7366	1957	1920	1920
query3	6712	213	209	209
query4	31311	20759	20746	20746
query5	4352	400	423	400
query6	276	183	195	183
query7	4623	310	295	295
query8	234	173	184	173
query9	9941	2319	2308	2308
query10	450	240	259	240
query11	17160	14074	14270	14074
query12	133	89	82	82
query13	1617	400	411	400
query14	10918	10380	10362	10362
query15	243	215	192	192
query16	7846	267	254	254
query17	1933	569	564	564
query18	2001	289	280	280
query19	208	166	159	159
query20	89	88	85	85
query21	198	129	126	126
query22	4800	4485	4530	4485
query23	31814	31209	31339	31209
query24	12367	2884	2869	2869
query25	667	384	394	384
query26	1886	155	160	155
query27	3193	347	338	338
query28	8069	1916	1898	1898
query29	1098	664	631	631
query30	309	149	143	143
query31	963	726	717	717
query32	100	59	52	52
query33	770	252	254	252
query34	1098	479	501	479
query35	846	611	602	602
query36	993	884	881	881
query37	283	81	79	79
query38	3294	3151	3154	3151
query39	1428	1382	1398	1382
query40	303	112	112	112
query41	51	46	45	45
query42	98	96	95	95
query43	493	460	459	459
query44	1242	729	720	720
query45	282	263	244	244
query46	1057	702	685	685
query47	1677	1597	1626	1597
query48	451	372	380	372
query49	1250	351	334	334
query50	766	417	388	388
query51	6704	6669	6533	6533
query52	111	93	101	93
query53	338	282	282	282
query54	322	234	266	234
query55	88	86	79	79
query56	259	229	236	229
query57	1118	1012	1006	1006
query58	232	207	216	207
query59	2821	2733	2452	2452
query60	273	249	249	249
query61	113	112	113	112
query62	603	403	425	403
query63	304	274	286	274
query64	6769	3977	3929	3929
query65	3091	2992	3011	2992
query66	1377	327	338	327
query67	15162	14448	14355	14355
query68	9869	542	548	542
query69	669	385	376	376
query70	1396	1120	1156	1120
query71	506	279	273	273
query72	6750	2645	2514	2514
query73	1607	318	319	318
query74	6999	6565	6530	6530
query75	10197	8085	8073	8073
query76	5770	863	1014	863
query77	649	254	269	254
query78	10415	9839	9706	9706
query79	9613	520	521	520
query80	1781	392	378	378
query81	492	215	213	213
query82	319	210	199	199
query83	214	149	139	139
query84	285	79	76	76
query85	1077	326	310	310
query86	333	315	317	315
query87	3438	3241	3197	3197
query88	4867	2375	2356	2356
query89	488	368	373	368
query90	2005	176	178	176
query91	170	131	126	126
query92	62	47	47	47
query93	6608	509	502	502
query94	1329	185	178	178
query95	442	337	328	328
query96	611	281	274	274
query97	4400	4274	4283	4274
query98	226	204	212	204
query99	968	762	737	737
Total cold run time: 324429 ms
Total hot run time: 187397 ms

doris-robot · 2024-03-12T06:27:26Z

ClickBench: Total hot run time: 30.03 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0115e46ebdbf94fba100719f599b9016814d70fc, data reload: false
query1	0.04	0.04	0.04
query2	0.05	0.02	0.02
query3	0.22	0.06	0.06
query4	1.68	0.10	0.10
query5	0.52	0.53	0.52
query6	1.18	0.62	0.61
query7	0.02	0.01	0.02
query8	0.03	0.02	0.03
query9	0.56	0.50	0.49
query10	0.55	0.56	0.55
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.57	0.56	0.57
query14	0.73	0.76	0.77
query15	0.80	0.79	0.77
query16	0.39	0.37	0.38
query17	0.99	0.99	0.99
query18	0.20	0.28	0.22
query19	1.82	1.74	1.74
query20	0.01	0.00	0.00
query21	15.45	0.57	0.56
query22	2.32	2.06	1.53
query23	17.13	0.86	0.87
query24	3.57	1.27	1.01
query25	0.14	0.16	0.03
query26	0.67	0.13	0.12
query27	0.06	0.03	0.03
query28	10.80	0.87	0.84
query29	12.50	3.16	3.13
query30	0.66	0.58	0.57
query31	2.80	0.34	0.33
query32	3.39	0.45	0.44
query33	2.77	2.84	2.76
query34	16.03	4.31	4.27
query35	4.34	4.26	4.30
query36	1.08	1.00	1.02
query37	0.07	0.05	0.06
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.18	0.13	0.14
query41	0.07	0.01	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 104.74 s
Total hot run time: 30.03 s

doris-robot · 2024-03-12T06:30:33Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 0115e46ebdbf94fba100719f599b9016814d70fc with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 17.2 seconds inserted 10000000 Rows, about 581K ops/s

RoanHeNaN

LGTM

related: #26278

github-actions · 2024-03-12T08:18:47Z

PR approved by anyone and no changes requested.

RoanHeNaN · 2024-03-12T08:22:02Z

Check token on local FE is hard to implement, some problems of token synchronization need to be solved.

mrhhsg

LGTM

github-actions · 2024-03-12T14:40:36Z

PR approved by at least one committer and no changes requested.

The check of the token should be forwarded to Master FE. I add a new RPC method `checkToken()` in Frontend for this logic. Otherwise, after enable the audit loader, the log from non-master FE can not be loaded to audit table with `Invalid token` error.

1
0115e46

morningman added the dev/2.0.x label Mar 12, 2024

github-actionsBot added the meta-change label Mar 12, 2024

RoanHeNaN approved these changes Mar 12, 2024
View reviewed changes

github-actionsBot added the reviewed label Mar 12, 2024

wm1581066 added the usercase Important user case type label label Mar 12, 2024

mrhhsg approved these changes Mar 12, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 12, 2024

morningman merged commit 1aa2d91 into apache:masterMar 12, 2024

wm1581066 removed the dev/2.0.x label Mar 14, 2024

Navigation Menu

Uh oh!

[fix](audit-loader) fix invalid token check logic - #32095

[fix](audit-loader) fix invalid token check logic#32095
morningman merged 1 commit into
apache:masterfrom
morningman:token_master

morningman commented Mar 12, 2024•
edited
Loading

Uh oh!

doris-robot commented Mar 12, 2024

Uh oh!

morningman commented Mar 12, 2024

Uh oh!

doris-robot commented Mar 12, 2024

Uh oh!

doris-robot commented Mar 12, 2024

Uh oh!

doris-robot commented Mar 12, 2024

Uh oh!

doris-robot commented Mar 12, 2024

Uh oh!

RoanHeNaN left a comment

Uh oh!

github-actionsBot commented Mar 12, 2024

Uh oh!

RoanHeNaN commented Mar 12, 2024

Uh oh!

mrhhsg left a comment

Uh oh!

github-actionsBot commented Mar 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

morningman commented Mar 12, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Mar 12, 2024

Uh oh!

morningman commented Mar 12, 2024

Uh oh!

doris-robot commented Mar 12, 2024

Uh oh!

doris-robot commented Mar 12, 2024

Uh oh!

doris-robot commented Mar 12, 2024

Uh oh!

doris-robot commented Mar 12, 2024

Uh oh!

RoanHeNaN left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Mar 12, 2024

Uh oh!

RoanHeNaN commented Mar 12, 2024

Uh oh!

mrhhsg left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Mar 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

morningman commented Mar 12, 2024•
edited
Loading