[improve](information schema) introduce routine load job system table (#48963) by sollhui · Pull Request #49284 · apache/doris

sollhui · 2025-03-20T03:11:17Z

Part IV of #48511

Introduce routine load job statistic system table:

mysql> show create table information_schema.routine_load_job\G
*************************** 1. row ***************************
Table: routine_load_job
Create Table: CREATE TABLE `routine_load_job` (
`JOB_ID` text NULL,
`JOB_NAME` text NULL,
`CREATE_TIME` text NULL,
`PAUSE_TIME` text NULL,
`END_TIME` text NULL,
`DB_NAME` text NULL,
`TABLE_NAME` text NULL,
`STATE` text NULL,
`CURRENT_TASK_NUM` text NULL,
`JOB_PROPERTIES` text NULL,
`DATA_SOURCE_PROPERTIES` text NULL,
`CUSTOM_PROPERTIES` text NULL,
`STATISTIC` text NULL,
`PROGRESS` text NULL,
`LAG` text NULL,
`REASON_OF_STATE_CHANGED` text NULL,
`ERROR_LOG_URLS` text NULL,
`USER_NAME` text NULL,
`CURRENT_ABORT_TASK_NUM` int NULL,
`IS_ABNORMAL_PAUSE` boolean NULL
) ENGINE=SCHEMA;
1 row in set (0.00 sec)

There are some benefits to empower job with SQL query capability for statistical information:

It can be used in conjunction with metrics add through [improve](routine load) add more metrics to observe the routine load job #48209 to roughly locate abnormal jobs when Grafana alarms, and the following SQL can be used:

SELECT JOB_NAME
FROM information_schema.routine_load_job_statistics
WHERE CURRENT_ABORT_TASK_NUM > 0
OR IS_ABNORMAL_PAUSE = TRUE;

User can use the select * from information_schema.routine_load_job instead of the show routine load. The advantage is that the show routine load can only be searched by name, but SQL can be very flexible in locating jobs

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…apache#48963) Part IV of apache#48511 doc apache/doris-website#2196 **Introduce routine load job statistic system table:** ``` mysql> show create table information_schema.routine_load_job\G *************************** 1. row *************************** Table: routine_load_job Create Table: CREATE TABLE `routine_load_job` ( `JOB_ID` text NULL, `JOB_NAME` text NULL, `CREATE_TIME` text NULL, `PAUSE_TIME` text NULL, `END_TIME` text NULL, `DB_NAME` text NULL, `TABLE_NAME` text NULL, `STATE` text NULL, `CURRENT_TASK_NUM` text NULL, `JOB_PROPERTIES` text NULL, `DATA_SOURCE_PROPERTIES` text NULL, `CUSTOM_PROPERTIES` text NULL, `STATISTIC` text NULL, `PROGRESS` text NULL, `LAG` text NULL, `REASON_OF_STATE_CHANGED` text NULL, `ERROR_LOG_URLS` text NULL, `USER_NAME` text NULL, `CURRENT_ABORT_TASK_NUM` int NULL, `IS_ABNORMAL_PAUSE` boolean NULL ) ENGINE=SCHEMA; 1 row in set (0.00 sec) ``` **There are some benefits to empower job with SQL query capability for statistical information:** - It can be used in conjunction with metrics add through apache#48209 to roughly locate abnormal jobs when Grafana alarms, and the following SQL can be used: ``` SELECT JOB_NAME FROM information_schema.routine_load_job_statistics WHERE CURRENT_ABORT_TASK_NUM > 0 OR IS_ABNORMAL_PAUSE = TRUE; ``` - User can use the `select * from information_schema.routine_load_job` instead of the `show routine load`. The advantage is that the `show routine load` can only be searched by name, but SQL can be very flexible in locating jobs

sollhui · 2025-03-20T03:11:26Z

run buildall

hello-stephen · 2025-03-20T03:11:37Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

doris-robot · 2025-03-20T04:56:57Z

TPC-H: Total hot run time: 40083 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d17cb53202aea830d9b246777928d2e1d0ae09b6, data reload: false
------ Round 1 ----------------------------------
q1	17576	6708	6680	6680
q2	2080	166	169	166
q3	10628	1093	1182	1093
q4	10578	733	745	733
q5	7731	2926	2839	2839
q6	224	137	137	137
q7	958	604	597	597
q8	9369	1971	2017	1971
q9	6617	6410	6423	6410
q10	6992	2260	2247	2247
q11	463	257	253	253
q12	399	206	208	206
q13	17777	2980	2993	2980
q14	234	206	209	206
q15	514	471	459	459
q16	644	585	591	585
q17	981	554	541	541
q18	7371	6736	6595	6595
q19	1390	1048	1098	1048
q20	476	209	198	198
q21	4014	3177	3156	3156
q22	1111	994	983	983
Total cold run time: 108127 ms
Total hot run time: 40083 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6558	6526	6582	6526
q2	331	237	234	234
q3	2938	2777	2924	2777
q4	2061	1822	1778	1778
q5	5781	5729	5753	5729
q6	206	127	126	126
q7	2240	1833	1828	1828
q8	3426	3567	3587	3567
q9	8801	8954	8899	8899
q10	3555	3543	3542	3542
q11	585	483	509	483
q12	819	589	618	589
q13	8476	3220	3158	3158
q14	303	272	278	272
q15	528	465	466	465
q16	699	655	658	655
q17	1823	1642	1604	1604
q18	8353	7912	7603	7603
q19	1667	1503	1534	1503
q20	2082	1897	1849	1849
q21	5549	5446	5374	5374
q22	1154	1059	1032	1032
Total cold run time: 67935 ms
Total hot run time: 59593 ms

doris-robot · 2025-03-20T05:09:32Z

TPC-DS: Total hot run time: 196662 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d17cb53202aea830d9b246777928d2e1d0ae09b6, data reload: false
query1	1296	916	917	916
query2	6280	2069	2060	2060
query3	10946	4583	4638	4583
query4	65759	28780	23354	23354
query5	4951	480	454	454
query6	419	174	175	174
query7	5543	312	327	312
query8	331	234	223	223
query9	8902	2662	2641	2641
query10	456	272	264	264
query11	17082	15071	15645	15071
query12	166	102	103	102
query13	1493	439	463	439
query14	9751	7335	7132	7132
query15	218	177	174	174
query16	7237	521	471	471
query17	1103	596	595	595
query18	1936	315	315	315
query19	211	162	164	162
query20	123	114	107	107
query21	206	105	104	104
query22	4585	4513	4395	4395
query23	34708	34015	34329	34015
query24	6068	2859	2995	2859
query25	552	447	428	428
query26	651	165	169	165
query27	1959	356	360	356
query28	4160	2513	2506	2506
query29	682	455	437	437
query30	244	165	159	159
query31	1013	811	851	811
query32	70	59	51	51
query33	416	291	295	291
query34	931	513	523	513
query35	854	714	737	714
query36	1084	938	960	938
query37	123	68	66	66
query38	4100	4048	3894	3894
query39	1522	1471	1447	1447
query40	203	94	94	94
query41	56	46	46	46
query42	109	97	98	97
query43	535	492	497	492
query44	1167	817	838	817
query45	181	163	168	163
query46	1127	732	723	723
query47	2021	1906	1948	1906
query48	489	395	408	395
query49	723	396	385	385
query50	855	417	428	417
query51	7390	7092	7109	7092
query52	100	87	86	86
query53	246	178	182	178
query54	550	453	450	450
query55	79	76	74	74
query56	260	238	242	238
query57	1262	1145	1126	1126
query58	212	202	202	202
query59	3147	2934	2787	2787
query60	273	248	256	248
query61	112	106	107	106
query62	758	684	685	684
query63	222	188	190	188
query64	1361	675	624	624
query65	3224	3164	3182	3164
query66	625	289	300	289
query67	15872	15646	15600	15600
query68	4129	601	580	580
query69	411	265	263	263
query70	1261	1139	1100	1100
query71	336	264	258	258
query72	6383	4081	3942	3942
query73	763	348	349	348
query74	10313	8977	8912	8912
query75	3346	2628	2591	2591
query76	1874	1025	990	990
query77	491	269	266	266
query78	10559	9636	9628	9628
query79	1672	606	606	606
query80	1371	417	415	415
query81	523	235	242	235
query82	473	88	86	86
query83	160	140	149	140
query84	278	84	83	83
query85	962	301	284	284
query86	394	296	293	293
query87	4416	4277	4215	4215
query88	3790	2431	2401	2401
query89	406	286	286	286
query90	1863	182	183	182
query91	181	150	148	148
query92	67	49	64	49
query93	1804	549	554	549
query94	755	299	286	286
query95	354	253	258	253
query96	603	280	286	280
query97	3308	3194	3146	3146
query98	213	205	198	198
query99	1621	1280	1307	1280
Total cold run time: 316370 ms
Total hot run time: 196662 ms

doris-robot · 2025-03-20T05:15:16Z

ClickBench: Total hot run time: 33.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d17cb53202aea830d9b246777928d2e1d0ae09b6, data reload: false
query1	0.03	0.03	0.03
query2	0.08	0.04	0.05
query3	0.22	0.06	0.05
query4	1.65	0.07	0.08
query5	0.52	0.52	0.52
query6	1.13	0.75	0.75
query7	0.02	0.01	0.01
query8	0.06	0.04	0.05
query9	0.54	0.51	0.51
query10	0.56	0.56	0.57
query11	0.18	0.11	0.12
query12	0.15	0.13	0.13
query13	0.60	0.61	0.60
query14	2.75	2.78	2.77
query15	0.91	0.83	0.83
query16	0.38	0.37	0.39
query17	1.04	1.05	1.07
query18	0.19	0.19	0.18
query19	1.87	1.82	2.04
query20	0.02	0.02	0.02
query21	15.36	0.67	0.66
query22	3.61	7.57	2.20
query23	18.24	1.44	1.31
query24	2.24	0.23	0.22
query25	0.16	0.08	0.08
query26	0.27	0.17	0.18
query27	0.08	0.08	0.08
query28	13.26	0.60	0.59
query29	12.78	3.36	3.36
query30	0.25	0.06	0.06
query31	2.88	0.40	0.40
query32	3.23	0.48	0.48
query33	2.97	3.07	3.02
query34	17.07	4.62	4.61
query35	4.60	4.63	4.54
query36	0.66	0.51	0.48
query37	0.20	0.15	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.06	0.04
query42	0.06	0.05	0.05
query43	0.06	0.05	0.04
Total cold run time: 111.35 s
Total hot run time: 33.32 s

dataroaring

LGTM

github-actions · 2025-03-21T01:28:21Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-03-21T01:28:23Z

PR approved by anyone and no changes requested.

doris-robot · 2025-03-21T01:50:00Z

TeamCity cloud ut coverage result:
Function Coverage: 83.00% (1074/1294)
Line Coverage: 65.92% (17739/26909)
Region Coverage: 65.32% (8728/13362)
Branch Coverage: 55.29% (4710/8518)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d17cb53202aea830d9b246777928d2e1d0ae09b6_d17cb53202aea830d9b246777928d2e1d0ae09b6_cloud/report/index.html

hello-stephen · 2025-03-21T03:24:40Z

BE UT Coverage Report

Increment line coverage 0.00% (0/130) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	38.87% (10169/26163)
Line Coverage	30.29% (86732/286380)
Region Coverage	29.32% (44557/151968)
Branch Coverage	25.84% (22667/87718)

sollhui requested a review from dataroaring as a code owner March 20, 2025 03:11

dataroaring approved these changes Mar 21, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 21, 2025

github-actionsBot added the reviewed label Mar 21, 2025

dataroaring merged commit a298b78 into apache:branch-3.0Mar 25, 2025

gavinchou mentioned this pull request Apr 23, 2025
Release Notes 3.0.5 #50333
Closed

Navigation Menu

Uh oh!

[improve](information schema) introduce routine load job system table (#48963) - #49284

[improve](information schema) introduce routine load job system table (#48963)#49284
dataroaring merged 1 commit into
apache:branch-3.0from
sollhui:3.0_job_system_table

sollhui commented Mar 20, 2025

Uh oh!

sollhui commented Mar 20, 2025

Uh oh!

hello-stephen commented Mar 20, 2025

Uh oh!

doris-robot commented Mar 20, 2025

Uh oh!

doris-robot commented Mar 20, 2025

Uh oh!

doris-robot commented Mar 20, 2025

Uh oh!

dataroaring left a comment

Uh oh!

github-actionsBot commented Mar 21, 2025

Uh oh!

github-actionsBot commented Mar 21, 2025

Uh oh!

doris-robot commented Mar 21, 2025

Uh oh!

hello-stephen commented Mar 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

sollhui commented Mar 20, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

sollhui commented Mar 20, 2025

Uh oh!

hello-stephen commented Mar 20, 2025

Uh oh!

doris-robot commented Mar 20, 2025

Uh oh!

doris-robot commented Mar 20, 2025

Uh oh!

doris-robot commented Mar 20, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Mar 21, 2025

Uh oh!

github-actionsBot commented Mar 21, 2025

Uh oh!

doris-robot commented Mar 21, 2025

Uh oh!

hello-stephen commented Mar 21, 2025

BE UT Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants