Skip to content

[Enhancement] support information_schema.partitions - #40153

Merged
morningman merged 3 commits into
apache:masterfrom
Vallishp:schema_partition
Sep 5, 2024
Merged

[Enhancement] support information_schema.partitions#40153
morningman merged 3 commits into
apache:masterfrom
Vallishp:schema_partition

Conversation

@Vallishp

@VallishpVallishp commented Aug 29, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx

select * from information_schema.partitions ; return empty.
this PR will fill the information based on partitions information available.
first only supported for olaptables.

@Vallishp
Vallishp marked this pull request as draft August 29, 2024 16:00
@doris-robot

doris-robot commented Aug 29, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions[bot]
github-actionsBot reviewed Aug 29, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/exec/schema_scanner/schema_scanner_helper.h
Comment threadbe/src/exec/schema_scanner/schema_scanner_helper.h
Comment threadbe/src/exec/schema_scanner/schema_scanner_helper.h
@zclllyybb

zclllyybb commented Aug 29, 2024

Copy link
Copy Markdown
Contributor

This will resolve a very common need. Thanks for your work.

@Vallishp
Vallishp marked this pull request as ready for review August 30, 2024 17:40
@Vallishp

Vallishp commented Aug 30, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

morningman
morningman reviewed Aug 31, 2024
View reviewed changes
Comment threadbe/src/exec/schema_scanner/schema_partitions_scanner.cpp Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java Outdated
ArrayList<Expr> expr = olapTable.getPartitionInfo().getPartitionExprs();
String partExpr = (expr != null) ? olapTable.getPartitionInfo().getPartitionExprs().toString() : "";

trow.addToColumnValue(new TCell().setStringVal(partExpr)); // PARTITION_EXPRESSION (not available)

@morningmanmorningmanAug 31, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
trow.addToColumnValue(newTCell().setStringVal(partExpr)); // PARTITION_EXPRESSION (not available)
trow.addToColumnValue(newTCell().setStringVal(partExpr)); // PARTITION_EXPRESSION

Comment threadfe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java Outdated
trow.addToColumnValue(new TCell().setStringVal(partExpr)); // PARTITION_EXPRESSION (not available)
trow.addToColumnValue(new TCell().setStringVal("")); // SUBPARTITION_EXPRESSION (not available)
trow.addToColumnValue(new TCell().setStringVal(
olapTable.getPartitionInfo().getPartitionRangeString(partition.getId())));

@morningmanmorningmanAug 31, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For partition range, how about refer to mysql?
image

@morningmanmorningmanAug 31, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

@VallishpVallishpSep 1, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

description is similar to mysql but since we support multiple partition column, still will come as a expression in the patition description. eg: [('4'), ('6'))

Comment threadfe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java Outdated
@Vallishp
Vallishp marked this pull request as draft August 31, 2024 21:17
@Vallishp
Vallishp marked this pull request as ready for review September 1, 2024 12:43
@Vallishp

Vallishp commented Sep 1, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

yiguolei
yiguolei reviewed Sep 2, 2024
View reviewed changes
Comment threadbe/src/exec/schema_scanner/schema_partitions_scanner.cpp
} else {
if (nullptr != _param->common_param->user) {
db_params.__set_user(*(_param->common_param->user));
}

@yiguoleiyiguoleiSep 2, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why remove these logic

@VallishpVallishpSep 2, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

user and user_ip params are deprecated in TGetDbsParams so removed it.

yiguolei
yiguolei reviewed Sep 2, 2024
View reviewed changes
Comment threadgensrc/thrift/FrontendService.thrift Outdated
TABLE_OPTIONS = 6,
WORKLOAD_GROUP_PRIVILEGES = 7,
TABLE_PROPERTIES = 8,
PARTITIONS,

@yiguoleiyiguoleiSep 2, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using PARTITIONS=9

@Vallishp
Vallishpforce-pushed the schema_partition branch 2 times, most recently from a79fb02 to 7e9b348CompareSeptember 2, 2024 15:03
morningman
morningman previously approved these changes Sep 3, 2024
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Vallishp
Vallishpforce-pushed the schema_partition branch 2 times, most recently from ee2f814 to 9d75e3eCompareSeptember 3, 2024 12:02
@Vallishp

Vallishp commented Sep 3, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@Vallishp

Vallishp commented Sep 3, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

github-actions[bot]
github-actionsBot reviewed Sep 3, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/exec/schema_scanner/schema_scanner_helper.h

#ifndef _SCHEMA_SCANNER_HELPER_H_

#include <stdint.h>

@github-actionsgithub-actionsBotSep 3, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: inclusion of deprecated C++ header 'stdint.h'; consider using 'cstdint' instead [modernize-deprecated-headers]

Suggested change
#include<stdint.h>
#include<cstdint>

@doris-robot

doris-robot commented Sep 3, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 36.82% (9385/25490)
Line Coverage: 28.25% (77399/273996)
Region Coverage: 27.66% (39952/144460)
Branch Coverage: 24.30% (20329/83654)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0672fe9b2fbe4987385c36f9fd5063999a5fbf06_0672fe9b2fbe4987385c36f9fd5063999a5fbf06/report/index.html

@doris-robot

doris-robot commented Sep 3, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 38823 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0672fe9b2fbe4987385c36f9fd5063999a5fbf06, data reload: false
------ Round 1 ----------------------------------
q1	18018	4707	4371	4371
q2	2034	190	175	175
q3	11807	1024	1181	1024
q4	10308	812	823	812
q5	7767	2911	2843	2843
q6	227	141	141	141
q7	964	611	601	601
q8	9339	2112	2136	2112
q9	7150	6577	6564	6564
q10	7015	2213	2235	2213
q11	437	249	242	242
q12	398	222	224	222
q13	17768	3108	3096	3096
q14	279	242	235	235
q15	524	492	492	492
q16	629	508	504	504
q17	996	716	788	716
q18	7396	6989	6950	6950
q19	1393	976	1052	976
q20	668	331	324	324
q21	3926	3169	3204	3169
q22	1158	1051	1041	1041
Total cold run time: 110201 ms
Total hot run time: 38823 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4524	4378	4320	4320
q2	389	272	269	269
q3	2905	2728	2649	2649
q4	1962	1642	1736	1642
q5	5716	5746	5795	5746
q6	232	139	137	137
q7	2179	1879	1870	1870
q8	3299	3475	3508	3475
q9	8948	8910	8843	8843
q10	3616	3385	3352	3352
q11	609	511	508	508
q12	854	652	636	636
q13	14874	3221	3251	3221
q14	324	285	275	275
q15	545	501	496	496
q16	651	590	608	590
q17	1841	1538	1568	1538
q18	8171	7890	7972	7890
q19	1727	1553	1412	1412
q20	2179	1960	1901	1901
q21	5808	5549	5602	5549
q22	1124	1078	1024	1024
Total cold run time: 72477 ms
Total hot run time: 57343 ms

@doris-robot

doris-robot commented Sep 3, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 194507 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0672fe9b2fbe4987385c36f9fd5063999a5fbf06, data reload: false
query1	1268	919	846	846
query2	6315	1992	2055	1992
query3	10766	4113	4008	4008
query4	60139	26820	23180	23180
query5	5429	531	526	526
query6	424	167	161	161
query7	5764	311	300	300
query8	291	213	208	208
query9	9004	2508	2517	2508
query10	507	283	281	281
query11	18016	14967	15480	14967
query12	158	99	98	98
query13	1561	403	391	391
query14	11081	7699	7827	7699
query15	244	182	200	182
query16	7358	503	481	481
query17	1094	593	556	556
query18	1621	303	298	298
query19	293	152	153	152
query20	127	114	119	114
query21	214	103	108	103
query22	4671	4462	4503	4462
query23	34685	33582	33820	33582
query24	5976	2962	3020	2962
query25	556	416	414	414
query26	692	167	158	158
query27	1772	280	293	280
query28	3708	2070	2047	2047
query29	680	443	446	443
query30	244	163	164	163
query31	936	773	800	773
query32	85	53	57	53
query33	478	305	305	305
query34	873	496	493	493
query35	859	733	749	733
query36	1059	934	967	934
query37	153	93	95	93
query38	3982	3884	3923	3884
query39	1455	1406	1394	1394
query40	206	128	121	121
query41	51	50	50	50
query42	115	96	97	96
query43	534	499	494	494
query44	1110	741	742	741
query45	201	170	169	169
query46	1110	760	750	750
query47	1906	1799	1798	1798
query48	386	301	298	298
query49	774	453	460	453
query50	831	427	432	427
query51	7042	6959	6863	6863
query52	102	89	88	88
query53	261	182	189	182
query54	595	469	466	466
query55	77	75	79	75
query56	290	275	283	275
query57	1218	1047	1062	1047
query58	227	257	246	246
query59	3228	2892	2933	2892
query60	308	287	277	277
query61	125	122	126	122
query62	752	679	667	667
query63	215	193	197	193
query64	2986	778	788	778
query65	3238	3179	3197	3179
query66	719	345	363	345
query67	15581	15205	15245	15205
query68	4376	557	563	557
query69	423	286	285	285
query70	1201	1054	1032	1032
query71	404	276	283	276
query72	6650	4037	3994	3994
query73	777	332	333	332
query74	9163	8848	8868	8848
query75	3383	2706	2750	2706
query76	1810	1007	970	970
query77	596	349	333	333
query78	9829	9659	9263	9263
query79	2785	545	547	545
query80	1286	537	523	523
query81	576	240	234	234
query82	974	144	149	144
query83	179	148	142	142
query84	264	79	80	79
query85	967	297	287	287
query86	398	307	344	307
query87	4432	4287	4285	4285
query88	3396	2350	2333	2333
query89	402	291	295	291
query90	1849	192	195	192
query91	131	107	104	104
query92	68	49	54	49
query93	2876	542	539	539
query94	794	319	281	281
query95	351	257	261	257
query96	628	270	266	266
query97	3215	3086	3073	3073
query98	225	208	204	204
query99	1528	1277	1255	1255
Total cold run time: 314662 ms
Total hot run time: 194507 ms

@Vallishp

Vallishp commented Sep 5, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Sep 5, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 36.83% (9392/25502)
Line Coverage: 28.26% (77446/274023)
Region Coverage: 27.65% (39952/144487)
Branch Coverage: 24.30% (20335/83676)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e4cf64da9ac99a54052b5f5e812f106a82b9df7b_e4cf64da9ac99a54052b5f5e812f106a82b9df7b/report/index.html

@doris-robot

doris-robot commented Sep 5, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 38276 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e4cf64da9ac99a54052b5f5e812f106a82b9df7b, data reload: false
------ Round 1 ----------------------------------
q1	18614	6505	4358	4358
q2	2026	192	179	179
q3	11930	933	1132	933
q4	10442	701	775	701
q5	7761	2879	3029	2879
q6	231	140	140	140
q7	989	630	604	604
q8	9331	2105	2096	2096
q9	7253	6575	6628	6575
q10	7017	2213	2284	2213
q11	466	238	246	238
q12	397	227	228	227
q13	17774	3127	3109	3109
q14	273	244	248	244
q15	532	492	505	492
q16	609	527	496	496
q17	999	704	721	704
q18	7499	6814	6965	6814
q19	1497	994	1049	994
q20	664	340	340	340
q21	3987	2934	3109	2934
q22	1123	1024	1006	1006
Total cold run time: 111414 ms
Total hot run time: 38276 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4352	4270	4296	4270
q2	393	264	270	264
q3	2876	2685	2640	2640
q4	1949	1678	1775	1678
q5	5735	5753	5697	5697
q6	226	135	138	135
q7	2284	1830	1834	1830
q8	3351	3548	3505	3505
q9	8891	8882	8871	8871
q10	3646	3325	3304	3304
q11	620	500	510	500
q12	888	684	672	672
q13	14526	3201	3309	3201
q14	311	292	290	290
q15	544	490	498	490
q16	624	588	609	588
q17	1883	1559	1614	1559
q18	8121	7888	7790	7790
q19	1795	1674	1558	1558
q20	2180	1913	1912	1912
q21	5757	5631	5486	5486
q22	1133	1069	1033	1033
Total cold run time: 72085 ms
Total hot run time: 57273 ms

@doris-robot

doris-robot commented Sep 5, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 193427 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e4cf64da9ac99a54052b5f5e812f106a82b9df7b, data reload: false
query1	1261	908	873	873
query2	6448	1947	1923	1923
query3	10600	3973	3992	3973
query4	59347	23908	23243	23243
query5	5286	504	487	487
query6	404	166	172	166
query7	5750	305	296	296
query8	331	227	232	227
query9	8933	2523	2521	2521
query10	467	272	266	266
query11	17931	15098	15238	15098
query12	153	108	100	100
query13	1534	417	380	380
query14	10953	7395	7723	7395
query15	236	177	178	177
query16	7164	500	505	500
query17	1095	586	563	563
query18	1887	289	286	286
query19	248	149	149	149
query20	117	112	109	109
query21	204	107	109	107
query22	4754	4333	4415	4333
query23	34433	33778	33426	33426
query24	5938	2840	2838	2838
query25	509	368	391	368
query26	671	154	151	151
query27	1762	278	282	278
query28	3607	2066	2041	2041
query29	653	409	406	406
query30	252	158	149	149
query31	955	762	741	741
query32	84	51	54	51
query33	421	293	296	293
query34	907	486	463	463
query35	820	714	731	714
query36	1056	957	960	957
query37	148	85	90	85
query38	4060	3799	3929	3799
query39	1450	1429	1373	1373
query40	192	112	110	110
query41	46	47	42	42
query42	114	95	94	94
query43	508	457	460	457
query44	1079	744	741	741
query45	194	169	167	167
query46	1083	723	747	723
query47	1897	1789	1818	1789
query48	371	300	298	298
query49	764	437	437	437
query50	814	417	414	414
query51	6923	6866	6904	6866
query52	101	88	89	88
query53	244	175	175	175
query54	557	453	459	453
query55	79	75	71	71
query56	267	254	245	245
query57	1200	1054	1060	1054
query58	265	232	232	232
query59	2910	2785	2706	2706
query60	311	275	266	266
query61	100	95	96	95
query62	754	654	674	654
query63	214	180	183	180
query64	2404	694	690	690
query65	3207	3177	3194	3177
query66	656	355	352	352
query67	15477	15272	15404	15272
query68	3022	596	604	596
query69	415	293	293	293
query70	1201	1119	1133	1119
query71	349	283	284	283
query72	5514	4164	4109	4109
query73	737	328	331	328
query74	9208	8920	8950	8920
query75	3422	2742	2698	2698
query76	1387	972	941	941
query77	538	332	331	331
query78	10160	9206	9653	9206
query79	1383	537	541	537
query80	902	533	516	516
query81	546	236	238	236
query82	1113	154	150	150
query83	179	160	158	158
query84	268	81	83	81
query85	885	361	347	347
query86	323	302	297	297
query87	4474	4221	4212	4212
query88	3283	2392	2518	2392
query89	391	281	282	281
query90	1949	189	189	189
query91	124	97	97	97
query92	59	48	47	47
query93	1262	541	537	537
query94	840	300	295	295
query95	351	250	250	250
query96	584	271	270	270
query97	3215	3105	3122	3105
query98	217	207	200	200
query99	1517	1253	1274	1253
Total cold run time: 305378 ms
Total hot run time: 193427 ms

@doris-robot

doris-robot commented Sep 5, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 32.14 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e4cf64da9ac99a54052b5f5e812f106a82b9df7b, data reload: false
query1	0.04	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.08
query5	0.54	0.50	0.50
query6	1.13	0.73	0.73
query7	0.01	0.01	0.01
query8	0.06	0.05	0.04
query9	0.55	0.50	0.48
query10	0.54	0.56	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.13
query13	0.61	0.59	0.60
query14	2.04	2.08	2.07
query15	0.85	0.84	0.80
query16	0.37	0.36	0.38
query17	1.00	0.99	1.01
query18	0.22	0.20	0.21
query19	1.86	1.82	1.72
query20	0.02	0.01	0.01
query21	15.40	0.66	0.65
query22	4.27	7.82	2.04
query23	18.24	1.53	1.30
query24	2.13	0.23	0.22
query25	0.16	0.08	0.08
query26	0.26	0.18	0.18
query27	0.07	0.08	0.07
query28	13.23	1.01	0.99
query29	12.58	3.38	3.36
query30	0.24	0.06	0.05
query31	2.87	0.41	0.41
query32	3.24	0.50	0.48
query33	2.99	2.99	3.13
query34	16.96	4.39	4.41
query35	4.48	4.40	4.39
query36	0.68	0.46	0.49
query37	0.17	0.16	0.15
query38	0.15	0.14	0.14
query39	0.05	0.04	0.04
query40	0.17	0.12	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 110.68 s
Total hot run time: 32.14 s

morningman
morningman approved these changes Sep 5, 2024
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 5, 2024
@github-actions

github-actionsBot commented Sep 5, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@morningman
morningman merged commit 271e1c4 into apache:masterSep 5, 2024
@morningmanmorningman mentioned this pull request Sep 9, 2024
Merged
@VallishpVallishp mentioned this pull request Sep 9, 2024
Merged
yiguolei pushed a commit that referenced this pull request Sep 10, 2024
@Vallishp
[fix](systable) refactor code to avoid wrong usage (#40568)
943ac04
## Proposed changes
Issue Number: close #xxx
refactor code to avoid wrong usage
Followup
#40153#40553 Test result:
mysql> select * from routines;
+---------------------+-----------------+----------------+---------------------+--------------+----------------+-----------------------------------------------------------------------------------------+--------------------+---------------+-------------------+-----------------+------------------+-----------------+----------+---------------+---------------------+---------------------+----------+-----------------+---------+----------------------+----------------------+--------------------+
| SPECIFIC_NAME | ROUTINE_CATALOG | ROUTINE_SCHEMA | ROUTINE_NAME |
ROUTINE_TYPE | DTD_IDENTIFIER | ROUTINE_BODY | ROUTINE_DEFINITION |
EXTERNAL_NAME | EXTERNAL_LANGUAGE | PARAMETER_STYLE | IS_DETERMINISTIC |
SQL_DATA_ACCESS | SQL_PATH | SECURITY_TYPE | CREATED | LAST_ALTERED |
SQL_MODE | ROUTINE_COMMENT | DEFINER | CHARACTER_SET_CLIENT |
COLLATION_CONNECTION | DATABASE_COLLATION |
+---------------------+-----------------+----------------+---------------------+--------------+----------------+-----------------------------------------------------------------------------------------+--------------------+---------------+-------------------+-----------------+------------------+-----------------+----------+---------------+---------------------+---------------------+----------+-----------------+---------+----------------------+----------------------+--------------------+
| TEST_PLSQL_ROUTINE2 | 0 | plsql_routine | TEST_PLSQL_ROUTINE2 |
PROCEDURE | | CREATE OR REPLACE PROCEDURE test_plsql_routine2() BEGIN
DECLARE a int = 1; print a; END | | NULL | | SQL | | | NULL | DEFINER |
2024-09-09 13:54:47 | 2024-09-09 13:54:47 | | | root | | | |
| TEST_PLSQL_ROUTINE3 | 0 | plsql_routine | TEST_PLSQL_ROUTINE3 |
PROCEDURE | | CREATE OR REPLACE PROCEDURE test_plsql_routine3() BEGIN
DECLARE a int = 1; print a; END | | NULL | | SQL | | | NULL | DEFINER |
2024-09-09 13:54:47 | 2024-09-09 13:54:47 | | | root | | | |
| TEST_PLSQL_ROUTINE4 | 0 | plsql_routine | TEST_PLSQL_ROUTINE4 |
PROCEDURE | | CREATE OR REPLACE PROCEDURE test_plsql_routine4() BEGIN
DECLARE a int = 1; print a; END | | NULL | | SQL | | | NULL | DEFINER |
2024-09-09 13:54:47 | 2024-09-09 13:54:47 | | | root | | | |
| TEST_PLSQL_ROUTINE5 | 0 | plsql_routine | TEST_PLSQL_ROUTINE5 |
PROCEDURE | | CREATE OR REPLACE PROCEDURE test_plsql_routine5() BEGIN
DECLARE a int = 1; print a; END | | NULL | | SQL | | | NULL | DEFINER |
2024-09-09 13:54:47 | 2024-09-09 13:54:47 | | | root | | | |
| TEST_PLSQL_ROUTINE1 | 0 | plsql_routine | TEST_PLSQL_ROUTINE1 |
PROCEDURE | | CREATE OR REPLACE PROCEDURE test_plsql_routine1() BEGIN
DECLARE a int = 1; print a; END | | NULL | | SQL | | | NULL | DEFINER |
2024-09-09 13:54:48 | 2024-09-09 13:54:48 | | | root | | | |
+---------------------+-----------------+----------------+---------------------+--------------+----------------+-----------------------------------------------------------------------------------------+--------------------+---------------+-------------------+-----------------+------------------+-------
mysql> select UPDATE_TIME from partitions;
+---------------------+
| UPDATE_TIME |
+---------------------+
| 2024-09-05 05:54:25 |
| 2024-09-05 05:54:25 |
| 2024-09-05 05:54:25 |
| 2024-09-09 13:27:46 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:27:46 |
| 2024-09-09 13:27:46 |
mysql> select * from active_queries;
+-----------------------------------+---------------------+---------------+-------------------+--------------------+-------------------+------------------+----------------+--------------+------------------------------+
| QUERY_ID | QUERY_START_TIME | QUERY_TIME_MS | WORKLOAD_GROUP_ID |
DATABASE | FRONTEND_INSTANCE | QUEUE_START_TIME | QUEUE_END_TIME |
QUERY_STATUS | SQL |
+-----------------------------------+---------------------+---------------+-------------------+--------------------+-------------------+------------------+----------------+--------------+------------------------------+
| 8c51705601bf4403-929f1b15051af48e | 2024-09-09 14:06:15 | 19 | 1 |
information_schema | 172.20.80.1 | | | | select * from active_queries |
+-----------------------------------+---------------------+---------------+-------------------+--------------------+-------------------+------------------+----------------+--------------+------------------------------+
This was referenced Sep 10, 2024
Merged
Closed
dataroaring pushed a commit that referenced this pull request Sep 11, 2024
@Vallishp@morningman
[branch 3.0] backport sys table related PR (#40568) (#40455) (#40456) (…
e9de19d
…#40153) (#40636)
## Proposed changes
#40568#40455#40456#40153
Test:
2024-09-10 14:46:14.206 INFO [suite-thread-1] (SuiteContext.groovy:299)
- Create new connection for user 'partitions_user'
2024-09-10 14:46:14.214 INFO [suite-thread-1] (Suite.groovy:1162) -
Execute tag: select_check_5, sql: select
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,PARTITION_NAME,SUBPARTITION_NAME,PARTITION_ORDINAL_POSITION,SUBPARTITION_ORDINAL_POSITION,PARTITION_METHOD,SUBPARTITION_METHOD,PARTITION_EXPRESSION,SUBPARTITION_EXPRESSION,PARTITION_DESCRIPTION,TABLE_ROWS,AVG_ROW_LENGTH,DATA_LENGTH,MAX_DATA_LENGTH,INDEX_LENGTH,DATA_FREE,CHECKSUM,PARTITION_COMMENT,NODEGROUP,TABLESPACE_NAME
from information_schema.partitions where
table_schema="test_partitions_schema_db" order by
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,PARTITION_NAME,SUBPARTITION_NAME,PARTITION_ORDINAL_POSITION,SUBPARTITION_ORDINAL_POSITION,PARTITION_METHOD,SUBPARTITION_METHOD,PARTITION_EXPRESSION,SUBPARTITION_EXPRESSION,PARTITION_DESCRIPTION,TABLE_ROWS,AVG_ROW_LENGTH,DATA_LENGTH,MAX_DATA_LENGTH,INDEX_LENGTH,DATA_FREE,CHECKSUM,PARTITION_COMMENT,NODEGROUP,TABLESPACE_NAME
2024-09-10 14:46:14.245 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-10 14:46:14.246 INFO [suite-thread-1] (ScriptContext.groovy:120)
- Run test_partitions_schema in
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_partitions_schema.groovy
succeed
2024-09-10 14:46:14.252 INFO [main] (RegressionTest.groovy:259) - Start
to run single scripts
2024-09-10 14:47:03.655 INFO [main] (RegressionTest.groovy:380) -
Success suites:
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_partitions_schema.groovy:
group=default,p0, name=test_partitions_schema
2024-09-10 14:47:03.656 INFO [main] (RegressionTest.groovy:459) - All
suites success.
____ _ ____ ____ _____ ____
| _ \ / \ / ___/ ___|| ____| _ \
| |_) / _ \ \___ \___ \| _| | | | |
| __/ ___ \ ___) |__) | |___| |_| |
|_| /_/ \_\____/____/|_____|____/
2024-09-10 14:47:03.656 INFO [main] (RegressionTest.groovy:410) - Test 1
suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts
2024-09-10 14:49:48.239 INFO [suite-thread-2] (Suite.groovy:1162) -
Execute tag: sql, sql: select * from triggers
2024-09-10 14:49:48.261 INFO [suite-thread-2] (Suite.groovy:1162) -
Execute tag: sql, sql: select * from parameters
2024-09-10 14:49:48.280 INFO [suite-thread-2] (Suite.groovy:1162) -
Execute tag: sql, sql: select * from profiling
2024-09-10 14:49:48.294 INFO [suite-thread-2] (ScriptContext.groovy:120)
- Run test_query_sys_tables in
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_query_sys_tables.groovy
succeed
2024-09-10 14:49:58.800 INFO [main] (RegressionTest.groovy:259) - Start
to run single scripts
2024-09-10 14:50:48.173 INFO [main] (RegressionTest.groovy:380) -
Success suites:
/root/doris/workspace/doris/regression-test/suites/nereids_p0/system/test_query_sys_tables.groovy:
group=query,p0, name=test_query_sys_tables
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_query_sys_tables.groovy:
group=query,p0, name=test_query_sys_tables
2024-09-10 14:50:48.174 INFO [main] (RegressionTest.groovy:459) - All
suites success.
____ _ ____ ____ _____ ____
| _ \ / \ / ___/ ___|| ____| _ \
| |_) / _ \ \___ \___ \| _| | | | |
| __/ ___ \ ___) |__) | |___| |_| |
|_| /_/ \_\____/____/|_____|____/
2024-09-10 14:50:48.175 INFO [main] (RegressionTest.groovy:410) - Test 2
suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts
2024-09-10 14:50:48.175 INFO [main] (RegressionTest.groovy:119) - Test
finished
---------
Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com>
@VallishpVallishp mentioned this pull request Sep 11, 2024
Merged
wangbo pushed a commit that referenced this pull request Sep 12, 2024
@morningman
[fix](systable) fix unstable case for partitions table (#40553)
9caefba
Followup #40153
yiguolei pushed a commit that referenced this pull request Sep 12, 2024
@Vallishp@morningman
[Branch 2.1] backport systable PR (#34384,#40153,#40456,#40455,#40568) (
3604d63
#40687)
backport
#40568#40455#40456#40153#34384
Test result:
2024-09-11 11:00:45.618 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:00:45.619 INFO [suite-thread-1] (Suite.groovy:359) -
Execute sql: REVOKE SELECT_PRIV ON
test_partitions_schema_db.duplicate_table FROM partitions_user
2024-09-11 11:00:45.625 INFO [suite-thread-1] (SuiteContext.groovy:299)
- Create new connection for user 'partitions_user'
2024-09-11 11:00:45.632 INFO [suite-thread-1] (Suite.groovy:1162) -
Execute tag: select_check_5, sql: select
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,PARTITION_NAME,SUBPARTITION_NAME,PARTITION_ORDINAL_POSITION,SUBPARTITION_ORDINAL_POSITION,PARTITION_METHOD,SUBPARTITION_METHOD,PARTITION_EXPRESSION,SUBPARTITION_EXPRESSION,PARTITION_DESCRIPTION,TABLE_ROWS,AVG_ROW_LENGTH,DATA_LENGTH,MAX_DATA_LENGTH,INDEX_LENGTH,DATA_FREE,CHECKSUM,PARTITION_COMMENT,NODEGROUP,TABLESPACE_NAME
from information_schema.partitions where
table_schema="test_partitions_schema_db" order by
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,PARTITION_NAME,SUBPARTITION_NAME,PARTITION_ORDINAL_POSITION,SUBPARTITION_ORDINAL_POSITION,PARTITION_METHOD,SUBPARTITION_METHOD,PARTITION_EXPRESSION,SUBPARTITION_EXPRESSION,PARTITION_DESCRIPTION,TABLE_ROWS,AVG_ROW_LENGTH,DATA_LENGTH,MAX_DATA_LENGTH,INDEX_LENGTH,DATA_FREE,CHECKSUM,PARTITION_COMMENT,NODEGROUP,TABLESPACE_NAME
2024-09-11 11:00:45.644 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:00:45.645 INFO [suite-thread-1] (ScriptContext.groovy:120)
- Run test_partitions_schema in
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_partitions_schema.groovy
succeed
2024-09-11 11:00:45.652 INFO [main] (RegressionTest.groovy:259) - Start
to run single scripts
2024-09-11 11:01:10.321 INFO [main] (RegressionTest.groovy:380) -
Success suites:
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_partitions_schema.groovy:
group=default,p0, name=test_partitions_schema
2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:459) - All
suites success.
____ _ ____ ____ _____ ____
| _ \ / \ / ___/ ___|| ____| _ \
| |_) / _ \ \___ \___ \| _| | | | |
| __/ ___ \ ___) |__) | |___| |_| |
|_| /_/ \_\____/____/|_____|____/
2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:410) - Test 1
suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts
2024-09-11 11:01:10.322 INFO [main] (RegressionTest.groovy:119) - Test
finished
2024-09-11 11:03:00.712 INFO [suite-thread-1] (Suite.groovy:1162) -
Execute tag: select_check_5, sql: select * from
information_schema.table_options ORDER BY
TABLE_CATALOG,TABLE_SCHEMA,TABLE_NAME,TABLE_MODEL,TABLE_MODEL_KEY,DISTRIBUTE_KEY,DISTRIBUTE_TYPE,BUCKETS_NUM,PARTITION_NUM;
2024-09-11 11:03:00.729 INFO [suite-thread-1] (SuiteContext.groovy:309)
- Recover original connection
2024-09-11 11:03:00.731 INFO [suite-thread-1] (ScriptContext.groovy:120)
- Run test_table_options in
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_table_options.groovy
succeed
2024-09-11 11:03:04.817 INFO [main] (RegressionTest.groovy:259) - Start
to run single scripts
2024-09-11 11:03:28.741 INFO [main] (RegressionTest.groovy:380) -
Success suites:
/root/doris/workspace/doris/regression-test/suites/query_p0/system/test_table_options.groovy:
group=default,p0, name=test_table_options
2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:459) - All
suites success.
____ _ ____ ____ _____ ____
| _ \ / \ / ___/ ___|| ____| _ \
| |_) / _ \ \___ \___ \| _| | | | |
| __/ ___ \ ___) |__) | |___| |_| |
|_| /_/ \_\____/____/|_____|____/
2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:410) - Test 1
suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts
2024-09-11 11:03:28.742 INFO [main] (RegressionTest.groovy:119) - Test
finished
*************************** 7. row ***************************
PartitionId: 18035
PartitionName: p100
VisibleVersion: 2
VisibleVersionTime: 2024-09-11 10:59:28
State: NORMAL
PartitionKey: col_1
Range: [types: [INT]; keys: [83647]; ..types: [INT]; keys: [2147483647];
)
DistributionKey: pk
Buckets: 10
ReplicationNum: 1
StorageMedium: HDD
CooldownTime: 9999-12-31 15:59:59
RemoteStoragePolicy: LastConsistencyCheckTime: NULL
DataSize: 2.872 KB
IsInMemory: false
ReplicaAllocation: tag.location.default: 1
IsMutable: true
SyncWithBaseTables: true
UnsyncTables: NULL
CommittedVersion: 2
RowCount: 4
7 rows in set (0.01 sec)
---------
Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com>
morningman added a commit to morningman/doris that referenced this pull request Sep 20, 2024
@morningman
[fix](systable) fix unstable case for partitions table (apache#40553)
5d40e54
Followup apache#40153
@morningmanmorningman mentioned this pull request Sep 20, 2024
Merged
morningman added a commit to morningman/doris that referenced this pull request Sep 25, 2024
@morningman
[fix](systable) fix unstable case for partitions table (apache#40553)
7978294
Followup apache#40153
@zclllyybb

zclllyybb commented Nov 4, 2024

Copy link
Copy Markdown
Contributor

Hi, I added document of this PR here. could u please review it? thx~

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@Vallishp@HappenLee
[Enhancement] support information_schema.partitions (apache#40153)
c5fe10a
select * from information_schema.partitions ; return empty.
this PR will fill the information based on partitions information
available.
first only supported for olaptables.
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@Vallishp@HappenLee
[fix](systable) refactor code to avoid wrong usage (apache#40568)
d323c09
## Proposed changes
Issue Number: close #xxx
refactor code to avoid wrong usage
Followup
apache#40153apache#40553 Test result:
mysql> select * from routines;
+---------------------+-----------------+----------------+---------------------+--------------+----------------+-----------------------------------------------------------------------------------------+--------------------+---------------+-------------------+-----------------+------------------+-----------------+----------+---------------+---------------------+---------------------+----------+-----------------+---------+----------------------+----------------------+--------------------+
| SPECIFIC_NAME | ROUTINE_CATALOG | ROUTINE_SCHEMA | ROUTINE_NAME |
ROUTINE_TYPE | DTD_IDENTIFIER | ROUTINE_BODY | ROUTINE_DEFINITION |
EXTERNAL_NAME | EXTERNAL_LANGUAGE | PARAMETER_STYLE | IS_DETERMINISTIC |
SQL_DATA_ACCESS | SQL_PATH | SECURITY_TYPE | CREATED | LAST_ALTERED |
SQL_MODE | ROUTINE_COMMENT | DEFINER | CHARACTER_SET_CLIENT |
COLLATION_CONNECTION | DATABASE_COLLATION |
+---------------------+-----------------+----------------+---------------------+--------------+----------------+-----------------------------------------------------------------------------------------+--------------------+---------------+-------------------+-----------------+------------------+-----------------+----------+---------------+---------------------+---------------------+----------+-----------------+---------+----------------------+----------------------+--------------------+
| TEST_PLSQL_ROUTINE2 | 0 | plsql_routine | TEST_PLSQL_ROUTINE2 |
PROCEDURE | | CREATE OR REPLACE PROCEDURE test_plsql_routine2() BEGIN
DECLARE a int = 1; print a; END | | NULL | | SQL | | | NULL | DEFINER |
2024-09-09 13:54:47 | 2024-09-09 13:54:47 | | | root | | | |
| TEST_PLSQL_ROUTINE3 | 0 | plsql_routine | TEST_PLSQL_ROUTINE3 |
PROCEDURE | | CREATE OR REPLACE PROCEDURE test_plsql_routine3() BEGIN
DECLARE a int = 1; print a; END | | NULL | | SQL | | | NULL | DEFINER |
2024-09-09 13:54:47 | 2024-09-09 13:54:47 | | | root | | | |
| TEST_PLSQL_ROUTINE4 | 0 | plsql_routine | TEST_PLSQL_ROUTINE4 |
PROCEDURE | | CREATE OR REPLACE PROCEDURE test_plsql_routine4() BEGIN
DECLARE a int = 1; print a; END | | NULL | | SQL | | | NULL | DEFINER |
2024-09-09 13:54:47 | 2024-09-09 13:54:47 | | | root | | | |
| TEST_PLSQL_ROUTINE5 | 0 | plsql_routine | TEST_PLSQL_ROUTINE5 |
PROCEDURE | | CREATE OR REPLACE PROCEDURE test_plsql_routine5() BEGIN
DECLARE a int = 1; print a; END | | NULL | | SQL | | | NULL | DEFINER |
2024-09-09 13:54:47 | 2024-09-09 13:54:47 | | | root | | | |
| TEST_PLSQL_ROUTINE1 | 0 | plsql_routine | TEST_PLSQL_ROUTINE1 |
PROCEDURE | | CREATE OR REPLACE PROCEDURE test_plsql_routine1() BEGIN
DECLARE a int = 1; print a; END | | NULL | | SQL | | | NULL | DEFINER |
2024-09-09 13:54:48 | 2024-09-09 13:54:48 | | | root | | | |
+---------------------+-----------------+----------------+---------------------+--------------+----------------+-----------------------------------------------------------------------------------------+--------------------+---------------+-------------------+-----------------+------------------+-------
mysql> select UPDATE_TIME from partitions;
+---------------------+
| UPDATE_TIME |
+---------------------+
| 2024-09-05 05:54:25 |
| 2024-09-05 05:54:25 |
| 2024-09-05 05:54:25 |
| 2024-09-09 13:27:46 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:26:20 |
| 2024-09-09 13:27:46 |
| 2024-09-09 13:27:46 |
mysql> select * from active_queries;
+-----------------------------------+---------------------+---------------+-------------------+--------------------+-------------------+------------------+----------------+--------------+------------------------------+
| QUERY_ID | QUERY_START_TIME | QUERY_TIME_MS | WORKLOAD_GROUP_ID |
DATABASE | FRONTEND_INSTANCE | QUEUE_START_TIME | QUEUE_END_TIME |
QUERY_STATUS | SQL |
+-----------------------------------+---------------------+---------------+-------------------+--------------------+-------------------+------------------+----------------+--------------+------------------------------+
| 8c51705601bf4403-929f1b15051af48e | 2024-09-09 14:06:15 | 19 | 1 |
information_schema | 172.20.80.1 | | | | select * from active_queries |
+-----------------------------------+---------------------+---------------+-------------------+--------------------+-------------------+------------------+----------------+--------------+------------------------------+
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@morningman@HappenLee
[fix](systable) fix unstable case for partitions table (apache#40553)
8183d43
Followup apache#40153
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actionsgithub-actions[bot]github-actions[bot] left review comments
@morningmanmorningmanmorningman approved these changes
@yiguoleiyiguoleiyiguolei approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.7-mergeddev/3.0.2-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Vallishp@doris-robot@zclllyybb@morningman@yiguolei@dataroaring