Skip to content

branch-3.1: [opt](catalog) support nested namespaces of iceberg #56415 #56874 - #57035

Merged
morningman merged 2 commits into
apache:branch-3.1from
morningman:31_bp56415_56874
Oct 16, 2025
Merged

branch-3.1: [opt](catalog) support nested namespaces of iceberg #56415 #56874#57035
morningman merged 2 commits into
apache:branch-3.1from
morningman:31_bp56415_56874

Conversation

@morningman

@morningmanmorningman commented Oct 16, 2025

Copy link
Copy Markdown
Contributor

@morningman
[opt](catalog) support nested namespaces of iceberg (apache#56415)
9401738
Iceberg has 3 levels of metadata: catalog, namespace and table, mapping
to Doris' catalog, database and table.
Iceberg support nested namespaces, which means the following namespaces
are valid:
```
ns1
ns1.ns2
ns1.ns2.ns3
```
So we need to support mapping nested namespace to Doris' database.
This PR add a global variable `enable_nested_namespace` to control this
behavior.
Default is `false`, and no logic is changed.
If set to true, Doris can support following statments:
```
mysql> switch iceberg;
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| nested |
| nested.db1 |
| nested.db2 |
+--------------------+
mysql> use iceberg.nested.db1;
ERROR 1049 (42000): Only one dot can be in the name: iceberg.nested.db1
mysql> use iceberg.`nested.db1`;
ERROR 5086 (42000): errCode = 2, detailMessage = Unknown catalog 'nested'
mysql> set global enable_nested_namespace=true;
mysql> use iceberg.nested.db1;
Database changed
mysql> select k1 from iceberg.`nested.db1`.nested1;
mysql> select nested1.k1 from `nested.db1`.nested1;
mysql> select `nested.db1`.nested1.k1 from iceberg.`nested.db1`.nested1;
mysql> select iceberg.`nested.db1`.nested1.k1 from nested1;
+------+
| k1 |
+------+
| 1 |
+------+
mysql> refresh catalog iceberg;
mysql> refresh database iceberg.`nested.db1`;
mysql> refresh table iceberg.`nested.db1`.nested1;
Query OK, 0 rows affected (0.01 sec)
```
But, I can execute statement like:
```
use iceberg.`nested.db1`;
```
I don't know why, there is a very strange behavior in MySQL client, when
adding back quota,
the INIT_DB command can only receive `nested.db1` part, but expect
`iceberg.nested.db1`.
Also support creating nested database name in internal catalog:
```
create database `db1.db2`
```
@morningman
[fix](iceberg) fix logic of getting namespaces (apache#56874)
3448cca
Followup apache#56415
Problem Summary:
1. The previous `getNamespace` logic is wrong, we should split the
`dbName` by `.` to create namespaces.
2. Allow not specify `oauth.uri` of iceberg rest catalog, to follow the
new spec of IRC
So we can connect Snowflake open catalog like this:
```
CREATE CATALOG ice PROPERTIES (
'type' = 'iceberg',
'warehouse' = 'yy_external_catalog3',
'iceberg.catalog.type' = 'rest',
'iceberg.rest.uri' =
'https://xxx.snowflakecomputing.com/polaris/api/catalog',
'iceberg.rest.security.type' = 'oauth2',
'iceberg.rest.oauth2.credential' = 'id:secrete,
'iceberg.rest.oauth2.scope' = 'PRINCIPAL_ROLE:yy_sn_principal_role',
'iceberg.rest.nested-namespace-enabled' = 'true',
's3.endpoint' = 'https://s3.us-west-2.amazonaws.com',
's3.region' = 'us-west-2',
'iceberg.rest.nested-namespace-enabled' = 'true'
);
```
@Thearas

Thearas commented Oct 16, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman

morningman commented Oct 16, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Oct 16, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 32676 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3448cca7506f25bfcdc72256043b98eac03df1ff, data reload: false
------ Round 1 ----------------------------------
q1	17577	5481	5340	5340
q2	2032	405	305	305
q3	12364	1221	755	755
q4	10564	873	468	468
q5	9601	2388	2115	2115
q6	190	160	130	130
q7	909	766	613	613
q8	9333	1489	1203	1203
q9	5224	5029	4944	4944
q10	6786	2280	1839	1839
q11	462	284	263	263
q12	339	353	210	210
q13	17775	3611	3022	3022
q14	245	227	209	209
q15	526	456	462	456
q16	419	423	371	371
q17	600	868	374	374
q18	6961	6381	6530	6381
q19	1429	950	523	523
q20	315	337	203	203
q21	2846	2187	1952	1952
q22	1072	1028	1000	1000
Total cold run time: 107569 ms
Total hot run time: 32676 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5487	5496	5440	5440
q2	231	323	227	227
q3	2250	2590	2299	2299
q4	1371	1806	1382	1382
q5	4410	5029	4969	4969
q6	173	165	133	133
q7	2040	2017	1787	1787
q8	2592	2833	2727	2727
q9	7250	7226	7183	7183
q10	3018	3204	2742	2742
q11	584	524	474	474
q12	636	766	605	605
q13	3341	3738	3120	3120
q14	271	301	291	291
q15	515	483	456	456
q16	460	478	443	443
q17	1218	1744	1279	1279
q18	7657	7396	7496	7396
q19	773	1093	1102	1093
q20	2072	2064	1906	1906
q21	5406	5074	4588	4588
q22	1088	1098	1038	1038
Total cold run time: 52843 ms
Total hot run time: 51578 ms

@doris-robot

doris-robot commented Oct 16, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192433 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3448cca7506f25bfcdc72256043b98eac03df1ff, data reload: false
query1	944	400	397	397
query2	6185	1936	1880	1880
query3	8680	209	198	198
query4	33640	23804	23225	23225
query5	3728	608	496	496
query6	296	211	171	171
query7	4206	488	321	321
query8	309	255	236	236
query9	9333	2630	2624	2624
query10	486	320	258	258
query11	17998	15420	15216	15216
query12	155	111	109	109
query13	1562	557	424	424
query14	8884	7135	7090	7090
query15	234	188	178	178
query16	8038	635	555	555
query17	1585	796	609	609
query18	2126	428	353	353
query19	235	186	189	186
query20	131	123	125	123
query21	210	134	115	115
query22	4579	4802	4475	4475
query23	35091	34318	33876	33876
query24	7192	2665	2681	2665
query25	552	495	447	447
query26	875	294	173	173
query27	2107	489	365	365
query28	5172	2248	2221	2221
query29	727	630	471	471
query30	249	199	176	176
query31	1006	938	852	852
query32	117	57	57	57
query33	478	364	315	315
query34	740	869	509	509
query35	807	817	719	719
query36	1028	1072	934	934
query37	98	94	80	80
query38	4045	4030	3909	3909
query39	1535	1467	1450	1450
query40	198	117	105	105
query41	49	52	51	51
query42	122	103	108	103
query43	502	518	489	489
query44	1330	830	840	830
query45	181	176	178	176
query46	904	1052	695	695
query47	2018	2021	1935	1935
query48	400	424	350	350
query49	729	514	412	412
query50	665	708	428	428
query51	7439	7255	7308	7255
query52	104	103	98	98
query53	229	257	195	195
query54	546	575	489	489
query55	81	84	82	82
query56	273	277	266	266
query57	1293	1266	1219	1219
query58	232	218	210	210
query59	3137	3193	3153	3153
query60	302	312	281	281
query61	143	134	116	116
query62	786	761	698	698
query63	232	193	189	189
query64	3879	1015	656	656
query65	3344	3269	3306	3269
query66	846	415	311	311
query67	16507	15872	15700	15700
query68	7692	817	550	550
query69	496	319	269	269
query70	1216	1158	1079	1079
query71	383	294	297	294
query72	5828	3741	3785	3741
query73	643	734	350	350
query74	10187	9520	9071	9071
query75	3253	3153	2715	2715
query76	3142	1161	756	756
query77	601	375	280	280
query78	10410	10496	9598	9598
query79	2887	902	604	604
query80	686	520	440	440
query81	485	264	220	220
query82	546	117	89	89
query83	161	179	147	147
query84	240	98	80	80
query85	793	390	292	292
query86	399	311	296	296
query87	4290	4278	4291	4278
query88	4964	2385	2485	2385
query89	409	323	296	296
query90	1813	187	186	186
query91	131	145	110	110
query92	64	59	51	51
query93	2634	884	550	550
query94	680	413	292	292
query95	350	285	273	273
query96	495	622	282	282
query97	3157	3247	3149	3149
query98	224	203	197	197
query99	1356	1421	1294	1294
Total cold run time: 292449 ms
Total hot run time: 192433 ms

@doris-robot

doris-robot commented Oct 16, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3448cca7506f25bfcdc72256043b98eac03df1ff, data reload: false
query1	0.04	0.03	0.02
query2	0.07	0.03	0.02
query3	0.23	0.07	0.06
query4	1.63	0.11	0.11
query5	0.52	0.51	0.51
query6	1.12	0.73	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.54
query10	0.55	0.56	0.55
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	0.76	0.85	0.79
query15	0.85	0.82	0.82
query16	0.38	0.41	0.38
query17	1.00	1.06	1.05
query18	0.23	0.24	0.21
query19	1.81	1.90	1.88
query20	0.01	0.01	0.01
query21	15.38	0.97	0.58
query22	0.76	0.75	0.68
query23	15.16	1.43	0.58
query24	3.54	1.24	1.54
query25	0.18	0.13	0.17
query26	0.41	0.16	0.13
query27	0.05	0.05	0.05
query28	13.44	0.95	0.43
query29	12.56	3.96	3.28
query30	0.26	0.09	0.07
query31	2.83	0.57	0.39
query32	3.21	0.54	0.46
query33	2.99	3.04	3.00
query34	16.52	5.18	4.50
query35	4.59	4.55	4.60
query36	0.65	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.77 s
Total hot run time: 29.35 s

@morrySnowmorrySnow changed the title [opt](catalog) support nested namespaces of iceberg #56415 #56874branch-3.1: [opt](catalog) support nested namespaces of iceberg #56415 #56874Oct 16, 2025
@morningman
morningman merged commit 545394e into apache:branch-3.1Oct 16, 2025
24 checks passed
@morrySnowmorrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnowmorrySnowmorrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@morningman@Thearas@doris-robot@morrySnow