(tools) Add show_segment_data operation to meta_tool by gavinchou · Pull Request #60608 · apache/doris

gavinchou · 2026-02-09T03:44:08Z

This is an improvement for debug tool meta_tool.

Add new operation show_segment_data to meta_tool for reading and displaying segment file column metadata and statistics.

Add get_field_type_string() to convert FieldType to readable string
Add get_encoding_string() to convert EncodingTypePB to readable string
Add get_compression_string() to convert CompressionTypePB to readable string
Add print_column_meta() to print column metadata recursively
Add show_segment_data() to display segment file info and statistics

This allows users to inspect segment file structure without requiring the full tablet schema.

meta_tool --operation=show_segment_data --file=020000000000004fd4412b500cb0833afcc5dd12e833c59b_0.dat
=== Segment File Info ===
File: 020000000000004fd4412b500cb0833afcc5dd12e833c59b_0.dat
Num Rows: 2
Num Columns: 16
Compression: LZ4F
=== column_id_0: type=VARCHAR, nullable=false, encoding=DICT_ENCODING ===
Dictionary Page: offset=56, size=53 bytes
Indexes: ORDINAL, ZONE_MAP
Data Values (2 of 2 rows, showing first 2):
[0] '10023--1-id'
[1] '10023--1-score'
=== column_id_1: type=VARCHAR, nullable=false, encoding=DICT_ENCODING ===
Dictionary Page: offset=164, size=25 bytes
Indexes: ORDINAL, ZONE_MAP
Data Values (2 of 2 rows, showing first 2):
[0] '0'
[1] '0'
...

Add new operation show_segment_data to meta_tool for reading and displaying segment file column metadata and statistics. - Add get_field_type_string() to convert FieldType to readable string - Add get_encoding_string() to convert EncodingTypePB to readable string - Add get_compression_string() to convert CompressionTypePB to readable string - Add print_column_meta() to print column metadata recursively - Add show_segment_data() to display segment file info and statistics This allows users to inspect segment file structure without requiring the full tablet schema. ``` meta_tool --operation=show_segment_data --file=020000000000004fd4412b500cb0833afcc5dd12e833c59b_0.dat === Segment File Info === File: 020000000000004fd4412b500cb0833afcc5dd12e833c59b_0.dat Num Rows: 2 Num Columns: 16 Compression: LZ4F === column_id_0: type=VARCHAR, nullable=false, encoding=DICT_ENCODING === Dictionary Page: offset=56, size=53 bytes Indexes: ORDINAL, ZONE_MAP Data Values (2 of 2 rows, showing first 2): [0] '10023--1-id' [1] '10023--1-score' === column_id_1: type=VARCHAR, nullable=false, encoding=DICT_ENCODING === Dictionary Page: offset=164, size=25 bytes Indexes: ORDINAL, ZONE_MAP Data Values (2 of 2 rows, showing first 2): [0] '0' [1] '0' ... ```

Thearas · 2026-02-09T03:44:16Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

gavinchou · 2026-02-09T03:44:17Z

run buildall

doris-robot · 2026-02-09T04:17:57Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.35% (1795/2262)
Line Coverage	64.87% (31962/49269)
Region Coverage	65.54% (15942/24323)
Branch Coverage	56.10% (8479/15114)

doris-robot · 2026-02-09T04:36:33Z

TPC-H: Total hot run time: 30049 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6631cdd17c2f908bcd5591663323bc43632d4e6b, data reload: false
------ Round 1 ----------------------------------
q1	17598	4538	4301	4301
q2	2073	345	247	247
q3	10123	1306	709	709
q4	10192	779	302	302
q5	7528	2230	1891	1891
q6	195	177	145	145
q7	868	733	603	603
q8	9265	1429	1188	1188
q9	4912	4664	4582	4582
q10	6798	1925	1544	1544
q11	541	318	278	278
q12	405	378	223	223
q13	17789	4032	3226	3226
q14	231	230	209	209
q15	913	814	795	795
q16	697	675	636	636
q17	695	847	480	480
q18	6646	5797	5649	5649
q19	1488	980	613	613
q20	512	502	389	389
q21	2561	1792	1762	1762
q22	349	310	277	277
Total cold run time: 102379 ms
Total hot run time: 30049 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4361	4359	4322	4322
q2	251	353	256	256
q3	2102	2694	2191	2191
q4	1343	1725	1267	1267
q5	4291	4182	4377	4182
q6	211	177	135	135
q7	1854	1815	1660	1660
q8	2439	2682	2402	2402
q9	7571	7432	7577	7432
q10	2805	3050	2711	2711
q11	559	494	484	484
q12	674	760	629	629
q13	3913	4463	3717	3717
q14	310	310	291	291
q15	847	861	820	820
q16	661	756	693	693
q17	1162	1339	1326	1326
q18	8222	8165	7987	7987
q19	886	816	854	816
q20	2029	2198	2046	2046
q21	5003	4487	4362	4362
q22	539	538	493	493
Total cold run time: 52033 ms
Total hot run time: 50222 ms

doris-robot · 2026-02-09T04:53:14Z

ClickBench: Total hot run time: 28.2 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6631cdd17c2f908bcd5591663323bc43632d4e6b, data reload: false
query1	0.05	0.05	0.05
query2	0.09	0.04	0.04
query3	0.26	0.09	0.08
query4	1.60	0.11	0.11
query5	0.25	0.26	0.26
query6	1.16	0.67	0.67
query7	0.03	0.03	0.03
query8	0.05	0.03	0.03
query9	0.56	0.49	0.51
query10	0.54	0.55	0.55
query11	0.14	0.10	0.09
query12	0.13	0.10	0.10
query13	0.63	0.63	0.61
query14	1.06	1.07	1.04
query15	0.87	0.87	0.88
query16	0.40	0.39	0.39
query17	1.14	1.09	1.12
query18	0.23	0.21	0.21
query19	2.04	2.02	2.09
query20	0.02	0.02	0.02
query21	15.39	0.26	0.15
query22	4.90	0.05	0.05
query23	15.80	0.30	0.11
query24	1.52	0.42	0.19
query25	0.09	0.09	0.07
query26	0.14	0.14	0.13
query27	0.08	0.06	0.06
query28	3.18	1.16	0.97
query29	12.59	3.88	3.17
query30	0.28	0.13	0.11
query31	2.81	0.64	0.40
query32	3.24	0.59	0.51
query33	3.20	3.21	3.33
query34	16.53	5.36	4.70
query35	4.76	4.73	4.87
query36	0.64	0.49	0.51
query37	0.10	0.07	0.07
query38	0.08	0.04	0.04
query39	0.05	0.03	0.03
query40	0.20	0.16	0.16
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 97.01 s
Total hot run time: 28.2 s

gavinchou · 2026-02-09T05:02:44Z

run buildall

doris-robot · 2026-02-09T05:22:09Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.35% (1795/2262)
Line Coverage	64.87% (31962/49269)
Region Coverage	65.57% (15949/24323)
Branch Coverage	56.09% (8477/15114)

doris-robot · 2026-02-09T05:25:27Z

TPC-H: Total hot run time: 30651 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 197a79bb5c3130315b494018b2fa1ebcbdccedea, data reload: false
------ Round 1 ----------------------------------
q1	17630	4471	4284	4284
q2	1999	335	228	228
q3	10200	1275	696	696
q4	10194	814	308	308
q5	7531	2168	1935	1935
q6	201	175	143	143
q7	870	709	597	597
q8	9256	1389	1111	1111
q9	4652	4603	4582	4582
q10	6779	1946	1534	1534
q11	521	294	279	279
q12	334	378	217	217
q13	17766	4062	3239	3239
q14	233	230	231	230
q15	868	815	805	805
q16	683	675	627	627
q17	680	765	532	532
q18	6698	6047	6014	6014
q19	1151	1098	696	696
q20	570	523	398	398
q21	2739	1985	1891	1891
q22	373	320	305	305
Total cold run time: 101928 ms
Total hot run time: 30651 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4618	4460	4571	4460
q2	368	335	258	258
q3	2555	2890	2495	2495
q4	1494	2061	1388	1388
q5	4557	4568	4428	4428
q6	232	189	141	141
q7	1947	1835	1772	1772
q8	2586	2415	2457	2415
q9	7444	7592	7515	7515
q10	2896	2928	2650	2650
q11	567	470	451	451
q12	660	756	581	581
q13	3871	4441	3505	3505
q14	316	346	300	300
q15	852	819	822	819
q16	655	689	639	639
q17	1069	1213	1203	1203
q18	7701	7421	7278	7278
q19	863	799	779	779
q20	1980	2020	1876	1876
q21	4503	4237	4055	4055
q22	572	533	526	526
Total cold run time: 52306 ms
Total hot run time: 49534 ms

doris-robot · 2026-02-09T05:42:10Z

ClickBench: Total hot run time: 28.16 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 197a79bb5c3130315b494018b2fa1ebcbdccedea, data reload: false
query1	0.06	0.05	0.05
query2	0.09	0.05	0.04
query3	0.25	0.08	0.08
query4	1.61	0.12	0.12
query5	0.27	0.24	0.26
query6	1.17	0.67	0.67
query7	0.03	0.03	0.02
query8	0.05	0.04	0.04
query9	0.56	0.50	0.49
query10	0.56	0.54	0.54
query11	0.15	0.09	0.10
query12	0.13	0.10	0.10
query13	0.63	0.61	0.62
query14	1.05	1.05	1.04
query15	0.88	0.86	0.88
query16	0.39	0.41	0.39
query17	1.09	1.09	1.08
query18	0.23	0.22	0.21
query19	2.03	2.01	2.00
query20	0.02	0.01	0.01
query21	15.44	0.29	0.15
query22	4.87	0.08	0.05
query23	15.82	0.28	0.11
query24	1.94	0.40	0.23
query25	0.13	0.07	0.06
query26	0.15	0.14	0.14
query27	0.07	0.09	0.05
query28	4.02	1.13	0.97
query29	12.58	3.88	3.17
query30	0.27	0.12	0.12
query31	2.82	0.64	0.40
query32	3.24	0.60	0.50
query33	3.17	3.27	3.24
query34	16.36	5.37	4.69
query35	4.85	4.84	4.72
query36	0.63	0.49	0.49
query37	0.10	0.07	0.07
query38	0.07	0.04	0.05
query39	0.04	0.03	0.03
query40	0.19	0.15	0.16
query41	0.10	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 98.2 s
Total hot run time: 28.16 s

doris-robot · 2026-02-09T06:15:15Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.73% (19445/36880)
Line Coverage	36.22% (181020/499841)
Region Coverage	32.63% (140669/431124)
Branch Coverage	33.64% (60893/181033)

gavinchou · 2026-02-09T10:19:01Z

run compile

gavinchou · 2026-02-10T07:52:58Z

run buildall

gavinchou · 2026-02-10T07:54:00Z

run buildall

doris-robot · 2026-02-10T08:24:16Z

TPC-H: Total hot run time: 30515 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 448730abdc4c89aea62aa0931c10b192fb51e516, data reload: false
------ Round 1 ----------------------------------
q1	17590	4600	4314	4314
q2	2003	348	256	256
q3	10152	1264	718	718
q4	10194	772	318	318
q5	7513	2191	1865	1865
q6	199	176	147	147
q7	898	746	607	607
q8	9263	1389	1148	1148
q9	4705	4621	4639	4621
q10	6855	1906	1520	1520
q11	541	305	288	288
q12	339	377	221	221
q13	17793	4016	3229	3229
q14	232	232	223	223
q15	876	812	799	799
q16	690	740	622	622
q17	714	807	507	507
q18	6289	5770	5920	5770
q19	1809	1094	646	646
q20	564	529	446	446
q21	2736	1974	1951	1951
q22	403	329	299	299
Total cold run time: 102358 ms
Total hot run time: 30515 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4561	4471	4672	4471
q2	308	350	259	259
q3	2294	2973	2489	2489
q4	1564	1839	1412	1412
q5	4568	4805	4539	4539
q6	246	193	146	146
q7	2073	1969	1829	1829
q8	2561	2426	2499	2426
q9	7835	7760	7337	7337
q10	2960	3176	2588	2588
q11	558	502	445	445
q12	652	713	572	572
q13	3675	4180	3293	3293
q14	273	299	277	277
q15	843	795	807	795
q16	641	699	660	660
q17	1109	1253	1264	1253
q18	7897	7465	7199	7199
q19	829	791	819	791
q20	1955	2076	1867	1867
q21	4461	4186	4160	4160
q22	569	573	533	533
Total cold run time: 52432 ms
Total hot run time: 49341 ms

doris-robot · 2026-02-10T08:40:54Z

ClickBench: Total hot run time: 28.28 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 448730abdc4c89aea62aa0931c10b192fb51e516, data reload: false
query1	0.06	0.04	0.04
query2	0.10	0.05	0.05
query3	0.26	0.09	0.08
query4	1.61	0.11	0.11
query5	0.27	0.26	0.25
query6	1.16	0.69	0.66
query7	0.04	0.02	0.03
query8	0.05	0.04	0.04
query9	0.57	0.50	0.49
query10	0.55	0.55	0.56
query11	0.13	0.09	0.09
query12	0.14	0.10	0.11
query13	0.63	0.61	0.62
query14	1.06	1.06	1.07
query15	0.88	0.87	0.85
query16	0.42	0.41	0.38
query17	1.14	1.16	1.18
query18	0.23	0.22	0.21
query19	2.11	1.97	2.06
query20	0.02	0.02	0.02
query21	15.41	0.28	0.16
query22	5.04	0.05	0.05
query23	15.90	0.30	0.11
query24	0.99	0.66	0.29
query25	0.06	0.06	0.07
query26	0.15	0.13	0.14
query27	0.09	0.05	0.06
query28	3.19	1.18	0.98
query29	12.55	3.89	3.14
query30	0.28	0.14	0.12
query31	2.82	0.64	0.40
query32	3.24	0.60	0.50
query33	3.36	3.33	3.23
query34	16.27	5.41	4.69
query35	4.79	4.79	4.76
query36	0.65	0.50	0.49
query37	0.11	0.07	0.07
query38	0.08	0.05	0.04
query39	0.05	0.03	0.04
query40	0.21	0.16	0.15
query41	0.08	0.03	0.04
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.84 s
Total hot run time: 28.28 s

doris-robot · 2026-02-10T09:08:45Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.71% (19450/36897)
Line Coverage	36.20% (181086/500190)
Region Coverage	32.60% (140648/431458)
Branch Coverage	33.63% (60948/181219)

hello-stephen · 2026-02-10T11:05:34Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.66% (25908/36152)
Line Coverage	54.27% (270794/498951)
Region Coverage	51.83% (225883/435838)
Branch Coverage	53.28% (96924/181923)

github-actions · 2026-02-11T09:10:09Z

PR approved by anyone and no changes requested.

github-actions · 2026-02-11T09:46:59Z

PR approved by at least one committer and no changes requested.

) ## Summary Pick #60608 to branch-4.1 Add new operation show_segment_data to meta_tool for reading and displaying segment file column metadata and statistics. ## Changes - Add get_field_type_string() to convert FieldType to readable string - Add get_encoding_string() to convert EncodingTypePB to readable string - Add get_compression_string() to convert CompressionTypePB to readable string - Add print_column_meta() to print column metadata recursively - Add show_segment_data() to display segment file info and statistics - Add common/cpp/private_member_accessor.hpp helper for accessing private members ## Test - [x] Compilation passed (BE meta_tool target) ## Related Issue close#60608 --------- Co-authored-by: gavinchou <gavinchou@apache.org>

gavinchou added the area/tools label Feb 9, 2026

Fix compile
197a79b

Fix namespace
eaae9d7

fix compile
789a640

format
448730a

wyxxxcat approved these changes Feb 11, 2026
View reviewed changes

github-actionsBot added the reviewed label Feb 11, 2026

w41ter approved these changes Feb 11, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 11, 2026

gavinchou added the dev/4.1.x label Feb 11, 2026

gavinchou merged commit d489a79 into apache:masterFeb 11, 2026
30 of 31 checks passed

gavinchou mentioned this pull request Apr 11, 2026
branch-4.1: (tools) Add show_segment_data operation to meta_tool #62385
Merged
1 task

yiguolei added dev/4.1.1-merged and removed dev/4.1.x labels Apr 17, 2026

yiguolei mentioned this pull request May 20, 2026
4.1.1 Release Notes #63426
Closed

Uh oh!

Conversation

gavinchou commented Feb 9, 2026

Uh oh!

Thearas commented Feb 9, 2026

Uh oh!

gavinchou commented Feb 9, 2026

Uh oh!

doris-robot commented Feb 9, 2026

Cloud UT Coverage Report

Uh oh!

doris-robot commented Feb 9, 2026

Uh oh!

doris-robot commented Feb 9, 2026

Uh oh!

gavinchou commented Feb 9, 2026

Uh oh!

doris-robot commented Feb 9, 2026

Cloud UT Coverage Report

Uh oh!

doris-robot commented Feb 9, 2026

Uh oh!

doris-robot commented Feb 9, 2026

Uh oh!

doris-robot commented Feb 9, 2026

BE UT Coverage Report

Uh oh!

gavinchou commented Feb 9, 2026

Uh oh!

gavinchou commented Feb 10, 2026

Uh oh!

gavinchou commented Feb 10, 2026

Uh oh!

doris-robot commented Feb 10, 2026

Uh oh!

doris-robot commented Feb 10, 2026

Uh oh!

doris-robot commented Feb 10, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Feb 10, 2026

BE Regression && UT Coverage Report

Uh oh!

github-actionsBot commented Feb 11, 2026

Uh oh!

github-actionsBot commented Feb 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants