[fix](decimalv2) fix scale of decimalv2 to string by jacktengg · Pull Request #35222 · apache/doris

jacktengg · 2024-05-22T10:23:46Z

Proposed changes

Issue Number: close #xxx

Before:

mysql [test]>create table decimalv2_cast_to_string_test2 (k1 decimalv2(10,3)) distributed by hash(k1) properties("replication_num"="1");
Query OK, 0 rows affected (0.01 sec)
mysql [test]>insert into decimalv2_cast_to_string_test2 values (0), (1), (1.234);
Query OK, 3 rows affected (0.03 sec)
{'label':'insert_50e1fb842cb34a47_b994e1dff25ad5fb', 'status':'VISIBLE', 'txnId':'9'}
mysql [test]>select k1, cast(k1 as varchar) from decimalv2_cast_to_string_test2 order by 1;
+-------+----------------------------+
| k1 | cast(k1 as VARCHAR(65533)) |
+-------+----------------------------+
| 0.000 | 0.000000000 |
| 1.000 | 1.000000000 |
| 1.234 | 1.234000000 |
+-------+----------------------------+
3 rows in set (0.04 sec)

After change:

mysql [test]>create table decimalv2_cast_to_string_test2 (k1 decimalv2(10,3)) distributed by hash(k1) properties("replication_num"="1");
Query OK, 0 rows affected (0.01 sec)
mysql [test]>insert into decimalv2_cast_to_string_test2 values (0), (1), (1.234);
Query OK, 3 rows affected (0.10 sec)
{'label':'label_7de3eae3df3a4d34_999dca1fa6ba6c13', 'status':'VISIBLE', 'txnId':'3026'}
mysql [test]>select k1, cast(k1 as varchar) from decimalv2_cast_to_string_test2 order by 1;
+-------+----------------------------+
| k1 | cast(k1 as VARCHAR(65533)) |
+-------+----------------------------+
| 0.000 | 0.000 |
| 1.000 | 1.000 |
| 1.234 | 1.234 |
+-------+----------------------------+
3 rows in set (0.07 sec)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-22T10:23:52Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

jacktengg · 2024-05-22T10:23:57Z

run buildall

jacktengg · 2024-05-22T13:29:05Z

run p0

jacktengg · 2024-05-23T09:10:26Z

run buildall

github-actions

clang-tidy made some suggestions

doris-robot · 2024-05-23T10:29:50Z

TPC-H: Total hot run time: 41999 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e31bb98d71348dc7afde6977a17732ecea8ec6ea, data reload: false
------ Round 1 ----------------------------------
q1	17599	4534	4270	4270
q2	2026	199	193	193
q3	10431	1292	1226	1226
q4	10201	859	783	783
q5	7470	2827	2725	2725
q6	226	132	138	132
q7	985	612	617	612
q8	9230	2161	2134	2134
q9	9577	6716	6786	6716
q10	9399	4000	3856	3856
q11	454	247	237	237
q12	442	231	219	219
q13	17206	3160	3268	3160
q14	295	229	231	229
q15	508	472	482	472
q16	463	400	372	372
q17	998	725	688	688
q18	8455	7864	7742	7742
q19	3259	1588	1541	1541
q20	662	317	316	316
q21	5227	4090	4168	4090
q22	355	286	289	286
Total cold run time: 115468 ms
Total hot run time: 41999 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4669	4500	4331	4331
q2	387	265	274	265
q3	3175	3000	2845	2845
q4	1874	1581	1598	1581
q5	5524	5525	5535	5525
q6	224	126	126	126
q7	2215	1843	1838	1838
q8	3252	3421	3415	3415
q9	8670	8763	8691	8691
q10	4009	3751	3879	3751
q11	606	506	487	487
q12	791	637	632	632
q13	16954	3157	3128	3128
q14	288	281	294	281
q15	552	487	484	484
q16	499	436	454	436
q17	1837	1495	1469	1469
q18	7783	7677	7439	7439
q19	1679	1524	1521	1521
q20	2000	1799	1811	1799
q21	8840	4838	4784	4784
q22	569	490	479	479
Total cold run time: 76397 ms
Total hot run time: 55307 ms

doris-robot · 2024-05-23T10:40:36Z

TPC-DS: Total hot run time: 169178 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e31bb98d71348dc7afde6977a17732ecea8ec6ea, data reload: false
query1	930	389	373	373
query2	6435	2415	2461	2415
query3	6649	205	206	205
query4	19797	17336	17335	17335
query5	4118	420	430	420
query6	247	158	159	158
query7	4572	302	289	289
query8	238	185	184	184
query9	8629	2459	2416	2416
query10	445	295	255	255
query11	10687	10220	10068	10068
query12	137	100	93	93
query13	1653	369	389	369
query14	10006	7344	6885	6885
query15	224	174	173	173
query16	7731	262	261	261
query17	1710	527	523	523
query18	1940	271	263	263
query19	216	147	158	147
query20	95	84	84	84
query21	196	133	131	131
query22	4218	4079	3867	3867
query23	33531	32990	32982	32982
query24	10846	2769	2889	2769
query25	574	356	359	356
query26	710	159	155	155
query27	2218	330	329	329
query28	5809	2081	2078	2078
query29	857	616	602	602
query30	257	170	185	170
query31	977	781	754	754
query32	95	53	55	53
query33	680	297	259	259
query34	878	484	493	484
query35	717	600	592	592
query36	1092	902	920	902
query37	105	73	69	69
query38	2886	2745	2734	2734
query39	841	791	785	785
query40	193	125	124	124
query41	45	45	43	43
query42	108	96	98	96
query43	604	563	563	563
query44	1114	716	737	716
query45	188	170	174	170
query46	1059	699	699	699
query47	1849	1782	1785	1782
query48	379	300	306	300
query49	853	379	392	379
query50	768	383	392	383
query51	6686	6739	6662	6662
query52	102	88	92	88
query53	355	283	291	283
query54	902	445	425	425
query55	75	73	73	73
query56	266	234	241	234
query57	1128	1067	1073	1067
query58	228	204	210	204
query59	3291	3119	3137	3119
query60	267	257	247	247
query61	89	85	90	85
query62	654	459	451	451
query63	312	291	284	284
query64	8531	2198	1711	1711
query65	3133	3101	3104	3101
query66	796	333	315	315
query67	15266	14651	14863	14651
query68	4633	538	547	538
query69	442	266	275	266
query70	1186	1115	1142	1115
query71	419	276	271	271
query72	6943	5482	2746	2746
query73	733	333	328	328
query74	6075	5645	5573	5573
query75	3385	2597	2634	2597
query76	2831	976	979	976
query77	406	266	269	266
query78	10224	9861	9638	9638
query79	2250	514	505	505
query80	1268	437	435	435
query81	541	240	239	239
query82	1214	100	95	95
query83	242	172	171	171
query84	246	90	95	90
query85	1453	267	284	267
query86	466	336	314	314
query87	3274	3087	3114	3087
query88	4424	2422	2412	2412
query89	472	390	389	389
query90	2040	186	190	186
query91	125	96	98	96
query92	57	54	48	48
query93	2007	505	495	495
query94	1250	193	187	187
query95	412	307	304	304
query96	601	264	266	264
query97	3214	2986	3025	2986
query98	244	218	212	212
query99	1140	840	856	840
Total cold run time: 267404 ms
Total hot run time: 169178 ms

doris-robot · 2024-05-23T10:46:06Z

ClickBench: Total hot run time: 30.2 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e31bb98d71348dc7afde6977a17732ecea8ec6ea, data reload: false
query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.07
query5	0.57	0.47	0.51
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.54	0.49	0.50
query10	0.55	0.56	0.55
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.60	0.59	0.60
query14	0.76	0.78	0.78
query15	0.82	0.81	0.79
query16	0.35	0.36	0.35
query17	1.04	0.92	0.99
query18	0.22	0.22	0.26
query19	1.78	1.68	1.69
query20	0.01	0.02	0.01
query21	15.56	0.68	0.65
query22	4.77	7.51	1.66
query23	18.30	1.42	1.29
query24	1.57	0.26	0.22
query25	0.15	0.08	0.08
query26	0.27	0.17	0.17
query27	0.08	0.08	0.07
query28	13.46	1.01	0.99
query29	13.42	3.36	3.31
query30	0.24	0.06	0.06
query31	2.86	0.39	0.37
query32	3.28	0.47	0.47
query33	2.83	2.93	2.84
query34	16.90	4.46	4.44
query35	4.46	4.50	4.55
query36	0.68	0.48	0.46
query37	0.18	0.15	0.15
query38	0.16	0.16	0.14
query39	0.04	0.04	0.03
query40	0.17	0.15	0.15
query41	0.09	0.05	0.04
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.33 s
Total hot run time: 30.2 s

github-actions · 2024-05-23T11:22:10Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-23T11:22:12Z

PR approved by anyone and no changes requested.

jacktengg · 2024-05-23T14:48:47Z

run buildall

doris-robot · 2024-05-23T16:09:17Z

TPC-H: Total hot run time: 41204 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2c08ce7fa744c460f5f65d973ec1c4f47787f69b, data reload: false
------ Round 1 ----------------------------------
q1	17590	4359	4294	4294
q2	2019	186	190	186
q3	10472	1310	1239	1239
q4	10208	800	741	741
q5	7487	2736	2740	2736
q6	220	134	138	134
q7	977	609	605	605
q8	9229	2162	2115	2115
q9	9425	6764	6708	6708
q10	9304	3998	3878	3878
q11	440	237	262	237
q12	516	222	218	218
q13	17245	3187	3278	3187
q14	299	238	232	232
q15	508	457	482	457
q16	461	383	387	383
q17	1002	648	666	648
q18	8464	7826	7822	7822
q19	4008	1553	1504	1504
q20	633	346	318	318
q21	5226	3291	4129	3291
q22	358	275	271	271
Total cold run time: 116091 ms
Total hot run time: 41204 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4578	4426	4454	4426
q2	368	277	284	277
q3	3178	2986	2953	2953
q4	1983	1669	1662	1662
q5	5487	5550	5504	5504
q6	220	125	123	123
q7	2248	1803	1828	1803
q8	3248	3428	3416	3416
q9	8683	8756	8761	8756
q10	4077	3675	3856	3675
q11	609	493	491	491
q12	799	625	604	604
q13	15959	3165	3123	3123
q14	299	286	253	253
q15	529	487	488	487
q16	520	449	456	449
q17	1813	1534	1527	1527
q18	7891	7711	7620	7620
q19	1953	1552	1602	1552
q20	2026	1780	1811	1780
q21	9719	4750	4794	4750
q22	584	502	497	497
Total cold run time: 76771 ms
Total hot run time: 55728 ms

doris-robot · 2024-05-23T16:10:04Z

TeamCity be ut coverage result:
Function Coverage: 35.66% (9019/25295)
Line Coverage: 27.31% (74583/273062)
Region Coverage: 26.55% (38608/145434)
Branch Coverage: 23.40% (19689/84136)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2c08ce7fa744c460f5f65d973ec1c4f47787f69b_2c08ce7fa744c460f5f65d973ec1c4f47787f69b/report/index.html

doris-robot · 2024-05-23T16:19:45Z

TPC-DS: Total hot run time: 170701 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2c08ce7fa744c460f5f65d973ec1c4f47787f69b, data reload: false
query1	923	382	371	371
query2	6434	2567	2345	2345
query3	6640	207	209	207
query4	20433	17211	17334	17211
query5	4106	425	410	410
query6	243	156	149	149
query7	4587	296	289	289
query8	255	192	182	182
query9	8612	2364	2356	2356
query10	456	283	270	270
query11	10453	10163	10067	10067
query12	134	96	85	85
query13	1638	353	354	353
query14	10960	7409	6797	6797
query15	218	170	167	167
query16	7133	269	260	260
query17	1011	522	515	515
query18	1909	272	267	267
query19	208	150	153	150
query20	92	95	83	83
query21	198	127	131	127
query22	4336	3998	3935	3935
query23	33530	33106	33127	33106
query24	5087	2807	2895	2807
query25	459	346	356	346
query26	692	161	167	161
query27	1819	314	323	314
query28	3650	2058	2035	2035
query29	843	604	602	602
query30	218	174	173	173
query31	920	721	724	721
query32	57	56	52	52
query33	448	262	259	259
query34	838	469	478	469
query35	696	589	606	589
query36	1036	927	890	890
query37	105	69	73	69
query38	2889	2792	2765	2765
query39	856	793	821	793
query40	199	121	125	121
query41	44	44	43	43
query42	106	96	95	95
query43	551	532	525	525
query44	1081	726	723	723
query45	177	172	161	161
query46	1052	716	695	695
query47	1839	1767	1760	1760
query48	366	297	281	281
query49	760	375	388	375
query50	762	377	379	377
query51	6962	6850	6840	6840
query52	106	87	87	87
query53	346	284	289	284
query54	528	440	420	420
query55	74	71	75	71
query56	301	240	243	240
query57	1104	1073	1036	1036
query58	264	212	212	212
query59	3251	3261	3143	3143
query60	268	253	253	253
query61	89	85	85	85
query62	547	482	436	436
query63	304	282	288	282
query64	2524	1786	1766	1766
query65	3128	3072	3111	3072
query66	657	332	331	331
query67	15215	14748	14644	14644
query68	4576	536	549	536
query69	443	268	267	267
query70	1145	1097	1152	1097
query71	411	269	264	264
query72	7599	5715	4229	4229
query73	702	321	321	321
query74	6051	5646	5644	5644
query75	3260	2625	2634	2625
query76	2291	1030	978	978
query77	391	266	270	266
query78	10414	9943	9806	9806
query79	1426	498	520	498
query80	1013	438	436	436
query81	513	237	241	237
query82	928	94	93	93
query83	242	167	168	167
query84	265	85	80	80
query85	907	264	255	255
query86	399	306	296	296
query87	3345	3165	3143	3143
query88	3495	2313	2333	2313
query89	466	381	379	379
query90	1933	182	184	182
query91	125	95	99	95
query92	55	49	50	49
query93	1097	515	500	500
query94	1098	181	186	181
query95	408	304	305	304
query96	573	265	261	261
query97	3228	3030	3018	3018
query98	238	220	222	220
query99	1246	856	868	856
Total cold run time: 248010 ms
Total hot run time: 170701 ms

doris-robot · 2024-05-23T16:25:15Z

ClickBench: Total hot run time: 30.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2c08ce7fa744c460f5f65d973ec1c4f47787f69b, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.50	0.49	0.49
query6	1.12	0.74	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.48
query10	0.54	0.55	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.58
query14	0.77	0.78	0.77
query15	0.84	0.81	0.81
query16	0.37	0.36	0.37
query17	1.05	1.01	1.03
query18	0.22	0.23	0.23
query19	1.89	1.74	1.75
query20	0.02	0.01	0.01
query21	15.49	0.67	0.66
query22	4.90	7.36	1.42
query23	18.28	1.35	1.20
query24	1.88	0.24	0.19
query25	0.14	0.09	0.09
query26	0.25	0.16	0.17
query27	0.07	0.08	0.07
query28	13.39	1.02	1.00
query29	13.30	3.33	3.33
query30	0.24	0.06	0.06
query31	2.89	0.37	0.38
query32	3.28	0.48	0.47
query33	2.83	2.90	2.91
query34	17.11	4.37	4.48
query35	4.52	4.48	4.57
query36	0.65	0.46	0.46
query37	0.17	0.16	0.15
query38	0.15	0.15	0.14
query39	0.05	0.03	0.04
query40	0.16	0.13	0.15
query41	0.10	0.06	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 110.79 s
Total hot run time: 30.02 s

jacktengg · 2024-05-23T16:52:30Z

run buildall

* [fix](decimalv2) fix scale of decimalv2 to string

…che#35332)

* [fix](decimalv2) fix scale of decimalv2 to string

…recision/frac (apache#63569) Pick: apache#63569 Problem Summary: After upgrading from 3.1.4 to 4.0.4, queries on DecimalV2 columns fail with: [INTERNAL_ERROR]meet invalid precision: real_precision=0, max_decimal_precision=27, min_decimal_precision=1 Root cause: Segments written by Doris < 2.1.0 (before apache#26572) do not persist precision/frac in ColumnMetaPB; they default to 0 when read back. Since apache#35222, DataTypeFactory passes those raw values as original_precision/original_scale into DataTypeDecimalV2, which calls check_type_precision(0) and throws. 3.1.4 was unaffected because the old code hardcoded (27, 9). Fix: In _create_primitive_data_type for OLAP_FIELD_TYPE_DECIMAL, when precision is not positive (legacy segment), pass UINT32_MAX to DataTypeDecimalV2 to signal that the original precision/scale are unknown. DecimalScaleInfo<TYPE_DECIMALV2> already treats UINT32_MAX as 'unknown' and falls back to the in-memory (27, 9) representation in get_original_precision()/get_original_scale(). Fix "meet invalid precision: real_precision=0" when querying DecimalV2 columns on segments written by Doris versions older than 2.1.0. - Test: Unit Test - Added DataTypeDecimalTest.create_decimalv2_from_legacy_tablet_column covering: legacy TabletColumn (unset precision/frac), modern TabletColumn with decimal(26,6), and legacy segment_v2::ColumnMetaPB with default-0 precision/frac. - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Issue Number: close #xxx Problem Summary: After upgrading from 3.1.4 to 4.0.4, queries on DecimalV2 columns fail with: [INTERNAL_ERROR]meet invalid precision: real_precision=0, max_decimal_precision=27, min_decimal_precision=1 Root cause: Segments written by Doris < 2.1.0 (before apache#26572) do not persist precision/frac in ColumnMetaPB; they default to 0 when read back. Since apache#35222, DataTypeFactory passes those raw values as original_precision/original_scale into DataTypeDecimalV2, which calls check_type_precision(0) and throws. 3.1.4 was unaffected because the old code hardcoded (27, 9). Fix: In _create_primitive_data_type for OLAP_FIELD_TYPE_DECIMAL, when precision is not positive (legacy segment), pass UINT32_MAX to DataTypeDecimalV2 to signal that the original precision/scale are unknown. DecimalScaleInfo<TYPE_DECIMALV2> already treats UINT32_MAX as 'unknown' and falls back to the in-memory (27, 9) representation in get_original_precision()/get_original_scale(). Fix "meet invalid precision: real_precision=0" when querying DecimalV2 columns on segments written by Doris versions older than 2.1.0. - Test: Unit Test - Added DataTypeDecimalTest.create_decimalv2_from_legacy_tablet_column covering: legacy TabletColumn (unset precision/frac), modern TabletColumn with decimal(26,6), and legacy segment_v2::ColumnMetaPB with default-0 precision/frac. PS: error callstack: ``` I20260525 16:41:28.185905 1945583 pipeline_fragment_context.cpp:177] PipelineFragmentContext::cancel|query_id=18982a6c19ee4166-ae3ba44bce094741|fragment_id=1|reason=[INTERNAL_ERROR]meet invalid precision: real_precision=0, max_decimal_precision=27, min_decimal_precision=1 0# doris::get_stack_trace(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/util/stack_util.cpp:59 1# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&, bool) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/common/exception.cpp:31 2# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/common/exception.h:38 3# doris::Exception::Exception<unsigned int const&, unsigned long>(int, std::basic_string_view<char, std::char_traits<char> > const&, unsigned int const&, unsigned long&&) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/common/exception.h:44 4# doris::vectorized::DataTypeDecimal<(doris::PrimitiveType)20>::check_type_precision(unsigned int) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/data_types/data_type_decimal.h:280 5# doris::vectorized::DataTypeDecimal<(doris::PrimitiveType)20>::DataTypeDecimal(unsigned int, unsigned int, unsigned int, unsigned int) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/data_types/data_type_decimal.h:144 6# void std::_Construct<doris::vectorized::DataTypeDecimal<(doris::PrimitiveType)20>, int, int, int&, int&>(doris::vectorized::DataTypeDecimal<(doris::PrimitiveType)20>*, int&&, int&&, int&, int&) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_construct.h:134 7# std::_Sp_counted_ptr_inplace<doris::vectorized::DataTypeDecimal<(doris::PrimitiveType)20>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<int, int, int&, int&>(std::allocator<void>, int&&, int&&, int&, int&) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/alloc_traits.h:805 8# std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<doris::vectorized::DataTypeDecimal<(doris::PrimitiveType)20>, std::allocator<void>, int, int, int&, int&>(doris::vectorized::DataTypeDecimal<(doris::PrimitiveType)20>*&, std::_Sp_alloc_shared_tag<std::allocator<void> >, int&&, int&&, int&, int&) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:970 9# std::__shared_ptr<doris::vectorized::DataTypeDecimal<(doris::PrimitiveType)20>, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<void>, int, int, int&, int&>(std::_Sp_alloc_shared_tag<std::allocator<void> >, int&&, int&&, int&, int&) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1719 10# std::shared_ptr<doris::vectorized::DataTypeDecimal<(doris::PrimitiveType)20> >::shared_ptr<std::allocator<void>, int, int, int&, int&>(std::_Sp_alloc_shared_tag<std::allocator<void> >, int&&, int&&, int&, int&) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr.h:464 11# std::shared_ptr<doris::vectorized::DataTypeDecimal<(doris::PrimitiveType)20> > std::make_shared<doris::vectorized::DataTypeDecimal<(doris::PrimitiveType)20>, int, int, int&, int&>(int&&, int&&, int&, int&) at /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr.h:1007 12# doris::vectorized::DataTypeFactory::_create_primitive_data_type(doris::FieldType const&, int, int, int) const at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/data_types/data_type_factory.cpp:196 13# doris::vectorized::DataTypeFactory::create_data_type(doris::segment_v2::ColumnMetaPB const&) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/data_types/data_type_factory.cpp:394 14# doris::segment_v2::ColumnReader::ColumnReader(doris::segment_v2::ColumnReaderOptions const&, doris::segment_v2::ColumnMetaPB const&, unsigned long, std::shared_ptr<doris::io::FileReader>) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/segment_v2/column_reader.cpp:290 15# doris::segment_v2::ColumnReader::create(doris::segment_v2::ColumnReaderOptions const&, doris::segment_v2::ColumnMetaPB const&, unsigned long, std::shared_ptr<doris::io::FileReader> const&, std::shared_ptr<doris::segment_v2::ColumnReader>*) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/segment_v2/column_reader.cpp:242 16# doris::segment_v2::ColumnReaderCache::get_column_reader(int, std::shared_ptr<doris::segment_v2::ColumnReader>*, doris::OlapReaderStatistics*) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/segment_v2/column_reader_cache.cpp:138 17# doris::segment_v2::Segment::get_column_reader(int, std::shared_ptr<doris::segment_v2::ColumnReader>*, doris::OlapReaderStatistics*) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/segment_v2/segment.cpp:738 18# doris::segment_v2::Segment::new_column_iterator(doris::TabletColumn const&, std::unique_ptr<doris::segment_v2::ColumnIterator, std::default_delete<doris::segment_v2::ColumnIterator> >*, doris::StorageReadOptions const*, std::unordered_map<int, std::unique_ptr<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<doris::segment_v2::SparseColumnCache>, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<doris::segment_v2::SparseColumnCache> > > >, std::default_delete<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<doris::segment_v2::SparseColumnCache>, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<doris::segment_v2::SparseColumnCache> > > > > >, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::unique_ptr<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<doris::segment_v2::SparseColumnCache>, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<doris::segment_v2::SparseColumnCache> > > >, std::default_delete<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<doris::segment_v2::SparseColumnCache>, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<doris::segment_v2::SparseColumnCache> > > > > > > > > const*) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/segment_v2/segment.cpp:679 19# doris::segment_v2::SegmentIterator::_init_return_column_iterators() at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1391 20# doris::segment_v2::SegmentIterator::init_iterators() at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/segment_v2/segment_iterator.cpp:396 21# doris::segment_v2::SegmentIterator::_init_impl(doris::StorageReadOptions const&) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/segment_v2/segment_iterator.cpp:372 22# doris::segment_v2::SegmentIterator::init(doris::StorageReadOptions const&) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/segment_v2/segment_iterator.cpp:290 23# doris::segment_v2::Segment::new_iterator(std::shared_ptr<doris::Schema const>, doris::StorageReadOptions const&, std::unique_ptr<doris::RowwiseIterator, std::default_delete<doris::RowwiseIterator> >*) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/segment_v2/segment.cpp:0 24# doris::segment_v2::LazyInitSegmentIterator::init(doris::StorageReadOptions const&) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.cpp:48 25# doris::BetaRowsetReader::_init_iterator() at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/beta_rowset_reader.cpp:330 26# doris::BetaRowsetReader::_init_iterator_once()::$_0::operator()() const at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/beta_rowset_reader.cpp:295 27# doris::Status doris::DorisCallOnce<doris::Status>::call<doris::BetaRowsetReader::_init_iterator_once()::$_0>(doris::BetaRowsetReader::_init_iterator_once()::$_0) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/util/once.h:92 28# doris::BetaRowsetReader::_init_iterator_once() at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/beta_rowset_reader.cpp:295 29# doris::Status doris::BetaRowsetReader::_next_batch<doris::vectorized::Block>(doris::vectorized::Block*) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/beta_rowset_reader.h:103 30# doris::BetaRowsetReader::next_batch(doris::vectorized::Block*) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/olap/rowset/beta_rowset_reader.h:56 31# doris::vectorized::VCollectIterator::Level0Iterator::_refresh() at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/olap/vcollect_iterator.h:0 32# doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row() at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/olap/vcollect_iterator.cpp:526 33# doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref() at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/olap/vcollect_iterator.cpp:497 34# doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref() at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/olap/vcollect_iterator.cpp:715 35# doris::vectorized::VCollectIterator::build_heap(std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > >&) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/olap/vcollect_iterator.cpp:190 36# doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/olap/block_reader.cpp:152 37# doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/olap/block_reader.cpp:226 38# doris::vectorized::OlapScanner::_open_impl(doris::RuntimeState*) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/exec/scan/olap_scanner.cpp:266 39# doris::vectorized::Scanner::open(doris::RuntimeState*) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/exec/scan/scanner.h:84 40# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/exec/scan/scanner_scheduler.cpp:187 41# doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()apache#1}::operator()() const::{lambda()apache#1}::operator()() const at /mnt/disk2/tengjianping/doris-4.0-dev/be/src/vec/exec/scan/scanner_scheduler.cpp:76 ``` Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…recision/frac (apache#63569) Pick: apache#63569 Problem Summary: After upgrading from 3.1.4 to 4.0.4, queries on DecimalV2 columns fail with: [INTERNAL_ERROR]meet invalid precision: real_precision=0, max_decimal_precision=27, min_decimal_precision=1 Root cause: Segments written by Doris < 2.1.0 (before apache#26572) do not persist precision/frac in ColumnMetaPB; they default to 0 when read back. Since apache#35222, DataTypeFactory passes those raw values as original_precision/original_scale into DataTypeDecimalV2, which calls check_type_precision(0) and throws. 3.1.4 was unaffected because the old code hardcoded (27, 9). Fix: In _create_primitive_data_type for OLAP_FIELD_TYPE_DECIMAL, when precision is not positive (legacy segment), pass UINT32_MAX to DataTypeDecimalV2 to signal that the original precision/scale are unknown. DecimalScaleInfo<TYPE_DECIMALV2> already treats UINT32_MAX as 'unknown' and falls back to the in-memory (27, 9) representation in get_original_precision()/get_original_scale(). Fix "meet invalid precision: real_precision=0" when querying DecimalV2 columns on segments written by Doris versions older than 2.1.0. - Test: Unit Test - Added DataTypeDecimalTest.create_decimalv2_from_legacy_tablet_column covering: legacy TabletColumn (unset precision/frac), modern TabletColumn with decimal(26,6), and legacy segment_v2::ColumnMetaPB with default-0 precision/frac. - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

#63569) ### What problem does this PR solve? Issue Number: close #xxx Problem Summary: After upgrading from 3.1.4 to 4.0.4, queries on DecimalV2 columns fail with: [INTERNAL_ERROR]meet invalid precision: real_precision=0, max_decimal_precision=27, min_decimal_precision=1 Root cause: Segments written by Doris < 2.1.0 (before #26572) do not persist precision/frac in ColumnMetaPB; they default to 0 when read back. Since #35222, DataTypeFactory passes those raw values as original_precision/original_scale into DataTypeDecimalV2, which calls check_type_precision(0) and throws. 3.1.4 was unaffected because the old code hardcoded (27, 9). Fix: In _create_primitive_data_type for OLAP_FIELD_TYPE_DECIMAL, when precision is not positive (legacy segment), pass UINT32_MAX to DataTypeDecimalV2 to signal that the original precision/scale are unknown. DecimalScaleInfo<TYPE_DECIMALV2> already treats UINT32_MAX as 'unknown' and falls back to the in-memory (27, 9) representation in get_original_precision()/get_original_scale(). ### Release note Fix "meet invalid precision: real_precision=0" when querying DecimalV2 columns on segments written by Doris versions older than 2.1.0. ### Check List (For Author) - Test: Unit Test - Added DataTypeDecimalTest.create_decimalv2_from_legacy_tablet_column covering: legacy TabletColumn (unset precision/frac), modern TabletColumn with decimal(26,6), and legacy segment_v2::ColumnMetaPB with default-0 precision/frac. - Behavior changed: No - Does this need documentation: No ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…recision/frac (#63569) (#63581) Pick: #63569 Problem Summary: After upgrading from 3.1.4 to 4.0.4, queries on DecimalV2 columns fail with: [INTERNAL_ERROR]meet invalid precision: real_precision=0, max_decimal_precision=27, min_decimal_precision=1 Root cause: Segments written by Doris < 2.1.0 (before #26572) do not persist precision/frac in ColumnMetaPB; they default to 0 when read back. Since #35222, DataTypeFactory passes those raw values as original_precision/original_scale into DataTypeDecimalV2, which calls check_type_precision(0) and throws. 3.1.4 was unaffected because the old code hardcoded (27, 9). Fix: In _create_primitive_data_type for OLAP_FIELD_TYPE_DECIMAL, when precision is not positive (legacy segment), pass UINT32_MAX to DataTypeDecimalV2 to signal that the original precision/scale are unknown. DecimalScaleInfo<TYPE_DECIMALV2> already treats UINT32_MAX as 'unknown' and falls back to the in-memory (27, 9) representation in get_original_precision()/get_original_scale(). Fix "meet invalid precision: real_precision=0" when querying DecimalV2 columns on segments written by Doris versions older than 2.1.0. - Test: Unit Test - Added DataTypeDecimalTest.create_decimalv2_from_legacy_tablet_column covering: legacy TabletColumn (unset precision/frac), modern TabletColumn with decimal(26,6), and legacy segment_v2::ColumnMetaPB with default-0 precision/frac. - Behavior changed: No - Does this need documentation: No

apache#63569) ### What problem does this PR solve? Issue Number: close #xxx Problem Summary: After upgrading from 3.1.4 to 4.0.4, queries on DecimalV2 columns fail with: [INTERNAL_ERROR]meet invalid precision: real_precision=0, max_decimal_precision=27, min_decimal_precision=1 Root cause: Segments written by Doris < 2.1.0 (before apache#26572) do not persist precision/frac in ColumnMetaPB; they default to 0 when read back. Since apache#35222, DataTypeFactory passes those raw values as original_precision/original_scale into DataTypeDecimalV2, which calls check_type_precision(0) and throws. 3.1.4 was unaffected because the old code hardcoded (27, 9). Fix: In _create_primitive_data_type for OLAP_FIELD_TYPE_DECIMAL, when precision is not positive (legacy segment), pass UINT32_MAX to DataTypeDecimalV2 to signal that the original precision/scale are unknown. DecimalScaleInfo<TYPE_DECIMALV2> already treats UINT32_MAX as 'unknown' and falls back to the in-memory (27, 9) representation in get_original_precision()/get_original_scale(). ### Release note Fix "meet invalid precision: real_precision=0" when querying DecimalV2 columns on segments written by Doris versions older than 2.1.0. ### Check List (For Author) - Test: Unit Test - Added DataTypeDecimalTest.create_decimalv2_from_legacy_tablet_column covering: legacy TabletColumn (unset precision/frac), modern TabletColumn with decimal(26,6), and legacy segment_v2::ColumnMetaPB with default-0 precision/frac. - Behavior changed: No - Does this need documentation: No ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jacktengg force-pushed the fix-0522-2 branch from 10dfe12 to c7381a5Compare May 23, 2024 09:09

jacktengg changed the title ~~[improvement](decimal) add test case of casting decimal to string~~[fix](decimalv2) fix scale of decimalv2 to stringMay 23, 2024

jacktengg force-pushed the fix-0522-2 branch from c7381a5 to e31bb98Compare May 23, 2024 09:10

github-actionsBot reviewed May 23, 2024
View reviewed changes

Comment threadbe/src/vec/data_types/data_type_decimal.h

yiguolei previously approved these changes May 23, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 23, 2024

github-actionsBot added the reviewed label May 23, 2024

jacktengg dismissed yiguolei’s stale review via 1622925May 23, 2024 14:48

jacktengg force-pushed the fix-0522-2 branch 2 times, most recently from 1622925 to 2c08ce7Compare May 23, 2024 14:48

jacktengg force-pushed the fix-0522-2 branch from 2c08ce7 to 53bcd2fCompare May 23, 2024 16:52

jacktengg added 3 commits May 24, 2024 00:52

[fix](decimalv2) fix scale of decimalv2 to string
4360368

fix be ut
3a1191b

add test cases
a955c59

jacktengg force-pushed the fix-0522-2 branch from 53bcd2f to a955c59Compare May 23, 2024 16:52

yiguolei approved these changes May 24, 2024
View reviewed changes

jacktengg merged commit 49c12b6 into apache:masterMay 24, 2024

jacktengg added a commit to jacktengg/incubator-doris that referenced this pull request May 24, 2024
[fix](decimalv2) fix scale of decimalv2 to string (apache#35222)
bfb740a

jacktengg mentioned this pull request May 24, 2024
[fix](decimalv2) fix scale of decimalv2 to string (#35222) #35332
Merged

jacktengg added a commit that referenced this pull request May 24, 2024
[fix](decimalv2) fix scale of decimalv2 to string (#35222)
f20eff2

yiguolei pushed a commit that referenced this pull request May 24, 2024
[fix](decimalv2) fix scale of decimalv2 to string (#35222) (#35332)
9ce4f74

yiguolei added dev/2.0.11-merged 2.1.0-conflict labels May 24, 2024

jacktengg added a commit to jacktengg/incubator-doris that referenced this pull request May 24, 2024
[fix](decimalv2) fix scale of decimalv2 to string (apache#35222)
8d25a43
* [fix](decimalv2) fix scale of decimalv2 to string

jacktengg mentioned this pull request May 24, 2024
[fix](decimalv2) fix scale of decimalv2 to string (#35222) #35359
Merged

yiguolei pushed a commit that referenced this pull request May 24, 2024
[fix](decimalv2) fix scale of decimalv2 to string (#35222) (#35359)
639c7ee
* [fix](decimalv2) fix scale of decimalv2 to string

yiguolei added dev/2.1.4-merged and removed 2.1.0-conflict labels May 24, 2024

dataroaring pushed a commit that referenced this pull request May 26, 2024
[fix](decimalv2) fix scale of decimalv2 to string (#35222)
e1237d4
* [fix](decimalv2) fix scale of decimalv2 to string

seawinde pushed a commit to seawinde/doris that referenced this pull request May 27, 2024
[fix](decimalv2) fix scale of decimalv2 to string (apache#35222)
f753dd1
* [fix](decimalv2) fix scale of decimalv2 to string

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[fix](decimalv2) fix scale of decimalv2 to string (apache#35222) (apa…
8e9a188
…che#35332)

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[fix](decimalv2) fix scale of decimalv2 to string (apache#35222)
aa1aeb2
* [fix](decimalv2) fix scale of decimalv2 to string

jacktengg mentioned this pull request May 24, 2026
[fix](be) Handle legacy DecimalV2 segments with missing precision/frac #63569
Merged
16 tasks

jacktengg mentioned this pull request May 25, 2026
branch-4.0: [fix](be) Handle legacy DecimalV2 segments with missing precision/frac (#63569) #63581
Merged
16 tasks

Uh oh!

Conversation

jacktengg commented May 22, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

jacktengg commented May 22, 2024

Uh oh!

jacktengg commented May 22, 2024

Uh oh!

jacktengg commented May 23, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

github-actionsBot commented May 23, 2024

Uh oh!

github-actionsBot commented May 23, 2024

Uh oh!

jacktengg commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

jacktengg commented May 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jacktengg commented May 22, 2024•
edited
Loading