[feature](hive)Support read hive4 transaction tables. by hubgeter · Pull Request #44001 · apache/doris

hubgeter · 2024-11-15T03:11:46Z

What problem does this PR solve?

Problem Summary:
Support read hive4 transaction tables.

Since there are compilation problems when directly using hive4 API to get the file need to read, a function called ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#getAcidState is implemented.
At the same time, it is forbidden to create, insert data, and delete aicd tables.
fix 'transactional_properties' = 'insert_only' table read Incorrect data read.

TODO : mege HMSTransaction ,HiveTransaction class
mege HiveTransactionMgr, HiveTransactionManager class

Release note

Support read hive4 transaction tables.
Fixed the problem that the result of reading insert only transaction table in Hive catalog is incorrect.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-15T03:11:51Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

github-actions · 2024-11-15T03:16:07Z

clang-tidy review says "All clean, LGTM! 👍"

hubgeter · 2024-12-11T03:27:01Z

+ //Since the hive3 library cannot read the hive4 transaction table normally, and there are many problems
+ // when using the Hive 4 library directly, this method is implemented.
+ //Ref: hive/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#getAcidState
+ public static FileCacheValue getAcidState(RemoteFileSystem fileSystem, HivePartition partition,


notice: fixfix hive table with cosn location issue (#23409)

github-actions · 2024-12-16T07:27:46Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-12-16T07:32:36Z

clang-tidy review says "All clean, LGTM! 👍"

hubgeter · 2024-12-16T07:35:04Z

run buildall

hubgeter · 2024-12-16T07:37:28Z

run buildall

github-actions · 2024-12-16T07:41:37Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-16T08:14:27Z

TeamCity be ut coverage result:
Function Coverage: 38.84% (10123/26065)
Line Coverage: 29.75% (85054/285897)
Region Coverage: 28.80% (43621/151462)
Branch Coverage: 25.34% (22154/87436)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3b4388790a35b415fe4c9feebc91b09018e35d06_3b4388790a35b415fe4c9feebc91b09018e35d06/report/index.html

morningman · 2024-12-17T09:34:23Z

- } else {
- throw new RuntimeException(status.getErrMsg());
- }
+ fileCacheValues.add(AcidUtil.getAcidState(


also need ugi.doAs

morningman · 2024-12-17T09:43:53Z

+ } catch (Exception e) {
+ // Release shared load (getValidWriteIds acquire Lock).
+ // If no exception is throw, the lock will be released when `finalizeQuery()`.
+ Env.getCurrentHiveTransactionMgr().deregister(hiveTransaction.getQueryId());


It is not a good design to deregister transaction here.
Looks like it is very error prone

hubgeter · 2024-12-19T12:57:06Z

run buildall

github-actions · 2024-12-19T13:01:08Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-12-19T13:52:44Z

TeamCity be ut coverage result:
Function Coverage: 38.88% (10132/26062)
Line Coverage: 29.78% (85171/285961)
Region Coverage: 28.90% (43456/150366)
Branch Coverage: 25.43% (22155/87126)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3fbe1a53c82d65a4b67eae27d40e2c879b9864ef_3fbe1a53c82d65a4b67eae27d40e2c879b9864ef/report/index.html

github-actions · 2024-12-20T05:44:25Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-20T05:44:27Z

PR approved by anyone and no changes requested.

hubgeter · 2024-12-20T19:33:25Z

run buildall

doris-robot · 2024-12-20T20:12:46Z

TeamCity be ut coverage result:
Function Coverage: 38.87% (10131/26065)
Line Coverage: 29.79% (85242/286170)
Region Coverage: 28.92% (43534/150532)
Branch Coverage: 25.43% (22166/87180)
Coverage Report: http://coverage.selectdb-in.cc/coverage/25d5978ed32fe514b9d4acd9e868e145ffbf0c95_25d5978ed32fe514b9d4acd9e868e145ffbf0c95/report/index.html

doris-robot · 2024-12-20T20:14:23Z

TPC-H: Total hot run time: 39528 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 25d5978ed32fe514b9d4acd9e868e145ffbf0c95, data reload: false
------ Round 1 ----------------------------------
q1	17743	7453	7344	7344
q2	2043	180	178	178
q3	11144	1108	1204	1108
q4	10530	783	746	746
q5	7630	2671	2616	2616
q6	232	147	145	145
q7	987	637	594	594
q8	9221	1822	1907	1822
q9	6498	6392	6416	6392
q10	7025	2325	2322	2322
q11	460	262	255	255
q12	426	220	224	220
q13	17775	2965	2891	2891
q14	240	210	217	210
q15	560	515	500	500
q16	659	571	601	571
q17	994	572	510	510
q18	7146	6560	6731	6560
q19	1362	1002	906	906
q20	456	180	180	180
q21	4085	3414	3145	3145
q22	368	319	313	313
Total cold run time: 107584 ms
Total hot run time: 39528 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7279	7228	7221	7221
q2	326	234	236	234
q3	2888	2808	2929	2808
q4	2063	1792	1841	1792
q5	5696	5669	5597	5597
q6	225	148	143	143
q7	2233	1806	1818	1806
q8	3329	3552	3491	3491
q9	8975	9112	8974	8974
q10	3598	3497	3525	3497
q11	598	495	513	495
q12	855	598	601	598
q13	13358	3181	3050	3050
q14	303	265	283	265
q15	570	511	513	511
q16	660	640	635	635
q17	1846	1656	1609	1609
q18	8404	7701	7783	7701
q19	3625	1582	1455	1455
q20	2078	1876	1908	1876
q21	5586	5458	5417	5417
q22	636	567	609	567
Total cold run time: 75131 ms
Total hot run time: 59742 ms

doris-robot · 2024-12-20T20:26:40Z

TPC-DS: Total hot run time: 196848 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 25d5978ed32fe514b9d4acd9e868e145ffbf0c95, data reload: false
query1	1302	931	943	931
query2	6254	2331	2301	2301
query3	10969	4713	4689	4689
query4	33353	23501	23343	23343
query5	4252	474	463	463
query6	298	189	193	189
query7	3979	302	309	302
query8	305	241	240	240
query9	9374	2712	2691	2691
query10	468	251	257	251
query11	17803	15300	15221	15221
query12	159	100	102	100
query13	1639	428	395	395
query14	10618	7079	7700	7079
query15	286	189	212	189
query16	8138	463	481	463
query17	1737	574	606	574
query18	2085	298	312	298
query19	384	162	163	162
query20	122	119	117	117
query21	210	105	107	105
query22	4752	4692	4635	4635
query23	34670	33627	33841	33627
query24	10245	2500	2454	2454
query25	574	400	392	392
query26	722	187	164	164
query27	2089	337	346	337
query28	6574	2471	2436	2436
query29	734	418	427	418
query30	231	151	157	151
query31	1070	877	852	852
query32	102	55	55	55
query33	770	334	311	311
query34	979	516	518	516
query35	895	784	756	756
query36	1129	960	968	960
query37	123	75	77	75
query38	4302	4243	4170	4170
query39	1536	1453	1458	1453
query40	204	108	101	101
query41	43	42	44	42
query42	130	101	102	101
query43	533	507	504	504
query44	1290	828	843	828
query45	191	171	170	170
query46	1185	698	718	698
query47	2009	1921	1947	1921
query48	439	326	319	319
query49	867	386	388	386
query50	843	407	408	407
query51	7363	7324	7156	7156
query52	104	93	92	92
query53	259	181	197	181
query54	1207	415	426	415
query55	89	76	82	76
query56	257	247	238	238
query57	1321	1186	1160	1160
query58	238	220	221	220
query59	3454	3125	3059	3059
query60	282	262	276	262
query61	112	109	113	109
query62	879	696	705	696
query63	225	236	189	189
query64	3637	704	638	638
query65	3299	3271	3309	3271
query66	774	302	309	302
query67	16469	15616	15519	15519
query68	4930	567	558	558
query69	470	249	248	248
query70	1135	1158	1148	1148
query71	476	244	245	244
query72	6620	4213	4282	4213
query73	780	359	372	359
query74	9727	8844	8813	8813
query75	3570	2638	2609	2609
query76	3650	1101	1062	1062
query77	586	285	270	270
query78	10384	9476	9465	9465
query79	1989	605	607	605
query80	1314	441	423	423
query81	504	236	226	226
query82	334	124	118	118
query83	196	146	150	146
query84	287	67	69	67
query85	932	302	299	299
query86	357	308	312	308
query87	4553	4365	4479	4365
query88	4070	2205	2185	2185
query89	432	297	287	287
query90	2182	183	184	183
query91	148	103	106	103
query92	64	51	52	51
query93	4153	568	537	537
query94	906	274	288	274
query95	360	257	264	257
query96	641	276	284	276
query97	2861	2728	2709	2709
query98	214	191	192	191
query99	1578	1324	1354	1324
Total cold run time: 303338 ms
Total hot run time: 196848 ms

doris-robot · 2024-12-26T08:02:39Z

TPC-H: Total hot run time: 32510 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 453bd6aa00e7e9dc67d7ec4f1c52e47ed7ba3844, data reload: false
------ Round 1 ----------------------------------
q1	17576	6630	6034	6034
q2	2047	289	167	167
q3	10446	1234	746	746
q4	10243	861	433	433
q5	8361	2176	1987	1987
q6	209	178	151	151
q7	897	761	610	610
q8	9211	1341	1167	1167
q9	5217	4887	4943	4887
q10	6959	2369	1902	1902
q11	494	281	262	262
q12	367	371	232	232
q13	17935	3671	2916	2916
q14	225	230	217	217
q15	567	515	499	499
q16	647	613	576	576
q17	562	838	328	328
q18	6838	6437	6351	6351
q19	1650	966	555	555
q20	308	327	190	190
q21	2901	2216	1985	1985
q22	365	333	315	315
Total cold run time: 104025 ms
Total hot run time: 32510 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6281	6239	6246	6239
q2	241	319	235	235
q3	2232	2611	2312	2312
q4	1393	1832	1355	1355
q5	4333	4781	4851	4781
q6	186	183	144	144
q7	2091	1950	1805	1805
q8	2634	2852	2676	2676
q9	7327	7299	7323	7299
q10	3095	3360	2747	2747
q11	577	512	492	492
q12	664	868	627	627
q13	3429	3749	3066	3066
q14	289	320	306	306
q15	561	518	511	511
q16	654	685	660	660
q17	1214	1720	1259	1259
q18	7550	7448	7328	7328
q19	842	1174	1066	1066
q20	1982	2095	1899	1899
q21	5721	5379	4878	4878
q22	647	643	594	594
Total cold run time: 53943 ms
Total hot run time: 52279 ms

doris-robot · 2024-12-26T08:14:59Z

TPC-DS: Total hot run time: 197590 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 453bd6aa00e7e9dc67d7ec4f1c52e47ed7ba3844, data reload: false
query1	1324	985	903	903
query2	6486	2487	2317	2317
query3	10988	4709	4850	4709
query4	32950	23778	23325	23325
query5	4120	615	466	466
query6	291	188	179	179
query7	3996	481	308	308
query8	299	241	235	235
query9	9429	2768	2774	2768
query10	463	301	245	245
query11	17861	15552	15256	15256
query12	148	111	104	104
query13	1567	527	401	401
query14	9707	7628	7670	7628
query15	242	204	201	201
query16	8401	634	477	477
query17	1565	763	581	581
query18	2074	446	321	321
query19	211	195	161	161
query20	122	116	117	116
query21	206	126	113	113
query22	4500	4753	4378	4378
query23	35206	33777	33750	33750
query24	6457	2325	2468	2325
query25	499	465	388	388
query26	783	277	158	158
query27	1996	470	328	328
query28	5541	2503	2513	2503
query29	579	559	422	422
query30	210	185	151	151
query31	1010	913	838	838
query32	72	60	57	57
query33	478	354	316	316
query34	785	862	516	516
query35	862	832	771	771
query36	1050	1085	994	994
query37	126	137	73	73
query38	4208	4167	4316	4167
query39	1541	1479	1477	1477
query40	205	115	108	108
query41	46	42	44	42
query42	121	109	105	105
query43	527	521	492	492
query44	1382	823	835	823
query45	191	179	175	175
query46	898	1066	655	655
query47	1989	2008	1908	1908
query48	398	413	332	332
query49	759	521	391	391
query50	646	686	387	387
query51	7406	7342	7082	7082
query52	110	99	92	92
query53	230	263	187	187
query54	501	509	415	415
query55	82	78	82	78
query56	254	254	246	246
query57	1266	1248	1166	1166
query58	233	239	227	227
query59	3335	3321	3056	3056
query60	282	256	260	256
query61	116	109	115	109
query62	836	810	786	786
query63	224	202	194	194
query64	2950	1040	681	681
query65	3324	3303	3244	3244
query66	708	401	322	322
query67	16474	15859	15615	15615
query68	7954	760	528	528
query69	489	300	257	257
query70	1202	1127	1147	1127
query71	429	290	254	254
query72	6119	3854	3889	3854
query73	674	762	368	368
query74	10363	9107	8957	8957
query75	4056	3155	2659	2659
query76	3682	1194	779	779
query77	765	361	271	271
query78	10176	10417	9578	9578
query79	3367	911	603	603
query80	599	537	542	537
query81	506	275	234	234
query82	629	151	124	124
query83	159	155	139	139
query84	240	93	68	68
query85	774	377	305	305
query86	390	317	300	300
query87	4658	4568	4619	4568
query88	4864	2234	2206	2206
query89	418	327	286	286
query90	1707	189	190	189
query91	134	134	105	105
query92	63	53	52	52
query93	1853	881	533	533
query94	628	377	287	287
query95	329	272	252	252
query96	488	614	284	284
query97	2746	2825	2644	2644
query98	217	207	197	197
query99	1691	1562	1445	1445
Total cold run time: 296215 ms
Total hot run time: 197590 ms

doris-robot · 2024-12-26T08:20:20Z

ClickBench: Total hot run time: 31.52 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 453bd6aa00e7e9dc67d7ec4f1c52e47ed7ba3844, data reload: false
query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.24	0.06	0.07
query4	1.62	0.11	0.11
query5	0.43	0.43	0.43
query6	1.15	0.64	0.65
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.55	0.55	0.54
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.61
query14	2.70	2.72	2.88
query15	0.90	0.82	0.82
query16	0.37	0.39	0.38
query17	1.00	0.97	1.02
query18	0.23	0.21	0.21
query19	1.95	1.85	1.91
query20	0.01	0.02	0.01
query21	15.36	0.95	0.59
query22	0.76	0.79	0.78
query23	15.20	1.48	0.59
query24	3.83	2.16	1.03
query25	0.23	0.20	0.12
query26	0.23	0.14	0.13
query27	0.04	0.05	0.08
query28	13.98	1.48	1.05
query29	12.57	3.94	3.32
query30	0.25	0.08	0.06
query31	2.83	0.61	0.37
query32	3.24	0.55	0.46
query33	3.12	3.18	3.06
query34	16.66	5.06	4.48
query35	4.46	4.52	4.49
query36	0.67	0.48	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.72 s
Total hot run time: 31.52 s

morningman · 2024-12-30T04:04:11Z

run buildall

doris-robot · 2024-12-30T04:44:42Z

TPC-H: Total hot run time: 32289 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dbf176541b8ff117c7791ec0614c2cc723de53f7, data reload: false
------ Round 1 ----------------------------------
q1	17571	6132	6034	6034
q2	2044	306	176	176
q3	10424	1198	732	732
q4	10220	847	432	432
q5	7720	2178	1937	1937
q6	209	183	151	151
q7	876	748	595	595
q8	9238	1360	1174	1174
q9	5196	4975	4887	4887
q10	6757	2324	1841	1841
q11	487	279	259	259
q12	350	359	217	217
q13	17752	3569	2890	2890
q14	233	228	204	204
q15	556	497	512	497
q16	627	613	596	596
q17	581	870	334	334
q18	6980	6640	6382	6382
q19	1903	952	529	529
q20	299	310	181	181
q21	2716	2103	1948	1948
q22	359	328	293	293
Total cold run time: 103098 ms
Total hot run time: 32289 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6318	6202	6185	6185
q2	230	320	227	227
q3	2276	2621	2274	2274
q4	1388	1850	1377	1377
q5	4350	4740	4721	4721
q6	183	176	143	143
q7	2105	1925	1812	1812
q8	2596	2837	2668	2668
q9	7310	7114	7288	7114
q10	3080	3347	2884	2884
q11	592	504	500	500
q12	641	724	571	571
q13	3441	3727	3156	3156
q14	297	301	263	263
q15	568	523	496	496
q16	652	683	631	631
q17	1233	1739	1311	1311
q18	7513	7505	7362	7362
q19	804	1083	1138	1083
q20	1995	2043	1920	1920
q21	5767	5233	4972	4972
q22	622	632	598	598
Total cold run time: 53961 ms
Total hot run time: 52268 ms

doris-robot · 2024-12-30T04:51:32Z

TeamCity be ut coverage result:
Function Coverage: 38.91% (10122/26017)
Line Coverage: 29.90% (85544/286118)
Region Coverage: 29.03% (43719/150616)
Branch Coverage: 25.56% (22304/87262)
Coverage Report: http://coverage.selectdb-in.cc/coverage/dbf176541b8ff117c7791ec0614c2cc723de53f7_dbf176541b8ff117c7791ec0614c2cc723de53f7/report/index.html

doris-robot · 2024-12-30T04:57:02Z

TPC-DS: Total hot run time: 197273 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dbf176541b8ff117c7791ec0614c2cc723de53f7, data reload: false
query1	1295	959	952	952
query2	6296	2247	2265	2247
query3	10992	4492	4706	4492
query4	32860	23588	23549	23549
query5	4752	612	464	464
query6	278	199	193	193
query7	3993	491	306	306
query8	313	258	246	246
query9	9481	2743	2735	2735
query10	478	314	246	246
query11	18164	15420	15232	15232
query12	157	108	106	106
query13	1639	575	417	417
query14	10600	7595	7479	7479
query15	239	203	177	177
query16	7363	601	441	441
query17	1476	780	563	563
query18	1909	381	339	339
query19	227	184	159	159
query20	125	122	119	119
query21	205	127	103	103
query22	4833	4588	4429	4429
query23	34748	34441	33904	33904
query24	6343	2273	2300	2273
query25	454	466	391	391
query26	727	255	146	146
query27	2472	500	335	335
query28	5370	2490	2478	2478
query29	551	538	423	423
query30	233	198	159	159
query31	1005	963	847	847
query32	67	60	56	56
query33	481	359	287	287
query34	779	850	507	507
query35	799	838	737	737
query36	1041	1052	960	960
query37	127	97	77	77
query38	4408	4188	4274	4188
query39	1503	1469	1455	1455
query40	221	114	98	98
query41	44	44	43	43
query42	121	104	110	104
query43	527	512	493	493
query44	1318	833	822	822
query45	187	179	166	166
query46	884	1095	671	671
query47	1982	2007	1998	1998
query48	390	413	327	327
query49	694	493	402	402
query50	644	694	395	395
query51	7378	7332	7143	7143
query52	108	98	94	94
query53	236	258	186	186
query54	486	499	417	417
query55	81	82	75	75
query56	252	255	234	234
query57	1283	1242	1163	1163
query58	232	233	229	229
query59	3044	3212	3177	3177
query60	277	251	247	247
query61	110	109	106	106
query62	887	845	753	753
query63	230	199	190	190
query64	3381	1034	660	660
query65	3324	3249	3292	3249
query66	747	413	307	307
query67	16272	16031	15516	15516
query68	8089	744	526	526
query69	459	304	252	252
query70	1136	1120	1141	1120
query71	483	281	272	272
query72	6167	3856	3853	3853
query73	640	752	356	356
query74	10294	9013	8787	8787
query75	3945	3151	2782	2782
query76	3648	1219	753	753
query77	771	367	281	281
query78	10223	10464	9439	9439
query79	3378	870	581	581
query80	784	510	439	439
query81	495	262	226	226
query82	600	158	120	120
query83	193	167	140	140
query84	289	88	80	80
query85	801	360	298	298
query86	406	314	301	301
query87	4585	4567	4560	4560
query88	4752	2231	2200	2200
query89	414	336	302	302
query90	1730	191	186	186
query91	146	149	119	119
query92	70	58	52	52
query93	1589	856	543	543
query94	664	405	291	291
query95	343	272	265	265
query96	481	604	283	283
query97	2736	2837	2744	2744
query98	221	208	196	196
query99	1671	1512	1444	1444
Total cold run time: 296326 ms
Total hot run time: 197273 ms

doris-robot · 2024-12-30T05:02:21Z

ClickBench: Total hot run time: 31.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dbf176541b8ff117c7791ec0614c2cc723de53f7, data reload: false
query1	0.03	0.03	0.02
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.60	0.10	0.11
query5	0.41	0.39	0.41
query6	1.15	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.59	0.51	0.50
query10	0.56	0.60	0.55
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.61	0.61	0.59
query14	2.87	2.91	2.81
query15	0.89	0.83	0.82
query16	0.39	0.38	0.38
query17	1.04	1.06	1.06
query18	0.23	0.20	0.20
query19	1.92	1.81	2.00
query20	0.01	0.01	0.01
query21	15.37	0.89	0.57
query22	0.75	0.76	0.69
query23	15.30	1.39	0.55
query24	3.02	0.99	0.77
query25	0.27	0.07	0.10
query26	0.31	0.16	0.13
query27	0.05	0.06	0.05
query28	13.70	1.56	1.04
query29	12.59	3.88	3.21
query30	0.25	0.09	0.07
query31	2.82	0.60	0.37
query32	3.22	0.54	0.47
query33	3.16	3.14	3.12
query34	16.61	5.06	4.48
query35	4.46	4.46	4.52
query36	0.67	0.50	0.53
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106 s
Total hot run time: 31.11 s

kaka11chen

LGTM

github-actions · 2025-01-02T03:58:46Z

PR approved by at least one committer and no changes requested.

bp apache#44001 , but no hive4 acid table. Problem Summary: 1. Fixed the issue that when reading insert translaction only tables, there was no acid check, which caused multiple data reads (i.e., reading data from the previous base_n). 2. Forbidden to create, insert data, and delete aicd tables.

Problem Summary: Support read hive4 transaction tables. 1. Since there are compilation problems when directly using hive4 API to get the file need to read, a function called `ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#getAcidState` is implemented. 2. At the same time, it is forbidden to create, insert data, and delete aicd tables. 3. fix `'transactional_properties' = 'insert_only'` table read Incorrect data read. TODO : merge HMSTransaction ,HiveTransaction class merge HiveTransactionMgr, HiveTransactionManager class [feature](hive)Support read hive4 transaction tables

hubgeter force-pushed the feature_hive4_acid branch from 7669f81 to 2dec82fCompare December 3, 2024 10:46

hubgeter commented Dec 11, 2024
View reviewed changes

hubgeter force-pushed the feature_hive4_acid branch from aacbf79 to 2a523cfCompare December 16, 2024 07:23

hubgeter changed the title ~~[feature](hive)support hive4 acid~~[feature](hive)Support read hive4 transaction tables.Dec 16, 2024

hubgeter marked this pull request as ready for review December 16, 2024 07:34

morningman added the dev/3.0.x label Dec 17, 2024

morningman reviewed Dec 17, 2024
View reviewed changes

morningman previously approved these changes Dec 20, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 20, 2024

github-actionsBot added the reviewed label Dec 20, 2024

hubgeter dismissed morningman’s stale review via 25d5978December 20, 2024 19:33

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Dec 20, 2024

hubgeter mentioned this pull request Dec 20, 2024
[fix](hive)fix hive insert only translaction table. #45753
Merged
16 tasks

hubgeter added 10 commits December 30, 2024 12:03

[feature](hive)support hive4 acid
869da2e

ban create insert drop translational tb.
4d2115f

refactor code.
da3c766

rm ut
7510fee

add ut && regression test
20d8a0e

rm unused code.
201ec62

fix formatted
65aaff2

fix hive acid insert only.
2ec6b6e

fix case
3fe927d

fix case2.
dbf1765

morningman force-pushed the feature_hive4_acid branch from 453bd6a to dbf1765Compare December 30, 2024 04:03

morningman added dev/3.0.x-experimental and removed dev/3.0.x labels Dec 30, 2024

kaka11chen approved these changes Jan 2, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 2, 2025

morningman approved these changes Jan 3, 2025
View reviewed changes

morningman merged commit c398cba into apache:masterJan 3, 2025

morningman added the dev/2.1.x-lh label Feb 17, 2025

Uh oh!

Conversation

hubgeter commented Nov 15, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 15, 2024

Uh oh!

github-actionsBot commented Nov 15, 2024

Uh oh!

hubgeterDec 11, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Dec 16, 2024

Uh oh!

github-actionsBot commented Dec 16, 2024

Uh oh!

hubgeter commented Dec 16, 2024

Uh oh!

hubgeter commented Dec 16, 2024

Uh oh!

github-actionsBot commented Dec 16, 2024

Uh oh!

doris-robot commented Dec 16, 2024

Uh oh!

morningmanDec 17, 2024

Choose a reason for hiding this comment

Uh oh!

morningmanDec 17, 2024

Choose a reason for hiding this comment

Uh oh!

hubgeter commented Dec 19, 2024

Uh oh!

github-actionsBot commented Dec 19, 2024

Uh oh!

doris-robot commented Dec 19, 2024

Uh oh!

github-actionsBot commented Dec 20, 2024

Uh oh!

github-actionsBot commented Dec 20, 2024

Uh oh!

hubgeter commented Dec 20, 2024

Uh oh!

doris-robot commented Dec 20, 2024

Uh oh!

doris-robot commented Dec 20, 2024

Uh oh!

doris-robot commented Dec 20, 2024

Uh oh!

doris-robot commented Dec 26, 2024

Uh oh!

doris-robot commented Dec 26, 2024

Uh oh!

doris-robot commented Dec 26, 2024

Uh oh!

morningman commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

doris-robot commented Dec 30, 2024

Uh oh!

kaka11chen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jan 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

hubgeter commented Nov 15, 2024•
edited
Loading