Skip to content

[Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. - #27563

Merged
yiguolei merged 1 commit into
apache:masterfrom
kaka11chen:fix_empty_hive_transactional_table
Nov 25, 2023
Merged

[Fix](hive-transactional-table) Fix NPE when query empty hive transactional table.#27563
yiguolei merged 1 commit into
apache:masterfrom
kaka11chen:fix_empty_hive_transactional_table

Conversation

@kaka11chen

@kaka11chenkaka11chen commented Nov 24, 2023

Copy link
Copy Markdown
Contributor

Proposed changes

Issue:

2023-11-24 17:59:16,171 WARN (mysql-nio-pool-0|236) [HiveScanNode.getSplits():211] get file split failed for table: lineitem
org.apache.doris.datasource.CacheException: failed to get input splits for write ids acid.lineitem:9223372036854775807:9223372036854775807:: in catalog regression1
at org.apache.doris.datasource.hive.HiveMetaStoreCache.getFilesByTransaction(HiveMetaStoreCache.java:863) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.external.HiveScanNode.getFileSplitByTransaction(HiveScanNode.java:308) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.external.HiveScanNode.getFileSplitByPartitions(HiveScanNode.java:223) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.external.HiveScanNode.getSplits(HiveScanNode.java:206) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.external.FileQueryScanNode.createScanRangeLocations(FileQueryScanNode.java:270) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.external.FileQueryScanNode.doFinalize(FileQueryScanNode.java:229) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.external.FileQueryScanNode.finalizeForNereids(FileQueryScanNode.java:221) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.util.Utils.execWithUncheckedException(Utils.java:67) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalFileScan(PhysicalPlanTranslator.java:522) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalFileScan(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.physical.PhysicalFileScan.accept(PhysicalFileScan.java:98) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalProject(PhysicalPlanTranslator.java:1568) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalProject(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.physical.PhysicalProject.accept(PhysicalProject.java:105) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashAggregate(PhysicalPlanTranslator.java:784) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashAggregate(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.physical.PhysicalHashAggregate.accept(PhysicalHashAggregate.java:185) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalDistribute(PhysicalPlanTranslator.java:265) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalDistribute(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.physical.PhysicalDistribute.accept(PhysicalDistribute.java:96) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashAggregate(PhysicalPlanTranslator.java:784) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashAggregate(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.physical.PhysicalHashAggregate.accept(PhysicalHashAggregate.java:185) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:355) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.physical.PhysicalResultSink.accept(PhysicalResultSink.java:70) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:239) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:134) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:580) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:451) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:441) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:358) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:511) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:762) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131]
Caused by: java.lang.NullPointerException
at org.apache.hadoop.fs.Path.<init>(Path.java:150) ~[hadoop-common-3.3.6.jar:?]
at org.apache.hadoop.fs.Path.<init>(Path.java:129) ~[hadoop-common-3.3.6.jar:?]
at org.apache.doris.datasource.hive.HiveMetaStoreCache.getFilesByTransaction(HiveMetaStoreCache.java:805) ~[doris-fe.jar:1.2-SNAPSHOT]
... 37 more

Solution

Fix NPE when query empty hive transactional table.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@kaka11chen

kaka11chen commented Nov 24, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

@kaka11chen
kaka11chenforce-pushed the fix_empty_hive_transactional_table branch from b5dd022 to 44eb203CompareNovember 24, 2023 12:20
@kaka11chen

kaka11chen commented Nov 24, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Nov 24, 2023

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.83 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17100689076 Bytes

@doris-robot

doris-robot commented Nov 24, 2023

Copy link
Copy Markdown
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 44eb2032557155872f57c793cf431f80b2288dca, data reload: false
run tpch-sf100 query with default conf and session variables
q1	4905	4636	4630	4630
q2	365	141	140	140
q3	2011	1898	1896	1896
q4	1383	1258	1236	1236
q5	3993	3922	3973	3922
q6	255	128	131	128
q7	1397	885	869	869
q8	2784	2789	2757	2757
q9	9644	9544	9403	9403
q10	3473	3524	3544	3524
q11	375	240	251	240
q12	435	293	292	292
q13	4577	3809	3805	3805
q14	321	290	297	290
q15	589	525	511	511
q16	660	581	582	581
q17	1134	939	907	907
q18	7845	7513	7447	7447
q19	1667	1689	1658	1658
q20	537	318	292	292
q21	4409	3933	4017	3933
q22	484	380	379	379
Total cold run time: 53243 ms
Total hot run time: 48840 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4617	4604	4572	4572
q2	351	244	260	244
q3	4040	4027	4025	4025
q4	2718	2700	2708	2700
q5	9860	9904	9914	9904
q6	248	119	124	119
q7	3023	2510	2459	2459
q8	4466	4438	4457	4438
q9	12990	12839	12834	12834
q10	4093	4187	4148	4148
q11	769	661	664	661
q12	971	807	830	807
q13	4313	3615	3574	3574
q14	381	369	342	342
q15	573	518	526	518
q16	738	655	700	655
q17	3847	3892	3827	3827
q18	9690	9144	9004	9004
q19	1816	1779	1800	1779
q20	2394	2046	2025	2025
q21	8812	8519	8521	8519
q22	920	852	778	778
Total cold run time: 81630 ms
Total hot run time: 77932 ms

morningman
morningman approved these changes Nov 24, 2023
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 24, 2023
@github-actions

github-actionsBot commented Nov 24, 2023

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Nov 24, 2023

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@kaka11chenkaka11chen mentioned this pull request Nov 24, 2023
Merged
@wm1581066wm1581066 added dev/2.0.4 usercase Important user case type label labels Nov 24, 2023
@yiguolei
yiguolei merged commit cc395f5 into apache:masterNov 25, 2023
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
@kaka11chen@seawinde
[Fix](hive-transactional-table) Fix NPE when query empty hive transac…
2522bed
…tional table. (apache#27563)
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@kaka11chen胥剑旭
[Fix](hive-transactional-table) Fix NPE when query empty hive transac…
e288fef
…tional table. (apache#27563)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes
@Jibing-LiJibing-LiJibing-Li approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.3-mergedreviewedusercaseImportant user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@kaka11chen@doris-robot@morningman@Jibing-Li@yiguolei@wm1581066