[fix](cache) fix sql cache throw npe in cloud mode by 924060929 · Pull Request #47221 · apache/doris

924060929 · 2025-01-20T03:44:32Z

What problem does this PR solve?

fix sql cache throw npe in cloud mode, when some partitions is dropped:

2025-01-15 18:18:54,811 WARN (mysql-nio-pool-101426|288) [ConnectProcessor.handleQueryException():537] Process one query failed because unknown reason:
java.lang.NullPointerException: Cannot invoke "org.apache.doris.cloud.catalog.CloudPartition.getDbId()" because "partition" is null
at org.apache.doris.cloud.catalog.CloudPartition.getSnapshotVisibleVersion(CloudPartition.java:196) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.OlapTable.getVersionInBatchForCloudMode(OlapTable.java:1190) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.cache.CacheAnalyzer.buildCacheTableForOlapScanNode(CacheAnalyzer.java:700) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.cache.CacheAnalyzer.buildCacheTableList(CacheAnalyzer.java:512) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.cache.CacheAnalyzer.innerCheckCacheModeForNereids(CacheAnalyzer.java:412) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.cache.CacheAnalyzer.getCacheData(CacheAnalyzer.java:522) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.handleCacheStmt(StmtExecutor.java:1725) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1831) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.handleQueryWithRetry(StmtExecutor.java:874) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:811) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:607) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:557) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:547) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:397) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:238) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:194) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:222) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:281) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]

This is an unstable exception, so I can not add test

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-01-20T03:44:38Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

924060929 · 2025-01-20T03:45:09Z

run buildall

924060929 · 2025-01-20T04:01:13Z

run buildall

doris-robot · 2025-01-20T04:53:41Z

TPC-H: Total hot run time: 32044 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba608388aa4a7bacca1296a0c774da928541a56d, data reload: false
------ Round 1 ----------------------------------
q1	17944	5520	5436	5436
q2	2038	310	170	170
q3	10717	1250	688	688
q4	10281	962	528	528
q5	8630	2390	2126	2126
q6	191	165	136	136
q7	912	785	622	622
q8	9215	1331	1167	1167
q9	5161	4856	4892	4856
q10	6844	2321	1860	1860
q11	475	279	258	258
q12	339	350	215	215
q13	17785	3689	3088	3088
q14	229	235	200	200
q15	512	462	477	462
q16	631	635	582	582
q17	550	851	320	320
q18	6975	6375	6383	6375
q19	1997	937	523	523
q20	312	319	189	189
q21	2919	2140	1926	1926
q22	358	326	317	317
Total cold run time: 105015 ms
Total hot run time: 32044 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5563	5436	5472	5436
q2	239	336	234	234
q3	2216	2658	2300	2300
q4	1399	1801	1333	1333
q5	4343	4760	4583	4583
q6	162	155	125	125
q7	2109	1965	1809	1809
q8	2620	2804	2667	2667
q9	7210	7231	7172	7172
q10	2983	3273	2764	2764
q11	588	517	503	503
q12	662	723	597	597
q13	3461	3900	3286	3286
q14	271	292	254	254
q15	515	479	480	479
q16	626	701	640	640
q17	1209	1745	1249	1249
q18	7625	7451	7189	7189
q19	783	1001	1101	1001
q20	1962	2028	1873	1873
q21	5587	5150	5064	5064
q22	602	581	549	549
Total cold run time: 52735 ms
Total hot run time: 51107 ms

doris-robot · 2025-01-20T05:05:29Z

TPC-DS: Total hot run time: 187319 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ba608388aa4a7bacca1296a0c774da928541a56d, data reload: false
query1	977	379	383	379
query2	6511	2037	1976	1976
query3	6802	215	219	215
query4	33549	23121	22993	22993
query5	4334	640	458	458
query6	292	201	177	177
query7	4616	475	305	305
query8	303	240	222	222
query9	9367	2613	2600	2600
query10	489	312	251	251
query11	17997	15553	14966	14966
query12	153	105	107	105
query13	1661	514	419	419
query14	9059	7633	7026	7026
query15	258	197	184	184
query16	8056	624	492	492
query17	1606	745	564	564
query18	2077	428	293	293
query19	223	172	144	144
query20	114	115	111	111
query21	211	122	99	99
query22	4302	4384	4239	4239
query23	33891	33035	33061	33035
query24	6484	2342	2261	2261
query25	503	454	373	373
query26	1185	270	154	154
query27	2037	452	326	326
query28	5082	2415	2417	2415
query29	720	539	415	415
query30	232	189	163	163
query31	976	865	796	796
query32	98	88	63	63
query33	512	354	298	298
query34	761	870	499	499
query35	824	825	716	716
query36	987	1033	957	957
query37	119	105	82	82
query38	4288	4057	4124	4057
query39	1438	1385	1374	1374
query40	203	114	101	101
query41	54	55	52	52
query42	115	104	100	100
query43	502	512	489	489
query44	1296	805	825	805
query45	179	174	164	164
query46	830	1048	637	637
query47	1838	1855	1796	1796
query48	379	402	306	306
query49	784	484	400	400
query50	630	654	407	407
query51	6752	6900	6760	6760
query52	99	102	91	91
query53	221	253	184	184
query54	480	503	407	407
query55	84	79	84	79
query56	256	250	232	232
query57	1173	1141	1061	1061
query58	238	228	235	228
query59	2910	3043	2838	2838
query60	280	266	259	259
query61	125	139	118	118
query62	808	702	686	686
query63	219	221	193	193
query64	4512	1019	635	635
query65	3253	3118	3113	3113
query66	1057	407	308	308
query67	15817	15704	15453	15453
query68	4306	815	522	522
query69	499	290	258	258
query70	1157	1170	1140	1140
query71	396	283	308	283
query72	5791	3774	3780	3774
query73	632	750	354	354
query74	10465	8867	8885	8867
query75	3162	3165	2716	2716
query76	3324	1170	765	765
query77	471	358	274	274
query78	9936	9897	9332	9332
query79	3076	795	587	587
query80	666	521	442	442
query81	528	266	232	232
query82	540	148	120	120
query83	171	191	151	151
query84	239	95	68	68
query85	766	362	292	292
query86	402	325	301	301
query87	4486	4438	4380	4380
query88	4966	2167	2137	2137
query89	403	342	308	308
query90	1807	191	196	191
query91	137	140	108	108
query92	69	59	53	53
query93	2659	857	520	520
query94	673	372	301	301
query95	335	270	258	258
query96	494	600	283	283
query97	2790	2869	2743	2743
query98	227	211	194	194
query99	1276	1371	1261	1261
Total cold run time: 285037 ms
Total hot run time: 187319 ms

doris-robot · 2025-01-20T05:10:48Z

ClickBench: Total hot run time: 31.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ba608388aa4a7bacca1296a0c774da928541a56d, data reload: false
query1	0.03	0.03	0.03
query2	0.08	0.04	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.44	0.44	0.38
query6	1.15	0.65	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.61	0.51	0.51
query10	0.55	0.55	0.56
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.59	0.60
query14	2.70	2.80	2.74
query15	0.90	0.82	0.83
query16	0.38	0.36	0.38
query17	1.04	1.03	1.07
query18	0.23	0.21	0.21
query19	1.88	1.78	1.99
query20	0.02	0.01	0.02
query21	15.36	0.94	0.60
query22	0.75	0.84	0.72
query23	15.17	1.39	0.60
query24	2.78	1.71	1.18
query25	0.21	0.15	0.13
query26	0.28	0.15	0.14
query27	0.07	0.05	0.04
query28	14.24	0.97	0.45
query29	12.55	4.01	3.34
query30	0.25	0.09	0.07
query31	2.81	0.60	0.39
query32	3.22	0.56	0.46
query33	3.09	3.02	3.00
query34	16.64	5.20	4.52
query35	4.63	4.48	4.52
query36	0.64	0.49	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.13	0.14
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.01 s
Total hot run time: 31.02 s

github-actions · 2025-01-21T02:44:42Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-01-21T02:44:44Z

PR approved by anyone and no changes requested.

fix sql cache throw npe in cloud mode, when some partitions is dropped: ``` 2025-01-15 18:18:54,811 WARN (mysql-nio-pool-101426|288) [ConnectProcessor.handleQueryException():537] Process one query failed because unknown reason: java.lang.NullPointerException: Cannot invoke "org.apache.doris.cloud.catalog.CloudPartition.getDbId()" because "partition" is null at org.apache.doris.cloud.catalog.CloudPartition.getSnapshotVisibleVersion(CloudPartition.java:196) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.OlapTable.getVersionInBatchForCloudMode(OlapTable.java:1190) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.cache.CacheAnalyzer.buildCacheTableForOlapScanNode(CacheAnalyzer.java:700) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.cache.CacheAnalyzer.buildCacheTableList(CacheAnalyzer.java:512) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.cache.CacheAnalyzer.innerCheckCacheModeForNereids(CacheAnalyzer.java:412) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.cache.CacheAnalyzer.getCacheData(CacheAnalyzer.java:522) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleCacheStmt(StmtExecutor.java:1725) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1831) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleQueryWithRetry(StmtExecutor.java:874) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:811) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:607) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:557) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:547) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:397) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:238) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:194) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:222) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:281) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] ``` This is an unstable exception, so I can not add test

…#47267) Cherry-picked from #47221 Co-authored-by: 924060929 <lanhuajian@selectdb.com>

…#47266) Cherry-picked from #47221 Co-authored-by: 924060929 <lanhuajian@selectdb.com>

fix sql cache throw npe in cloud mode, when some partitions is dropped: ``` 2025-01-15 18:18:54,811 WARN (mysql-nio-pool-101426|288) [ConnectProcessor.handleQueryException():537] Process one query failed because unknown reason: java.lang.NullPointerException: Cannot invoke "org.apache.doris.cloud.catalog.CloudPartition.getDbId()" because "partition" is null at org.apache.doris.cloud.catalog.CloudPartition.getSnapshotVisibleVersion(CloudPartition.java:196) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.OlapTable.getVersionInBatchForCloudMode(OlapTable.java:1190) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.cache.CacheAnalyzer.buildCacheTableForOlapScanNode(CacheAnalyzer.java:700) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.cache.CacheAnalyzer.buildCacheTableList(CacheAnalyzer.java:512) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.cache.CacheAnalyzer.innerCheckCacheModeForNereids(CacheAnalyzer.java:412) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.cache.CacheAnalyzer.getCacheData(CacheAnalyzer.java:522) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleCacheStmt(StmtExecutor.java:1725) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1831) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleQueryWithRetry(StmtExecutor.java:874) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:811) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:607) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:557) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:547) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:397) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:238) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:194) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:222) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:281) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] ``` This is an unstable exception, so I can not add test

fix sql cache throw npe in cloud mode
37b8e85

fix sql cache throw npe in cloud mode
ba60838

924060929 added dev/2.1.x dev/3.0.x labels Jan 20, 2025

morrySnow approved these changes Jan 21, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 21, 2025

github-actionsBot added the reviewed label Jan 21, 2025

starocean999 approved these changes Jan 21, 2025
View reviewed changes

starocean999 merged commit fb1ad0c into apache:masterJan 21, 2025

github-actionsBot mentioned this pull request Jan 21, 2025
branch-3.0: [fix](cache) fix sql cache throw npe in cloud mode #47221 #47266
Merged

github-actionsBot mentioned this pull request Jan 21, 2025
branch-2.1: [fix](cache) fix sql cache throw npe in cloud mode #47221 #47267
Merged

924060929 added the usercase Important user case type label label Jan 24, 2025

yiguolei pushed a commit that referenced this pull request Jan 24, 2025
branch-2.1: [fix](cache) fix sql cache throw npe in cloud mode#47221(…
2c47675
…#47267) Cherry-picked from #47221 Co-authored-by: 924060929 <lanhuajian@selectdb.com>

wm1581066 added dev/2.1.9-merged and removed dev/2.1.x labels Jan 24, 2025

dataroaring pushed a commit that referenced this pull request Feb 5, 2025
branch-3.0: [fix](cache) fix sql cache throw npe in cloud mode#47221(…
b1e6ee2
…#47266) Cherry-picked from #47221 Co-authored-by: 924060929 <lanhuajian@selectdb.com>

dataroaring added dev/3.0.4-merged and removed dev/3.0.x labels Feb 5, 2025

924060929 deleted the fix-sql-cache-npe branch March 25, 2025 08:22

Navigation Menu

Uh oh!

[fix](cache) fix sql cache throw npe in cloud mode - #47221

[fix](cache) fix sql cache throw npe in cloud mode#47221
starocean999 merged 2 commits into
apache:masterfrom
924060929:fix-sql-cache-npe

924060929 commented Jan 20, 2025•
edited
Loading

Uh oh!

hello-stephen commented Jan 20, 2025

Uh oh!

924060929 commented Jan 20, 2025

Uh oh!

924060929 commented Jan 20, 2025

Uh oh!

doris-robot commented Jan 20, 2025

Uh oh!

doris-robot commented Jan 20, 2025

Uh oh!

doris-robot commented Jan 20, 2025

Uh oh!

github-actionsBot commented Jan 21, 2025

Uh oh!

github-actionsBot commented Jan 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

924060929 commented Jan 20, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jan 20, 2025

Uh oh!

924060929 commented Jan 20, 2025

Uh oh!

924060929 commented Jan 20, 2025

Uh oh!

doris-robot commented Jan 20, 2025

Uh oh!

doris-robot commented Jan 20, 2025

Uh oh!

doris-robot commented Jan 20, 2025

Uh oh!

github-actionsBot commented Jan 21, 2025

Uh oh!

github-actionsBot commented Jan 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

924060929 commented Jan 20, 2025•
edited
Loading