[enhancement](cloud) support BE http action: list_cache and clear - #41037

Merged

gavinchou merged 8 commits into

apache:masterfrom

yagagagaga:core-4634

Nov 7, 2024

yagagagaga commented Sep 20, 2024•
edited
Loading

Contributor

Proposed changes

Add a http action which is useful when you debug.

API

GET /api/file_cache

request parameter

request parameter1

param	type	desc	require
op	string	the value must be `list_cache`, other value you can refer to #40831 #37484	yes
value	string	the segment file name	yes

request parameter2

param	type	desc	require
op	string	the value must be `clear`, other value you can refer to #40831 #37484	yes
value	string	the segment file name	yes
sync	bool	clean local cache in sync	no

response

response1

if success

param	type	desc
	array	return the segment file cache in local path

if fail

param	type	desc
	array	empty array

response2

if success

param	type	desc
status	string
msg	string

example

case 1

curl '172.100.0.4:8040/api/file_cache?op=list_cache&value=0200000000000001bf42c14374fff491ffb7c89a1a65c5bb_0.dat'

return

["/opt/doris/be/file_cache/c6a/c6a599f453f67f0949f80ad9990fa3dd/0"]

case 2

curl '127.0.0.1:8040/api/file_cache?op=clear&sync=true&value=0200000000000001284b68fea3dcfe8a83e65cd88426b081_0.dat'

return

{
"status": "OK",
"msg": "OK"
}

doris-robot commented Sep 20, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

yagagagaga marked this pull request as draft

September 20, 2024 03:54

gavinchou reviewed

View reviewed changes

be/src/io/cache/block_file_cache.cpp Outdated

gavinchou reviewed

View reviewed changes

be/src/io/cache/block_file_cache.cpp Outdated

freemandealer requested changes

View reviewed changes

be/src/http/action/file_cache_action.cpp Outdated

be/src/io/cache/block_file_cache_factory.h Outdated

regression-test/suites/cloud_p0/cache/http/test_calc_cache_file_hash.groovy

be/src/io/cache/block_file_cache.cpp Outdated

yagagagaga force-pushed the core-4634 branch from 4d2c05a to 62ee150Compare

September 30, 2024 09:25

yagagagaga added 2 commits

October 21, 2024 16:29

update

53834e5

update

c55ce0c

yagagagaga force-pushed the core-4634 branch from 62ee150 to c55ce0cCompare

October 22, 2024 09:12

Merge branch 'master' into core-4634

2b2dabb

github-actionsBot commented Oct 22, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment

github-actionsBot commented Oct 22, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

yagagagaga marked this pull request as ready for review

October 22, 2024 12:54

update

dbb6a78

github-actionsBot commented Oct 22, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

yagagagaga commented Oct 22, 2024

ContributorAuthor

run buidlall

yagagagaga added 2 commits

October 22, 2024 21:24

clang-tidy code format

6ae1c44

code format

d779985

github-actionsBot commented Oct 22, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment

github-actionsBot commented Oct 22, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

yagagagaga changed the title ~~[enhancement](cloud) support BE http action: hash_path and clear~~[enhancement](cloud) support BE http action: list_cache and clear

yagagagaga force-pushed the core-4634 branch from f906fef to d779985Compare

October 22, 2024 15:24

add regression test

2e528d9

yagagagaga requested review from freemandealer and gavinchou

October 22, 2024 15:28

github-actionsBot commented Oct 22, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment

github-actionsBot commented Oct 22, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

add more message

166281b

github-actionsBot commented Oct 22, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

yagagagaga commented Oct 22, 2024

ContributorAuthor

run buildall

github-actionsBot commented Oct 22, 2024

Contributor

clang-tidy review says "All clean, LGTM! 👍"

doris-robot commented Oct 22, 2024

TPC-H: Total hot run time: 41499 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 166281bcefca5e6980e799a69f9f12e17b0ba255, data reload: false
------ Round 1 ----------------------------------
q1	17685	7511	7267	7267
q2	2014	290	306	290
q3	11590	1082	1181	1082
q4	10593	891	754	754
q5	7752	3137	3142	3137
q6	244	155	152	152
q7	1055	642	610	610
q8	9357	1974	2027	1974
q9	6677	6548	6453	6453
q10	7091	2457	2466	2457
q11	447	247	234	234
q12	410	215	217	215
q13	17754	2994	3018	2994
q14	238	213	213	213
q15	584	520	522	520
q16	642	602	598	598
q17	998	535	528	528
q18	7273	6799	6836	6799
q19	1372	992	926	926
q20	483	192	191	191
q21	4124	3340	3104	3104
q22	1104	1008	1001	1001
Total cold run time: 109487 ms
Total hot run time: 41499 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7305	7295	7249	7249
q2	327	229	232	229
q3	3061	3036	2876	2876
q4	2100	1841	1799	1799
q5	5779	5791	5781	5781
q6	241	146	152	146
q7	2263	1866	1796	1796
q8	3401	3593	3467	3467
q9	9020	9002	8898	8898
q10	3587	3602	3570	3570
q11	605	491	503	491
q12	820	635	663	635
q13	9941	3179	3149	3149
q14	302	280	300	280
q15	578	532	536	532
q16	686	647	631	631
q17	1876	1613	1615	1613
q18	8365	7805	7654	7654
q19	1746	1511	1603	1511
q20	2123	1923	1899	1899
q21	5625	5423	5458	5423
q22	1162	1035	1027	1027
Total cold run time: 70913 ms
Total hot run time: 60656 ms

doris-robot commented Oct 22, 2024

TeamCity be ut coverage result:
Function Coverage: 37.46% (9707/25915)
Line Coverage: 28.73% (80529/280319)
Region Coverage: 28.15% (41634/147906)
Branch Coverage: 24.72% (21151/85574)
Coverage Report: http://coverage.selectdb-in.cc/coverage/166281bcefca5e6980e799a69f9f12e17b0ba255_166281bcefca5e6980e799a69f9f12e17b0ba255/report/index.html

doris-robot commented Oct 22, 2024

TPC-DS: Total hot run time: 191663 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 166281bcefca5e6980e799a69f9f12e17b0ba255, data reload: false
query1	967	420	406	406
query2	6262	2088	2030	2030
query3	8688	197	214	197
query4	34200	23562	23627	23562
query5	3608	490	510	490
query6	289	169	184	169
query7	4201	294	283	283
query8	285	227	225	225
query9	9542	2713	2710	2710
query10	472	276	266	266
query11	17982	15193	15356	15193
query12	153	104	100	100
query13	1550	422	426	422
query14	9407	7476	6981	6981
query15	263	175	186	175
query16	8036	472	466	466
query17	1641	623	592	592
query18	2164	320	327	320
query19	367	161	179	161
query20	137	115	122	115
query21	213	108	103	103
query22	4941	4659	4738	4659
query23	35332	34067	33825	33825
query24	10854	2805	2724	2724
query25	626	425	409	409
query26	1138	162	165	162
query27	2271	280	283	280
query28	7415	2453	2429	2429
query29	837	445	417	417
query30	272	152	153	152
query31	1154	752	804	752
query32	104	56	54	54
query33	769	289	310	289
query34	936	527	514	514
query35	871	779	767	767
query36	1132	949	962	949
query37	159	98	93	93
query38	4070	3879	3927	3879
query39	1473	1423	1414	1414
query40	213	105	100	100
query41	49	49	47	47
query42	130	99	103	99
query43	529	472	489	472
query44	1250	825	806	806
query45	196	165	170	165
query46	1160	711	704	704
query47	1946	1835	1854	1835
query48	424	324	320	320
query49	918	441	431	431
query50	801	391	383	383
query51	7091	7047	6946	6946
query52	100	89	90	89
query53	252	181	180	180
query54	1183	436	440	436
query55	78	80	75	75
query56	314	295	284	284
query57	1330	1136	1162	1136
query58	227	236	238	236
query59	3219	2968	2948	2948
query60	303	275	266	266
query61	108	100	101	100
query62	896	668	665	665
query63	218	188	175	175
query64	4013	627	597	597
query65	3275	3187	3238	3187
query66	833	296	303	296
query67	16240	15671	15629	15629
query68	4550	565	557	557
query69	466	287	308	287
query70	1192	1109	1127	1109
query71	382	294	277	277
query72	7213	4184	3974	3974
query73	779	354	349	349
query74	10102	9093	8968	8968
query75	3408	2670	2684	2670
query76	2881	992	937	937
query77	417	297	298	297
query78	10435	9650	9642	9642
query79	1302	588	584	584
query80	2074	441	451	441
query81	565	243	238	238
query82	732	141	139	139
query83	280	139	132	132
query84	272	70	69	69
query85	1310	292	312	292
query86	368	302	310	302
query87	4519	4313	4398	4313
query88	3008	2192	2152	2152
query89	405	285	288	285
query90	2196	184	184	184
query91	152	103	103	103
query92	65	48	52	48
query93	1061	525	530	525
query94	1053	291	285	285
query95	345	239	254	239
query96	611	279	285	279
query97	3262	3190	3147	3147
query98	216	198	194	194
query99	1573	1319	1304	1304
Total cold run time: 299271 ms
Total hot run time: 191663 ms

doris-robot commented Oct 22, 2024

ClickBench: Total hot run time: 33.03 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 166281bcefca5e6980e799a69f9f12e17b0ba255, data reload: false
query1	0.03	0.04	0.03
query2	0.05	0.03	0.03
query3	0.23	0.07	0.07
query4	1.67	0.10	0.10
query5	0.51	0.51	0.50
query6	1.15	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.55	0.49	0.50
query10	0.57	0.55	0.54
query11	0.14	0.12	0.11
query12	0.14	0.11	0.10
query13	0.61	0.62	0.60
query14	2.84	2.86	2.73
query15	0.89	0.83	0.83
query16	0.38	0.40	0.38
query17	1.00	1.00	1.03
query18	0.20	0.20	0.20
query19	1.87	1.87	2.04
query20	0.01	0.01	0.01
query21	15.35	0.60	0.56
query22	2.75	2.76	1.33
query23	17.07	0.89	0.94
query24	2.49	1.46	2.20
query25	0.18	0.17	0.19
query26	0.41	0.15	0.15
query27	0.05	0.05	0.05
query28	9.63	1.10	1.06
query29	12.55	3.36	3.36
query30	0.25	0.07	0.06
query31	2.87	0.38	0.39
query32	3.28	0.46	0.46
query33	3.01	3.00	3.05
query34	16.98	4.47	4.47
query35	4.49	4.52	4.51
query36	0.67	0.50	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.41 s
Total hot run time: 33.03 s

gavinchou commented Oct 30, 2024

Contributor

list what has been cleared, a file list with size

freemandealer approved these changes

View reviewed changes

freemandealer left a comment

Member

LGTM

github-actionsBot added the reviewed label

github-actionsBot commented Nov 5, 2024

Contributor

PR approved by anyone and no changes requested.

gavinchou approved these changes

View reviewed changes

github-actionsBot added the approved label

github-actionsBot commented Nov 5, 2024

Contributor

PR approved by at least one committer and no changes requested.

gavinchou merged commit 99d0748 into apache:master

gavinchou added the dev/3.0.x label

yagagagaga added a commit to yagagagaga/doris that referenced this pull request

[enhancement](cloud) support BE http action: list_cache and clear (ap…

fb1036e

…ache#41037)
## Proposed changes
Add a http action which is useful when you debug.
### API
```http
GET /api/file_cache
```
### request parameter
#### request parameter1
|param|type|desc|require|
|:---|:---|:---|:---|
|op|string|the value must be `list_cache`, other value you can refer to
apache#40831apache#37484 |yes|
|value|string|the segment file name |yes|
#### request parameter2
|param|type|desc|require|
|:---|:---|:---|:---|
|op|string|the value must be `clear`, other value you can refer to
apache#40831apache#37484 |yes|
|value|string|the segment file name |yes|
|sync|bool|clean local cache in sync |no|
### response
#### response1
if success
|param|type|desc|
|:---|:---|:---|
||array|return the segment file cache in local path|
if fail
|param|type|desc|
|:---|:---|:---|
||array|empty array|
#### response2
if success
|param|type|desc|
|:---|:---|:---|
|status|string||
|msg|string||
### example
#### case 1
```bash
curl '172.100.0.4:8040/api/file_cache?op=list_cache&value=0200000000000001bf42c14374fff491ffb7c89a1a65c5bb_0.dat'
```
return
```json
["/opt/doris/be/file_cache/c6a/c6a599f453f67f0949f80ad9990fa3dd/0"]
```
#### case 2
```bash
curl '127.0.0.1:8040/api/file_cache?op=clear&sync=true&value=0200000000000001284b68fea3dcfe8a83e65cd88426b081_0.dat'
```
return
```json
{
"status": "OK",
"msg": "OK"
}
```
(cherry picked from commit 99d0748)

yagagagaga mentioned this pull request

[cherry-pick][enhancement](cloud) support BE http action: list_cache and clear (#41037) #43412

Merged

16 tasks

gavinchou pushed a commit that referenced this pull request

[cherry-pick][enhancement](cloud) support BE http action: list_cache …

098b66e

…and clear (#41037) (#43412)
## Proposed changes
Add a http action which is useful when you debug.
### API
```http
GET /api/file_cache
```
### request parameter
#### request parameter1
|param|type|desc|require|
|:---|:---|:---|:---|
|op|string|the value must be `list_cache`, other value you can refer to
#40831#37484 |yes|
|value|string|the segment file name |yes|
#### request parameter2
|param|type|desc|require|
|:---|:---|:---|:---|
|op|string|the value must be `clear`, other value you can refer to
#40831#37484 |yes|
|value|string|the segment file name |yes|
|sync|bool|clean local cache in sync |no|
### response
#### response1
if success
|param|type|desc|
|:---|:---|:---|
||array|return the segment file cache in local path|
if fail
|param|type|desc|
|:---|:---|:---|
||array|empty array|
#### response2
if success
|param|type|desc|
|:---|:---|:---|
|status|string||
|msg|string||
### example
#### case 1
```bash
curl '172.100.0.4:8040/api/file_cache?op=list_cache&value=0200000000000001bf42c14374fff491ffb7c89a1a65c5bb_0.dat'
```
return
```json
["/opt/doris/be/file_cache/c6a/c6a599f453f67f0949f80ad9990fa3dd/0"]
```
#### case 2
```bash
curl '127.0.0.1:8040/api/file_cache?op=clear&sync=true&value=0200000000000001284b68fea3dcfe8a83e65cd88426b081_0.dat'
```
return
```json
{
"status": "OK",
"msg": "OK"
}
```
(cherry picked from commit 99d0748)

gavinchou added dev/3.0.3-merged and removed dev/3.0.x labels

gavinchou mentioned this pull request

Release Note 3.0.3 #44522

Closed

yagagagaga deleted the core-4634 branch

February 25, 2025 02:41

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request

[enhancement](cloud) support BE http action: list_cache and clear (ap…

e3a0a37

…ache#41037)
## Proposed changes
Add a http action which is useful when you debug.
### API
```http
GET /api/file_cache
```
### request parameter
#### request parameter1
|param|type|desc|require|
|:---|:---|:---|:---|
|op|string|the value must be `list_cache`, other value you can refer to
apache#40831apache#37484 |yes|
|value|string|the segment file name |yes|
#### request parameter2
|param|type|desc|require|
|:---|:---|:---|:---|
|op|string|the value must be `clear`, other value you can refer to
apache#40831apache#37484 |yes|
|value|string|the segment file name |yes|
|sync|bool|clean local cache in sync |no|
### response
#### response1
if success
|param|type|desc|
|:---|:---|:---|
||array|return the segment file cache in local path|
if fail
|param|type|desc|
|:---|:---|:---|
||array|empty array|
#### response2
if success
|param|type|desc|
|:---|:---|:---|
|status|string||
|msg|string||
### example
#### case 1
```bash
curl '172.100.0.4:8040/api/file_cache?op=list_cache&value=0200000000000001bf42c14374fff491ffb7c89a1a65c5bb_0.dat'
```
return
```json
["/opt/doris/be/file_cache/c6a/c6a599f453f67f0949f80ad9990fa3dd/0"]
```
#### case 2
```bash
curl '127.0.0.1:8040/api/file_cache?op=clear&sync=true&value=0200000000000001284b68fea3dcfe8a83e65cd88426b081_0.dat'
```
return
```json
{
"status": "OK",
"msg": "OK"
}
```

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment