Skip to content

[fix](catalog)Disable Hadoop FileSystem cache for multi-configurationobject storage catalogs - #57063

Merged
CalvinKirs merged 2 commits into
apache:masterfrom
CalvinKirs:master-disable-cache
Oct 20, 2025
Merged

[fix](catalog)Disable Hadoop FileSystem cache for multi-configurationobject storage catalogs#57063
CalvinKirs merged 2 commits into
apache:masterfrom
CalvinKirs:master-disable-cache

Conversation

@CalvinKirs

@CalvinKirsCalvinKirs commented Oct 16, 2025•
edited
Loading

Copy link
Copy Markdown
Member

…

Iceberg and Paimon catalogs access object storage (OSS, OBS, etc.) using Hadoop FileSystem libraries. In environments with multiple storage configurations, the FileSystem cache may cause instances to be shared across configurations, leading to potential conflicts or incorrect access.

This PR disables the Hadoop FileSystem cache (fs.oss.impl.disable.cache=true) for each OSS to ensure configuration isolation and prevent cache overwrites.

If necessary, the cache can also be explicitly enabled for specific catalogs
by setting fs.oss.impl.disable.cache=false.

@CalvinKirs
[fix](catalog)Disable Hadoop FileSystem cache for multi-configuration…
741aa46
… object storage catalogs
Iceberg and Paimon catalogs access object storage (OSS, OBS, etc.) using Hadoop FileSystem libraries.
In environments with multiple storage configurations, the FileSystem cache may cause instances
to be shared across configurations, leading to potential conflicts or incorrect access.
This PR disables the Hadoop FileSystem cache (`fs.oss.impl.disable.cache=true`)
for each OSS to ensure configuration isolation and prevent cache overwrites.
@hello-stephen

hello-stephen commented Oct 16, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

morningman
morningman previously approved these changes Oct 16, 2025
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman

morningman commented Oct 16, 2025

Copy link
Copy Markdown
Contributor

run buildall

@github-actions

github-actionsBot commented Oct 16, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Oct 16, 2025
@github-actions

github-actionsBot commented Oct 16, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@CalvinKirs

CalvinKirs commented Oct 17, 2025

Copy link
Copy Markdown
MemberAuthor

run buildall

@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label Oct 17, 2025
@doris-robot

doris-robot commented Oct 17, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.97 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02f0f03787bd99cbaac3355cc0564728b9e1085a, data reload: false
query1	0.06	0.05	0.05
query2	0.09	0.06	0.05
query3	0.26	0.09	0.09
query4	1.62	0.13	0.12
query5	0.30	0.26	0.25
query6	1.19	0.67	0.65
query7	0.03	0.02	0.03
query8	0.06	0.04	0.05
query9	0.64	0.55	0.54
query10	0.59	0.58	0.57
query11	0.20	0.11	0.11
query12	0.16	0.12	0.12
query13	0.64	0.61	0.61
query14	1.03	1.05	1.04
query15	0.88	0.89	0.87
query16	0.41	0.39	0.40
query17	1.02	1.10	1.07
query18	0.22	0.20	0.21
query19	1.99	1.79	1.87
query20	0.02	0.01	0.02
query21	15.41	0.94	0.57
query22	0.77	1.15	0.70
query23	14.91	1.44	0.64
query24	7.00	1.60	1.07
query25	0.48	0.27	0.09
query26	0.59	0.16	0.16
query27	0.07	0.06	0.05
query28	9.62	1.36	0.94
query29	12.59	4.08	3.37
query30	0.31	0.15	0.11
query31	2.83	0.59	0.40
query32	3.24	0.56	0.47
query33	3.09	3.14	3.13
query34	16.23	5.44	4.87
query35	4.98	4.97	4.91
query36	0.70	0.53	0.51
query37	0.11	0.08	0.07
query38	0.07	0.05	0.05
query39	0.04	0.03	0.03
query40	0.20	0.15	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.04
query43	0.05	0.05	0.04
Total cold run time: 104.83 s
Total hot run time: 30.97 s

@hello-stephen

hello-stephen commented Oct 17, 2025

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (19/19) 🎉
Increment coverage report
Complete coverage report

@github-actions

github-actionsBot commented Oct 20, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 20, 2025
@CalvinKirs
CalvinKirs merged commit 017f9e4 into apache:masterOct 20, 2025
26 checks passed
@CalvinKirs
CalvinKirs deleted the master-disable-cache branch October 20, 2025 07:44
github-actionsBot pushed a commit that referenced this pull request Oct 20, 2025
@CalvinKirsYour Name
[fix](catalog)Disable Hadoop FileSystem cache for multi-configuration…
f98f604
…object storage catalogs (#57063)
… Iceberg and Paimon catalogs access object storage (OSS, OBS, etc.) using
Hadoop FileSystem libraries. In environments with multiple storage
configurations, the FileSystem cache may cause instances to be shared
across configurations, leading to potential conflicts or incorrect
access.
This PR disables the Hadoop FileSystem cache
(`fs.oss.impl.disable.cache=true`) for each OSS to ensure configuration
isolation and prevent cache overwrites.
If necessary, the cache can also be explicitly enabled for specific
catalogs
by setting `fs.oss.impl.disable.cache=false`.
@github-actionsgithub-actionsBot mentioned this pull request Oct 20, 2025
Merged
github-actionsBot pushed a commit that referenced this pull request Oct 20, 2025
@CalvinKirsYour Name
[fix](catalog)Disable Hadoop FileSystem cache for multi-configuration…
1b603b2
…object storage catalogs (#57063)
… Iceberg and Paimon catalogs access object storage (OSS, OBS, etc.) using
Hadoop FileSystem libraries. In environments with multiple storage
configurations, the FileSystem cache may cause instances to be shared
across configurations, leading to potential conflicts or incorrect
access.
This PR disables the Hadoop FileSystem cache
(`fs.oss.impl.disable.cache=true`) for each OSS to ensure configuration
isolation and prevent cache overwrites.
If necessary, the cache can also be explicitly enabled for specific
catalogs
by setting `fs.oss.impl.disable.cache=false`.
@github-actionsgithub-actionsBot mentioned this pull request Oct 20, 2025
Merged
morrySnow pushed a commit that referenced this pull request Oct 21, 2025
@github-actions@CalvinKirs
branch-3.1: [fix](catalog)Disable Hadoop FileSystem cache for multi-c…
0a2fc5c
…onfigurationobject storage catalogs #57063 (#57153)
Cherry-picked from #57063
Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
yiguolei pushed a commit that referenced this pull request Oct 23, 2025
@github-actions@CalvinKirs
branch-4.0: [fix](catalog)Disable Hadoop FileSystem cache for multi-c…
5492afb
…onfigurationobject storage catalogs #57063 (#57154)
Cherry-picked from #57063
Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
@yiguoleiyiguolei mentioned this pull request Nov 5, 2025
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zy-kkkzy-kkkzy-kkk approved these changes
@hubgeterhubgeterhubgeter approved these changes
@morningmanmorningmanmorningman left review comments

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.1.2-mergeddev/4.0.1-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@CalvinKirs@hello-stephen@morningman@doris-robot@zy-kkk@hubgeter@yiguolei@morrySnow