Uh oh!
There was an error while loading. Please reload this page.
[feature](multi-catalog) support hive metastore more events - #15702
Merged
Conversation
added 2 commits
January 4, 2023 21:37
Contributor
TeamCity pipeline, clickbench performance test result: |
added 2 commits
January 7, 2023 22:55
| } | ||
| } | ||
| public void refreshExternalTable(String dbName, String tableName, String catalogName) throws DdlException { |
Contributor
There was a problem hiding this comment.
We had a same method in RefreshManager, called refreshExternalCtlTable(), why not using that?
ContributorAuthor
There was a problem hiding this comment.
That method is not abstract enough. I abstracted a method and put it in CatalogMgr, and then deleted the method in RefreshMgr
| @Override | ||
| protected void process() throws MetastoreNotificationException { | ||
| // only can change properties,we do nothing | ||
| infoLog("catalogName:[{}],dbName:[{}]", catalogName, dbName); |
Contributor
There was a problem hiding this comment.
You can print the modified properties in this log. so we can get more info.
ContributorAuthor
There was a problem hiding this comment.
current api version we used can not support we do this
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
morningman pushed a commit
that referenced
this pull request
Jan 16, 2023
support hive metastore more events
Merged
7 tasks
924060929 added a commit
that referenced
this pull request
Jul 13, 2026
…tition cache misses (#65334) Problem Summary: Partition-level refresh / invalidation (`HiveExternalMetaCache.invalidatePartitionCache`) returned early when the target partition was absent from the partition metadata cache — evicted under `max_hive_partition_cache_num` pressure, or simply never loaded — and left that partition's file listing cache untouched. The partition cache (`partition`) and the file listing cache (`file`) are independent caches with independent size limits (`max_hive_partition_cache_num` vs `max_external_file_cache_num`) and evict separately, so the partition entry can be evicted while its file listing survives. In that state a partition-level refresh cannot rebuild the exact `FileCacheKey` (it needs the partition path / input format that only the cached `HivePartition` carries), so the old code gave up and the stale file listing remained. A later query would reload the partition metadata and re-hit the same stale file listing — i.e. "the partition exists, but Doris keeps seeing the old files/data". Fix: when the exact `FileCacheKey` cannot be rebuilt, fall back to a predicate-based invalidation on the file cache keyed by `table id + partition values`, so the stale listing for that partition is cleared without over-invalidating other partitions. This mirrors table-level invalidation, which already invalidates the file cache by predicate. The partition invalidation coordinator is also decoupled from `ExternalTable` to `NameMapping` (mirroring the existing `invalidateTableCache(NameMapping)`) so the path can be unit tested directly. This behavior has existed since partition-level invalidation was first introduced in #15702 (the file cache was only invalidated inside the `if (partition != null)` branch), and was later carried into the current `HiveExternalMetaCache` by the #60937 cache-framework refactor. ### Release note Fix a Hive external catalog consistency issue where a partition-level refresh could leave a stale file listing cached when the partition metadata cache had been evicted, causing queries to keep seeing outdated files for that partition.
yiguolei pushed a commit
that referenced
this pull request
Jul 30, 2026
…esh when partition cache misses #65334 (#66259) ### What problem does this PR solve? Cherry-pick from master #65334. Problem Summary: Partition-level refresh / invalidation (`HiveExternalMetaCache.invalidatePartitionCache`) returned early when the target partition was absent from the partition metadata cache — evicted under `max_hive_partition_cache_num` pressure, or simply never loaded — and left that partition's file listing cache untouched. The partition cache (`partition`) and the file listing cache (`file`) are independent caches with independent size limits (`max_hive_partition_cache_num` vs `max_external_file_cache_num`) and evict separately, so the partition entry can be evicted while its file listing survives. In that state a partition-level refresh cannot rebuild the exact `FileCacheKey` (it needs the partition path / input format that only the cached `HivePartition` carries), so the old code gave up and the stale file listing remained. A later query would reload the partition metadata and re-hit the same stale file listing — i.e. "the partition exists, but Doris keeps seeing the old files/data". Fix: when the exact `FileCacheKey` cannot be rebuilt, fall back to a predicate-based invalidation on the file cache keyed by `table id + partition values`, so the stale listing for that partition is cleared without over-invalidating other partitions. This mirrors table-level invalidation, which already invalidates the file cache by predicate. The partition invalidation coordinator is also decoupled from `ExternalTable` to `NameMapping` (mirroring the existing `invalidateTableCache(NameMapping)`) so the path can be unit tested directly. This behavior has existed since partition-level invalidation was first introduced in #15702 (the file cache was only invalidated inside the `if (partition != null)` branch), and was later carried into the current `HiveExternalMetaCache` by the #60937 cache-framework refactor. #### Backport notes Conflicts resolved by hand, because branch-4.1 does not have #64867 ("Key Hive partition cache by name"): - `HivePartitionValues` on this branch is still keyed by local partition id (`partitionNameToIdMap` / `partitionValuesMap`) instead of master's `nameToPartitionValues`, so the removed block looks different. The replacement is unchanged: `HiveUtil.toPartitionValues(partitionName)`. This is behavior-preserving on the happy path — on this branch `partitionValuesMap` is built from `PartitionKey.createListPartitionKeyWithTypes(..., isHive=true)`, whose `getPartitionValuesAsStringList()` returns `originHiveKeys`, i.e. exactly the strings `HiveUtil.toPartitionValues(partitionName)` produced in the first place. - The two master-only unit tests that git pulled into the conflict hunk as context (`testDefaultSpecsFollowConfig`, `testHivePartitionValuesCopyKeepsIndependentNameMaps`) are not part of #65334 and were dropped; only the new test from #65334 is added. ### Release note Fix a Hive external catalog consistency issue where a partition-level refresh could leave a stale file listing cached when the partition metadata cache had been evicted, causing queries to keep seeing outdated files for that partition. ### Check List (For Author) - Test - [x] Unit Test - [ ] Regression test - [ ] Manual test - [ ] No need to test - Behavior changed: - [ ] No. - [x] Yes. Partition-level refresh now also invalidates the file listing cache when the partition metadata cache entry is missing (previously it was left stale). - Does this need documentation? - [x] No.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close #xxx
Problem summary
support hive metastore more events
Checklist(Required)
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...