branch-4.1: (fix)[fe] pick some fe memory improve prs by mymeiyi · Pull Request #61455 · apache/doris

mymeiyi · 2026-03-18T03:39:07Z

pick:
#59327
#59460
#59648
#59683
#59759
#59757
#59776
#59814
#59932
#59934
#61080
#59791

…t used in CloudTablet (apache#59327) 1. some fields are not used in cloud mode, but `CloudTablet` extends `Tablet` and inherits these fields, which consumes too much memory 2. this pr add `LocalTablet` subclass to keep these fields which are only used in local mode 3. this pr only move cooldown related fileds, more fileds will be moved later in a cloud cluster with one million tablets, the test results are as follows: ||before|after|| |--------|--------|--------|--------| |total memory of `CloudTablet`|<img width="1280" height="351" alt="1-1" src="https://github.com/user-attachments/assets/b20cb1dd-da76-420d-8093-53891bbd5063" />|<img width="1280" height="358" alt="2-1" src="https://github.com/user-attachments/assets/00791566-83b7-4605-b7ac-1e25d7274223" />| reduce 272.93 MB | |memory of one `CloudTablet`|<img width="1386" height="762" alt="1-2" src="https://github.com/user-attachments/assets/424a01eb-d276-463d-9a8d-ff3a9ed185b2" />|<img width="1396" height="692" alt="2-2" src="https://github.com/user-attachments/assets/2bedff14-7bcc-42ab-bf7b-d5a8f993b8dd" />|reduce 272 B| |fields of `CloudTablet`|<img width="1388" height="620" alt="1-3" src="https://github.com/user-attachments/assets/cbc7fca3-5e39-4a44-b1c2-1e21d8328769" />|<img width="1388" height="512" alt="2-3" src="https://github.com/user-attachments/assets/cfc094f9-46e1-4fc3-8f1d-37dad0d04a8a" />|| ||old FE upgrade to new FE| new FE downgrade to old FE| |--------|--------|--------| |cloud mode| supported | supported| |local mode| supported | unsupported (need modify code in old FE to support it) | None - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…not used in CloudReplica to reduce memory (apache#59460)

…he#59648)

…ory (apache#59683)

…nsactionMgr when drop table (apache#59757) ### What problem does this PR solve? 1. `CloudGlobalTransactionMgr` does not remove `waitToCommitTxnCountMap` if table is dropped. 2. simplify `lastTxnIdMap` and `txnLastSignatureMap`

…ce memory (apache#59932)

### What problem does this PR solve? for about 1340000 tablets in cloud mode, the fe memory is: before: <img width="1834" height="174" alt="image" src="https://github.com/user-attachments/assets/f2392442-bb4c-4a85-b506-559400932b8e" /> after: <img width="1842" height="312" alt="image" src="https://github.com/user-attachments/assets/f3e1351f-3bc8-4994-89fb-8e423f77aff9" /> Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

hello-stephen · 2026-03-18T03:39:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

mymeiyi · 2026-03-18T03:42:09Z

run buildall

Copilot

Pull request overview

This branch-pick aggregates several upstream FE changes primarily aimed at reducing memory usage in cloud mode by splitting “local” vs “cloud” metadata representations and switching to more memory-efficient primitive collections.

Changes:

Introduce LocalTablet/LocalReplica and refactor Tablet/Replica into abstract bases with mode-specific implementations.
Split TabletInvertedIndex into LocalTabletInvertedIndex and CloudTabletInvertedIndex, and wire selection via EnvFactory / CloudEnvFactory.
Add fastutil-core and adopt fastutil primitive maps in hot metadata structures; adjust cloud table stats storage and related metric export.

Reviewed changes

Copilot reviewed 51 out of 52 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
fe/pom.xml	Adds `fastutil.version` and manages `fastutil-core` dependency.
fe/fe-core/pom.xml	Adds `fastutil-core` as a module dependency.
dist/LICENSE-dist.txt	Updates fastutil entry (needs alignment with actual artifact).
fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java	Updates Gson runtime subtype adapters for `Tablet`/`Replica` local vs cloud classes and compatibility.
fe/fe-core/src/main/java/org/apache/doris/metric/PrometheusMetricVisitor.java	Switches cloud table stats iteration to new `getCloudTableStats()` API.
fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java	Uses `LocalReplica` when creating replicas from BE reports (local mode).
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java	Uses `LocalReplica` for local catalog tablet/replica creation paths; updates cloud drop-table hook.
fe/fe-core/src/main/java/org/apache/doris/consistency/ConsistencyChecker.java	Refactors tablet selection queue typing; adds tablet comparator.
fe/fe-core/src/main/java/org/apache/doris/common/proc/TabletsProcDir.java	Proc output switches cooldown replica id accessor.
fe/fe-core/src/main/java/org/apache/doris/common/proc/ReplicasProcNode.java	Proc output switches cooldown replica id accessor.
fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java	Simplifies tracking maps; adds `afterDropTable()` cleanup hook.
fe/fe-core/src/main/java/org/apache/doris/cloud/datasource/CloudInternalCatalog.java	Avoids `getReplicas()` in cloud tablet paths by using `CloudTablet.getCloudReplica()`.
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java	Avoids `getReplicas()` in cloud balancing paths by using `CloudTablet.getCloudReplica()`.
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletInvertedIndex.java	New cloud-specific inverted index using fastutil map and single-replica assumption.
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTablet.java	Refactors cloud tablet to store a single `replica` (with backward-compatible `replicas` field).
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudReplica.java	Refactors primary BE mapping and adds in-memory segment/rowset counters.
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnvFactory.java	Creates `CloudTabletInvertedIndex` via factory override.
fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java	Uses `LocalReplica` for clone replica creation (local mode).
fe/fe-core/src/main/java/org/apache/doris/catalog/TabletStatMgr.java	Switches wrapper `Long` counters to primitive `long` for less overhead.
fe/fe-core/src/main/java/org/apache/doris/catalog/TabletInvertedIndex.java	Makes base class abstract; moves local implementation out; uses fastutil for tablet meta map.
fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java	Makes base class abstract; moves local-only fields/behaviors to `LocalTablet`.
fe/fe-core/src/main/java/org/apache/doris/catalog/Replica.java	Makes base class abstract; moves local-only fields/behaviors to `LocalReplica`.
fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java	Uses `LocalReplica` where applicable; converts `Statistics` fields/ctor to primitive `long`.
fe/fe-core/src/main/java/org/apache/doris/catalog/LocalTabletInvertedIndex.java	New local inverted index implementation (includes async tablet report processing).
fe/fe-core/src/main/java/org/apache/doris/catalog/LocalTablet.java	New local tablet implementation containing replicas/cooldown/health timing state.
fe/fe-core/src/main/java/org/apache/doris/catalog/LocalReplica.java	New local replica implementation containing backend id, version tracking, cooldown, etc.
fe/fe-core/src/main/java/org/apache/doris/catalog/EnvFactory.java	Centralizes creation of tablets/replicas/inverted index for local mode.
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java	Uses `EnvFactory.createTabletInvertedIndex()` for initialization.
fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java	Refactors stat collection flow and stores per-table stats in a list for lower overhead.
fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java	Uses `LocalReplica` for restore reset paths (local mode).
fe/fe-core/src/test/java/org/apache/doris/statistics/OlapAnalysisTaskTest.java	Updates mocks to use `LocalTablet`.
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/CreateTableCommandTest.java	Removes tablet meta table assertions and unused imports.
fe/fe-core/src/test/java/org/apache/doris/http/DorisHttpTestCase.java	Updates test fixtures to use `LocalTablet`/`LocalReplica`/`LocalTabletInvertedIndex`.
fe/fe-core/src/test/java/org/apache/doris/cooldown/CooldownConfHandlerTest.java	Uses `LocalTablet` and casts retrieved tablet accordingly.
fe/fe-core/src/test/java/org/apache/doris/common/util/UnitTestUtil.java	Uses `LocalTablet`/`LocalReplica` in helper table creation.
fe/fe-core/src/test/java/org/apache/doris/cluster/SystemInfoServiceTest.java	Uses `LocalTabletInvertedIndex` in setup.
fe/fe-core/src/test/java/org/apache/doris/cluster/DecommissionBackendTest.java	Uses `LocalReplica` for fake replica creation.
fe/fe-core/src/test/java/org/apache/doris/cloud/cache/CacheHotspotManagerTest.java	Uses `CloudTablet` for cloud-mode tablet fixture.
fe/fe-core/src/test/java/org/apache/doris/clone/TabletSchedCtxTest.java	Uses `LocalReplica` in comparator test.
fe/fe-core/src/test/java/org/apache/doris/clone/TabletReplicaTooSlowTest.java	Uses `LocalTabletInvertedIndex`.
fe/fe-core/src/test/java/org/apache/doris/clone/TabletHealthTest.java	Uses `LocalReplica` when adding a replica in tests.
fe/fe-core/src/test/java/org/apache/doris/clone/RebalancerTestUtil.java	Uses `LocalTablet`/`LocalReplica` in rebalancer test utilities.
fe/fe-core/src/test/java/org/apache/doris/clone/RebalanceTest.java	Uses `LocalTabletInvertedIndex`.
fe/fe-core/src/test/java/org/apache/doris/clone/DiskRebalanceTest.java	Uses `LocalTabletInvertedIndex`.
fe/fe-core/src/test/java/org/apache/doris/clone/DecommissionTest.java	Uses `LocalTabletInvertedIndex`.
fe/fe-core/src/test/java/org/apache/doris/clone/ClusterLoadStatisticsTest.java	Uses `LocalTabletInvertedIndex` and `LocalReplica` fixtures.
fe/fe-core/src/test/java/org/apache/doris/catalog/TabletTest.java	Uses `LocalTabletInvertedIndex`, `LocalTablet`, and `LocalReplica`; removes `clearReplica()` assertion.
fe/fe-core/src/test/java/org/apache/doris/catalog/ReplicaTest.java	Uses `LocalReplica` in serialization/version tests.
fe/fe-core/src/test/java/org/apache/doris/catalog/CreateTableTest.java	Removes tablet meta table assertions and unused imports.
fe/fe-core/src/test/java/org/apache/doris/catalog/CatalogTestUtil.java	Uses `LocalTablet`/`LocalReplica` for non-cloud test DB creation.
fe/fe-core/src/test/java/org/apache/doris/backup/CatalogMocker.java	Uses `LocalTablet`/`LocalReplica` in backup/restore mocks.
fe/fe-core/src/test/java/org/apache/doris/backup/BackupHandlerTest.java	Uses `LocalTabletInvertedIndex` in test setup.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

 (first, second) -> Long.signum(first.getLastCheckTime() - second.getLastCheckTime());
+ private static final Comparator<Tablet> TABLET_COMPARATOR =
+ (first, second) -> Long.signum(first.getLastCheckTime() - second.getLastCheckTime());


+ CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {
+ for (int j = start; j < end; j++) {
+ Map.Entry<Long, Replica> entry = entries.get(j);
+ processTabletEntry(backendId, backendTablets, storageMediumMap, tabletSyncMap,
+ tabletDeleteFromMeta, tabletFoundInMeta, tabletMigrationMap,
+ transactionsToPublish, transactionsToClear, tabletRecoveryMap,
+ tabletToUpdate, cooldownTablets, entry);
+ }
+ }, taskPool);
+
+ tabletFutures.add(future);
+ }
+
+ // Process partition versions in parallel
+ CompletableFuture<Void> partitionFuture = CompletableFuture.runAsync(() -> {
+ processPartitionVersions(backendPartitionsVersion, partitionVersionSyncMap);
+ }, taskPool);
+
+ // Wait for all tasks to complete
+ CompletableFuture.allOf(tabletFutures.toArray(new CompletableFuture[0])).join();
+ partitionFuture.join();


+ }
+ }
+ // return replica with max remoteDataSize
+ return replicas.stream().max(Comparator.comparing(Replica::getRemoteDataSize)).get().getRemoteDataSize();


+ if (memClusterToBackends == null) {
+ synchronized (this) {
+ if (memClusterToBackends == null) {
+ memClusterToBackends = new ConcurrentHashMap<>();
+ }


…g is confusing (apache#59791)

mymeiyi · 2026-03-18T03:59:30Z

run buildall

hello-stephen · 2026-03-18T05:15:08Z

FE UT Coverage Report

Increment line coverage 52.13% (623/1195) 🎉
Increment coverage report
Complete coverage report

mymeiyi · 2026-03-18T06:27:37Z

run cloud_p0

mymeiyi added 11 commits March 18, 2026 11:12

[fix](fe) move some variables from Replica to LocalReplica which are …
45d7d60
…not used in CloudReplica to reduce memory (apache#59460)

[fix](fe) move variables in Replica and Tablet to reduce memory (apac…
59b16ad
…he#59648)

[opt](fe) modify TabletInvertedIndex to local and cloud to reduce mem…
797ec08
…ory (apache#59683)

[fix](fe) modify tablet and replica to reduce memory (apache#59759)
b58beba

[fix](fe) modify CloudTabletStatMgr to reduce memory (apache#59776)
f38d106

[fix](fe) modify replicas to replica in CloudTablet (apache#59814)
7d2b143

[fix](cloud) modify primary cluster bes to be in CloudReplica to redu…
f1360d7
…ce memory (apache#59932)

[fix](cloud) skip call getReplicas in cloud tablet (apache#59934)
9d5cfce

mymeiyi requested a review from yiguolei as a code owner March 18, 2026 03:39

CopilotAI review requested due to automatic review settings March 18, 2026 03:39

Copilotstarted reviewing on behalf of mymeiyi March 18, 2026 03:39 View session

CopilotAI reviewed Mar 18, 2026
View reviewed changes

mymeiyiand others added 2 commits March 18, 2026 11:51

fix
ae9394b

[fix](cloud) Fix getNormalReplicaBackendPathMapCloud function namin…
3b2958d
…g is confusing (apache#59791)

mymeiyi changed the title ~~branch-4.1: pick some fe memory improves prs~~branch-4.1: (fix)[fe] pick some fe memory improves prsMar 18, 2026

mymeiyi changed the title ~~branch-4.1: (fix)[fe] pick some fe memory improves prs~~branch-4.1: (fix)[fe] pick some fe memory improve prsMar 18, 2026

yiguolei approved these changes Mar 18, 2026
View reviewed changes

yiguolei merged commit f7a4908 into apache:branch-4.1Mar 18, 2026
25 of 31 checks passed

Navigation Menu

Uh oh!

branch-4.1: (fix)[fe] pick some fe memory improve prs - #61455

branch-4.1: (fix)[fe] pick some fe memory improve prs#61455
yiguolei merged 13 commits into
apache:branch-4.1from
mymeiyi:branch-4.1-pick

mymeiyi commented Mar 18, 2026•
edited
Loading

Uh oh!

hello-stephen commented Mar 18, 2026

Uh oh!

mymeiyi commented Mar 18, 2026

Uh oh!

CopilotAI left a comment

Uh oh!

mymeiyi commented Mar 18, 2026

Uh oh!

hello-stephen commented Mar 18, 2026

Uh oh!

mymeiyi commented Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

mymeiyi commented Mar 18, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hello-stephen commented Mar 18, 2026

Uh oh!

mymeiyi commented Mar 18, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

mymeiyi commented Mar 18, 2026

Uh oh!

hello-stephen commented Mar 18, 2026

FE UT Coverage Report

Uh oh!

mymeiyi commented Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mymeiyi commented Mar 18, 2026•
edited
Loading