Skip to content

HDDS-5821 Container cache management for closing RockDB - #3426

Merged
jojochuang merged 5 commits into
apache:masterfrom
kerneltime:HDDS-5821
Jun 3, 2022
Merged

HDDS-5821 Container cache management for closing RockDB #3426
jojochuang merged 5 commits into
apache:masterfrom
kerneltime:HDDS-5821

Conversation

@kerneltime

@kerneltimekerneltime commented May 18, 2022

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add container cache management logic on top of #3091

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5821

How was this patch tested?

updated unit tests.

@kerneltime

kerneltime commented May 18, 2022

Copy link
Copy Markdown
ContributorAuthor

This can be merged post #3091
cc @szetszwo@jojochuang

@kerneltimekerneltime mentioned this pull request May 18, 2022
Merged
@kerneltimekerneltime changed the title Hdds 5821 Container cache management for closing RockDB HDDS-5821 Container cache management for closing RockDB May 18, 2022
@adoroszlai
adoroszlai marked this pull request as draft May 18, 2022 06:58
@adoroszlai

adoroszlai commented May 18, 2022

Copy link
Copy Markdown
Contributor

@kerneltime please enable the build-branch workflow in your fork to get CI feedback on this.

@kerneltime
kerneltime marked this pull request as ready for review May 18, 2022 17:47
@kerneltimekerneltime mentioned this pull request May 20, 2022
Merged
@adoroszlai
adoroszlai marked this pull request as draft May 20, 2022 06:09
jojochuang
jojochuang reviewed May 23, 2022
View reviewed changes
Comment threadhadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabase.java
public void batchDelete(WriteBatch writeBatch, byte[] key)
throws IOException {
try {
writeBatch.delete(getHandle(), key);

@jojochuangjojochuangMay 23, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WriteBatch needs to be closed if unused. See PR #3400

@kerneltimekerneltimeMay 24, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jojochuang I will be filling additional PRs to address remaining leaks. This PR is to get the RocksDatabase.java abstraction in and address the cache handling for it.

jojochuang
jojochuang reviewed May 23, 2022
View reviewed changes
Comment threadhadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TableConfig.java
Comment threadhadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TableConfig.java
@kerneltime
kerneltime marked this pull request as ready for review May 24, 2022 21:37
jojochuang
jojochuang reviewed May 24, 2022
View reviewed changes
Comment threadhadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabase.java
Comment thread...ainer-service/src/test/java/org/apache/hadoop/ozone/container/common/TestContainerCache.java Outdated
Comment thread...-service/src/main/java/org/apache/hadoop/ozone/container/metadata/AbstractDatanodeStore.java Outdated
Comment thread...-service/src/main/java/org/apache/hadoop/ozone/container/metadata/AbstractDatanodeStore.java
@kerneltime
Address review comments
84acf7b
Change-Id: I112b424fd280bb3ca9d370bef9dc611e2aef67e1
jojochuang
jojochuang approved these changes Jun 2, 2022
View reviewed changes

@jojochuangjojochuang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kerneltime
kerneltimeforce-pushed the HDDS-5821 branch 2 times, most recently from 9c98d54 to d74b298CompareJune 2, 2022 22:26
@kerneltime
Merge branch 'master' into HDDS-5821
2f68f8e
Change-Id: I239872c604da6eb107e61bae75695f942044fc15
@jojochuang

jojochuang commented Jun 3, 2022

Copy link
Copy Markdown
Contributor

The last test failures don't look related. The test run before it passed the same set of tests. I'll merge it now.

@jojochuang
jojochuang merged commit 4a0d01e into apache:masterJun 3, 2022
errose28 added a commit to errose28/ozone that referenced this pull request Jun 7, 2022
@errose28
Merge branch 'master' into HDDS-6760
049d7bc
* master: (87 commits)
HDDS-6686. Do Leadship check before SASL token verification. (apache#3382)
HDDS-4364: [FSO]List FileStatus : startKey can be a non-existed path (apache#3481)
HDDS-6091. Add file checksum to OmKeyInfo (apache#3201)
HDDS-6706. Exposing Volume Information Metrics to the DataNode UI (apache#3478)
HDDS-6759: Add listblock API in MockDatanodeStorage (apache#3452)
HDDS-5821 Container cache management for closing RockDB (apache#3426)
HDDS-6683. Refactor OM server bucket layout configuration usage (apache#3477)
HDDS-6824. Revert changes made in proto.lock by HDDS-6768. (apache#3480)
HDDS-6811. Bucket create message with layout type (apache#3479)
HDDS-6810. Add a optional flag to trigger listStatus as part of listKeys for FSO buckets. (apache#3461)
HDDS-6828. Revert RockDB version pending leak fixes (apache#3475)
HDDS-6764: EC: DN ability to create RECOVERING containers for EC reconstruction. (apache#3458)
HDDS-6795: EC: PipelineStateMap#addPipeline should not have precondition checks post db updates (apache#3453)
HDDS-6823. Intermittent failure in TestOzoneECClient#testExcludeOnDNMixed (apache#3476)
HDDS-6820. Bucket Layout Post-Finalization Validators for ACL Requests. (apache#3472)
HDDS-6819. Add LEGACY to AllowedBucketLayouts in CreateBucketHandler (apache#3473)
HDDS-4859. [FSO]ListKeys: seek all the files/dirs from startKey to keyPrefix (apache#3466)
HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved (apache#3430)
HDDS-6474. Add test to cover the FSO bucket list status with beyond batch boundary and cache. (apache#3379). Contributed by aswinshakil
HDDS-6280. Support Container Balancer HA (apache#3423)
...
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jojochuangjojochuangjojochuang approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kerneltime@adoroszlai@jojochuang@szetszwo