Skip to content

HDDS-5332. Add a new column family and a service provider in Recon DB for Namespace Summaries - #2366

Merged
avijayanhwx merged 15 commits into
apache:masterfrom
yuangu002:HDDS-5332-2
Jul 6, 2021
Merged

HDDS-5332. Add a new column family and a service provider in Recon DB for Namespace Summaries#2366
avijayanhwx merged 15 commits into
apache:masterfrom
yuangu002:HDDS-5332-2

Conversation

@yuangu002

@yuangu002yuangu002 commented Jun 24, 2021

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Table schema:

objectId --> class NSSummary
New class NSSummary should at least include these fields:

  1. Total number of files directly[1] under this directory,
  2. Total size of files directly under this directory,
  3. File Size Buckets of files directly under this directory.

Note: [1] directly means not counting files under any subdirectories, for now. We could further optimize the speed later by trading more writes (propagating the statistics any layers deep upwards) with speed. This is for reducing potential write-amplification for now. The stats could be extended into arbitrary layers deep later.

[06/21] Renamed ReconContainerDBProvider to ReconDBProvider
[06/21] Expanded the scope of this issue by designing a service provider interface for namespace summary, which wraps up all operations on the new schema.

[06/23] Merge with HDDS-5379
Because we are intending to add a new CF to the Recon container DB, the existing ContainerDBServiceProvider must be refactored to resolve conflict with a new service provider sharing the same RDB as the current one directly operates on the DB not just the CF/tables.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5332

How was this patch tested?

Unit test: TestReconNamespaceSummaryManagerImpl

avijayanhwx
avijayanhwx requested changes Jun 24, 2021
View reviewed changes

@avijayanhwxavijayanhwx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @yuangu002, great work on this refactoring. A few comments inline

Comment threadhadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconRocksDB.java Outdated
Comment threadhadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconRocksDB.java Outdated
Comment thread.../src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconContainerMetadataManagerImpl.java Outdated
Comment threadhadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/codec/NSSummaryCodec.java Outdated
Comment threadhadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServer.java Outdated
Comment threadhadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServer.java Outdated
Comment threadhadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServer.java Outdated
Comment thread...zone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/ReconNamespaceSummaryManager.java
Comment thread...n/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconNamespaceSummaryManagerImpl.java Outdated
Comment threadhadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconRocksDB.java Outdated
@smengcl

smengcl commented Jun 24, 2021

Copy link
Copy Markdown
Contributor

pls check the findbugs warning:

M V EI: org.apache.hadoop.ozone.recon.api.types.NSSummary.getFileSizeBucket() may expose internal representation by returning NSSummary.fileSizeBucket At NSSummary.java:[line 57]

smengcl
smengcl reviewed Jun 24, 2021
View reviewed changes
Comment threadhadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconRocksDB.java Outdated
smengcl
smengcl reviewed Jun 24, 2021
View reviewed changes
Comment threadhadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconRocksDB.java Outdated
smengcl
smengcl reviewed Jun 24, 2021
View reviewed changes

@smengclsmengcl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @yuangu002. Overall looks good. Pls address the comments. And I'll take another look after that.

@yuangu002@smengcl
Update hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon…
fb09152
…/ReconConstants.java
Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
smengcl
smengcl reviewed Jun 25, 2021
View reviewed changes
Comment threadhadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconConstants.java
Comment threadhadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconConstants.java Outdated
smengcl
smengcl reviewed Jun 25, 2021
View reviewed changes
Comment threadhadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServer.java Outdated
smengcl
smengcl reviewed Jun 25, 2021
View reviewed changes
Comment threadhadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/codec/NSSummaryCodec.java Outdated
@yuangu002@smengcl
Update hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon…
bb65535
…/ReconConstants.java
Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
yuangu002and others added 3 commits June 25, 2021 15:26
@yuangu002@smengcl
Update hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon…
7fd9bec
…/codec/NSSummaryCodec.java
Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
@yuangu002
codec length change
0cb9d60
@yuangu002
codec length change, minor fix
28985e5
smengcl
smengcl approved these changes Jun 25, 2021
View reviewed changes

@smengclsmengcl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. @avijayanhwx would you take another look?

yuangu002and others added 2 commits June 25, 2021 18:23
@yuangu002@smengcl
Update hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon…
ec1fadb
…/codec/NSSummaryCodec.java
Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
@yuangu002
assert fix
4922315
avijayanhwx
avijayanhwx approved these changes Jul 6, 2021
View reviewed changes

@avijayanhwxavijayanhwx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me. NSSummary classes can be refined in follow up patches.

@avijayanhwx
avijayanhwx merged commit b9a86c3 into apache:masterJul 6, 2021
errose28 added a commit that referenced this pull request Jul 6, 2021
@errose28
Merge remote-tracking branch 'upstream/master' intoHDDS-3698-nonroll…
01e8811
…ing-upgrade
* upstream/master:
HDDS-5332. Add a new column family and a service provider in Recon DB for Namespace Summaries (#2366)
HDDS-5405. Refactor pom files for HadoopRpc and Grpc/Ratis compilation properties. (#2386)
HDDS-5406. add proto version to all the proto files. (#2385)
errose28 added a commit to errose28/ozone that referenced this pull request Jul 7, 2021
@errose28
Merge branch 'master' into HDDS-5337
fda5ca4
* master: (28 commits)
HDDS-5332. Add a new column family and a service provider in Recon DB for Namespace Summaries (apache#2366)
HDDS-5405. Refactor pom files for HadoopRpc and Grpc/Ratis compilation properties. (apache#2386)
HDDS-5406. add proto version to all the proto files. (apache#2385)
HDDS-5398. Avoid object creation in ReplicationManger debug log statements (apache#2379)
HDDS-5396. Fix negligence issue conditional expressions in MockCRLStore.java (apache#2380)
HDDS-5395. Avoid unnecessary numKeyOps.incr() call in OMMetrics (apache#2374)
HDDS-5389. Include ozoneserviceid in fs.defaultFS when configuring o3fs (apache#2370)
HDDS-5383. Eliminate expensive string creation in debug log messages (apache#2372)
HDDS-5380. Get more accurate space info for DedicatedDiskSpaceUsage. (apache#2365)
HDDS-5341. Container report processing is single threaded (apache#2338)
HDDS-5387. ProfileServlet to move the default output location to an ozone specific directory (apache#2368)
HDDS-5289. Update container's deleteTransactionId on creation of the transaction in SCM. (apache#2361)
HDDS-5369. Cleanup unused configuration related to SCM HA (apache#2359)
HDDS-5381. SCM terminated with exit status 1: null. (apache#2362)
HDDS-5353. Avoid unnecessary executeBatch call in insertAudits (apache#2342)
HDDS-5350 : Add allocate block support in MockOmTransport (apache#2341). Contributed by Uma Maheswara Rao G.
HDDS-4926. Support start/stop for container balancer via command line (apache#2278)
HDDS-5269. Datandoe with low ratis log volume space should not be considered for new pipeline allocation. (apache#2344)
HDDS-5367. Update modification time when updating quota/storageType/versioning (apache#2355)
HDDS-5352. java.lang.ClassNotFoundException: org/eclipse/jetty/alpn/ALPN (apache#2347)
...
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@avijayanhwxavijayanhwxavijayanhwx approved these changes
@smengclsmengclsmengcl approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yuangu002@smengcl@avijayanhwx@web-flow