Skip to content

HDDS-4254. Bucket space: add usedBytes and update it when create and delete key. - #1431

Merged
ChenSammi merged 3 commits into
apache:masterfrom
captainzmc:bucket-quota-1
Sep 23, 2020
Merged

HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.#1431
ChenSammi merged 3 commits into
apache:masterfrom
captainzmc:bucket-quota-1

Conversation

@captainzmc

@captainzmccaptainzmc commented Sep 17, 2020

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

In addition, the current Quota setting does not take effect. HDDS-541 gives all the work needed to perfect Quota.
This PR is a subtask of HDDS-541.
First, we increase usedBytes of Bucket. Later, we will judge whether the Bucket can be written based on this when we write the key.
Volume has implemented increase usedBytes when write, and this is based on HDDS-4053.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-4254

How was this patch tested?

added UT

@captainzmc

captainzmc commented Sep 17, 2020

Copy link
Copy Markdown
MemberAuthor

Here are the freon test results:
Using 100 threads to write data to different buckets under the same volume, adding quota Usage has little impact on performance. (LongAdder disperses the concurrency of a single cell to each cell, improving concurrency efficiency compared to atomicLong). In my test, I used three virtual machines, each with a key size of 10K.
image

@captainzmccaptainzmc reopened this Sep 17, 2020
@captainzmc

captainzmc commented Sep 17, 2020

Copy link
Copy Markdown
MemberAuthor

hi @ChenSammi@xiaoyuyao, this PR is base on #1296. Can you help to review this?

ChenSammi
ChenSammi reviewed Sep 22, 2020
View reviewed changes
Comment thread...e-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMAllocateBlockResponse.java Outdated
omMetadataManager.getBucketTable().putWithBatch(batchOperation,
omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
omBucketInfo.getBucketName()),
omBucketInfo);

@ChenSammiChenSammiSep 22, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the last two lines can be merged into one line.

ChenSammi
ChenSammi reviewed Sep 22, 2020
View reviewed changes
Comment thread...ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCommitResponse.java Outdated
omMetadataManager.getBucketTable().putWithBatch(batchOperation,
omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
omBucketInfo.getBucketName()),
omBucketInfo);

@ChenSammiChenSammiSep 22, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above.

ChenSammi
ChenSammi reviewed Sep 22, 2020
View reviewed changes
Comment thread...ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCreateResponse.java Outdated
omMetadataManager.getBucketTable().putWithBatch(batchOperation,
omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
omBucketInfo.getBucketName()),
omBucketInfo);

@ChenSammiChenSammiSep 22, 2020•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same. please double check the rest files.

@captainzmc

captainzmc commented Sep 23, 2020

Copy link
Copy Markdown
MemberAuthor

Thanks @ChenSammi for the review. Review issues has been fixed.

@ChenSammi

ChenSammi commented Sep 23, 2020

Copy link
Copy Markdown
Contributor

LGTM, +1.

@ChenSammi
ChenSammi merged commit a78a4b7 into apache:masterSep 23, 2020
errose28 added a commit to errose28/ozone that referenced this pull request Sep 28, 2020
@errose28
Merge branch 'master' into HDDS-4122-remove-code-consolidation
82e70a6
* master:
HDDS-4102. Normalize Keypath for lookupKey. (apache#1328)
HDDS-4263. ReplicatiomManager shouldn't consider origin node Id for CLOSED containers. (apache#1438)
HDDS-4282. Improve the emptyDir syntax (apache#1450)
HDDS-4194. Create a script to check AWS S3 compatibility (apache#1383)
HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance (apache#1443)
HDDS-2660. Create insight point for datanode container protocol (apache#1272)
HDDS-3297. Enable TestOzoneClientKeyGenerator. (apache#1442)
HDDS-4324. Add important comment to ListVolumes logic (apache#1417)
HDDS-4236. Move "Om*Codec.java" to new project hadoop-ozone/interface-storage (apache#1424)
HDDS-4254. Bucket space: add usedBytes and update it when create and delete key. (apache#1431)
HDDS-2766. security/SecuringDataNodes.md (apache#1175)
HDDS-4206. Attempt pipeline creation more frequently in acceptance tests (apache#1389)
HDDS-4233. Interrupted exeception printed out from DatanodeStateMachine (apache#1416)
HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs (apache#1385)
HDDS-3102. ozone getconf command should use the GenericCli parent class (apache#1410)
HDDS-3981. Add more debug level log to XceiverClientGrpc for debug purpose (apache#1214)
HDDS-4255. Remove unused Ant and Jdiff dependency versions (apache#1433)
HDDS-4247. Fixed log4j usage in some places (apache#1426)
HDDS-4241. Support HADOOP_TOKEN_FILE_LOCATION for Ozone token CLI. (apache#1422)
errose28 added a commit to errose28/ozone that referenced this pull request Sep 28, 2020
@errose28
Merge branch 'HDDS-4122-remove-code-consolidation' into HDDS-4122
3cb1e2c
* HDDS-4122-remove-code-consolidation: (21 commits)
Restore files that had deduplicated code from master
Revert other delete request/response files back to their original states on master
HDDS-4102. Normalize Keypath for lookupKey. (apache#1328)
HDDS-4263. ReplicatiomManager shouldn't consider origin node Id for CLOSED containers. (apache#1438)
HDDS-4282. Improve the emptyDir syntax (apache#1450)
HDDS-4194. Create a script to check AWS S3 compatibility (apache#1383)
HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance (apache#1443)
HDDS-2660. Create insight point for datanode container protocol (apache#1272)
HDDS-3297. Enable TestOzoneClientKeyGenerator. (apache#1442)
HDDS-4324. Add important comment to ListVolumes logic (apache#1417)
HDDS-4236. Move "Om*Codec.java" to new project hadoop-ozone/interface-storage (apache#1424)
HDDS-4254. Bucket space: add usedBytes and update it when create and delete key. (apache#1431)
HDDS-2766. security/SecuringDataNodes.md (apache#1175)
HDDS-4206. Attempt pipeline creation more frequently in acceptance tests (apache#1389)
HDDS-4233. Interrupted exeception printed out from DatanodeStateMachine (apache#1416)
HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs (apache#1385)
HDDS-3102. ozone getconf command should use the GenericCli parent class (apache#1410)
HDDS-3981. Add more debug level log to XceiverClientGrpc for debug purpose (apache#1214)
HDDS-4255. Remove unused Ant and Jdiff dependency versions (apache#1433)
HDDS-4247. Fixed log4j usage in some places (apache#1426)
...
@captainzmccaptainzmc mentioned this pull request Sep 30, 2020
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ChenSammiChenSammiChenSammi left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@captainzmc@ChenSammi