Add support for Ceph RGW Object Store by wido · Pull Request #8389 · apache/cloudstack

wido · 2023-12-20T16:22:56Z

Description

This PR adds support for the Ceph RGW as an Object Store Driver in CloudStack

Types of changes

New feature (non-breaking change which adds functionality)

Screenshots (if appropriate):

How Has This Been Tested?

I have a local Ceph environment (IPv6 only) running with the RADOS Gateway (S3 server) running against which I tested this PR.

It also has SSL enabled so it could verify that this works as expected.

How did you try to break this feature and the system with this change?

I manually tried to create and remove buckets and that worked as expected.

codecov · 2023-12-20T16:31:14Z

Codecov Report

Attention: Patch coverage is 13.02083% with 334 lines in your changes missing coverage. Please review.

Project coverage is 15.64%. Comparing base (72d0546) to head (a448992).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...ge/datastore/driver/CephObjectStoreDriverImpl.java	13.08%	185 Missing and 1 partial ⚠️
...astore/lifecycle/CephObjectStoreLifeCycleImpl.java	0.00%	57 Missing ⚠️
...e/datastore/driver/MinIOObjectStoreDriverImpl.java	11.11%	16 Missing ⚠️
...loudstack/storage/object/BucketApiServiceImpl.java	0.00%	16 Missing ⚠️
...atastore/provider/CephObjectStoreProviderImpl.java	40.00%	15 Missing ⚠️
...src/main/java/com/cloud/agent/api/to/BucketTO.java	22.22%	14 Missing ⚠️
...oudstack/storage/object/store/ObjectStoreImpl.java	0.00%	14 Missing ⚠️
...tastore/driver/SimulatorObjectStoreDriverImpl.java	0.00%	11 Missing ⚠️
...hema/src/main/java/com/cloud/storage/BucketVO.java	66.66%	3 Missing ⚠️
...ain/java/com/cloud/storage/StorageManagerImpl.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #8389 +/- ##
============================================
+ Coverage 15.57% 15.64% +0.06% - Complexity 12056 12118 +62 
============================================
Files 5506 5539 +33 Lines 482997 485358 +2361 Branches 59483 58983 -500 ============================================
+ Hits 75236 75941 +705 - Misses 399450 401043 +1593 - Partials 8311 8374 +63

Flag	Coverage Δ
uitests	`4.12% <ø> (-0.05%)`	⬇️
unittests	`16.44% <13.02%> (+0.08%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-01-09T12:29:15Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

vishesh92

clgtm

DaanHoogland · 2024-01-18T09:02:29Z

@blueorangutan package

blueorangutan · 2024-01-18T09:04:03Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-01-18T09:32:01Z

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 8356

yadvr · 2024-02-05T09:02:40Z

@wido I'll help kick tests again - can you check for build failures?

@blueorangutan package

blueorangutan · 2024-02-05T09:04:03Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-02-05T11:07:31Z

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 8524

weizhouapache · 2024-02-06T12:26:17Z

@blueorangutan package

blueorangutan · 2024-02-06T12:28:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-02-06T13:18:16Z

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 8548

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr · 2024-09-05T07:26:26Z

@blueorangutan package

blueorangutan · 2024-09-05T07:28:04Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-09-05T07:33:29Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 10984

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

DaanHoogland · 2024-09-05T08:36:15Z

@blueorangutan package

blueorangutan · 2024-09-05T08:38:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-09-05T08:41:06Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 10987

blueorangutan · 2024-09-05T09:24:04Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 10989

DaanHoogland · 2024-09-05T09:30:15Z

@wido :

11:03:41 [ERROR] /jenkins/workspace/acs-deb-pkg-builder/plugins/storage/object/ceph/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/CephObjectStoreLifeCycleImpl.java:[30,23] error: package org.apache.log4j does not exist
11:03:41 [ERROR] /jenkins/workspace/acs-deb-pkg-builder/plugins/storage/object/ceph/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/CephObjectStoreLifeCycleImpl.java:[41,25] error: cannot find symbol
11:03:41 symbol: class Logger
11:03:41 location: class CephObjectStoreLifeCycleImpl

strange this error would only happen on the debian build and the el build is successful, but there you have it ???

Packaging result [SF]: ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 10989

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr · 2024-09-05T09:34:21Z

Fixed some issue, tested LGTM but some cases don't work will raise a separate issue for @wido to explore further:

Bucket creation fails when any of the options are specified or these options aren't supported: quota, versioning & encryption, bucket policy
During Bucket creation, locking feature/option is ignored
User quota not enforced/configured
Bucket policy fails and throws exception:

Error while updating bucket. Error while updating bucket: admin-test-bucket. SignatureDoesNotMatch (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx00000c2e2357a96c47d4c-0066d97a79-42778b-default; S3 Extended Request ID: 42778b-default-default; Proxy: null)

UI browser doesn't work (due to cors issue, couldn't figure out how to configure radosgw for that)

yadvr · 2024-09-05T09:34:32Z

@blueorangutan package

blueorangutan · 2024-09-05T09:36:03Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

yadvr

LGTM, tested this plugins with Ceph Reef rgw, doesn't support all options but good to merge this.

yadvr · 2024-09-05T09:39:50Z

Testing notes:

Setup ceph cluster & rgw service:

Next, was able to add this as object storage pool in cloudstack using url & api/secret credentials by creating a system 'admin' user in the rgw:

Following screenshots show, when cloudstack bucket is created, the relevant user is created in rgw & the bucket:

Was able to create bucket without any options or bucket policy, but then was able to use S3 client like Cyberduck to create/configure access permissions on the bucket & upload/download objects/files in it:

Bucket policies don't work via cloudstack UI though and throw exception:

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr · 2024-09-05T11:34:57Z

@blueorangutan package

blueorangutan · 2024-09-05T11:36:04Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr · 2024-09-05T12:26:38Z

Merging this based on manual QA with limitations and followup logged at #9641 (cc @wido), reviews, earlier smoketests & GH build/smoketests.

blueorangutan · 2024-09-05T13:07:28Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11001

This feature adds support for Ceph's RADOS Gateway (RGW) support for the Object Store feature of CloudStack. The RGW of Ceph is Amazon S3 compliant and is therefor an easy and straigforward implementation of basic S3 features. Existing Ceph environments can have the RGW added as an additional feature to a cluster already providing RBD (Block Device) to a CloudStack environment. Introduce the BucketTO to pass to the drivers. This replaces just passing the bucket's name. Some upcoming drivers require more information then just the bucket name to perform their actions, for example they require the access and secret key which belong to the account of this bucket. This is leftover code from a long time ago and this validation test has nu influence on the end result on how a URL will be used afterwards. We should support hosts pointing to an IPv6(-only) address out of the box. For the code it does not matter if it's IPv4 or IPv6. This is the admin's choice. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com> Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>

wido added component:ceph component:object-storage Object Storage labels Dec 20, 2023

wido requested a review from kishankavala December 20, 2023 16:22

boring-cyborgBot added component:api component:build component:storage component:UI labels Dec 20, 2023

DaanHoogland requested a review from vishesh92 December 21, 2023 08:16

github-actionsBot added the status:has-conflicts label Jan 9, 2024

wido force-pushed the ceph-object-store branch from 209332f to 7eb5966Compare January 9, 2024 15:13

github-actionsBot removed the status:has-conflicts label Jan 9, 2024

DaanHoogland added this to the 4.20.0.0 milestone Jan 10, 2024

vishesh92 approved these changes Jan 17, 2024
View reviewed changes

yadvr requested review from harikrishna-patnala and sureshanaparti February 5, 2024 09:01

yadvr closed this Feb 5, 2024

yadvr reopened this Feb 5, 2024

weizhouapache reviewed Feb 5, 2024
View reviewed changes

Comment threadplugins/storage/object/ceph/pom.xml Outdated

wido force-pushed the ceph-object-store branch from 7eb5966 to a0b0471Compare February 6, 2024 10:11

yadvr added 2 commits September 5, 2024 12:53

fix failing unit test
04905c5
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

remove unnecessary stubbing
2f4ac5f
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

get latest bucketTO in case credentials are changed/updated
c86eb5a
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

fix user creation and bucket creation issue on rgw
9d5c436
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr approved these changes Sep 5, 2024
View reviewed changes

yadvr mentioned this pull request Sep 5, 2024
Ceph RADOS-GW object storage plugin issue #9641
Closed

yadvr closed this Sep 5, 2024

yadvr reopened this Sep 5, 2024

apache deleted a comment from blueorangutan Sep 5, 2024

yadvr added 2 commits September 5, 2024 17:03

Merge remote-tracking branch 'origin/main' into ceph-object-store
e5a7405

fix logger build issue
3d58be1
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

fix unit test :(
a448992
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr merged commit c3f0d14 into apache:mainSep 5, 2024

Uh oh!

Conversation

wido commented Dec 20, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecovBot commented Dec 20, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actionsBot commented Jan 9, 2024

Uh oh!

vishesh92 left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Jan 18, 2024

Uh oh!

blueorangutan commented Jan 18, 2024

Uh oh!

blueorangutan commented Jan 18, 2024

Uh oh!

yadvr commented Feb 5, 2024

Uh oh!

blueorangutan commented Feb 5, 2024

Uh oh!

blueorangutan commented Feb 5, 2024

Uh oh!

Uh oh!

weizhouapache commented Feb 6, 2024

Uh oh!

blueorangutan commented Feb 6, 2024

Uh oh!

blueorangutan commented Feb 6, 2024

Uh oh!

yadvr commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

DaanHoogland commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

DaanHoogland commented Sep 5, 2024

Uh oh!

yadvr commented Sep 5, 2024

Uh oh!

yadvr commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

yadvr commented Sep 5, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yadvr commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

yadvr commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

wido commented Dec 20, 2023•
edited
Loading

codecovBot commented Dec 20, 2023•
edited
Loading

yadvr commented Sep 5, 2024•
edited
Loading