allow setting s3 concurrency, proxy, timeout, uploadPartSize, putSizeLimit, version, and verify_bucket_exists by jessebot · Pull Request #2271 · nextcloud/docker

jessebot · 2024-07-26T12:39:51Z

Adds missing S3 related environment variables. Fixes#2270

Per the docs here's the missing supported S3 parameters:

concurrency defaults to 5 [Note: This defines the maximum number of concurrent multipart uploads]
proxy defaults to false
timeout defaults to 15
uploadPartSize defaults to 524288000
putSizeLimit defaults to 104857600
version defaults to latest
useMultipartCopy defaults to true
copySizeLimit defaults to 5242880000
verify_bucket_exists defaults to true [Note: Setting this to false after confirming the bucket has been created may provide a performance benefit, but may not be possible in multibucket scenarios.]

This also helps us get ready to merge nextcloud/helm#614 downstream 🙏

Let me know if you need anything else :)

jessebot · 2024-10-14T12:14:08Z

@joshtrichards wanted to gently follow up and ask if anything else is needed here?

jessebot · 2024-12-13T09:02:27Z

@J0WI or @joshtrichards can I ask if there's anything else needed here to get this merged into the default branch? Sorry to be a pest. I just want to make sure this gets merged so we can support it in the helm chart properly.

jessebot · 2025-01-05T09:36:14Z

@joshtrichards or @J0WI could you take another look at this when you have a moment?

jessebot · 2025-01-09T13:43:12Z

+- `OBJECTSTORE_S3_CONCURRENCY` defines the maximum number of concurrent multipart uploads
+- `OBJECTSTORE_S3_PROXY` (default: `false`)
+- `OBJECTSTORE_S3_TIMEOUT` (not set by default)
+- `OBJECTSTORE_S3_UPLOADPARTSIZE` (not set by default)
+- `OBJECTSTORE_S3_PUTSIZELIMIT` (not set by default)
+- `OBJECTSTORE_S3_USEMULTIPARTCOPY` (default: `false`)
+- `OBJECTSTORE_S3_COPYSIZELIMIT` (not set by default)
+- `OBJECTSTORE_S3_VERSION` (default: `latest`)
+- `OBJECTSTORE_S3_VERIFY_BUCKET_EXISTS` (default: `true`) Setting this to `false` after confirming the bucket has been created may provide a performance benefit, but may not be possible in multibucket scenarios.


@J0WI how's this? I now only set an int value if it is provided, this way, no one has to find out if empty string breaks stuff. I still set default strings and default booleans though, as we do that everywhere else as well and I want to be consistent. I know that if these new int s3 values are not set, it doesn't break anything, so this should be the best of both worlds right? :)

jessebot · 2025-01-09T13:43:44Z

+ if $concurrency {
+ $CONFIG['objectstore']['arguments']['concurrency'] = $concurrency;
+ }
+
+ if $timeout {
+ $CONFIG['objectstore']['arguments']['timeout'] = $timeout;
+ }
+
+ if $upload_part_size {
+ $CONFIG['objectstore']['arguments']['uploadPartSize'] = $upload_part_size;
+ }
+
+ if $put_size_limit {
+ $CONFIG['objectstore']['arguments']['putSizeLimit'] = $put_size_limit;
+ }
+
+ if $copy_size_limit {
+ $CONFIG['objectstore']['arguments']['copySizeLimit'] = $copy_size_limit;
+ }


these are now all conditionally set by the variables defined at the top of the file that check the env vars.

I like this approach but all tests are failing

hmmm, I don't know why... I am terrible at php. I'll try to take a look this weekend

yaleman · 2025-04-10T23:25:51Z

Any chance of an update on this? It's blocking using the helm chart 😢

…Limit, version, and verify_bucket_exists Signed-off-by: jessebot <jessebot@linux.com>

Signed-off-by: Jesse Hitch <jessebot@linux.com>

Signed-off-by: jessebot <jessebot@linux.com>

J0WI

Something appears to be wrong in the syntax. all tests are failing.

J0WI · 2025-04-18T15:18:16Z

 'use_path_style' => $use_path == true && strtolower($use_path) !== 'false',
 // required for older protocol versions
- 'legacy_auth' => $use_legacyauth == true && strtolower($use_legacyauth) !== 'false'
+ 'useMultipartCopy' => strtolower($useMultipartCopy) !== 'true',


Suggested change
'useMultipartCopy' => strtolower($useMultipartCopy) !== 'true',
'useMultipartCopy' => strtolower($use_multipart_copy) !== 'true',

Or even just strtolower($use_multipart_copy) === 'true',?

J0WI · 2025-04-18T15:20:32Z

 // required for older protocol versions
- 'legacy_auth' => $use_legacyauth == true && strtolower($use_legacyauth) !== 'false'
+ 'useMultipartCopy' => strtolower($useMultipartCopy) !== 'true',
+ 'legacy_auth' => $use_legacyauth == true && strtolower($use_legacyauth) !== 'false',


Suggested change
'legacy_auth' => $use_legacyauth == true && strtolower($use_legacyauth) !== 'false',
'legacy_auth' => strtolower($use_path) === 'true',

J0WI · 2025-04-18T15:22:08Z

 )
 );

+ if $concurrency {


missing brackets around conditions.

SamyDjemai · 2025-11-07T17:56:25Z

Hi @jessebot, any update on this? Not being able to set OBJECTSTORE_S3_VERIFY_BUCKET_EXISTS is currently blocking for me

jessebot commented Jul 29, 2024
View reviewed changes

Comment thread.config/s3.config.php Outdated

This was referenced Jul 29, 2024
Add missing S3 parameters nextcloud/helm#614
Open
New S3 Configuration breaks prior S3 configuration and is missing uploadPartSize nextcloud/helm#612
Open

jessebot force-pushed the add-missing-s3-variables branch from c355acb to 040b741Compare July 29, 2024 10:48

joshtrichards added enhancement feature: auto config (environment variables) Auto configuring via environment variables labels Sep 10, 2024

joshtrichards reviewed Sep 10, 2024
View reviewed changes

Comment thread.config/s3.config.php Outdated

joshtrichards added the 3. to review Waiting for reviews label Sep 10, 2024

joshtrichards added this to the Nextcloud 29.0.8 milestone Sep 10, 2024

joshtrichards requested a review from J0WI September 10, 2024 13:48

jessebot commented Sep 20, 2024
View reviewed changes

Comment threadREADME.md Outdated

jessebot mentioned this pull request Sep 20, 2024
Kubernetes - Redeploying not working when using S3 as default storage backend nextcloud/helm#590
Open

jessebot requested a review from joshtrichards September 24, 2024 09:14

joshtrichards modified the milestones: Nextcloud 29.0.8, Nextcloud 30.0.2 Oct 9, 2024

jessebot mentioned this pull request Oct 20, 2024
Setting nextcloud.objectStore.s3.legacyAuth is not being set as string env var nextcloud/helm#637
Closed

joshtrichards self-assigned this Nov 15, 2024

joshtrichards modified the milestones: Nextcloud 30.0.2, Nextcloud 30.0.3 Nov 24, 2024

jessebot force-pushed the add-missing-s3-variables branch from 10d6edd to 2b545eaCompare December 17, 2024 18:13

jessebot commented Jan 2, 2025
View reviewed changes

Comment threadREADME.md Outdated

jessebot commented Jan 2, 2025
View reviewed changes

Comment threadREADME.md Outdated

jessebot force-pushed the add-missing-s3-variables branch from d7baf2f to 1581991Compare January 2, 2025 16:19

jessebot commented Jan 9, 2025
View reviewed changes

jessebot force-pushed the add-missing-s3-variables branch from c29fea3 to e35ea1eCompare January 9, 2025 13:44

allow setting s3 concurrency, proxy, timeout, uploadPartSize, putSize…
bb0354f
…Limit, version, and verify_bucket_exists Signed-off-by: jessebot <jessebot@linux.com>

jessebot added 4 commits April 18, 2025 14:25

Update .config/s3.config.php - don't set defaults for new s3 values
3aed7a1
Signed-off-by: Jesse Hitch <jessebot@linux.com>

Update README.md
547a09d
Signed-off-by: Jesse Hitch <jessebot@linux.com>

Update README.md
49e139c
Signed-off-by: Jesse Hitch <jessebot@linux.com>

only set variables if they're actively in use
1bfb735
Signed-off-by: jessebot <jessebot@linux.com>

J0WI force-pushed the add-missing-s3-variables branch from e35ea1e to 1bfb735Compare April 18, 2025 14:25

J0WI requested changes Apr 18, 2025
View reviewed changes

J0WI reviewed Apr 18, 2025
View reviewed changes

Comment thread.config/s3.config.php
)
);

if $concurrency {
J0WIApr 18, 2025
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
missing brackets around conditions.

	'useMultipartCopy' => strtolower($useMultipartCopy) !== 'true',
	'useMultipartCopy' => strtolower($use_multipart_copy) !== 'true',

	'legacy_auth' => $use_legacyauth == true && strtolower($use_legacyauth) !== 'false',
	'legacy_auth' => strtolower($use_path) === 'true',

Uh oh!

Conversation

jessebot commented Jul 26, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jessebot commented Oct 14, 2024

Uh oh!

jessebot commented Dec 13, 2024

Uh oh!

Uh oh!

Uh oh!

jessebot commented Jan 5, 2025

Uh oh!

jessebotJan 9, 2025

Choose a reason for hiding this comment

Uh oh!

jessebotJan 9, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

J0WIJan 9, 2025

Choose a reason for hiding this comment

Uh oh!

jessebotJan 10, 2025

Choose a reason for hiding this comment

Uh oh!

yaleman commented Apr 10, 2025

Uh oh!

J0WI left a comment

Choose a reason for hiding this comment

Uh oh!

J0WIApr 18, 2025

Choose a reason for hiding this comment

Uh oh!

J0WIApr 18, 2025

Choose a reason for hiding this comment

Uh oh!

J0WIApr 18, 2025

Choose a reason for hiding this comment

Uh oh!

J0WIApr 18, 2025

Choose a reason for hiding this comment

Uh oh!

SamyDjemai commented Nov 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jessebot commented Jul 26, 2024•
edited
Loading

jessebotJan 9, 2025•
edited
Loading