perf(s3): Provide direct pre-signed download link

[CarlSchwan]

Summary

This is faster than going back to nextcloud and setting up the file system to download the files.

TODO

• Currently in my dev docker setup, the minio service is not available from outside, so the url can't be accessed. There should be a way to configure whether the s3 service is available or not from outsideMinio + pre-signed urls nextcloud-docker-dev#431
• Also expose the preview url like this (let's move that to a seperate PR)

Checklist

• Code is properly formatted
• Sign-off message is added to all commits
• Tests (unit, integration, api and/or acceptance) are included
• Screenshots before/after for front-end changes
• Documentation (manuals or wiki) has been updated or is not required
• Backports requested where applicable (ex: critical bugfixes)

[invario]

FYI I've been working on #53634 to enable previews to generate by passing a presigned URL to ffmpeg.

[ldpr]

very excited for this! it'll be a gamechanger for nextcloud users no longer being "choked" behind nextcloud proxying everything. can't wait!
thanks so much @CarlSchwan !!

[ser]

I am also super excited, as it would allow to distribute data geographically with several auto mirroring minio instances across continents. Web browsers are automatically getting the file from the shortest round trip servers.

[volf3n]

Oh, this would be amazing for something I've been cooking up on the side!
Any idea how the development is going?

[lakema17]

Is there any way we can get a basic guide to testing this PR? I'm looking to get some basic performance data so I can plan my setup ahead of the release (and also have a bit of fun with it)

[CarlSchwan]

Is there any way we can get a basic guide to testing this PR? I'm looking to get some basic performance data so I can plan my setup ahead of the release (and also have a bit of fun with it)

Best way is to test this PR is with nextcloud/nextcloud-docker-dev#431 just change in .env PRIMARY to minio and start it up with docker compose up minio nextcloud

[artonge]

1. How is this behaving for files without read or download permissions?
2. What happens if the object store is not available publicly?

[CarlSchwan]

• How is this behaving for files without read or download permissions?

I added a check for read. For the download permissions, the downloadUrl dav properties is already completely disabled for shares.

• What happens if the object store is not available publicly?

This is opt-in an behavior. You need to mark 'use_presigned_url' to true in your config.php for the object store. I need to add it to the documentation

[artonge]

This is opt-in an behavior. You need to mark 'use_presigned_url' to true in your config.php for the object store. I need to add it to the documentation

Should we add a setup check to hint to this new possibility?

[artonge]

Looks good as is.

[ser]

• What happens if the object store is not available publicly?

This is opt-in an behavior. You need to mark 'use_presigned_url' to true in your config.php for the object store. I need to add it to the documentation

I would propose to specify the URL instead, as the URL of the front S3 server might be and often is different than the one set internally

[ldpr]

Quick question.

Also expose the preview url like this (let's move that to a seperate PR)

Does this mean the actual download url provided to the user within nextcloud while using this PR is still proxied?
The reason I ask is i have a spare install with s3 so i updated it and disabled extensions, then pulled down and applied the patch but downloads are still proxied. It is offsite s3 if that helps, maybe i'm misinterpreting this PR and this is focused on local s3 instead?
Sorry also, I'm just pretty excited to test it out so i'm probably just far too early to be testing. Thanks :)

[ser]

The reason I ask is i have a spare install with s3 so i updated it and disabled extensions, then pulled down and applied the patch but downloads are still proxied.

@ldpr Have you set use_presigned_url?

[ldpr]

The reason I ask is i have a spare install with s3 so i updated it and disabled extensions, then pulled down and applied the patch but downloads are still proxied.

@ldpr Have you set use_presigned_url?

Yes, I really should have mentioned that, sorry. It's set within the S3 config block.

[nickvergessen]

This is a breaking change for apps that provide a storage wrapper 🙈
Please document accordingly

[joshtrichards]

Indeed. Surprised me a bit while doing some groupfolders changes; thought it was my fault. ;-)

[AndyScherzinger]

^ @CarlSchwan

[joshtrichards]

Correction: the real issue was outdated stubs in gf repo that lacked this new method. All good now (at least in terms of fallback behavior).

[bjo81]

• How is this behaving for files without read or download permissions?

I added a check for read. For the download permissions, the downloadUrl dav properties is already completely disabled for shares.

• What happens if the object store is not available publicly?

This is opt-in an behavior. You need to mark 'use_presigned_url' to true in your config.php for the object store. I need to add it to the documentation

What needs to be done further to use it? I upgraded to 33.0.0beta1 and enabled use_presigned_url within the S3 block.

[CarlSchwan]

• How is this behaving for files without read or download permissions?

I added a check for read. For the download permissions, the downloadUrl dav properties is already completely disabled for shares.

• What happens if the object store is not available publicly?

This is opt-in an behavior. You need to mark 'use_presigned_url' to true in your config.php for the object store. I need to add it to the documentation

What needs to be done further to use it? I upgraded to 33.0.0beta1 and enabled use_presigned_url within the S3 block.

If you do a raw PROPFIND request with e.g. curl and include <oc:downloadURL /> in the requested d:prop, you should see the direct links. This is atm only used by the desktop clients and the video preview provider. Not yet the web client or other clients.

In my dev setup, I do curl -k -X PROPFIND -H "Depth: 1" -u 'admin:admin' https://nextcloud.local/remote.php/dav/files/admin/Media -d @propfind

with a file propfind with the following content:

<?xml version="1.0"?>
<d:propfind xmlns:d="DAV:" xmlns:nc="http://nextcloud.org/ns" xmlns:oc="http://owncloud.org/ns" xmlns:ocs="http://open-collaboration-services.org/ns">
<d:prop>
<d:getcontentlength /> <d:getcontenttype /> <d:getetag /> <d:getlastmodified /> <d:creationdate /> <d:displayname /> <d:quota-available-bytes /> <d:resourcetype /> <nc:has-preview /> <nc:is-encrypted /> <nc:mount-type /> <oc:comments-unread /> <oc:favorite /> <oc:fileid /> <oc:owner-display-name /> <oc:owner-id /> <oc:permissions /> <oc:size /> <nc:hidden /> <nc:is-mount-root /> <nc:metadata-blurhash /> <oc:downloadURL /> <nc:download-url-expiration /> <nc:metadata-files-live-photo /> <nc:note /> <nc:sharees /> <nc:hide-download /> <nc:share-attributes /> <oc:share-types /> <ocs:share-permissions /> <nc:system-tags /> <nc:preview-metadata />
</d:prop>
</d:propfind>
``

[ser]

Hmmm, I have set use_presigned_url to true but using above curl command I do not see that they are included.

I am getting <oc:downloadURL></oc:downloadURL><nc:download-url-expiration></nc:download-url-expiration>

The desktop app is also not using the S3 server directly.

How can I diagnose it further? Nextcloud 33.0.0.6 from NixOS patched with PR498820

[ser]

Are you really sure it is working at all? When I add $this->usePresignedUrl = $this->params['use_presigned_url'] ?? false; to function __construct(array $parameters) in file lib/private/Files/ObjectStore/S3.php the URLs are getting signed.

[ser]

#59249