Return hashes of uploaded content for dav uploads by icewind1991 · Pull Request #19351 · nextcloud/server

icewind1991 · 2020-02-07T13:24:04Z

hashes are set in "X-Hash-MD5", "X-Hash-SHA1" and "X-Hash-SHA256" headers.

these headers are set for file uploads and the MOVE request at the end of a multipart upload.

cc @tobiasKaminsky

tobiasKaminsky · 2020-02-10T08:10:36Z

Nice :-)
I unfortunately cannot answer my questions via looking into code:

will this fail upload, if checksum mismatch? (I do not think so)
how can I get the hash value for existing files?

icewind1991 · 2020-02-13T13:30:45Z

no
this pr doesn't add that

This only returns the hash of the uploaded content so clients can detect corrupted transfers

rullzer · 2020-02-13T13:35:17Z

To give a bit more context ;)

will this fail upload, if checksum mismatch? (I do not think so)

No but you can detect it at least client side. And do something.

how can I get the hash value for existing files?

Adding this is tricky. Just imagine on a big instance and everybody starts to request the hash of several GB files. This could easily kill the server.

Long story short. This is a step in the right direction but not the full fix at all.

tobiasKaminsky · 2020-02-13T14:45:26Z

will this fail upload, if checksum mismatch? (I do not think so)
No but you can detect it at least client side. And do something.

So I would upload the file, then download info of it, compare hash.
If not same, I have to remove it, and show an error to user.

That sounds a bit too complicated… Can't we adjust it so that server will compare hash and if it is not the same refuses the file to be moved and returns an error.

Adding this is tricky. Just imagine on a big instance and everybody starts to request the hash of several GB files. This could easily kill the server.

True. So we can reliable use this for all files, once each upload and change on server computes a hash sum?
(background of this this to have a reliable way that a file to upload does not exist in a folder)

icewind1991 · 2020-02-13T15:09:23Z

Keeping hashes for files stored on the server is a much more complex problem then just returning a hash from uploaded content.
In order to keep hashes for stored files we need to be 100% sure that we update (or invalidate) the hash for every change made to the file on the server

rullzer · 2020-02-13T15:33:33Z

That sounds a bit too complicated… Can't we adjust it so that server will compare hash and if it is not the same refuses the file to be moved and returns an error.

That is a next step. But a lot more complicated and not covered in this PR.

tobiasKaminsky · 2020-02-14T11:46:13Z

Keeping hashes for files stored on the server is a much more complex problem then just returning a hash from uploaded content.
In order to keep hashes for stored files we need to be 100% sure that we update (or invalidate) the hash for every change made to the file on the server

Totally understandable.

That is a next step. But a lot more complicated and not covered in this PR.

Then I will wait until the next step, as we otherwise would implement the error handling logic on client side and then later can remove it again.
But really nice to see improvement in this area :-)

rullzer · 2020-04-04T08:44:54Z

@icewind1991 mind to fix ci so we can get this in as well?

kesselb · 2020-04-12T19:06:19Z

So I would upload the file, then download info of it, compare hash.

@tobiasKaminsky the calculated hash is part of the response of the move or put operation. No need to download the file info.

kesselb

LGTM 👍

Needs a rebase due those code style changes.

hashes are set in "X-Hash-MD5", "X-Hash-SHA1" and "X-Hash-SHA256" headers. these headers are set for file uploads and the MOVE request at the end of a multipart upload. Signed-off-by: Robin Appelman <robin@icewind.nl>

Signed-off-by: Robin Appelman <robin@icewind.nl>

tobiasKaminsky · 2020-04-15T15:52:44Z

@tobiasKaminsky the calculated hash is part of the response of the move or put operation. No need to download the file info.

True, but still this needs to be done on client side.
In my opinion a wrong hash sum must lead to an upload error. But as Roeland said, this will happen hopefully in next step.

tobiasKaminsky · 2020-04-21T14:14:57Z

@tobiasKaminsky the calculated hash is part of the response of the move or put operation.

I only see it as a response header in put operation, but not in final move (of chunked upload)?

tobiasKaminsky · 2020-04-21T15:38:23Z

@tobiasKaminsky the calculated hash is part of the response of the move or put operation.
I only see it as a response header in put operation, but not in final move (of chunked upload)?