Skip to content

Fix folder size contained in S3 buckets - #28185

Merged
juliusknorr merged 1 commit into
masterfrom
investigate/pending_size_for_s3_buckets_master
Aug 20, 2021
Merged

Fix folder size contained in S3 buckets#28185
juliusknorr merged 1 commit into
masterfrom
investigate/pending_size_for_s3_buckets_master

Conversation

@artonge

@artongeartonge commented Jul 27, 2021

Copy link
Copy Markdown
Collaborator

Folders contained in S3 bucket's are displayed with a size of "Pending".

The following command can get the real size to be displayed, but only for one refresh: php occ files:scan --path "<S3 bucket path>"

Rational (I might be wrong):

  1. When running the stat method on the S3 folder, the mtime is always set to 'now'.
  2. This leads to the hasUpdated method from Files/Storage/Common.php to always returns true,
  3. And to the needsUpdate method in Files/View.php to always returns true too.
  4. So the cache gets updated by Files/Cache/Scanner.php every time the folder is acceceed. But it gets updated with a size of -1 because the size for S3 folders is not getting computed and default to -1

Need guidance on how and where we should address this issue :).

@artonge
artongeforce-pushed the investigate/pending_size_for_s3_buckets_master branch 4 times, most recently from 1c04f22 to e05aa2cCompareJuly 27, 2021 09:24
Comment threadlib/private/Files/Cache/Scanner.php Outdated
Comment threadapps/files_external/lib/Lib/Storage/AmazonS3.php Outdated
Comment threadapps/files_external/lib/Lib/Storage/AmazonS3.php
@CarlSchwan

Copy link
Copy Markdown
Member

I investigated a similar issue with using Group Folders with the S3 storage with Vincent last week:

  • OC\File\Cache\Updater: the update function is never called for the folder itself, only for the preview. It's correctly called for folders outside the group folder
  • A S3ObjectStorage uses a NoopScanner, but the GroupFolderStorage uses a normal Scanner. This will probably cause troubles when the Updater::update method is called. Maybe it's the reason why Update is not called but I doubt it.

@PVince81

Copy link
Copy Markdown
Member

ref: nextcloud/groupfolders#1630

but probably not related because the latter is for group folders and the problem with S3 primary happens also without

@artonge
artongeforce-pushed the investigate/pending_size_for_s3_buckets_master branch 2 times, most recently from aea7a2e to bdd1a35CompareAugust 19, 2021 10:28
@artongeartonge changed the title Investigate pending size for s3 buckets Fix folder size contained in S3 bucketsAug 19, 2021
@artongeartonge added this to the Nextcloud 23 milestone Aug 19, 2021
@artonge

Copy link
Copy Markdown
CollaboratorAuthor

/backport to stable22

@artonge

Copy link
Copy Markdown
CollaboratorAuthor

/backport to stable21

If 'filesystem_check_changes' was set to never, the cached size was alway set to -1 (Pending) on every access
Signed-off-by: Louis Chemineau <louis@chmn.me>
@artonge
artongeforce-pushed the investigate/pending_size_for_s3_buckets_master branch from bdd1a35 to 735fd94CompareAugust 19, 2021 13:03

@PVince81PVince81 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, since the solution covers all the interesting cases let's move forward 👍

@juliusknorr
juliusknorr merged commit 041fbda into masterAug 20, 2021
@juliusknorr
juliusknorr deleted the investigate/pending_size_for_s3_buckets_master branch August 20, 2021 13:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@artonge@CarlSchwan@PVince81@juliusknorr