Skip to content

feat: remove orphaned entries from filecache_extended - #38933

Merged
icewind1991 merged 2 commits into
masterfrom
orphaned-entries-filecache-extended
Apr 15, 2024
Merged

feat: remove orphaned entries from filecache_extended#38933
icewind1991 merged 2 commits into
masterfrom
orphaned-entries-filecache-extended

Conversation

@kesselb

@kesselbkesselb commented Jun 21, 2023

Copy link
Copy Markdown
Contributor
  • Resolves: #

Summary

Test scenario:

  • Run sql: select * from oc_filecache_extended left join oc_filecache on oc_filecache_extended.fileid = oc_filecache.fileid where oc_filecache.fileid is null;
  • Mount an external storage (e.g. local)
  • Upload some files to the external storage
  • Delete external storage
  • Run sql: select * from oc_filecache_extended left join oc_filecache on oc_filecache_extended.fileid = oc_filecache.fileid where oc_filecache.fileid is null;
  • See files without a matching record in filecache

publicstaticfunctioncleanByMountId(int$mountId) {

Best would be to clear filecache_extended when deleting the storage.

TODO

  • OCI
  • CI

Checklist

@kesselbkesselb added 2. developing Work in progress technical debt 🧱 🤔🚀 labels Jun 21, 2023
@kesselbkesselb added this to the Nextcloud 28 milestone Jun 21, 2023
@kesselbkesselb self-assigned this Jun 21, 2023
Comment threadapps/files/lib/Command/DeleteOrphanedFiles.php Fixed
@kesselb
kesselbforce-pushed the orphaned-entries-filecache-extended branch from 4c3a6c7 to c84aef5CompareJune 21, 2023 16:55
@kesselb
kesselb requested review from a team, ArtificialOwl, icewind1991, juliusknorr and nfebe and removed request for a teamJune 21, 2023 16:55
@kesselbkesselb added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Jun 21, 2023
->setName('files:cleanup')
->setDescription('cleanup filecache');
->setDescription('cleanup filecache')
->addOption('filecache-extended', null, InputOption::VALUE_NONE, 'remove orphaned entries from filecache_extended');

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to not make this run by default?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically, no. The execution for the left join seems quick, but I only tested with ~1000 files.

My motivation, to hide it behind a flag, is my incomplete knowledge about filecache and especially external storages. However, I can't think of a case, why we would have a record in filecache_extended without a matching one in filecache.

We can run it by default if you think it should ;)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched to running it by default and having the flag disable it in case we ever find cases where that is required

@juliusknorrjuliusknorr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good, small question inside :)

@skjnldsvskjnldsv mentioned this pull request Nov 1, 2023
This was referenced Nov 6, 2023
@kesselb

Copy link
Copy Markdown
ContributorAuthor

Moving to 29

@icewind1991
icewind1991force-pushed the orphaned-entries-filecache-extended branch from c84aef5 to da57476CompareFebruary 9, 2024 13:50
@kesselb
kesselbforce-pushed the orphaned-entries-filecache-extended branch from da57476 to 9fb2258CompareFebruary 17, 2024 16:14
@kesselb

Copy link
Copy Markdown
ContributorAuthor

Howdy 👋

Thank you for taking care, I almost forgot about it 🙈

I don't remember if I tested it with OCI back then.
The chunking (for the OCI in clauses limit) is there.

@skjnldsv
skjnldsvforce-pushed the orphaned-entries-filecache-extended branch from 9fb2258 to 3b73c89CompareFebruary 24, 2024 16:11
@skjnldsvskjnldsv added 4. to release Ready to be released and/or waiting for tests to finish and removed 3. to review Waiting for reviews labels Feb 24, 2024
@skjnldsv

Copy link
Copy Markdown
Member
1) OCA\Files\Tests\Command\DeleteOrphanedFilesTest::testClearFiles
Expectation failed for method name is "writeln" when invoked 2 time(s)
Parameter 0 for invocation #1 Symfony\Component\Console\Output\OutputInterface::writeln('0 orphaned file cache extende...eleted', 0) does not match expected value.
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'1 orphaned mount entries deleted'
+'0 orphaned file cache extended entries deleted'

@skjnldsvskjnldsv added 2. developing Work in progress and removed 4. to release Ready to be released and/or waiting for tests to finish labels Feb 24, 2024
This was referenced Mar 12, 2024
This was referenced Mar 20, 2024
@kesselb
kesselbforce-pushed the orphaned-entries-filecache-extended branch 2 times, most recently from 63353ba to 0635c24CompareMarch 26, 2024 10:46
@kesselbkesselb added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Mar 26, 2024
@skjnldsvskjnldsv mentioned this pull request Mar 28, 2024
81 tasks
@kesselb

Copy link
Copy Markdown
ContributorAuthor

Moving to 30

@kesselb
kesselbforce-pushed the orphaned-entries-filecache-extended branch from 0635c24 to 3e0299fCompareApril 8, 2024 19:43
kesselband others added 2 commits April 15, 2024 20:30
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
Signed-off-by: Robin Appelman <robin@icewind.nl>
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
@kesselb
kesselbforce-pushed the orphaned-entries-filecache-extended branch from 3e0299f to 1d34f0aCompareApril 15, 2024 18:30
@icewind1991
icewind1991 merged commit b5e3508 into masterApr 15, 2024
@icewind1991
icewind1991 deleted the orphaned-entries-filecache-extended branch April 15, 2024 19:07
@blizzzblizzz mentioned this pull request Jul 24, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3. to reviewWaiting for reviewstechnical debt🧱 🤔🚀

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@kesselb@skjnldsv@icewind1991@juliusknorr@github-advanced-security