perf(preview): Split preview data to new table by CarlSchwan · Pull Request #54543 · nextcloud/server

CarlSchwan · 2025-08-20T15:38:25Z

Resolves: Split previews out of the filecache table #54513

Summary

The new oc_previews table is optimized for storing previews and should decrease significantly the space taken by previews in the filecache table.

This attend to reuse the IObjectStore abstraction over S3/Swift/Azure but currently only support one single bucket configuration.

TODO

Implement multibucket configuration
Implement multibucket configuration where previews are stored in different bucket than the original files
Add migration for local storage
Add migration for object store storage
Add migration for multibucket object store storage
Make migration efficient
Adapt/Add unit tests
Re-thing BackgroundCleanupJob and other jobs and commands

TODO moved to seperate MR

Sharding: we want to do the sharding on the file_id table. Likely could be done with a tool like citus or vitess. Need some research
Server side encryption: currently we don't use SSE on the previews. If we end up implementing SSE for previews, we should use the same key file as the original files and add a seperate IPreviewStorage implementation using the IEncryptionModule. I already added an encrypted bool to the oc_previews table in preparation as it would be needed anyway for the migration

Checklist

Code is properly formatted
Sign-off message is added to all commits
Tests (unit, integration, api and/or acceptance) are included
~~Screenshots before/after for front-end changes~~ Should not contains any visual changes ;)
Documentation (manuals or wiki) has been updated or is not required
~~Backports requested where applicable (ex: critical bugfixes)~~

Altahrim

Quick thoughts on the new table

CarlSchwan · 2025-09-01T18:36:21Z

@icewind1991 some questions for you :)

With allow configuring multiple object store backends #52786 there is now the possibility of using multiple backends per user. The current mapping is done using user configs (oc_preferences) and previews are still stored in the 'root' object store. Should we add a column to the preview table for the backend used (e.g. 'default', 'new', 'old', 'root', ...) to allow an easier migration of the previews in the future? Or do we keeping using only the 'root' backend for previews?
Multi bucket support. Currently either all the previews are stored on the root bucket or with an opt-in flag can be spread out in 256. I keep the exact same mapping in the new system, does it make sense? Or do we want to use the opportunity of having a breaking change to change that?
Sharding: I added a storage_id field to the column with sharding in mind. I didn't configure yet anything else or test it. Should I had it to SHARD_PRESETS in Connection.php?
Generally how do you feel about the direction this is going?

github-actions · 2025-09-10T08:58:15Z

Possible performance regression detected

Show Output

An unhandled exception has been thrown:
TypeError: array_map(): Argument #2 ($array) must be of type array, null given in /home/runner/work/server/server/apps/profiler/lib/Command/Compare.php:35
Stack trace:
#0 /home/runner/work/server/server/apps/profiler/lib/Command/Compare.php(35): array_map()
#1 /home/runner/work/server/server/3rdparty/symfony/console/Command/Command.php(326): OCAProfilerCommandCompare->execute()
#2 /home/runner/work/server/server/core/Command/Base.php(218): SymfonyComponentConsoleCommandCommand->run()
#3 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(1078): OCCoreCommandBase->run()
#4 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(324): SymfonyComponentConsoleApplication->doRunCommand()
#5 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(175): SymfonyComponentConsoleApplication->doRun()
#6 /home/runner/work/server/server/lib/private/Console/Application.php(187): SymfonyComponentConsoleApplication->run()
#7 /home/runner/work/server/server/console.php(91): OCConsoleApplication->run()
#8 /home/runner/work/server/server/occ(33): require_once('...')
#9 {main}

github-actions · 2025-09-11T13:33:27Z

Possible performance regression detected

Show Output

An unhandled exception has been thrown:
TypeError: array_map(): Argument #2 ($array) must be of type array, null given in /home/runner/work/server/server/apps/profiler/lib/Command/Compare.php:35
Stack trace:
#0 /home/runner/work/server/server/apps/profiler/lib/Command/Compare.php(35): array_map()
#1 /home/runner/work/server/server/3rdparty/symfony/console/Command/Command.php(326): OCAProfilerCommandCompare->execute()
#2 /home/runner/work/server/server/core/Command/Base.php(218): SymfonyComponentConsoleCommandCommand->run()
#3 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(1078): OCCoreCommandBase->run()
#4 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(324): SymfonyComponentConsoleApplication->doRunCommand()
#5 /home/runner/work/server/server/3rdparty/symfony/console/Application.php(175): SymfonyComponentConsoleApplication->doRun()
#6 /home/runner/work/server/server/lib/private/Console/Application.php(187): SymfonyComponentConsoleApplication->run()
#7 /home/runner/work/server/server/console.php(91): OCConsoleApplication->run()
#8 /home/runner/work/server/server/occ(33): require_once('...')
#9 {main}

artonge

~~- Do we have a way to garbage collect orphan previews?~~ Was answered during our call.

Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>

This work similarly to the move preview job to migrate the previews to the new DB table and also reuse some code. So when we are finding files in appdata/preview, try adding them to the oc_previews table and delete them from the oc_filecache table. Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>

Allow to quickly query all the files from a specific mimetype like in the ResetRenderedTexts command. Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>

Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>

And move it to a different table so that we don't have to pay the storage cost when not using it (most of the times). Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>

Simplify logic Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>

szaimen · 2025-10-08T14:39:35Z

🎉🎉🎉🎉🎉

ChristophWurst · 2025-10-08T15:13:46Z

+ if (!$schema->hasTable('preview_versions')) {
+ $table = $schema->createTable('preview_versions');
+ $table->addColumn('id', Types::BIGINT, ['autoincrement' => true, 'notnull' => true, 'length' => 20, 'unsigned' => true]);
+ $table->addColumn('file_id', Types::BIGINT, ['notnull' => true, 'length' => 20, 'unsigned' => true]);


Uneducated guess, but an this smells like it will be the primary column to look up rows. Add an index?

The mapper confirms that an index on file_id+version would be beneficial :)

^ @CarlSchwan

keestux · 2026-04-14T11:06:47Z

+/**
+ *
+ */
+#[CreateTable(table: 'preview', description: 'Holds the preview data')]


preview or previews ?

keestux · 2026-04-14T11:07:31Z

+ *
+ */
+#[CreateTable(table: 'preview', description: 'Holds the preview data')]
+#[CreateTable(table: 'preview_locations', description: 'Holds the preview location in an object store')]


We're missing preview_versions

CarlSchwan added this to the Nextcloud 33 milestone Aug 20, 2025

CarlSchwan self-assigned this Aug 20, 2025

CarlSchwan added the 2. developing Work in progress label Aug 20, 2025

Altahrim reviewed Aug 21, 2025
View reviewed changes

Comment threadcore/Migrations/Version33000Date20250819110529.php Outdated
Comment threadcore/Migrations/Version33000Date20250819110529.php Outdated
Comment threadcore/Migrations/Version33000Date20250819110529.php Outdated
Comment threadcore/Migrations/Version33000Date20250819110529.php Outdated

CarlSchwan force-pushed the preview-db-rework branch 3 times, most recently from 43b105b to 0447f56Compare August 27, 2025 15:14

CarlSchwan force-pushed the preview-db-rework branch 2 times, most recently from 663dfd5 to e55c56dCompare August 28, 2025 17:42

AndyScherzinger added the performance 🚀 label Aug 30, 2025

CarlSchwan force-pushed the preview-db-rework branch from f6ddec7 to 24dc8a1Compare September 1, 2025 08:14

CarlSchwan force-pushed the preview-db-rework branch 5 times, most recently from 9b52d71 to 7d3c7e8Compare September 17, 2025 09:06

CarlSchwan added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Sep 17, 2025

CarlSchwan marked this pull request as ready for review September 17, 2025 09:14

CarlSchwan requested a review from a team as a code owner September 17, 2025 09:14

CarlSchwan requested review from icewind1991, provokateurin and yemkareems and removed request for a teamSeptember 17, 2025 09:14

CarlSchwan force-pushed the preview-db-rework branch from 7d3c7e8 to 296b349Compare September 17, 2025 09:21

artonge reviewed Sep 17, 2025
View reviewed changes

Comment threadcore/BackgroundJobs/MovePreviewJob.php Outdated

CarlSchwan force-pushed the preview-db-rework branch from 296b349 to f90075fCompare September 17, 2025 12:03

CarlSchwan added 6 commits October 6, 2025 13:37

fix(preview): Fix some tests
6f56dcf
Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>

refactor(preview): Cleanup the implementation of the new preview backend
324b54b
Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>

feat(preview): Store original file mimetype in preview table
5802378
Allow to quickly query all the files from a specific mimetype like in the ResetRenderedTexts command. Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>

refactor: Use Override annotation in new preview code
bd001c9
Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>

refactor(preview): Use same mimetype ids as filecache
66f50bd
Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>

CarlSchwan force-pushed the preview-db-rework branch 4 times, most recently from b1c9a14 to 9b63d58Compare October 8, 2025 08:52

artonge approved these changes Oct 8, 2025
View reviewed changes

CarlSchwan force-pushed the preview-db-rework branch from 9b63d58 to 12b77ebCompare October 8, 2025 09:43

leftybournes approved these changes Oct 8, 2025
View reviewed changes

CarlSchwan force-pushed the preview-db-rework branch from 12b77eb to f91ea9eCompare October 8, 2025 11:24

CarlSchwan added 2 commits October 8, 2025 14:07

fix(preview): Make version column a string
bef3996
And move it to a different table so that we don't have to pay the storage cost when not using it (most of the times). Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>

refactor(preview-object-store): Refactor object store backend
fed7a33
Simplify logic Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>

CarlSchwan force-pushed the preview-db-rework branch from f91ea9e to fed7a33Compare October 8, 2025 12:07

icewind1991 approved these changes Oct 8, 2025
View reviewed changes

Altahrim merged commit 3aa0c23 into masterOct 8, 2025
213 of 221 checks passed

Altahrim deleted the preview-db-rework branch October 8, 2025 14:39

ChristophWurst reviewed Oct 8, 2025
View reviewed changes

artonge mentioned this pull request Oct 14, 2025
fix(Viewer): Provide id instead of fileid to File constructor nextcloud/viewer#2988
Merged

artonge mentioned this pull request Oct 29, 2025
Split previews out of the filecache table #54513
Open

nextcloud-bot mentioned this pull request Jan 6, 2026
33.0.0 beta 1 #57361
Merged

joshtrichards mentioned this pull request Feb 27, 2026
[Bug]: Error "An index with name "unique_bucket_store" was already defined" while updateing from 32.0.2 to 33.0.0 #58591
Closed
8 tasks

Whisprin mentioned this pull request Mar 31, 2026
Support Nextcloud 33 ariselseng/camerarawpreviews#124
Closed

keestux reviewed Apr 14, 2026
View reviewed changes

joshtrichards mentioned this pull request Jul 30, 2026
Fix: [Preview] Generator::getPreviewFolder fails to create preview folder in empty preview-structure #52946
Closed
4 tasks

Uh oh!

Uh oh!

Conversation

CarlSchwan commented Aug 20, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

TODO

TODO moved to seperate MR

Checklist

Uh oh!

Altahrim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CarlSchwan commented Sep 1, 2025

Uh oh!

github-actionsBot commented Sep 10, 2025

Uh oh!

github-actionsBot commented Sep 11, 2025

Uh oh!

artonge left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

szaimen commented Oct 8, 2025

Uh oh!

ChristophWurstOct 8, 2025

Choose a reason for hiding this comment

Uh oh!

ChristophWurstOct 8, 2025

Choose a reason for hiding this comment

Uh oh!

AndyScherzingerOct 8, 2025

Choose a reason for hiding this comment

Uh oh!

keestuxApr 14, 2026

Choose a reason for hiding this comment

Uh oh!

keestuxApr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

CarlSchwan commented Aug 20, 2025•
edited
Loading

artonge left a comment •
edited
Loading