Perf: Optimize pages loading (Filecache path like approach)

[Koc]

📝 Summary

This is alternative approach to #2390 that fixes same performance issue (closes#2380).

Benefits comparing to previous implementation:

• no extra columns
• no migration to re-process already existent page
• no listeners
• much simpler implementation

So, we're just load all necessary pages via simple query SELECT * FROM filecache WHERE storage_id = <storageId> AND path LIKE 'appdata_<instanceId>/collectives/<collectiveId>/%'

🖼️ Screenshots

Collective with 390 pages with various nesting level

🏚️ Before	🏡 After
6.31s	5.20s
747 queries - depends on pages count and nesting level	50 queries - more or less constant

🚧 TODO

• As a future improvement we can consider to add index to filecache table to storage_id, path columns (but this requires extra PR to nextcloud/server) - see comment Perf: Optimize pages loading (Filecache path like approach) #2549 (comment)

🏁 Checklist

• Code is properly formatted (npm run lint / npm run stylelint / composer run cs:check)
• Sign-off message is added to all commits
• Tests (unit, integration and/or end-to-end) passing and the changes are covered with tests
• Documentation (README or documentation) has been updated or is not required

🤖 AI (if applicable)

• The content of this PR was partly or fully generated using AI tools
• The AI-generated content was reviewed, comprehended and tested by a human

[mejo-]

Thanks a lot @Koc, this looks really promising 🤩

I have some comments, but I'm genuinely curious what you think about the comments.

If you do further changes to the PR, could you do them in separate fixup commits (and don't force-push changes to the existing commit for now) so it's easier to review your changes?

[Koc]

We have green pipeline 🎉

[max-nextcloud]

Proposing a refactoring of the getPagesFromFolder / buildPageInfoTree functions.

I'm pretty sure your approach works as well. I think this refactoring would still clarify things a lot. Would you be up for it or shall I give it a try and open a PR against this one?

[max-nextcloud]

Just tested this locally and performance wise it's really great. Managed to handle a collective with 2k pages well. Became somewhat slow from 3k on.
In comparison - on the same machine 700 pages were getting problematic before. So this seems to provide for approx 4-5 times as many pages.

[Koc]

@max-nextcloud@mejo- WDYT about adding a composite index to filecache table to columns storage_id, path? This can improve performance even more or large instances with millions files

[max-nextcloud]

@Koc there's the fs_storage_path_prefix cache (except for postgres dbs):
https://github.com/nextcloud/server/blob/master/core/Migrations/Version13000Date20170718121200.php#L244-L246

Might be interesting to confirm it's being used though. On my local instance requests for > 1k files were way below 1 second. That seems good enough.

[max-nextcloud]

From my point of view the next step would be to fetch the pages for even larger collectives in multiple requests.
I sketched my ideas about that here: #2581

[Koc]

@max-nextcloud heh we're using Postgres 😕 Why it's excluded?

I will try to work on your other comments next week

[max-nextcloud]

@max-nextcloud heh we're using Postgres 😕 Why it's excluded?

• Originally explained here: add a prefix index to filecache.path, attempt 2 server#28541 (comment)
• See the discussion in Database index fs_storage_path_prefix is not created with Postgresql server#38483

The latter seems to have some discussions on creating the index by hand to avoid the DBAL limitations.

[max-nextcloud]

Thanks a lot for your patience and commitment @Koc
I'll take another look to be sure and hope I can approve the PR then.

[max-nextcloud]

Thanks a lot! 💙

[max-nextcloud]

Tracking behat failure in #2614

Issues have been adressed. Follow up issue created.