feat(dav): introduce paginate with custom headers

[Altahrim]

• Ref: 🔢 Implement server-side DAV pagination #45102

Summary

• Implements WebDav pagination by reusing code from A dav plugin to allow pagination of multipart responses #10880
• Need a release of [4.6] Add event to allow inspecting and changing multipart responses sabre-io/dav#1549

Checklist

• Code is properly formatted
• Sign-off message is added to all commits
• Tests (unit, integration, api and/or acceptance) are included
• Screenshots before/after for front-end changes
• Documentation (manuals or wiki) has been updated or is not required
• Backports requested where applicable (ex: critical bugfixes)

[icewind1991]

pushed a minor change to better type return arrays

[icewind1991]

Changed the queries to use DateTimes instead of timestamps for the insert_time to make them work with postgresql

[come-nc]

Should this also be added to ServerFactory?
(in apps/dav/lib/Connector/Sabre)

[Altahrim]

I don't know if there is a clear rule for that :/

[come-nc]

I do not remember the details, I think @skjnldsv knows.
From what I recall one is used for public pages and the other one for authenticated webdav or something like that? Which is why they do not have the exact same list of plugins loaded.
But quite frankly it should all be moved to the server factory, even if it stays 2 separate methods if needed.

[come-nc]

Can you detail the performance and resources implication of this?
Any (paginated) propfind will add all its results to the cache (redis?), how big of an impact is that? That can be a huge list depending on the number of files in the folder, right?

[Altahrim]

Difficult to give a precise impact. It will depends of number of items in a directory, number of paginated requests… We implement this as a test to have first answers about it.
It's stored in cache to have faster results and also limit the impact on Nextcloud (compared to database)

[come-nc]

Difficult to give a precise impact. It will depends of number of items in a directory, number of paginated requests… We implement this as a test to have first answers about it. It's stored in cache to have faster results and also limit the impact on Nextcloud (compared to database)

But there is no trigger to enable/disable it?

[Altahrim]

Only client will use it for now, and not everywhere

[ChristophWurst]

@Altahrim since offsets are typically not scalable, at least for database operations, I'm wondering how the file list pagination behaves as the page number increases? Do you still have to build and discard entries lower than the offset or does the caching avoid rebuilding of the list?
For example, you have a directory with 1M files, and request page 101 with pages of 100 entries. Are the previous 10k entries still fetched?
I couldn't figure it out from just reading the code.

https://use-the-index-luke.com/sql/partial-results/fetch-next-page

[Altahrim]

Once the cache is built, we'll fetch only the corresponding results from cache: https://github.com/nextcloud/server/pull/48662/files#diff-7dd44509fa2becb8b56ada8a254c14836090dd56baba7f4a10abb040cfa85351R62-R65

It's trickier when we build the cache since we always fetch all matching entries to cache them…

[Altahrim]

Documentation: nextcloud/documentation#12977