A dav plugin to allow pagination of multipart responses

[icewind1991]

• requires dav: Add event to allow inspecting and changing multipart responses 3rdparty#139

Adds the option for clients to get paginated results for PROPFIND, REPORT and SEARCH, the server will still query everything at once, but only sending the first page worth of data to the client while caching the rest.

Proper pagination is needed to be able to properly handle large responses on mobile clients that can otherwise run into memory issues while handling the response.

---

How to use

Initial request

• prepare a PROPFIND, REPORT or SEARCH request as usual to the dav endpoint (the old webdav endpoint isn't supported by this)
• add X-NC-Paginate: true header to tell the server we want pagination
• optionally add a X-NC-Paginate-Count: X header with the number of results to return (defaults to 100)

The result will contain the first X items and the following headers

• X-NC-Paginate: true: to signify that the response is paginated
• X-NC-Paginate-Token: XXXXXXX: the token to request the rest of the results
• X-NC-Paginate-Total: X: the total number of items

Follow up requests

• While sending the same base dav request as above
• add X-NC-Paginate-Token: XXXXXXX header with the returned token
• add X-NC-Paginate-Offset: X the first item to return
• optionally add X-NC-Paginate-Count: X to control the number of items returned.

The result will have the requested items and the X-NC-Paginate: true header.

Note that any follow up request will not reflect any changes that are made on the server since the original request.

---

Support for pagination on the server can be determined from the DAV header of OPTIONS requests, which will contain nc-paginate if the feature is supported.

Cached results are cleaned up after an hour.

[MorrisJobke]

Proper pagination is needed to be able to properly handle large responses on mobile clients that can otherwise run into memory issues while handling the response.

Usually XML can be parsed while reading the stream as well. 🤔

[icewind1991]

The problem is not parsing the xml, the problem is that the mobile client will have to keep the full file listing in memory.

[MorrisJobke]

The problem is not parsing the xml, the problem is that the mobile client will have to keep the full file listing in memory.

True point 👍

[MorrisJobke]

Could you rebase this one?

[MorrisJobke]

Moved to 16.

[mario]

So um, what happened with this one? Can I help to make it happen? It was done so early in the cycle that I really thought it'd get in :(

[MorrisJobke]

Getting it rebased and reviewed tomorrow morning would make it happen. Otherwise I’m sorry, but sometimes things go under the radar and are forgotten somehow.

[mario]

I have now rebased this. Please let me know if I can assist any further @icewind1991.

[marinofaggiana]

Yes yes yesssss, for me this is very very gooooodddddd

[tobiasKaminsky]

@icewind1991 this will not speed up first search?
On my productive system it takes 35s to generate a response (with cURL):

• 27k photos
• out of 30476 files

Is there something I can do from client side to get this faster? I already ask for as few properties as possible.

[MorrisJobke]

@icewind1991 Any news here?

[marinofaggiana]

@icewind1991 Any news here? for mobile its very important

[icewind1991]

@tobiasKaminsky can you give me the xml for the search request you're making

[MorrisJobke]

@icewind1991 What is the status here? We are close to the beta 1. Should this go into 17 or 18?

[marinofaggiana]

😴

[tobiasKaminsky]

Cached results are cleaned up after an hour.

One hour after last usage, or one hour after creating the first query?

[ChristophWurst]

One year without any update? GUess we can close this 🤡

[rullzer]

I'm going to close this due to lack of activity.
Feel free to reopen if anybody wants to continue.

[marinofaggiana]

Can we reopen ? After 5 years the clients still no have a pagination ...