Implement webdav SEARCH by icewind1991 · Pull Request #3360 · nextcloud/server

icewind1991 · 2017-02-02T17:28:36Z

Still WIP, but submitting so the clients can start with it, functionality wise it should be ready for clients to start using the API

requires add icewind/searchdav 3rdparty#34
needs proper test coverage
Adds a mechanism to describe more complex search queries
Extend the cache to support doing these searches
Expose the new search mechanism in \OCP\Files\Folder
Implement rfc5323 SEARCH to expose this to the world
fixesSearch API #676

Clients can do a search by firing a SEARCH request at nextcloud/remote.php/dav with a search body as described by the rfc

For an example of a somewhat complex query that is possible with this api see this example request

cc @mario

mention-bot · 2017-02-02T17:28:39Z

@icewind1991, thanks for your PR! By analyzing the history of the files in this pull request, we identified @MorrisJobke, @butonic and @nickvergessen to be potential reviewers.

icewind1991 · 2017-02-08T16:56:13Z

Fixes #3253

MorrisJobke

Doesn't work:

$ curl -X SEARCH http://192.168.99.100/server/remote.php/dav -u admin:admin <?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
<s:exception>Sabre\DAV\Exception\NotImplemented</s:exception>
<s:message>There was no plugin in the system that was willing to handle this SEARCH method.</s:message>
</d:error>

icewind1991 · 2017-02-15T13:10:03Z

@MorrisJobke you need to send a request body

curl -u test:test -X SEARCH https://localcloud.icewind.me/remote.php/dav -H "content-Type: text/xml" --data '<?xml version="1.0"?>
<d:searchrequest xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns">
<d:basicsearch>
<d:select>
<d:prop>
<oc:fileid/>
<d:getcontenttype/>
<d:getetag/>
<oc:size/>
</d:prop>
</d:select>
<d:from>
<d:scope>
<d:href>/files/test</d:href>
<d:depth>infinity</d:depth>
</d:scope>
</d:from>
<d:where>
<d:like>
<d:prop>
<d:getcontenttype/>
</d:prop>
<d:literal>text/%</d:literal>
</d:like>
</d:where>
<d:orderby/>
</d:basicsearch>
</d:searchrequest>'

MorrisJobke · 2017-02-15T13:16:53Z

I send it but it seems I forgot the text/xml header - let me retry then

LukasReschke · 2017-02-15T13:21:55Z

+ /** @var Folder $folder $results */
+ $results = $folder->search($query);
+
+ var_dump(array_map(function (Node $node) {


expected? 😉

Leaving debug code in production saves me from having to create a debug patch in the future 😄

MorrisJobke · 2017-02-15T16:09:27Z

If I create a folder files/test I only get this as answer:

<?xml version="1.0" encoding="utf-8"?>
<d:errorxmlns:d="DAV:"xmlns:s="http://sabredav.org/ns">
<s:exception>InvalidArgumentException</s:exception>
<s:message>Search is only supported on directories</s:message>
</d:error>

icewind1991 · 2017-02-15T16:18:47Z

@MorrisJobke can you paste the full request you made?

MorrisJobke · 2017-02-15T16:37:38Z

$ curl -X SEARCH https://example.org/remote.php/dav -u admin:admin -i -H "content-Type: text/xml" --data '<?xml version="1.0"?> <d:searchrequest xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns">
<d:basicsearch>
<d:select>
<d:prop>
<oc:fileid/>
<d:getcontenttype/>
<d:getetag/>
<oc:size/>
</d:prop>
</d:select>
<d:from>
<d:scope>
<d:href>/files/test</d:href>
<d:depth>infinity</d:depth>
</d:scope>
</d:from>
<d:where>
<d:like>
<d:prop>
<d:getcontenttype/>
</d:prop>
<d:literal>text/%</d:literal>
</d:like>
</d:where>
<d:orderby/>
</d:basicsearch>
</d:searchrequest>'
HTTP/1.1 404 Not Found
Date: Wed, 15 Feb 2017 16:36:52 GMT
...
Content-Length: 226
Content-Type: application/xml; charset=utf-8
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
<s:exception>Sabre\DAV\Exception\NotFound</s:exception>
<s:message>Principal with name test not found</s:message>
</d:error>

MorrisJobke · 2017-02-15T16:40:58Z

icewind1991 · 2017-02-17T21:59:32Z

@MorrisJobke the scope paths are relative to the dav root (remote.php/dav), so settings the scope to files/test searches in the root folder of the test user

MorrisJobke · 2017-02-20T19:04:06Z

@MorrisJobke the scope paths are relative to the dav root (remote.php/dav), so settings the scope to files/test searches in the root folder of the test user

Tested and works 👍

cc @marinofaggiana I tested it against your test setup and it works 😃

codecov-io · 2017-02-21T18:39:19Z

Codecov Report

Merging #3360 into master will increase coverage by 0.54%.
The diff coverage is 68.96%.

@@ Coverage Diff @@## master #3360 +/- ##
===========================================
+ Coverage 54.15% 54.7% +0.54% - Complexity 21023 21630 +607 
===========================================
Files 1306 1316 +10 Lines 80264 82610 +2346 Branches 1250 1300 +50 ===========================================
+ Hits 43466 45190 +1724 - Misses 36798 37420 +622

Impacted Files	Coverage Δ	Complexity Δ
lib/private/Files/Search/SearchOrder.php	`0% <ø> (ø)`	`3 <3> (?)`
apps/dav/lib/Server.php	`45.31% <ø> (-2.63%)`	`12 <ø> (ø)`
lib/private/Lockdown/Filesystem/NullCache.php	`90.9% <ø> (-3.44%)`	`22 <1> (+1)`
lib/private/Files/Cache/FailedCache.php	`0% <ø> (ø)`	`26 <1> (+1)`	✅
lib/private/Files/Search/SearchComparison.php	`100% <100%> (ø)`	`4 <4> (?)`
lib/private/Files/Cache/Wrapper/CacheJail.php	`85.48% <100%> (+0.11%)`	`55 <1> (+18)`	✅
lib/private/Files/Cache/Wrapper/CacheWrapper.php	`78.08% <100%> (+0.93%)`	`32 <1> (+1)`	✅
lib/private/Files/Node/Folder.php	`88.05% <50%> (-0.83%)`	`49 <ø> (+1)`
lib/private/Files/Search/SearchBinaryOperator.php	`57.14% <57.14%> (ø)`	`3 <3> (?)`
lib/private/Files/Cache/QuerySearchHelper.php	`67.12% <67.12%> (ø)`	`23 <23> (?)`
... and 52 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cd49eb3...7c7abf7. Read the comment docs.

MorrisJobke · 2017-02-22T01:04:23Z

For this body:

<?xml version="1.0"?>
<d:searchrequestxmlns:d="DAV:"xmlns:oc="http://owncloud.org/ns">
<d:basicsearch>
<d:select>
<d:prop>
<d:displayname/>
</d:prop>
</d:select>
<d:from>
<d:scope>
<d:href>/files/admin</d:href>
<d:depth>infinity</d:depth>
</d:scope>
</d:from>
<d:where>
<d:like>
<d:prop>
<d:displayname/>
</d:prop>
<d:literal>Documents</d:literal>
</d:like>
</d:where>
</d:basicsearch>
</d:searchrequest>

I get following response:

<?xml version="1.0" encoding="utf-8"?>
<d:errorxmlns:d="DAV:"xmlns:s="http://sabredav.org/ns">
<s:exception>Sabre\DAV\Exception\ServiceUnavailable</s:exception>
<s:message>TypeError: Argument 4 passed to OC\Files\Search\SearchQuery::__construct() must be of the type array, null given, called in /var/www/nextcloud/apps/dav/lib/Files/FileSearchBackend.php on line 177</s:message>
</d:error>

LukasReschke · 2017-02-22T09:12:03Z

<s:message>TypeError: Argument 4 passed to OC\Files\Search\SearchQuery::__construct() must be of the type array, null given, called in /var/www/nextcloud/apps/dav/lib/Files/FileSearchBackend.php on line 177</s:message>

General note: Can we avoid leaking exception messages via DAV? They can sometimes contain sensitive content :)

icewind1991 · 2017-02-22T17:25:17Z

@MorrisJobke can you retry with the updated 3rdparty

MorrisJobke · 2017-02-24T06:13:40Z

@MorrisJobke can you retry with the updated 3rdparty

I now get:

HTTP/1.1 207 Multi-Status
<?xml version="1.0"?>
<d:multistatusxmlns:d="DAV:"xmlns:s="http://sabredav.org/ns"xmlns:cal="urn:ietf:params:xml:ns:caldav"xmlns:cs="http://calendarserver.org/ns/"xmlns:card="urn:ietf:params:xml:ns:carddav"xmlns:oc="http://owncloud.org/ns"xmlns:nc="http://nextcloud.org/ns"/>

😕

icewind1991 · 2017-02-24T14:13:23Z

@MorrisJobke what is the request you're trying?

MorrisJobke · 2017-02-28T05:25:05Z

@MorrisJobke what is the request you're trying?

#3360 (comment)

mario · 2017-02-28T11:38:58Z

@icewind1991 when doing OPTIONS, search doesn't seem to be among supported methods?

icewind1991 · 2017-02-28T12:18:30Z

Works for me:

curl -I -X OPTIONS -u test:test https://localcloud.icewind.me/remote.php/dav
...
Allow: OPTIONS, GET, HEAD, DELETE, PROPFIND, PUT, PROPPATCH, COPY, MOVE, REPORT, SEARCH

mario · 2017-02-28T12:25:17Z

Ok will check once again.

…

On Tue, 28 Feb 2017 at 13:18, Robin Appelman ***@***.***> wrote: Works for me: curl -I -X OPTIONS -u test:test https://localcloud.icewind.me/remote.php/dav ... Allow: OPTIONS, GET, HEAD, DELETE, PROPFIND, PUT, PROPPATCH, COPY, MOVE, REPORT, SEARCH — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3360 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAWsubNeTyFPZjT7PiHSMFwDzucPUUKks5rhBCYgaJpZM4L1ZwO> .

mario · 2017-02-28T13:01:59Z

@icewind1991 would something like this work with your implementation, since it doesn't look like the current libs supports that draft entirely:

<d:searchrequest xmlns:d="DAV:" dcr:="http://www.day.com/jcr/webdav/1.0" >
dcr:xpath//sv:node[@sv:name='myapp:paragraph'][1]</dcr:xpath>
</d:searchrequest>

cc @AndyScherzinger @tobiasKaminsky

Otherwise we might look into how we can either extend it use or use (possibly?) another lib just for this particular use case if such a thing exists.

Signed-off-by: Robin Appelman <robin@icewind.nl>

marinofaggiana · 2017-03-04T15:43:50Z

Is changed API ??? i have a response error with my driver ...

marinofaggiana · 2017-03-06T14:43:53Z

ping @icewind1991

icewind1991 · 2017-03-06T14:43:56Z

@marinofaggiana please describe the error you're getting instead of having us guess

marinofaggiana · 2017-03-06T14:46:53Z

curl -u admin:XXXXXXXX -X SEARCH https://ios-demo.weasel.rocks/remote.php/dav -H "content-Type: text/xml" --data '<d:searchrequest xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns"><d:basicsearch><d:select><d:prop>oc:fileid/<d:getcontenttype/><d:getetag/>oc:size/oc:favorite/</d:prop></d:select><d:from><d:scope><d:href>/files/admin</d:href><d:depth>infinity</d:depth></d:scope></d:from><d:where><d:like><d:prop><d:displayname/></d:prop><d:literal>Nextcloud%</d:literal></d:like></d:where><d:orderby/></d:basicsearch></d:searchrequest>'

marinofaggiana · 2017-03-06T14:49:44Z

@icewind1991 this is the response :

<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
<s:exception>Sabre\DAV\Exception\NotImplemented</s:exception>
<s:message>There was no plugin in the system that was willing to handle this SEARCH method.</s:message>
</d:error>

marinofaggiana · 2017-03-07T10:44:22Z

@icewind1991 @MorrisJobke please info for this issue.

mario · 2017-03-07T14:32:49Z

So search basically works, so please +1 all of you.

After that, I just need it extended to support oc:favorite in a query because filter-files won't work for Android atm.

marinofaggiana · 2017-03-07T14:47:01Z

@mario
"<oc:filter-files xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns\" xmlns:nc="http://nextcloud.org/ns\">oc:filter-rulesoc:favorite1</oc:favorite></oc:filter-rules><d:prop>"

mario · 2017-03-07T14:50:21Z

@marinofaggiana using that our Android lib doesn't want to parse the propstat so I don't get the properties back.

marinofaggiana · 2017-03-07T14:52:37Z

Modify the lib ?

mario · 2017-03-07T15:09:03Z

Not something I feel comfortable doing under these kind of deadlines.

MorrisJobke · 2017-03-08T00:03:24Z

Sorry - I screwed up the instance. It should work now. :( @marinofaggiana @icewind1991

ghost · 2018-01-15T13:28:14Z

@mario good work. I have been playing around with WebDAV SEARCH on nextcloud 12.0.4 and integrate it in Zimbra Nextcloud Client (https://github.com/Zimbra-Community/owncloud-zimlet)

Mostly I can make it work, but I have issues searching for files with more than 2 search words.
So I would like to search for all files that match: word1, word2 and word3. So far I can only make Nextcloud look for word1 and word2. The 3rd word is ignored. Where can I find good documentation on webdav search?

 curl -u user:password -X SEARCH https://nextcloud.example.com/nextcloud/remote.php/dav -H "content-Type: text/xml" --data '<d:searchrequest xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns"><d:basicsearch><d:select><d:prop>oc:fileid/<d:getcontentlength/><d:getlastmodified/><d:getcontenttype/><d:getetag/>oc:size/oc:favorite/</d:prop></d:select><d:from><d:scope><d:href>/files/user</d:href><d:depth>infinity</d:depth></d:scope></d:from><d:where><d:and><d:like><d:prop><d:displayname/></d:prop><d:literal>%word1%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word2%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word3%</d:literal></d:like></d:and></d:where><d:orderby><d:order><d:prop><d:displayname/></d:prop><d:ascending/></d:order><d:order><d:prop><d:getlastmodified/></d:prop><d:ascending/></d:order></d:orderby></d:basicsearch></d:searchrequest>'
curl -u user:password -X SEARCH https://nextcloud.domain.com/nextcloud/remote.php/dav -H "content-Type: text/xml" --data '<d:searchrequest xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns"><d:basicsearch><d:select><d:prop>oc:fileid/<d:getcontentlength/><d:getlastmodified/><d:getcontenttype/><d:getetag/>oc:size/oc:favorite/</d:prop></d:select><d:from><d:scope><d:href>/files/user</d:href><d:depth>infinity</d:depth></d:scope></d:from><d:where><d:and><d:like><d:prop><d:displayname/></d:prop><d:literal>%word1%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word2%</d:literal></d:like></d:and></d:where><d:orderby><d:order><d:prop><d:displayname/></d:prop><d:ascending/></d:order><d:order><d:prop><d:getlastmodified/></d:prop><d:ascending/></d:order></d:orderby></d:basicsearch></d:searchrequest>'

So here only the first 2 words are matched, the 3rd is ignored... is there a limit of 2 in AND?

 <d:where><d:and><d:like><d:prop><d:displayname/></d:prop><d:literal>%word1%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word2%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word3%</d:literal></d:like></d:and></d:where>

ghost · 2018-01-15T13:37:47Z

https://www.ietf.org/rfc/rfc5323.txt
5.7. Boolean Operators: DAV:and, DAV:or, and DAV:not

The DAV:and operator performs a logical AND operation on the
expressions it contains.

Can anyone help me to do a WebDAV Search to nextcloud with more than 2 words to look for in displayname, I think the XML example, is according to the spec, but maybe I misread.

 <d:where><d:and><d:like><d:prop><d:displayname/></d:prop><d:literal>%word1%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word2%</d:literal></d:like><d:like><d:prop><d:displayname/></d:prop><d:literal>%word3%</d:literal></d:like></d:and></d:where>

icewind1991 · 2018-01-15T13:40:20Z

How I understood the spec (and implemented it) is that DAV:and and DAV:or only take 2 arguments,
you can nest multiple and/or statements to get your desired effect

ghost · 2018-01-15T13:50:57Z

@icewind1991 thanks, that makes it a bit harder to implement on the client side. It would be nice if the rfc said The operator takes two arguments. instead of The DAV:and operator performs a logical AND operation on the expressions it contains. I would not read 2 in that.

ghost · 2018-01-15T14:05:48Z

@icewind1991 I am thinking about asking julian reschke (his name is on the rfc) and see what he says.

But should the server return a 501 Not Implemented error if I do something that is outside the webdav spec? Aka an AND with 3 criteria = 501. Now it parses only the first 2.

reschke · 2018-01-15T14:34:26Z

I agree that RFC 5323 could be clearer, but AFAIR, we would have said "two" if we had intended to restrict it to two operands.

icewind1991 · 2018-01-15T20:04:02Z

I will see if I can extend the implementation to allow for arbitrary number of arguments

ghost · 2018-01-16T08:29:42Z

@icewind1991 thanks!

rullzer · 2018-01-16T11:55:21Z

I just noticed this as well.

Since and is associative you can just do

a AND b AND c AND d

and transform it into

((a AND b) AND c) AND d)

icewind1991 · 2018-01-16T12:25:16Z

Fix is here

rullzer · 2018-02-25T18:39:52Z

@pinkylabs please open a new issue.

icewind1991 mentioned this pull request Feb 2, 2017
add icewind/searchdav nextcloud/3rdparty#34
Merged

MorrisJobke mentioned this pull request Feb 2, 2017
Search API #676
Closed

MorrisJobke requested changes Feb 15, 2017
View reviewed changes

LukasReschke reviewed Feb 15, 2017
View reviewed changes

icewind1991 force-pushed the dav-search branch from a7535fc to 2d4cb4dCompare February 15, 2017 13:28

mario mentioned this pull request Feb 28, 2017
File search nextcloud/android#558
Closed

icewind1991 added this to the Nextcloud 12.0 milestone Feb 28, 2017

add icewind/searchdav
706131b
Signed-off-by: Robin Appelman <robin@icewind.nl>

icewind1991 force-pushed the dav-search branch from caab1f9 to 56f84d6Compare February 28, 2017 13:24

MorrisJobke approved these changes Mar 3, 2017
View reviewed changes

rullzer approved these changes Mar 8, 2017
View reviewed changes

icewind1991 merged commit 2a8e922 into masterMar 8, 2017

icewind1991 deleted the dav-search branch March 8, 2017 12:09

MorrisJobke mentioned this pull request Jun 16, 2017
Document WebDAV search nextcloud/documentation#504
Closed

icewind1991 mentioned this pull request Jan 16, 2018
Support arbitrary number of arguments for d:or and d:and in search queries #7882
Merged

pinkylabs mentioned this pull request Feb 25, 2018
WebDAV SEARCH request requires 'where' to be present, which is not actually required by rfc #8531
Open

Uh oh!

Uh oh!

Conversation

icewind1991 commented Feb 2, 2017• edited by MorrisJobke Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mention-bot commented Feb 2, 2017

Uh oh!

icewind1991 commented Feb 8, 2017

Uh oh!

MorrisJobke left a comment

Choose a reason for hiding this comment

Uh oh!

icewind1991 commented Feb 15, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MorrisJobke commented Feb 15, 2017

Uh oh!

LukasReschkeFeb 15, 2017

Choose a reason for hiding this comment

Uh oh!

icewind1991Feb 15, 2017

Choose a reason for hiding this comment

Uh oh!

MorrisJobke commented Feb 15, 2017

Uh oh!

icewind1991 commented Feb 15, 2017

Uh oh!

MorrisJobke commented Feb 15, 2017

Uh oh!

MorrisJobke commented Feb 15, 2017

Uh oh!

icewind1991 commented Feb 17, 2017

Uh oh!

MorrisJobke commented Feb 20, 2017

Uh oh!

codecov-io commented Feb 21, 2017

Codecov Report

Uh oh!

MorrisJobke commented Feb 22, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LukasReschke commented Feb 22, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

icewind1991 commented Feb 22, 2017

Uh oh!

MorrisJobke commented Feb 24, 2017

Uh oh!

icewind1991 commented Feb 24, 2017

Uh oh!

MorrisJobke commented Feb 28, 2017

Uh oh!

mario commented Feb 28, 2017

Uh oh!

icewind1991 commented Feb 28, 2017

Uh oh!

mario commented Feb 28, 2017 via email

Uh oh!

mario commented Feb 28, 2017

Uh oh!

marinofaggiana commented Mar 4, 2017

Uh oh!

marinofaggiana commented Mar 6, 2017

Uh oh!

icewind1991 commented Mar 6, 2017

Uh oh!

marinofaggiana commented Mar 6, 2017

Uh oh!

marinofaggiana commented Mar 6, 2017

Uh oh!

marinofaggiana commented Mar 7, 2017

Uh oh!

mario commented Mar 7, 2017

Uh oh!

marinofaggiana commented Mar 7, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mario commented Mar 7, 2017

Uh oh!

icewind1991 commented Feb 2, 2017•
edited by MorrisJobke
Loading

icewind1991 commented Feb 15, 2017•
edited
Loading

MorrisJobke commented Feb 22, 2017•
edited
Loading

LukasReschke commented Feb 22, 2017•
edited
Loading

marinofaggiana commented Mar 7, 2017•
edited
Loading

ghost commented Jan 15, 2018•
edited by ghost
Loading