Feat - add custom filter parser by kodumbeats · Pull Request #3 · utopia-php/database

kodumbeats · 2021-04-22T18:06:43Z

New Feature

This PR introduces a new Query class which will hold all related information for custom document filters.

Note - due to $filter being used already, I refer to a custom filter as a $query until we can find something more clear.

Test plan

Tests included with PR

Todo

Implement filter parser with no nesting
Create new Query class
Add tests
Unit test parseExpression()

Should we...

Check types against collection schema
Expand tests to check all query types

kodumbeats · 2021-04-22T18:13:00Z

Testing locally against MariaDB has 🟢

$ docker-compose exec tests vendor/bin/phpunit --debug /usr/src/code/tests/Database/Adapter/MariaDBTest.php
PHPUnit 9.5.4 by Sebastian Bergmann and contributors.
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDelete' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDelete' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDeleteCollection' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDeleteCollection' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDeleteAttribute' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDeleteAttribute' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDeleteIndex' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDeleteIndex' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDocument' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDocument' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testGetDocument' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testGetDocument' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testUpdateDocument' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testUpdateDocument' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testDeleteDocument' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testDeleteDocument' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testFindFirst' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testFindFirst' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testFindLast' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testFindLast' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testParseQueries' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testParseQueries' ended
Time: 00:03.764, Memory: 6.00 MB
OK (11 tests, 101 assertions)

eldadfux · 2021-04-23T05:38:42Z

Thinking about this a bit here are some of my ideas:

We will create a class called Database\Query to hold all query related operations/data
The class constructor will create a query instance with all relevant metadata that will allow us to build a native SQL/NOSQL query (key, operator, value/values), an array of query objects will allow us to build complex queries.
Create a static method called Database\Query::parse([STRING]) that will parse a string query (director.equals('brandon')) and return a Database\Query instance.
The Database\Query class can have multiple getters and setters to make DX nicer.

kodumbeats · 2021-04-23T16:32:00Z

As of e802153

# kodumbeats @ magician in ~/w/kodumbeats/database on git:v0 x [12:28:00]
$ dc exec tests vendor/bin/phpunit tests/Database/Adapter/MariaDBTest.php PHPUnit 9.5.4 by Sebastian Bergmann and contributors.
.......... 10 / 10 (100%)
Time: 00:00.711, Memory: 6.00 MB
OK (10 tests, 85 assertions)
# kodumbeats @ magician in ~/w/kodumbeats/database on git:v0 x [12:28:17] 
$ dc exec tests vendor/bin/phpunit tests/Database/QueryTest.php PHPUnit 9.5.4 by Sebastian Bergmann and contributors.
.. 2 / 2 (100%)
Time: 00:00.002, Memory: 6.00 MB
OK (2 tests, 10 assertions)

eldadfux · 2021-04-23T16:44:26Z

+ return [$operator, $operand];
+ }
+
+ // /**


Love this idea! we should probably have a validator class for queries syntax...

How's this for a start? I figure this is a good place to check if the operator is supported too:
179a0d5

Looks like a good start, we can have it as a standalone PR

This reverts commit f581824.

This reverts commit 1c6e60c.

eldadfux · 2021-04-25T15:32:12Z

We also need to support multiple params for or queries, a few examples:

director.equals('eldad', 'brandon')
director.equals('eldad','brandon')

kodumbeats · 2021-04-26T15:53:56Z

As of 631ecfd

$ docker-compose exec tests vendor/bin/phpunit --debug tests/Database/QueryTest.php
PHPUnit 9.5.4 by Sebastian Bergmann and contributors.
Test 'Utopia\Tests\QueryTest::testParse' started
Test 'Utopia\Tests\QueryTest::testParse' ended
Test 'Utopia\Tests\QueryTest::testGetAttribute' started
Test 'Utopia\Tests\QueryTest::testGetAttribute' ended
Test 'Utopia\Tests\QueryTest::testGetOperator' started
Test 'Utopia\Tests\QueryTest::testGetOperator' ended
Test 'Utopia\Tests\QueryTest::testGetValue' started
Test 'Utopia\Tests\QueryTest::testGetValue' ended
Test 'Utopia\Tests\QueryTest::testGetQuery' started
Test 'Utopia\Tests\QueryTest::testGetQuery' ended
Time: 00:00.002, Memory: 6.00 MB
OK (5 tests, 22 assertions)

eldadfux · 2021-04-26T16:23:28Z

Looks good! I would have added more tests to make sure casting works as expected with all types

eldadfux · 2021-04-26T19:00:34Z

+ return null;
+ } else {
+ // strip quotes from queries of type string
+ return str_replace(['"',"'"], "", $value);


We need to remove only the first and last chars if they are " or '

Maybe we should also handle escaping of \' and \"

Used trim() and stripslashes() for this

#1 Drop $enable flag on skipDuplicates() scope guard The $enable param made every non-skipDuplicates createDocuments call pay for a closure allocation + extra function call. Branch at the call site instead so the cost only applies when the flag is actually set. - Adapter::skipDuplicates(callable, bool) → skipDuplicates(callable) - Database::skipDuplicates(callable, bool) → skipDuplicates(callable) - Database::createDocuments, Mirror::createDocuments, Pool::delegate, Pool::withTransaction now branch inline. #2 Drop fetchExistingByIds helper, inline find() The helper's per-tenant grouping defended a hypothetical multi-tenant batching scenario that no caller exercises (relationships are intra- tenant, callers always batch per tenant). Existing patterns in the same file (refetchDocuments, relationship loading) just call find() directly. Match that idiom and drop ~70 lines. #4 Mirror: only capture inserted docs in skipDuplicates mode The captureOnNext accumulator paid the cost (closure + per-doc array push) on every createDocuments call, including the common non-skip path. Branch at the entry of Mirror::createDocuments so the capture only happens when skipDuplicates is set; the non-skip path passes through to source/destination unchanged. #5 Move getInsertKeyword/Suffix/PermissionsSuffix to getters cluster Were sitting next to createDocuments(); moved to the getSupport* cluster around line 1030 where other adapter-capability shims live. Not addressed: - #2 partial: the existing patterns (refetchDocuments etc.) don't handle tenant-per-document multi-tenant batches either, so this is consistent. - #3 (drop the pre-filter): rejected. createDocumentRelationships runs in the encoding loop BEFORE the adapter's INSERT IGNORE no-ops the parent, so dropping the pre-filter would deterministically duplicate child rows on every CSV re-import of a collection with relationships (not a race window — every call). The relationships test verifies this. Reverting would require reintroducing the deferred-relationships scaffolding we just removed, and the adapter still couldn't tell us which parents were actually inserted (SQL INSERT IGNORE has no per-row reporting). Pre-filter stays.

kodumbeats added 3 commits April 22, 2021 14:00

Add function to parse query expressions
3cf58ce

Add method to parse custom filter queries
514bf70

Add tests for parsing queries
80eec64

kodumbeats marked this pull request as draft April 22, 2021 18:09

Add test for parsing query expressions
69e6bc7

kodumbeats added 9 commits April 23, 2021 11:21

Refactor query into standalone class
cdf5267

Add getters for object properties
3f75885

Add method to get all query details
2a8a5b3

Add return typing
f763ba4

Clean up code
06f02f7

Call parseExpression as a static method
e01fcc4

Add tests for query class
65a2166

Refactor parsing code into query class and tests
1bb848d

Add type hinting
e802153

Merge branch 'v0' into v0
83a09a3

kodumbeats marked this pull request as ready for review April 23, 2021 16:37

eldadfux requested changes Apr 23, 2021
View reviewed changes

kodumbeats added 10 commits April 23, 2021 12:48

Add types to constructor params
c18b485

Rename operand to value
b2f6603

Change to protected method
f581824

Use bracket syntax for switch
fe6becb

Treat query expression as kv pair
1c6e60c

Create query validator
179a0d5

Remove references to old var
8ada10e

Revert "Change to protected method"
8934ce7
This reverts commit f581824.

Revert "Treat query expression as kv pair"
38b02cf
This reverts commit 1c6e60c.

Use value instead of operand
c2b9dc2

kodumbeats added 6 commits April 26, 2021 11:41

Typecast expression values and handle comma-separated values
6b3bcd2

Test with integer param
a1b472d

Test for typecasted bool expressions
e74da85

Test multiple query values
f7e86b6

Cleanup for readability
a14a730

Reserve query validator for separate PR
631ecfd

kodumbeats added 3 commits April 26, 2021 13:53

Only consider periods oudside parentheses
4d6189b

Test for float values
1f4c64f

Handle null as value
fa28286

eldadfux requested changes Apr 26, 2021
View reviewed changes

kodumbeats added 5 commits April 26, 2021 15:21

Refactor elseifs into switch
2ec2fc8

Use single array_map to trim whitespace and typecast
ab2a6b7

Remove unneeded switch case
b01d9dd

Trim whitespace and remove escape slashes
7db5c8f

Test for whitespace and escaped character
40cd42c

TorstenDittmann reviewed Apr 26, 2021
View reviewed changes

Comment threadsrc/Database/Query.php Outdated
Comment threadsrc/Database/Query.php Outdated

kodumbeats added 3 commits April 26, 2021 16:29

Use strrpos to find last occurrence of parenthesis
53cf8dc

Correct comments
ba852c2

Test constructor function
87dd5b3

eldadfux requested changes Apr 26, 2021
View reviewed changes

Comment threadtests/Database/QueryTest.php Outdated

Properly handle whitespace between values
1522284

eldadfux merged commit 60391ec into utopia-php:v0Apr 26, 2021

coderabbitaiBot mentioned this pull request Aug 8, 2025
Implement find method caching #656
Closed

coderabbitaiBot mentioned this pull request Oct 17, 2025
Add vector attribute support for PostgreSQL with pgvector extension #665
Merged

coderabbitaiBot mentioned this pull request Dec 2, 2025
Joins #763
Open

coderabbitaiBot mentioned this pull request Dec 18, 2025
Handle non utf chars #773
Merged

Uh oh!

Conversation

kodumbeats commented Apr 22, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New Feature

Test plan

Todo

Should we...

Uh oh!

kodumbeats commented Apr 22, 2021

Uh oh!

eldadfux commented Apr 23, 2021

Uh oh!

kodumbeats commented Apr 23, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eldadfuxApr 23, 2021

Choose a reason for hiding this comment

Uh oh!

kodumbeatsApr 23, 2021

Choose a reason for hiding this comment

Uh oh!

eldadfuxApr 23, 2021

Choose a reason for hiding this comment

Uh oh!

eldadfux commented Apr 25, 2021

Uh oh!

kodumbeats commented Apr 26, 2021

Uh oh!

eldadfux commented Apr 26, 2021

Uh oh!

Uh oh!

eldadfuxApr 26, 2021

Choose a reason for hiding this comment

Uh oh!

eldadfuxApr 26, 2021

Choose a reason for hiding this comment

Uh oh!

kodumbeatsApr 26, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kodumbeats commented Apr 22, 2021•
edited
Loading