"Array contains element" translation

[havotto]

According to the documentation array element containment is translated like this:

.Where(c => c.SomeArray.Contains(3))

to

WHERE 3 = ANY("c"."SomeArray")

As I understand this filter cannot use GIN index created on the array column.
So what about changing the translation to:

WHERE "c"."SomeArray" @> ARRAY[3]

or

WHERE "c"."SomeArray" @> '{3}'

There is a difference with NULL element handling though.

[roji]

Can you please point to the documentation about this? Translating array.Contains(scalar) to a PostgreSQL array-contains-array set operation is quite strange, I'm hoping there's a way to keep the array/scalar types without compromising on the index.

Also, it's important for SQL translations to be as close as possible to their C# source, so a difference in behavior (with regards to NULL for example) is likely not acceptable.

[havotto]

Yeah, I agree that this NULL behavior difference is problematic.
Anyway, here are the operator classes that GIN indexes support: GIN built-in operator classes
And these are the array operators: Array operators
And here is a discussion (the second answer is relevant): ANY vs @>

What kind of approach do you think would be good to take advantage of GIN indexes on arrays?

[roji]

I'm not an expert on PostgreSQL indices, but isn't it possible to use an expression index to speed up ANY searches?

[havotto]

I don't this that expression based indexes can make a difference here.
I am thinking about supporting these operators in a different, and more direct way.
Is it possible to translate custom static functions (like EF.Functions.Like) to SQL in EF Core?
If it is, then static such static functions could be defined for these operators (@>, &&, and others).
What do think about this?

[roji]

I don't this that expression based indexes can make a difference here.

Why not? Do you have anything to back this up? The whole point of expression-based indexes is, well, that you can use them for any expression...

Is it possible to translate custom static functions (like EF.Functions.Like) to SQL in EF Core?
If it is, then static such static functions could be defined for these operators (@>, &&, and others).
What do think about this?

Of course it's possible to translate custom static functions to SQL - this is exactly what happens with EF.Functions.Like. But how would that help here? The question isn't how the C# code looks like - is it if (arr.Contains(x)) or is it if (EF.Functions.Foo(arr, x)) - but rather what the generated SQL looks like (does it use ANY, @>...).

[havotto]

Expression-based indexes can only be used if that exact same expression is present in the SQL. For example 'lower(name)'. But here we have value = ANY(array_column). There is no expression here that can be used in an index expression.

As for the static functions, these would be mapped directly to these special PostgreSQL operators, so the user would not confuse Array.Contains(x) with these. @> is one of these operators, it can be used with several PostgreSQL types, so there could be several overloads of this static function:

EF.PgOperator.Contains<T>(T[] container, T[] contained)
EF.PgOperator.Contains<T>(NpgsqlRange<T> container, NpgsqlRange<T> contained)
EF.PgOperator.Contains<T>(NpgsqlRange<T> range, T value)

There are other operators, like overlaps (&&). This could also used in a similar way:

EF.PgOperator.Overlaps<T>(T[] array1, T[] array2) //true if have common value
EF.PgOperator.Overlaps<T>(NpgsqlRange<T> range1, NpgsqlRange<T> range2)

And with the appropriate index these expressions would use that index.

[roji]

OK, thanks for the links and the explanation - it's clear to me why value = ANY(array) is insufficient.

It still seems that ANY should be the translation for .NET array.Contains(value), rather than PostgreSQL @>. Aside from the behavior difference around NULL, value = ANY(array) does work well if value is the one being indexed and array is a constant or parameter, and is a more natural translation for "scalar in array" (whereas @> is does "array in array").

But it does seem clear that some way is needed to generate a @> expression instead, and adding an EF.Functions.Contains<T>(T[] container, T[] contained) seems like the right way to go, and indeed it's a good idea to add all the array operators supported by PostgreSQL. This should actually be quite simple to do, do you intend to submit a PR for this?

Note that the range operators have already been added in version 2.1.0 (see #311). You can look at that implementation as the basis for your own PR for array operations.

[austindrenski]

I have some work in progress on general translation support for array operators in #431, which currently supports this type of translation:

src.Where(x => x.SomeList.All(y => x.SomeArray.Contains(y)));

WHERE x."SomeList" <@ x."SomeArray"

This should also work with an inline array:

src.Where(x => new int[] { 3 }.All(y => x.SomeArray.Contains(y)));

WHERE '{3}'::int[] <@ x."SomeArray"

@havotto Would this form of translation support your use case?

[roji]

@austindrenski thanks, I forgot about your ongoing work, which I'll try to review ASAP :) Do you plan to provide translations for all of the array operations?

[havotto]

@austindrenski Yes, this looks great. But I think that with array input syntax SQL parameters cannot be supported, like this:

src.Where(x => new int[] { myIntValue }.All(y => x.SomeArray.Contains(y)));

There is an alternative syntax for arrays, but I am not sure if it supports SQL parameters for elements:

WHERE ARRAY[3]::int[] <@ x."SomeArray"

[austindrenski]

@roji The big goal is to provide translations for the major ones by 2.2.0. But if there's time, I would like to add support for all of them.

I would recommend reviewing with #444 before #431, as it splits out some of the generalized components that could be used by related features.

@havotto You are correct that we translate arrays with the constructor syntax, rather than the literal syntax:

-- What I wrote:
WHERE '{3}'::int[] <@ x."SomeArray"

-- What I should have written:
WHERE ARRAY[3]::int[] <@ x."SomeArray"

I believe that we handle new int[] { myIntValue, 3, 4 } by generating ARRAY[myIntValue, 3, 4].

But, I'll add a test to confirm.

[roji]

@austindrenski and @havotto, note the specific behavior around NULL which the PostgreSQL array operators have (e.g. @>): SELECT ARRAY[1,NULL] @> ARRAY[1,NULL]; will return false although its C# counterpart will return true. We need to decide whether this discrepancy is enough to not translate C# native constructs (whether it's array.Contains(value) or array1.All(x => array2.Contains(x)) and have a dedicated function under EF.Functions instead.

I don't think there should be an issue with supporting SQL parameters for array elements - we should definitely be able to translate that correctly.