ARROW-9160: [C++] Implement contains for exact matches

[xhochy]

No description provided.

[xhochy]

This currently fails for chunked arrays. I thought that they should be handled by the kernel framework automatically but it seems, they aren't.

[----------] 1 test from TestStringKernels/1, where TypeParam = arrow::LargeStringType
[ RUN ] TestStringKernels/1.ContainsExact
../src/arrow/testing/gtest_util.cc:149: Failure
Failed
Got:
[
[
false,
null,
false
]
]
Expected:
[
[
true,
null
],
[
false
]
]
../src/arrow/testing/gtest_util.cc:149: Failure
Failed
Got:
[
[
false,
false,
false,
null,
false
]
]
Expected:
[
[
true,
false
],
[
true,
null,
false
]
]
[ FAILED ] TestStringKernels/1.ContainsExact, where TypeParam = arrow::LargeStringType (2 ms)

@wesm@pitrou Any pointers what I'm missing?

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9160

[wesm]

@xhochy chunked arrays should be handled automatically by the function executors. I will take a look.

[xhochy]

Implemented Knuth-Morris-Pratt, got roughly a 2x speedup in the benchmark.

Before:
ContainsExact 53673203 ns 53323692 ns 13 bytes_per_second=297.061M/s items_per_second=19.6644M/s
After:
ContainsExact 30150889 ns 29982087 ns 23 bytes_per_second=528.329M/s items_per_second=34.9734M/s

[wesm]

@xhochy I'm fixing a couple issues with the implementation:

• The function executor allocates memory for you unless you explicitly disable it. The idea is that you don't want to do allocations in-kernel unless it is necessary. The preallocation settings affect how execution with chunked arrays work
• The slice offset of the Datum* out parameter must be respected (it may be non-zero -- the executor does not preserve the chunk layout of ChunkedArrays and will merge chunks to make larger output arrays -- this can be configured in ExecContext)

[wesm]

I just opened https://issues.apache.org/jira/browse/ARROW-9285 -- it should be easy to check if a kernel has mistakenly replaced a preallocated data buffer (which may be a slice of a larger output that is being populated)

[wesm]

I also propose to rename the function from "contains_exact" to "utf8_contains". I'm pushing that change here shortly. Or should we call it "binary_contains" / "string_contains" since it will work with any base binary types? I'll hold off on making any changes until there is consensus about the name. I think it would be useful to put string functions in a "pseudo-namespace" so they appear next to each other when sorted alphabetically.

Ideas:

• string_contains
• string_lower_utf8
• string_lower_ascii
• ...

@maartenbreddels@pitrou any opinions about this?

[xhochy]

I think prefixes make sense. We will have sometime similar kernel names that act quite different depending on the types they work on. I would differentiate in the string/binary case the kernels between the categories:

• binary_: works on any binary data
• ascii_: works on string data but is limited ASCII-only characters
• utf8_: works on string data and respects UTF8-encoding or even makes use of an UTF8-symbol table (i.e. utf8proc).

With that, I would rename the kernel here to binary_contains_exact. The exact suffix is important for me as I intent to implement also ut8_contains_case_insensitve, binary_contains_regex (or should this be string/utf8?), utf8_contains_regex_case_insensitve.

[pitrou]

You match a regex, you don't contain it.

[xhochy]

You match a regex, you don't contain it.

That is one of the name clashes, we already have a match kernel.

[pitrou]

match_regex then? :-)

[maartenbreddels]

I like the prefixing by string. I'm a big fan of ordering 'words' in snake or camel casing for good tab completion and alphabetic ordering, so I agree with @wesm 's proposal. As a user, I first think of the type I work with ('string'), next, what I want to do with it (upper/lower casing). Tab-completion/ordering would then reveal 2 variants, very intuitive.
To make this concrete:

• string_lower_utf8
• string_lower_ascii
• binary_contains_exact
• string_contains_regex_ascii
• string_contains_regex_utf8

(or s/contains/match 😄 )

There might be kernels that work on binary data, but do not work well with utf8, e.g. a binary_slice, which would slice each binary-string on a byte basis. It could cut a multibyte encoded codepoint to result in an invalid utf8 string. I'd say that's acceptable, we could check that depending on the type we get it.

Regarding case insensitive variants, I think we should expose functions to do normalization (eg replace the single codepoint ë by the letter e and the diaeresis combining character(the dots above the ë)), case folding, and removal of combining characters. That allows users to remove e.g. the diaeresis from ë to do pattern matching without diacritics.

[maartenbreddels]

While at the difficult topic of naming, is there a conversion (agreed or emerging) for naming the functors/ArrayKernelExec implementations? I see in scalar_string.cc we use Transform, which is probably redundant (as I understand scalar kernels basically map/transform elementwise). But I think it would be useful to include the type name, e.g.

• Utf8ToUtf8 (which is now Utf8Transform) (hence Utf8Upper : Utf8ToUtf8<Type, Utf8Upper<Type>>)
• Utf8ToBool / Utf8Predicate (which is now StringBoolTransform)
• Utf8ToScalar
• BinaryToBool

[maartenbreddels]

• The function executor allocates memory for you unless you explicitly disable it. The idea is that you don't want to do allocations in-kernel unless it is necessary. The preallocation settings affect how execution with chunked arrays work

Could you elaborate? Why is this not a problem with the lower/upper kernels?

[pitrou]

This is a nice addition. A couple comments.

[pitrou]

I don't think defaulting to the empty string makes sense here.

[xhochy]

This is needed to be able to construct an instance of it in Cython. I can remove this and switch the Cython instantiation to a pointer.

[pitrou]

That's rather vague. Does contains_exact(pa.array([1,2,3]), pa.array([1,2])) return true? I don't think so.

[wesm]

Could you elaborate? Why is this not a problem with the lower/upper kernels?

The data preallocation is only for fixed size outputs (eg boolean, integers, floating point, etc).

[kiszk]

We can move this line into then clause before line 327.

[wesm]

I think prefixes make sense. We will have sometime similar kernel names that act quite different depending on the types they work on. I would differentiate in the string/binary case the kernels between the categories:

• binary_: works on any binary data
• ascii_: works on string data but is limited ASCII-only characters
• utf8_: works on string data and respects UTF8-encoding or even makes use of an UTF8-symbol table (i.e. utf8proc).

With that, I would rename the kernel here to binary_contains_exact. The exact suffix is important for me as I intent to implement also ut8_contains_case_insensitve, binary_contains_regex (or should this be string/utf8?), utf8_contains_regex_case_insensitve.

This seems reasonable to me

[pitrou]

"binary_contains_exact" still works for utf8 data, though.

[wesm]

OK, do you have an idea for a prefix for a function that works either on BinaryArray or StringArray? This could also be handled with aliases so we could have utf8_contains_exact as an alias for the same function

[maartenbreddels]

String is a subclass of binary right? So I expect binary_contains_exact to work on strings right?

[pitrou]

Right, I was just mentioned that "binary" isn't exclusive of "utf8" in this particular instance.
However, I'm not sure why we need prefixes. "contains" seems sufficient to me :-)

[wesm]

Other types (e.g. list) will need to have a "contains" operation, so I think it's more clear to have binary_contains and list_contains than simply contains.

[xhochy]

Adressed all comments and CI is happy, too.

[wesm]

+1. I'll fix the benchmark function rename and then merge this

[wesm]

Looks like one rename missed here