ARROW-15545: [Python][C++] Support casting to extension type

[milesgranger]

Fixes ARROW-15545, fixes ARROW-14500

[github-actions]

https://issues.apache.org/jira/browse/ARROW-15545

[jorisvandenbossche]

I think ideally we handle this on the C++ side, so that it for example also works within a Acero query / dataset scan.

[rok]

This would indeed be useful in c++ too. Reading scalar_cast_numeric.cc and scalar_cast_test.cc might be a good way to get familiar with casting there.

[milesgranger]

Thanks @rok, just saw your comment, I'll take a deeper look at that. Now in b9d8cb6 I loop over id or storage id. This creates the same result as the Python casting in e4db00b which fixes the reported issue (dictionary casting ends up as the ExtensionArray UuidType). However, when casting the test's IntegerType it comes back as Int64Array instead of extension array.. (maybe that's okay?)

[rok]

@milesgranger My guess is you will want to instantiate an ExtensionArray from a 'regular' Array by passing the underlaying buffers, kind of like what CastFromExtension does and then add c++ test like this one.

[milesgranger]

@pitrou (and others already subscribed), this is ready for a look now. Couple things I'm not sure about is if I'm adding the supported types in GetCastToExtension correctly and if I should better the C++ tests to cover more type varieties like the Python tests do.

[pitrou]

Thanks @milesgranger . Some comments below.

[pitrou]

Also cc @lidavidm for potential opinions.

[lidavidm]

Don't think I have much to add to Antoine's comments here.

It may be good to add an entry to "Generic conversions" in the docs: https://github.com/apache/arrow/blob/master/docs/source/cpp/compute.rst#conversions

[lidavidm]

In general that sounds interesting. Especially if that would also allow to change the behaviour of a specific cast (and not just allowing a pre-defined cast to/from the storage type). For example casting to string could give some actual string representation instead of just the string representation of the storage.
But at the same time this also sounds quite complicated for a maybe limited use case?

Cast-to-string is how things like the CSV writer work, right? So maybe the use case isn't all that limited?

[mavam]

Glad to see this being worked on!

I'm currently running in the not-yet-implemented error, per https://issues.apache.org/jira/browse/ARROW-17839, where I have a nested struct with an extension type.

[jorisvandenbossche]

My interpretation was to (for now) disallow any cast unless the input array has exactly a type equal to the target storage type (in which case basically no actual cast is needed, only the creation of the extension array ..)

So something like if (!array->type()->Equals(out_ty)) { ...

(the error message you now wrote would cover that as well)

[milesgranger]

Ah okay.. then it's updated, and also does not cast unless it's needed.

[jorisvandenbossche]

For future reference, Miles and I chatted about this and were thinking to keep the above, and thus only disallow Extension->Extension casts for now in this PR, while still allow Any->Extension casts based on the storage type casting rules (so if Any can be cast to storage type, then it can also be cast to the Extension type).

This would make it consistent with the existing Extension->Any cast, which also just defers to the casting rules of the storage type at the moment (and does not limit to strictly casting Extension only to the exact storage type). And I think this is also what others were OK with (based on the inline discussion in #14106 (comment))

Personally I still think it's a bit inconsistent to allow casting any unrelated type to ExtensionType, but not allow an unrelated extension array to ExtensionType. But we can see to allow registering such casts in the follow-up https://issues.apache.org/jira/browse/ARROW-17890

[pitrou]

Implicit conversions always have to end up inconsistent in practice, so this is not a problem.

For example, in Python you can convert a dict to bool, a bool to float, but not a dict to float.

[jorisvandenbossche]

Yes, but you can use that analogy for the "any -> extension" casts as well (for the case where "any" is not equal to "storage"), that the user needs to do "any -> storage -> extension" explicitly instead .

[pitrou]

Yes, but you can use that analogy for the "any -> extension" casts as well (for the case where "any" is not equal to "storage"), that the user needs to do "any -> storage -> extension" explicitly instead .

Well, I wouldn't mind this :-)

[jorisvandenbossche]

I opened a follow-up JIRA about allowing more casts / ability to register casts -> https://issues.apache.org/jira/browse/ARROW-17890

[milesgranger]

Is there anything else we ought to address?
I don't think the two failing jobs are related.

[jorisvandenbossche]

Thanks for the last updates @milesgranger !

Planning to merge this later today (the failures seem unrelated)

[jorisvandenbossche]

Thanks @milesgranger !

[ursabot]

Benchmark runs are scheduled for baseline = 5fb02bd and contender = 06c99f7. 06c99f7 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️1.11% ⬆️0.0%] test-mac-arm
[Failed ⬇️0.82% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.32% ⬆️0.0%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 06c99f7a ec2-t3-xlarge-us-east-2
[Failed] 06c99f7a test-mac-arm
[Failed] 06c99f7a ursa-i9-9960x
[Finished] 06c99f7a ursa-thinkcentre-m75q
[Finished] 5fb02bdb ec2-t3-xlarge-us-east-2
[Failed] 5fb02bdb test-mac-arm
[Failed] 5fb02bdb ursa-i9-9960x
[Finished] 5fb02bdb ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[ursabot]

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

[jorisvandenbossche]

FWIW the regressions all seems to be flaky runs (they didn't continue to be slower on the subsequent merged commits)