ARROW-8917: [C++] Formalize "metafunction" concept. Add Take and Filter metafunctions, port R and Python bindings

[wesm]

A "metafunction" is one that dispatches to other functions based on the argument types. It does not contain any kernels.

Other stuff in this PR:

• Set up to remove all but two versions of arrow::compute::Take. Other ones are still there but will be deprecated or removed after the GLib bindings are ported to use either CallFunction or the Take with Datum-Datum.
• Make "take" and "filter" metafunctions that also deal with RecordBatch, Table arguments
• Delete tons of now unnecessary binding code from Python and R. Hence the significant LOC reduction

There is one failing R test that I wasn't able to debug easily.

Connected JIRAs: ARROW-7009

[wesm]

@nealrichardson I will need your help on the one failing R test

@kou I removed 6 of the 8 Take APIs -- I think it would be better for GLib to use the Datum/Datum or CallFunction API if possible and perhaps net a code reduction like Python and R have achieved here. What do you think?

[github-actions]

https://issues.apache.org/jira/browse/ARROW-8917

[wesm]

I figured out the R issue, I'm trying to fix. It's this hack here

https://github.com/apache/arrow/blob/master/r/src/compute.cpp#L180

[wesm]

Fixed

[kou]

I'll take a look this in a few days.

[wesm]

@kou thanks. I'll try to add deprecated APIs for the functions that were removed so that merging need not be blocked on the refactoring

[wesm]

Done. So GLib can be refactored later.

[kou]

OK.
I'll work on the GLib part as a separated pull request and add ARROW_DEPRECATED in the pull request.

[wesm]

OK.
I'll work on the GLib part as a separated pull request and add ARROW_DEPRECATED in the pull request.

Perfect

[wesm]

The CI is passing except for this s390x issue

src/arrow/CMakeFiles/arrow_objlib.dir/vendored/uriparser/UriShorten.c.o /usr/lib/s390x-linux-gnu/libcrypto.so /usr/lib/s390x-linux-gnu/libssl.so /usr/lib/s390x-linux-gnu/libbrotlienc.so /usr/lib/s390x-linux-gnu/libbrotlidec.so /usr/lib/s390x-linux-gnu/libbrotlicommon.so orc_ep-install/lib/liborc.a protobuf_ep-install/lib/libprotobuf.a /usr/lib/s390x-linux-gnu/libglog.so -ldl jemalloc_ep-prefix/src/jemalloc_ep/dist//lib/libjemalloc_pic.a -lrt /usr/lib/s390x-linux-gnu/libcrypto.so /usr/lib/s390x-linux-gnu/libssl.so /usr/lib/s390x-linux-gnu/libbrotlienc.so /usr/lib/s390x-linux-gnu/libbrotlidec.so /usr/lib/s390x-linux-gnu/libbrotlicommon.so /usr/lib/s390x-linux-gnu/libbz2.so /usr/lib/s390x-linux-gnu/liblz4.so /usr/lib/s390x-linux-gnu/libsnappy.so.1.1.8 /usr/lib/s390x-linux-gnu/libz.so /usr/lib/s390x-linux-gnu/libzstd.so orc_ep-install/lib/liborc.a protobuf_ep-install/lib/libprotobuf.a /usr/lib/s390x-linux-gnu/libglog.so /usr/lib/s390x-linux-gnu/libcrypto.so -pthread && :
/usr/bin/ld: final link failed: No space left on device
collect2: error: ld returned 1 exit status

[jorisvandenbossche]

Looks good!
(some nice code removal in the python bindings indeed ;))

Added some small comments.

One more conceptual aspect I am wondering: compute functions on tables that basically just apply the function on each column (like take, but also if eg support for Table would be added to the sum registered function), wouldn't that start to be part of the (higher level) execution plan?

[wesm]

One more conceptual aspect I am wondering: compute functions on tables that basically just apply the function on each column (like take, but also if eg support for Table would be added to the sum registered function), wouldn't that start to be part of the (higher level) execution plan?

Perhaps, but it's not mutually exclusive. Some applications will just want to do CallFunction(name, args) while others might build an expression pipeline with multiple functions and then request that it be evaluated (once we have this capability, which will be relatively soon)

[wesm]

@kiszk the s390x machine on Travis appears to be in bad shape.

It would be good to get this merged sometime today, let me know if there are other comments on the C++ or R side

[kiszk]

Oh, I saw https://travis-ci.org/github/apache/arrow/builds one hour ago. And, I got good feeling...

According to my experience, this type of disk full error may disappear when we will run the test after 30 min. or 1 hour.

[bkietz]

This seems like a clean refactor, thanks for doing this. Would you comment here or on the JIRA about other metafunctions (aside from take and filter)?

[wesm]

Would you comment here or on the JIRA about other metafunctions (aside from take and filter)?

These are the only ones I can think of aside from cast at the moment

[wesm]

Will merge this once the build passes

[wesm]

@kou this test failure https://github.com/apache/arrow/pull/7318/checks?check_run_id=732734225 seems unrelated to this patch, thoughts? It seems like the test case may be malformed

https://github.com/apache/arrow/blob/master/c_glib/test/test-struct-array.rb#L41

[wesm]

Merging as it doesn't seem like the test failure could be related to this patch (the error showed up after rebasing)

[kou]

We can ignore the failure.
It's a test code problem. It's a GC related problem. We need to keep reference to the raw buffer data.
I'll fix it later.