[C++] Determine the feasibility and build a prototype to replace compute/kernels with gandiva kernels

[asfimport]

See discussion on #23330

 

Requirements:

1. No hard runtime dependency on LLVM

2. Ability to run without LLVM static/shared libraries.

 

Open questions:

1. What dependencies does this add to the build tool chain?

Reporter: Micah Kornfield / @emkornfield

Related issues:

• [C++] C++ array kernels framework and execution buildout (umbrella issue) (is a child of)
• [C++] Refactor AddKernel to support other operations and types (relates to)

_{Note: This issue was originally created as ARROW-7083. Please see the migration documentation for further details.}

[asfimport]

Yuan Zhou / @zhouyuan:
Hi [~emkornfield@gmail.com]

For the coming AQE what kernels will Arrows use, Is it using 100% C++ kernels? or a combination of C++ and Gandiva kernels?

In the design draft there seems to make a combination of these two kernels.
https://docs.google.com/document/d/10RoUZmiMQRi_J1FcPeVAUAMJ6d_ZuiEbaM2Y33sNPu4/edit#heading=h.2k6k5a4y9b8y

Cheers, -yuan

[asfimport]

Micah Kornfield / @emkornfield:
@zhouyuan that is what this Jira is about. Currently all the kernels are 100% C++ and don't use Gandiva. The question is how feasible is it to reuse Gandiva kernels in a non-JIT environment. It would be nice to not duplicate code but in some contexts JIT isn't an option.

[asfimport]

Wes McKinney / @wesm:
Note that no query engine development has been done so that design document is simply a proposal until actual work happens.

[asfimport]

Wes McKinney / @wesm:
As part of the kernel API revamp, I am looking into how to utilize the code in gandiva/precompiled both for pre-compiled kernels (for use in interpretered expr evaluation) as well as cross-compiling for use in Gandiva. Probably a couple weeks at minimum until I have any patches land about this

[asfimport]

Wes McKinney / @wesm:
I'm inclined to close this issue. After much study, I believe the best that we can do is to to take the single-value kernel implementations found in

https://github.com/apache/arrow/tree/master/cpp/src/gandiva/precompiled

and move them to inline-able header files. Then two things happen:

• These inline functions are translated to LLVM IR for use in Gandiva
• The inline functions form the basis for pre-compiled array kernels in arrow/compute

[asfimport]

Wes McKinney / @wesm:
Note that we should be able to add Gandiva-generated kernels (with some glue) to arrow::compute::Function instances. Perhaps we can create an arrow::compute::GandivaFunction that provides the wrapping magic