ARROW-17980: [C++] As-of-Join Substrait extension

[rtpsw]

Replacing #14385

[github-actions]

https://issues.apache.org/jira/browse/ARROW-17980

[westonpace]

I'm a little torn on this PR. It seems that it includes #14415 which hasn't merged yet, although that is primarily because it blocked waiting for substrait-io/substrait#342 . At this point I think @bkietz remaining comments were fairly minor and I had addressed them. Also, since things are a bit squashed, it is unclear to me how much of #14415 this includes. Do you know if you were able to squash in all the commits from #14415 ?

Also, this effectively means we are temporarily based on a Substrait PR while we wait for Substrait to catch up.

I'm not a big fan of the fact that we need the using statements for the substrait namespace but if we really want that package name then it seems protobuf gives us little choice.

That being said, this PR works, I don't see anything functionally wrong with it, it is moving in the right direction, and it will make adopting the official Substrait release a smaller task. We should probably proceed by merging in #14415 properly, instead of forcing it through integrated as part of this PR though.

[rtpsw]

I'm a little torn on this PR. It seems that it includes #14415 which hasn't merged yet, although that is primarily because it blocked waiting for substrait-io/substrait#342 . At this point I think @bkietz remaining comments were fairly minor and I had addressed them. Also, since things are a bit squashed, it is unclear to me how much of #14415 this includes. Do you know if you were able to squash in all the commits from #14415 ?

Unfortunately, I can't say whether this PR includes all of #14415. To avoid confusion, I'm fine with marking this PR a draft. I'll hold to see how you and @bkietz suggest we should proceed.

[anjakefala]

I left one comment about whether we want to require protobuf headers in order to build libarrow_python with substrait.

Additionally, find_package(Protobuf) seems to be missing from the CMakeLists

I would add the following here https://github.com/rtpsw/arrow/blob/fce4dff7df8577731cccb220457ade5e24a33e60/python/CMakeLists.txt#L232

if(ARROW_PROTOBUF_USE_SHARED)
find_package(Protobuf)
include_directories(${PROTOBUF_INCLUDE_DIRS})
get_property(dirs DIRECTORY PROPERTY INCLUDE_DIRECTORIES)
foreach(dir ${dirs})
message("dir='${dir}'")
endforeach()
get_cmake_property(_variableNames VARIABLES)
foreach (_variableName ${_variableNames})
message("${_variableName}=${${_variableName}}")
endforeach()
endif()

But, there might be a better place for it.

[westonpace]

I think before merging we should address if this is behaviour we want to keep.

@anjakefala It is not. I would like to hide protobuf from python (I suspect an options_internal.h is doable somehow).

[westonpace]

Now that #14415 has merged we can move forwards on this. However, the code here is out of date (it was based on an older revision of #14415) and I worry it might not merge so cleanly. Let me know if you have trouble and I can take a stab at it.

[rtpsw]

Now that #14415 has merged we can move forwards on this. However, the code here is out of date (it was based on an older revision of #14415) and I worry it might not merge so cleanly. Let me know if you have trouble and I can take a stab at it.

I will try to create a replacement PR.

[rtpsw]

I ended up merging. @westonpace, does it look like what you expected?

[rtpsw]

How to deal with the following CI job's error?

/arrow/cpp/src/arrow/engine/substrait/options.h:37: error: documented symbol 'enum ARROW_ENGINE_EXPORT arrow::engine::arrow::engine::ConversionStrictness' was not declared or defined. (warning treated as error, aborting now)

[rtpsw]

I'm not a big fan of the fact that we need the using statements for the substrait namespace but if we really want that package name then it seems protobuf gives us little choice.

@westonpace, note that this PR is currently using a substrait_ext namespace

[rtpsw]

How to deal with the following CI job's error?

/arrow/cpp/src/arrow/engine/substrait/options.h:37: error: documented symbol 'enum ARROW_ENGINE_EXPORT arrow::engine::arrow::engine::ConversionStrictness' was not declared or defined. (warning treated as error, aborting now)

Fixed in https://issues.apache.org/jira/browse/ARROW-18424

[westonpace]

@rtpsw Looks like the merge didn't go through cleanly. Let me know if you want me to clean this up. Have you been able to address the concern here: #14485 (review) ?

[rtpsw]

@rtpsw Looks like the merge didn't go through cleanly. Let me know if you want me to clean this up.

Note that this is a result of a GitHub automated rebase. @westonpace, if you can easily clean this up, that would be helpful.

Have you been able to address the concern here: #14485 (review) ?

I have removed the dependence on protobuf in the header, but I haven't added the proposed cmake snippet. It sounded like the place for this snippet is under discussion. I'm also not sure whether it is required in this PR or can be deferred.

[rtpsw]

@rtpsw I had to rebase this again. I've looked through it and I think this is in good shape now. If you're ok with it I will merge once the tests pass.

I compared the codes, mostly manually, and couldn't find anything missing.

[westonpace]

The tests were failing after a rebase due to a recent change to CheckRoundTrip (doesn't need a schema). I pushed a small fix.

Unfortunately, the test is now failing for me because the process thread marks the plan complete and then tries to join itself. I copied over the change we made in b9cda43 to address this but that change relies on an executor always being present which won't be true until ARROW-15732 merges.

[rtpsw]

@westonpace, I see you approved - is there anything holding up this PR?

[westonpace]

@rtpsw I believe it's the same failure you noticed before, which is that there is a segfault because schedule task is synchronous. In the other PR you fixed it by requiring an executor but that introduces ordering problems if I remember correctly. Although it may not matter for a test and will allow us to move forward on this. I will try this real quick.

[rtpsw]

@rtpsw I believe it's the same failure you noticed before, which is that there is a segfault because schedule task is synchronous. In the other PR you fixed it by requiring an executor but that introduces ordering problems if I remember correctly. Although it may not matter for a test and will allow us to move forward on this. I will try this real quick.

If indeed it introduces these ordering problems, there are alternatives fixes for the segfault, e.g., use a standalone thread for that task or use a new ExecContext::async_executor() method that is guaranteed to return an asynchronous executor.

[westonpace]

If indeed it introduces these ordering problems, there are alternatives fixes for the segfault, e.g., use a standalone thread for that task or use a new ExecContext::async_executor() method that is guaranteed to return an asynchronous executor.

Agreed. I think we can address this later though. I've rebased (had to address the query context change) and would like to merge this when it is green.

[westonpace]

Agreed. I think we can address this later though. I've rebased (had to address the query context change) and would like to merge this when it is green.

The problem was still happening. I've fixed it properly in #15104 using the same fix that was in the demo branch (ensuring that we always have an executor and the async scheduler is never synchronous).

I've rebased this (once more) and the problem should no longer occur.

[westonpace]

CI failure seems surprising but unrelated. I filed #15137 to follow up.

[westonpace]

Failure appears unrelated. I will merge.

[westonpace]

@rtpsw thanks for your persistence on this one. Sorry for the rebase troubles at the end.

[ursabot]

Benchmark runs are scheduled for baseline = 9857891 and contender = 85db6f7. 85db6f7 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️7.22% ⬆️6.67%] test-mac-arm
[Finished ⬇️2.04% ⬆️0.26%] ursa-i9-9960x
[Failed ⬇️0.0% ⬆️0.0%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 85db6f7b ec2-t3-xlarge-us-east-2
[Failed] 85db6f7b test-mac-arm
[Finished] 85db6f7b ursa-i9-9960x
[Failed] 85db6f7b ursa-thinkcentre-m75q
[Finished] 98578916 ec2-t3-xlarge-us-east-2
[Failed] 98578916 test-mac-arm
[Finished] 98578916 ursa-i9-9960x
[Failed] 98578916 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[ursabot]

['Python', 'R'] benchmarks have high level of regressions.
test-mac-arm
ursa-i9-9960x

[rtpsw]

@westonpace, thanks for wrapping this up.