Uh oh!
There was an error while loading. Please reload this page.
ARROW-8853: [Rust] [Integration Testing] Enable Flight tests - #9049
ARROW-8853: [Rust] [Integration Testing] Enable Flight tests #9049carols10cents wants to merge 9 commits into
Conversation
ce2519b to
2c5899cCompare
alamb
left a comment
There was a problem hiding this comment.
I reviewed this code commit by commit (thanks @carols10cents for splitting it up into a easy-to-follow chain).
While I am not an expert in Arrow Flight, I did find the code easy to follow and understand and it made sense to me.
As predicted, some java tests failed on this integration test and I have pinged some people on #8963 and hopefully we can get that merged soon. The error was
RuntimeError: Command failed: java -Dio.netty.tryReflectionSetAccessible=true -Darrow.struct.conflict.policy=CONFLICT_APPEND -cp /arrow/java/flight/flight-core/target/flight-core-3.0.0-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.flight.example.integration.IntegrationTestClient -port 39459 -j /tmp/arrow-integration-xu49kq36/generated_null_trivial.json
With output:
--------------
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Verifying location grpc+tcp://0.0.0.0:39459
Unknown error
java.lang.IllegalArgumentException: A batch can only be consumed if it contains a single ArrowBuf.
When I looked at the rest of the logs, I see the rust producer is running and passing. ❤️ ❤️
2020-12-30T20:04:00.8176245Z ##########################################################
2020-12-30T20:04:00.8176859Z IPC: C++ producing, Rust consuming
2020-12-30T20:04:00.8177378Z ##########################################################
2020-12-30T20:05:45.7634027Z ##########################################################
2020-12-30T20:05:45.7634638Z IPC: Java producing, Rust consuming
2020-12-30T20:05:45.7635156Z ##########################################################
And there are others too.
Really nice work 🎉
alamb
commented
Dec 31, 2020
The full set of Rust CI tests did not run on this PR :( Can you please rebase this PR against apache/master to pick up the changes in #9056 so that they do? I apologize for the inconvenience. |
2c5899c to
84d9b72Comparecarols10cents
commented
Dec 31, 2020
No problem at all, rebased! Thank you for the review ❤️ |
Codecov Report
@@ Coverage Diff @@## master #9049 +/- ##
==========================================
- Coverage 82.57% 81.81% -0.77%
==========================================
Files 204 214 +10 Lines 50879 51352 +473 ==========================================
- Hits 42014 42013 -1 - Misses 8865 9339 +474
Continue to review full report at Codecov.
|
nevi-me
commented
Jan 7, 2021
@carols10cents this needs a rebase :) |
nevi-me
left a comment
There was a problem hiding this comment.
I've gone through the PR, and I don't have any comments. We can merge this after the blocking PR is merged, and the PR's been rebased.
alamb
commented
Jan 8, 2021
@carols10cents -- #8963 was just merged - so hopefully after a rebase this PR will be ready too |
84d9b72 to
a36694bCompareThis function returns, potentially, a few dictionaries, and will always return one record batch. Return these as a tuple rather than putting them together in one vec to clarify.
This doesn't seem necessary
Tracked in ARROW-10961. There's a bug in tonic that doesn't handle headers and trailers correctly; it has been fixed but a new version of tonic needs to be released and used to get the fix.
a36694b to
863420dComparecarols10cents
commented
Jan 8, 2021
Rebased, and CI is looking good! |
alamb
commented
Jan 9, 2021
🎉 |
This PR has a few refactorings and then the main commit contains a new Flight integration test client and server 🎉
The middleware scenario tests are currently skipped because they will fail until
toniccan be updated to a version containing a fix having to do with trailers; this is tracked in ARROW-10961.Some Rust <-> Java integration tests will fail until this PR is merged; I'm happy to rebase once that goes in, but I wanted to get code review started on this. Thank you!!