Uh oh!
There was an error while loading. Please reload this page.
ARROW-16996: [Java] Configure Netty/GRPC/Protobuf base on BOM configuration + upgrade of dependencies by CVE - #13544
Conversation
davisusanibar
commented
Jul 8, 2022
- Configure Netty/GRPC/Protobuf base on BOM Bill Of Material configuration to dependencies versions be added by configuration (Bill Of Materials (BOM) pom netty/netty#5994).
- Upgrade Netty/GRPC/Protobuf dependencies. Netty CVE
…Netty/GRPC/Protobuf dependencies
lidavidm
commented
Jul 8, 2022
Hmm, something about the build is off though. |
| <dependency> | ||
| <groupId>io.netty</groupId> | ||
| <artifactId>netty-transport-native-unix-common</artifactId> | ||
| <version>${dep.netty.version}</version> |
There was a problem hiding this comment.
Seems this is required, at least judging by the build?
There was a problem hiding this comment.
From some searching, it seems dependencyManagement may not get inherited by profiles, so we'll need to explicitly reference the version here still.
There was a problem hiding this comment.
What will be the impact if we only use io.netty:netty-transport-native-unix-common:4.1.78.Final instead of io.netty:netty-transport-native-unix-common:osx-x86_64:4.1.78.Final? Is there some reason to why we use that?
I am building and testing code without -osx / -linux and it finished without problems.
There was a problem hiding this comment.
You need the platform specific code since this is a JNI library.
There was a problem hiding this comment.
That said our tests might not actually hit any of the platform-specific paths
There was a problem hiding this comment.
That said our tests might not actually hit any of the platform-specific paths
Does it mean that we still need the platform-specific JNI library for actual usages? Or, do we ever need no the JNI libraries for both test and actual usages?
There was a problem hiding this comment.
We currently do not need the Netty JNI libraries for tests, because we aren't properly testing those paths (ARROW-17024). For actual use, they are only required if the application is using domain sockets, or wants to manually configure Netty to use epoll/kqueue. The application can always explicitly add the necessary dependencies.
There was a problem hiding this comment.
I see. I will remove this code to download the Netty JNI binaries from the build script for s390x.
lidavidm
commented
Jul 8, 2022
If we're going to remove the profiles please update the docs too. |
davisusanibar
commented
Jul 8, 2022
Is github-actions crossbow submit java-jars run needed? How do you evaluate if this is needed? |
lidavidm
commented
Jul 8, 2022
It's usually fine unless we're changing something about how the JARs are built |
Added |
lidavidm
commented
Jul 8, 2022
I just noticed. Why are all tests skipped? |
lidavidm
commented
Jul 8, 2022
Never mind, they get run at the end. However: The test that exercises the Netty-with-native-utils path is getting skipped |
lidavidm
commented
Jul 8, 2022
That said they're getting skipped on HEAD too so I'll file a ticket to investigate that. |
lidavidm
commented
Jul 8, 2022
kou
commented
Jul 15, 2022
@kiszk Could you upload protoc-gen-grpc-java 1.47.0 for s390x to https://apache.jfrog.io/ui/native/arrow/protoc-gen-grpc-java-binary/ to fix a Travis CI job failure? e.g.: https://app.travis-ci.com/github/apache/arrow/jobs/576822591#L2933 |
kiszk
commented
Jul 16, 2022
I will do it this weekend. |
When I prepare the required binaries at my end, the build process on s390x goes forward. But, I got the following error while the build on x86_64 did not fail. I am investigating why this error occurs. |
lidavidm
commented
Jul 17, 2022
lidavidm
commented
Jul 17, 2022
Oh wait. I read it backwards, sorry. Hmm, it looks like something is generating gRPC/Protobuf code inside flight-sql? Or maybe we need to make those dependencies platform-dependent. |
kiszk
commented
Jul 17, 2022
When I execute only analyze-only, an error does not occur... |
kiszk
commented
Jul 17, 2022
Sorry for my bad. After executing |
kiszk
commented
Jul 18, 2022
Submitted a PR #13632 |
kiszk
commented
Jul 19, 2022
Now, the test passed. |
…ration + upgrade of dependencies by CVE (apache#13544) - Configure Netty/GRPC/Protobuf base on BOM Bill Of Material configuration to dependencies versions be added by configuration (netty/netty#5994). - Upgrade Netty/GRPC/Protobuf dependencies. Netty [CVE](GHSA-269q-hmxg-m83q) Authored-by: david dali susanibar arce <davi.sarces@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com>
… configuration + upgrade of dependencies by CVE (apache#13544)" This reverts commit 9b18459.
* DX-86156: Cherry pick a4a3d3f for apacheGH-39265 * Update README_DREMIO.md * ARROW-16996: [Java] Configure Netty/GRPC/Protobuf base on BOM configuration + upgrade of dependencies by CVE (apache#13544) - Configure Netty/GRPC/Protobuf base on BOM Bill Of Material configuration to dependencies versions be added by configuration (netty/netty#5994). - Upgrade Netty/GRPC/Protobuf dependencies. Netty [CVE](GHSA-269q-hmxg-m83q) Authored-by: david dali susanibar arce <davi.sarces@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com> * Revert "ARROW-16996: [Java] Configure Netty/GRPC/Protobuf base on BOM configuration + upgrade of dependencies by CVE (apache#13544)" This reverts commit 9b18459. * Update pom.xml Remove unused dependency. * apacheGH-38996: [Java] Update dependencies and plugins for JPMS modules (apache#38994) Update dependencies and plugins to versions that work correctly with JPMS modules. * Update several plugins to use module-enabled versions: * maven-compiler-plugin is updated to 3.11.0 * maven-shade-plugin is updated to 3.2.4 * maven-dependency-plugin is updated to 3.1.2 * CycloneDX is updated to 2.7.10 * Update grpc-java to 1.59 for module support * Update jackson to 2.16.0 as 2.15.1 had corrupt module-info.class files that broke module support. Existing tests only Users may need to update their own dependencies if they have the same ones. * Closes: apache#38996 Authored-by: James Duong <james.duong@improving.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com> * Update README_DREMIO.md --------- Signed-off-by: David Li <li.davidm96@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com> Co-authored-by: panbingkun <pbk1982@gmail.com> Co-authored-by: david dali susanibar arce <davi.sarces@gmail.com> Co-authored-by: James Duong <james.duong@improving.com>
…ration + upgrade of dependencies by CVE (apache#13544) - Configure Netty/GRPC/Protobuf base on BOM Bill Of Material configuration to dependencies versions be added by configuration (netty/netty#5994). - Upgrade Netty/GRPC/Protobuf dependencies. Netty [CVE](GHSA-269q-hmxg-m83q) Authored-by: david dali susanibar arce <davi.sarces@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com>