Uh oh!
There was an error while loading. Please reload this page.
[SPARK-27733][CORE] Upgrade Avro to version 1.10.1 - #31232
Conversation
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
This is an improvement because Nullable in that class is used to designate class attributes that are nullable (like javax.annotation) and not to refer to nullability of an Avro field.
wangyum
commented
Jan 18, 2021
ok to test. |
SparkQA
commented
Jan 18, 2021
Kubernetes integration test starting |
SparkQA
commented
Jan 18, 2021
Kubernetes integration test status success |
HyukjinKwon
commented
Jan 18, 2021
@iemejia, please fill the PR description. |
SparkQA
commented
Jan 18, 2021
Test build #134193 has finished for PR 31232 at commit
|
iemejia
commented
Jan 18, 2021
Done thanks for the reminder @HyukjinKwon |
iemejia
commented
Jan 18, 2021
All green now PTAL |
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Thank you for pinging me, @iemejia .
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
gengliangwang
commented
Jan 19, 2021
@iemejia Thanks for the work! |
cb3b0ce to
6c6b137CompareSparkQA
commented
Jan 19, 2021
Kubernetes integration test starting |
SparkQA
commented
Jan 19, 2021
Kubernetes integration test status success |
SparkQA
commented
Jan 19, 2021
Test build #134221 has finished for PR 31232 at commit
|
iemejia
commented
Jan 19, 2021
Hive error looks like a flake is there a way to re run that CI job/action? |
mridulm
commented
Jan 19, 2021
+CC @xkrogen |
6c6b137 to
f0ea522CompareSparkQA
commented
Jan 20, 2021
Kubernetes integration test starting |
SparkQA
commented
Jan 20, 2021
Kubernetes integration test status failure |
iemejia
commented
Jan 20, 2021
Yay! green again after rebasing. PTAL again |
SparkQA
commented
Jan 20, 2021
Test build #134260 has finished for PR 31232 at commit
|
gengliangwang
left a comment
There was a problem hiding this comment.
LGTM. Again, thanks for the work!
gengliangwang
commented
Jan 20, 2021
Also cc @bozhang2820 |
wangyum
commented
Jan 20, 2021
AvroReadBenchmark result. Avro 1.10.1: |
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Thank you, @iemejia and all!
Merged to master for Apache Spark 3.2.0.
dongjoon-hyun
commented
Jan 20, 2021
@iemejia I added you to the Apache Spark contributor group and assigned SPARK-27733 to you. Thanks! |
iemejia
commented
Jan 21, 2021
Thanks to you @dongjoon-hyun and @wangyum! This PR seems 'trivial' but when you have lived through the multi project issues/fixes and dependencies you realize this would not have been possible without everybody's work. |
iemejia
commented
Jan 21, 2021
@wangyum just for learning's sake what command do you use to calculate the benchmarks? |
wangyum
commented
Jan 21, 2021
iemejia
commented
Jan 21, 2021
Thanks for the ref 👍 @wangyum ! |
### What changes were proposed in this pull request? Update Avro dependency to version 1.10.1 ### Why are the changes needed? To catch up multiple improvements of Avro as well as fix security issues on transitive dependencies. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Since there were no API changes required we just run the tests Closesapache#31232 from iemejia/SPARK-27733-avro-upgrade. Authored-by: Ismaël Mejía <iemejia@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Update Avro dependency to version 1.10.1 To catch up multiple improvements of Avro as well as fix security issues on transitive dependencies. No Since there were no API changes required we just run the tests Closesapache#31232 from iemejia/SPARK-27733-avro-upgrade. Authored-by: Ismaël Mejía <iemejia@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Update Avro dependency to version 1.10.1 To catch up multiple improvements of Avro as well as fix security issues on transitive dependencies. No Since there were no API changes required we just run the tests Closesapache#31232 from iemejia/SPARK-27733-avro-upgrade. Authored-by: Ismaël Mejía <iemejia@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Update Avro dependency to version 1.10.1 To catch up multiple improvements of Avro as well as fix security issues on transitive dependencies. No Since there were no API changes required we just run the tests Closesapache#31232 from iemejia/SPARK-27733-avro-upgrade. Authored-by: Ismaël Mejía <iemejia@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
What changes were proposed in this pull request?
Update Avro dependency to version 1.10.1
Why are the changes needed?
To catch up multiple improvements of Avro as well as fix security issues on transitive dependencies.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Since there were no API changes required we just run the tests