[SPARK-13120] [test-maven] Shade protobuf-java

[tedyu]

https://groups.google.com/forum/#!topic/protobuf/wAqvtPLBsE8

PB2 and PB3 are wire compatible, but, protobuf-java is not compatible so dependency will be a problem.
Shading protobuf-java would provide better experience for downstream projects.

This PR shades com.google.protobuf:protobuf-java as org.spark-project.protobuf

[tedyu]

Would like some feedback before creating a JIRA.

Thanks

[SparkQA]

Test build #50447 has finished for PR 10995 at commit 21cbc45.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

Since Hadoop doesn't shade protobuf, I think this won't fix the issue in the PR description. Right?

[tedyu]

This would still benefit Spark standalone and Spark on Mesos, right ?

For Spark on YARN, status quo is maintained.

[zsxwing]

@tedyu could you create a JIRA and add it to the title? This change is worth to have a JIRA to track.

Ping @srowen@JoshRosen to take a look.

[JoshRosen]

Can you give this a descriptive PR description rather than a link to some mailing list thread?

[tedyu]

@zsxwing@JoshRosen@srowen
Mind taking another look ?

Thanks

[vanzin]

@zsxwing is right. Because Hadoop libraries don't shade protobuf, this will not help. You'd have to create custom Hadoop jars referencing the shaded protobuf classes, and that's a much bigger change.

(Both standalone and mesos still need hadoop jars, btw.)