Add hadoop-2.5 profile with upgraded jets3t

[zh4ngx]

This contribution is my original work and I license the work to the project under the project's open source license.

Adding support for hadoop-2.5.

Also bumped the jets3t version because there are conflicts with the java-aws-sdk under version 0.9.0.

JIRA: https://issues.apache.org/jira/browse/SPARK-4807

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

This has been discussed a few times - the hadoop-2.4 profile covers 2.4+. You don't need a new profile. I think the goal is to match jets3t distributed with Hadoop. It might be fine to update from 0.9.0 to 0.9.2 but if it's not actually a problem, might not be worth touching it just now.

[zh4ngx]

Thanks for the response! I'll try out the hadoop-2.4 profile again. For jets3t, I'll take another look to see if it's Spark or Hadoop that needs to do some shading.

[pwendell]

@ZhangBanger yeah did the jet3s version change between Hadoop 2.4 and 2.5? Feel free to open a separate issue for that.

[JoshRosen]

@ZhangBanger Did you end up trying out the Hadoop 2.4 profile? Did it work or do we need to do further work for this?

[zh4ngx]

It doesn't change between the Hadoop version profiles, so I should separate the jets3t into a separate issue/PR.

The jets3t issue arises from transitive dependencies that clash with java-aws-sdk. Is that something we should be resolving on a one-off basis, is there some shading that needs to happen, or is there a better solution altogether?

[srowen]

How about including the updated jets3t in your app and ensuring it's used with userClassPathFirst?

[zh4ngx]

That seems like a good approach. Thanks for the help!