[DRAFT] Build: remove hadoop 2 support

[steveloughran]

Now that the minimum java version is 11, it's impossible for Iceberg to work on a Hadoop release less than 3.3.0.a

Removing the hadoop2 version and libraries forces all building and testing onto a compatible version, and permits followup work using modern hadoop APIs.

[steveloughran]

hive2 tests are failing with what looks like guava version/access issues.

TestHiveIcebergStorageHandlerWithEngine > testDescribeTable() > fileFormat=PARQUET, engine=mr, catalog=CUSTOM_CATALOG, isVectorized=false FAILED
java.lang.IllegalArgumentException: Failed to execute Hive query 'DESCRIBE default.customers': Error running query: java.lang.IllegalAccessError: class org.apache.hadoop.hive.ql.exec.FetchOperator tried to access method 'com.google.common.collect.UnmodifiableIterator com.google.common.collect.Iterators.emptyIterator()' (org.apache.hadoop.hive.ql.exec.FetchOperator and com.google.common.collect.Iterators are in unnamed module of loader 'app')
at org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:164)
at org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine.testDescribeTable(TestHiveIcebergStorageHandlerWithEngine.java:253)
Caused by:
org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.IllegalAccessError: class org.apache.hadoop.hive.ql.exec.FetchOperator tried to access method 'com.google.common.collect.UnmodifiableIterator com.google.common.collect.Iterators.emptyIterator()' (org.apache.hadoop.hive.ql.exec.FetchOperator and com.google.common.collect.Iterators are in unnamed module of loader 'app')
at app//org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:238)
at app//org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
at app//org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
at app//org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
at app//org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:500)
at app//org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:265)
at app//org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:150)
... 1 more
Caused by:
java.lang.IllegalAccessError: class org.apache.hadoop.hive.ql.exec.FetchOperator tried to access method 'com.google.common.collect.UnmodifiableIterator com.google.common.collect.Iterators.emptyIterator()' (org.apache.hadoop.hive.ql.exec.FetchOperator and com.google.common.collect.Iterators are in unnamed module of loader 'app')
at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:108)
at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1295)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)
... 7 more

apparently Hive 2 also needs to be cut as it is only a hadoop 2.x runtime.

https://hive.apache.org/general/downloads/

[Fokko]

@steveloughran Thanks for taking a stab at this. I've created an issue here #10940 to track this. I'm also a bit constrained on time, so I'm open for someone else to take this on.

[manuzhang]

@Fokko@steveloughran I'd love to take this on if you guys are time constrained.

[Fokko]

@manuzhang That would be great, thank you! 🙌

[steveloughran]

@manuzhang a key issue is hive2 has to go -which is not just a build profile, its the CI test setup.

That can be done in the same PR as this, or as a PR done in advance.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.