IcebergSource cannot load data from hiveTable

[manishmalhotrawork]

While testing HiveTable read using Iceberg DataSource API, looks like, its not possible.

read like this:
sparkSession.read().format("iceberg").load("db.tbl")

as the findTable(...) method in IcebergSource uses HadoopTables:

protected Table findTable(DataSourceOptions options, Configuration conf) {
 Optional<String> location = options.get("path");
 Preconditions.checkArgument(location.isPresent(),
 "Cannot open table without a location: path is not set");

 HadoopTables tables = new HadoopTables(conf);

 return tables.load(location.get());
 }

May be Im missing something and didnt understand the flow.

[eskrm]

Hi @manishmalhotrawork,

You're not missing anything. This is tracked by #8.

@rdblue How should different table implementations be supported by the Spark module? I'd like to contribute a DynamoDB-based implementation, if it's wanted, and be able to switch to that implementation in the Iceberg data source. Since the module only depends on the Tables interface can we have an option to specify the Tables FQCN or something to that effect?

[aokolnychyi]

I will leave the actual answer to @rdblue. I believe I had a similar question as @eskrm.

For now, one can extend IcebergSource and override findTable to use HiveTables. However, there might be version conflicts as iceberg-hive uses Hive 3.1.0 and it includes Hadoop 3.1.0 as its dependency. Spark 2.4 still relies on Hadoop 2.x.

[rdblue]

@aokolnychyi is right. The version conflict is something we need to take care of as part of #8. The right way to plug in your own metastore implementation for Spark is to extend IcebergSource and override findTable.

No need to do this for Hive tables, since we want to add support for using the Hive metastore as the default for named tables in IcebergSource (path-based tables would continue to use HadoopTables).

[manishmalhotrawork]

thanks @eskrm @aokolnychyi @rdblue
yes, I extended IcebergSource to test it.

@rdblue so for HiveTables we should not extend,
If Im not missing, then HiveTables, HiveTableOperations depends on the the ThriftClient of HiveMetastore, so internally its using Metastore.
So, we still changes to use the metastore in IcebergDatasource.

[eskrm]

@manishmalhotrawork I believe @rdblue simply meant that once #8 is closed you will no longer have to extend IcebergSource to get it working with HiveTables. For now you do.

[manishmalhotrawork]

@eskrm thanks, sure.
At our place also, we would love to contribute for #8 as well.

[rdblue]

Fixed by #239.