Accessing Iceberg tables without catalog

[asheeshgarg]

Query engine

Spark

Question

I have a scenario where in I have loss access to catalog data. But I have access to iceberg metastore metadata and data. Is the a way to access the tables without catalog.

Do I need to create external tables in some other catalogs and point it to the location of warehouse?

[kbendick]

If I understand correctly, you have lost your catalog data (e.g. the data in HMS or in your dynamodb table). Is that correct?

There's a RegisterTableProcedure that can be used to register an existing table into a catalog.

I'm not sure if the procedure is available in Iceberg 0.14.0, though you might be able to use the action directly from spark code (see the associated unit tests for the "action", which actually comes from BaseMetastoreCatalog). But you could otherwise try running one of the import procedures.

I think that's what you're looking for.

[asheeshgarg]

@kbendick yeah I lost access to HMS. But I have access to the warehouse folder in the s3 with data and metadata folders for all the tables generate by iceberg. Just looking at the ways to recreate the tables as well how to read the data correctly using the information available in the warehouse directory.
I have enabled spark extensions and able to call the stored procedure. So what I am trying to call now is spark.sql("CALL register_table()") with parameters to table and metadata file.
Table is something local I have created in my local catalog and giving it the metadata directory of the original warehouse.

spark-shell --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.local.type=hadoop
--conf spark.sql.catalog.local.warehouse=$PWD/warehouse
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

is these correct the right steps?

[asheeshgarg]

Icreated a local catalog and trying to register a table I get not supported error
I am on spak 3.3 and iceberg 0.14
spark.sql("CALL local.system.register_table('table','s3a://test/')").show()
java.lang.UnsupportedOperationException: Registering tables is not supported
at org.apache.iceberg.catalog.Catalog.registerTable(Catalog.java:363)
at org.apache.iceberg.CachingCatalog.registerTable(CachingCatalog.java:187)
at org.apache.iceberg.spark.procedures.RegisterTableProcedure.call(RegisterTableProcedure.java:84)

[nastra]

The registerTable() functionality from #5037 didn't make it into 0.14.0. However, we do publish nightly snapshot versions off of master (0.15.0-SNAPSHOT) so as a workaround you could try and use that version. Otherwise you might have to wait for the next version to be released with this functionality.

[asheeshgarg]

not able to find the artifactory location for nightly build. Could you please point me to that.

[nastra]

You need to use the snapshot repository mentioned in https://infra.apache.org/repository-faq.html. The artifacts themselves are under https://repository.apache.org/content/groups/snapshots/org/apache/iceberg/

[asheeshgarg]

Thanks I was able to test it out one issue is if I move the data backed by s3 bucket in a DR location and try to recreate it the s3 buckets has different names are there any tools to fix the locations references used in metadata with the new locations.

[Hoeze]

Hi, I would like to do (kind of) the same.
Catalogs are very hard to maintain in my use case.

Is it possible to open Iceberg tables like simple Parquet stores in PySpark?
spark.read.format("iceberg").load(iceberg_path)

[nastra]

Hi, I would like to do (kind of) the same. Catalogs are very hard to maintain in my use case.

Is it possible to open Iceberg tables like simple Parquet stores in PySpark? spark.read.format("iceberg").load(iceberg_path)

Currently you would always have to go through a catalog

[Hoeze]

Currently you would always have to go through a catalog

@nastra Will this change in the near future?

[Hoeze]

My aim is to provide people with a large sorted dataset that they can simply download, read and query, while avoiding unnnecessary sort shuffles when reading + joining on it.

Having to setup Spark catalogs makes this impossible or at least very hard.

[Hoeze]

@nastra How would I register an Iceberg dataset to the in-memory PySpark catalog?

[nastra]

@nastra How would I register an Iceberg dataset to the in-memory PySpark catalog?

@Hoeze you would have to create a catalog and then register the tables within that catalog (similar to #5512 (comment)). However, note that the registerTable() functionality from #5037 has not been part of an official release yet.

[dennishuo]

@Hoeze For your use case is it sufficient to have a "temp view" on your iceberg table that's just available in the spark session that registers it?

Assuming your Iceberg metadata is intact with a metadata/v#.metadata.json file under the base Iceberg table path, you should be able to do:

spark.read.format("iceberg").load(iceberg_path).createOrReplaceTempView("my_view")
spark.sql("select * from my_view")

The iceberg_path should be the parent directory of metadata/.

[Hoeze]

@dennishuo I am not sure if I understand your suggestion.
Isn't your code snippet equal to mine?

# yours
spark.read.format("iceberg").load(iceberg_path).createOrReplaceTempView("my_view")
df1 = spark.table("my_view")
# mine
df2 = spark.read.format("iceberg").load(iceberg_path)

Now, if df1 == df2 it would be absolutely sufficient.

As I said, I just want to send an iceberg store to people and let them load it in PySpark in a single line of code.