Support custom catalog for Iceberg StorageHandler

[rdsr]

@massdosage, @rdblue . Is anyone workin on this? I wanted to take this up and we are only using HiveCatalog.

[guilload]

I have some thoughts regarding this. Now that the SerDe has been merged, there are two pieces of code that do essentially the same thing:

• IcebergInputFormat.findTable(...)
• TableResolver.resolveTableFromConfiguration(...)

1. Those two classes should probably be consolidated into one.
2. The ability to define a custom catalog loader is neat. We actually use our own catalog at Airbnb and that feature comes in handy.
3. At the same time, most users are going to use either the default Hadoop or Hive catalogs, so let's make it easy for them.
4. I'm personally not a fan of using a "table path" configuration property to pass either a file path or a table identifier and have to look for the / character to know which use is intended.

Roughly, I have something like that in mind:

def find_table(conf)
  assert conf.get('table.identifier') is None or conf.get('table.path') is None
  assert conf.get('catalog') is None or conf.get('catalog.loader.class') is None

  if conf.get('table.path'):
    return HadoopTables.load(conf.get('table.path'))

  if conf.get('catalog.loader.class'):
    loader = load_class(conf.get('catalog.loader.class'))
    return loader.load(conf)

  identifier = conf.get('table.identifier')

  if (conf.get('catalog') == 'hadoop'):
    return HadoopCatalog.load(identifier)
  elif (conf.get('catalog') == 'hive'):
    return HiveCatalog.load(identifier)
  else:
    raise ...

[rdsr]

I had something similar in mind.

[massdosage]

This is on our TODO list but we were only going to start looking at it properly after the mapred InputFormat is merged. The above suggestion to merge the code and the logic outlined above looks good. If someone else wants to start working on this now that's fine by me.

[guilload]

Has anybody started working on this issue? Happy to get this going otherwise.

[rdsr]

@guilload I might not be able to get to this this week. So I'm cool if you'd like to finish this up. Happy to review the code!

[guilload]

Ok, I'll take a stab at it!