`catalog.table_exists()` returns 'False' when table exists in Polaris catalog

[JasperHG90]

Apache Iceberg version

0.8.0 (latest release)

Please describe the bug 🐞

Hi 👋

I see quite a few issues/PRs centered around the table_exists() method when using REST catalogs, e.g.

• Fix table_exists behavior in REST catalog #1096
• Adjust the "table_exists" behavior in the REST Catalog #1018
• pyiceberg always return false for catalog.table_exists when used with Polaris catalog #1006
• Rest Catalog: table_exist does not succeed on 204 response code #735

When using Polaris OSS, I also experience issues with this method. Even though a table exists the catalog.table_exists() method will return 'False'.

I have a devcontainer setup that you can find here to make the example below reproducible.

Python code:

#%%

import httpx
import polars as pl
from pyiceberg.catalog import load_catalog

# Assumes this catalog exists and credentials stored in
# .pyiceberg.yaml
# Possibly created using https://github.com/JasperHG90/dagster-pyiceberg-example-polaris/blob/main/scripts/set_up_catalog.py
catalog = load_catalog(
 name="dagster_example_catalog",
)

# %% Create a namespace

catalog.create_namespace("example")

# %% Create data and add to table

df = pl.from_dict({
 "a": [1, 2, 3],
 "b": [4, 5, 6]
}).to_arrow()

catalog.create_table(
 identifier="example.table",
 schema=df.schema
)

# %% This will return 'False'

catalog.table_exists("example.table")

# %% But querying the 'table exists' Polaris endpoint returns 200

catalog_client = httpx.Client(
 base_url="http://polaris:8181/api/catalog/v1",
 headers={
 "Authorization": "Bearer principal:root;realm:default-realm",
 "Content-Type": "application/json",
 }
)

# See endpoint here: https://polaris.apache.org/in-dev/unreleased/rest-catalog-open-api/#tag/Catalog-API/operation/tableExists
resp = catalog_client.head(
 url="dagster_example_catalog/namespaces/example/tables/table"
)

resp.status_code # 200

[kevinjqliu]

Thank you for bringing this up. Its been on my todo list to investigate this as part of #1018 (comment)

can you help me go through the same exercise and step through the table_exists call? In #1018, you can see the response code is 404, which should return False according to the implementation and the spec

I want to see if this is a polaris issue or pyiceberg issue

[JasperHG90]

I've annotated this method with what I see in the debugger

 @retry(**_RETRY_ARGS)
 def table_exists(self, identifier: Union[str, Identifier]) -> bool:
 """Check if a table exists.

 Args:
 identifier (str | Identifier): Table identifier.

 Returns:
 bool: True if the table exists, False otherwise.
 """
 identifier_tuple = self._identifier_to_tuple_without_catalog(identifier) # ('example', 'test')
 # Url: 'http://polaris:8181/api/catalog/v1/dagster_example_catalog/namespaces/example/tables/table'
 response = self._session.head(
 self.url(Endpoints.load_table, prefixed=True, **self._split_identifier_for_path(identifier_tuple))
 )

 # response.status_code is 200

 # This code block is skipped
 if response.status_code == 404:
 return False
 elif response.status_code == 204:
 return True

 # This code block is skipped
 try:
 response.raise_for_status()
 except HTTPError as exc:
 self._handle_non_200_response(exc, {})

 # ... And 'False' is returned because 200 != 204
 return False

Polaris is returning a <200> status code, so that's not correct according to your spec.

This method is somewhat confusing because the code is checking explicitly for two status codes, and raises in case of a non-2xx status code, but will return 'False' in case of 404/200 and 'True' in case of 204. Although it’s probably fair not to expect another 2xx status code, it is confusing for the end-user to not at least raise a warning.

[kevinjqliu]

Thanks @JasperHG90 for the breakdown. The server spec above describes what the server should send and what the client should expect.
According to the spec, Polaris should be sending a 204 for success, not 200.

Do you mind opening an issue with Polaris?

[JasperHG90]

FYI PR that fixes status code for this and other endpoints: apache/polaris#472

[Fokko]

To make PyIceberg a bit friendlier to the catalogs, we could also allow for HTTP 200

[kevinjqliu]

apache/polaris#472 is merged @JasperHG90 do you mind testing it again with the latest code?

[JasperHG90]

Tested and works. Closing this issue. Thanks @kevinjqliu !

[kevinjqliu]

cc @djouallah from #1018 (comment)