Python Iceberg REST Catalog

A Python-based implementation of the Apache Iceberg REST Catalog API.

Features

• Standard Iceberg REST API: Supports namespaces, tables, and config endpoints.
• Multi-Catalog Support: Run multiple isolated catalogs on the same server using URL prefixes.
• Pluggable Storage: Comes with SQLite storage for metadata, extensible to other DBs.
• Pluggable I/O: Supports S3 and local filesystem via PyIceberg's FileIO.
• Custom Metadata: Extension endpoints to store arbitrary JSON metadata.
• Extensible Auth: Interface for custom authentication logic.

Getting Started

Prerequisites

• Python 3.9+
• Docker (optional)

Configuration

The catalog can be configured via Environment Variables (global) or Catalog Properties (per-catalog).

Environment Variables

Variable	Description	Default
CATALOG_WAREHOUSE	Base path for table data (e.g. s3://my-bucket/warehouse or /tmp/warehouse)	/tmp/warehouse
CATALOG_PORT	Port to run the server on	8000
S3_ENDPOINT_URL	Global S3 Endpoint URL (for MinIO/S3-compatible)	None
AWS_ACCESS_KEY_ID	Global AWS Access Key	None
AWS_SECRET_ACCESS_KEY	Global AWS Secret Key	None
AWS_REGION	Global AWS Region	None

Catalog Properties (Per-Catalog IO Config)

You can configure IO settings per catalog using the /v1/{catalog_name}/config/properties endpoint. These properties override environment variables.

Property	Description	Example
s3.endpoint	S3 Endpoint URL	http://minio:9000
s3.access-key-id	AWS Access Key	minioadmin
s3.secret-access-key	AWS Secret Key	minioadmin
s3.region	AWS Region	us-east-1
warehouse	Warehouse Location (overrides env var)	s3://warehouse/my_cat

Example: Setting properties via API

curl -X POST http://localhost:8000/v1/my_catalog/config/properties \
-H "Content-Type: application/json" \
-d '{ "s3.endpoint": "http://minio:9000", "s3.access-key-id": "minioadmin", "s3.secret-access-key": "minioadmin" }'

Important: If you want the server to use a specific warehouse location for a catalog (e.g. for create_table), you must set the warehouse property via this API. Passing it to pyiceberg.load_catalog only configures the client, not the server.

Running Locally

1. Install Dependencies:

   pip install -e .

2. Run the Server:

   export CATALOG_WAREHOUSE=/tmp/warehouse
   uvicorn main:app --reload --host 0.0.0.0 --port 8000

   The server will start at http://127.0.0.1:8000.

Running via Docker

You can run the pre-built image alexmerced/iceberg-catalog:

docker run -p 8000:8000 -e CATALOG_WAREHOUSE=/tmp/warehouse alexmerced/iceberg-catalog

To mount a local directory for the warehouse (persistence):

docker run -p 8000:8000 \
-v $(pwd)/warehouse:/tmp/warehouse \
-e CATALOG_WAREHOUSE=/tmp/warehouse \
alexmerced/iceberg-catalog

Running with MinIO (Docker Compose)

To run the catalog with MinIO (S3-compatible storage) using Docker Compose:

1. Create docker-compose.yml (provided in the repo).
2. Run:

   docker-compose up

   This starts:
   • Catalog: http://localhost:8000
   • MinIO Console: http://localhost:9001 (User: minioadmin, Pass: minioadmin)
   • MinIO API: http://localhost:9000
   • mc: Automatically creates a warehouse bucket.

Usage with PyIceberg

You can connect to this catalog using pyiceberg. The URL prefix determines the catalog name.

Example: Connecting to a catalog named my_team_catalog

frompyiceberg.catalogimportload_catalogcatalog=load_catalog(
"my_team_catalog",
**{
"uri": "http://127.0.0.1:8000",
"prefix": "my_team_catalog", # IMPORTANT: Matches the URL prefix"warehouse": "s3://my-bucket/warehouse",
}
)
# Create a namespacecatalog.create_namespace("analytics")
# Create a tablefrompyiceberg.schemaimportSchemafrompyiceberg.typesimportNestedField, StringTypeschema=Schema(NestedField(1, "data", StringType(), required=True))
catalog.create_table("analytics.logs", schema=schema)

Example: Using MinIO (S3)

If running via Docker Compose with MinIO:

frompyiceberg.catalogimportload_catalogcatalog=load_catalog(
"minio_catalog",
**{
"uri": "http://localhost:8000",
"prefix": "minio_catalog",
"warehouse": "s3://warehouse/minio_catalog",
"s3.endpoint": "http://localhost:9000",
"s3.access-key-id": "minioadmin",
"s3.secret-access-key": "minioadmin",
}
)
# Create namespace and tablecatalog.create_namespace("test_ns")
# ...

Extensibility

This catalog is designed to be easily extended.

1. Storage Backend (Database)

By default, metadata is stored in SQLite. To use Postgres, MySQL, or others:

1. Open catalog/storage.py.
2. Create a new class inheriting from CatalogStorage.
3. Implement the abstract methods (create_namespace, get_table, etc.).
4. In catalog/api/routes.py, update the get_storage dependency to return your new storage class.

# catalog/storage.pyclassPostgresStorage(CatalogStorage):
def__init__(self, connection_string):
# Setup SQLAlchemy engine with Postgrespass# ... implement methods ...

2. Authentication

By default, no authentication is enforced. To add Auth (e.g., OAuth2, Basic Auth):

1. Open catalog/auth.py.
2. Create a new class inheriting from AuthenticationProvider.
3. Implement get_user(request: Request).
4. In catalog/api/routes.py, update the get_auth dependency.

# catalog/auth.pyclassBasicAuthProvider(AuthenticationProvider):
defget_user(self, request: Request) ->str:
# Check Authorization headerreturnuser_id

3. Custom Metadata

The catalog exposes endpoints to store arbitrary JSON metadata associated with a catalog.

• GET /v1/{catalog_name}/ext/metadata/{key}
• POST /v1/{catalog_name}/ext/metadata/{key}
• DELETE /v1/{catalog_name}/ext/metadata/{key}

This is useful for storing UI configurations, tags, or other auxiliary data not covered by the Iceberg spec.

API Endpoints

Standard Iceberg

• GET /v1/{prefix}/config
• GET /v1/{prefix}/namespaces
• POST /v1/{prefix}/namespaces
• GET /v1/{prefix}/namespaces/{namespace}
• DELETE /v1/{prefix}/namespaces/{namespace}
• POST /v1/{prefix}/namespaces/{namespace}/properties
• GET /v1/{prefix}/namespaces/{namespace}/tables
• POST /v1/{prefix}/namespaces/{namespace}/tables
• GET /v1/{prefix}/namespaces/{namespace}/tables/{table}
• DELETE /v1/{prefix}/namespaces/{namespace}/tables/{table}
• POST /v1/{prefix}/tables/rename

Extensions

• GET /v1/{prefix}/ext/metadata/{key}
• POST /v1/{prefix}/ext/metadata/{key}
• DELETE /v1/{prefix}/ext/metadata/{key}