IceFrame (Alpha)

A DataFrame-like library for working with Apache Iceberg tables using REST catalogs with local execution.

IceFrame provides a simple, intuitive API for creating, reading, updating, and deleting Iceberg tables, as well as performing maintenance operations and exporting data.

Upgrading from 0.12?0.13.0 fixes silent data loss and silently wrong query results.compact_data_files(filter_expr=...) used to replace the whole table with the filtered subset; a compound filter whose operand couldn't be pushed down had that operand silently dropped; null rows passed every data-quality constraint; expire_snapshots called a PyIceberg API that doesn't exist; and create_table(sort_order=["col"]) raised AttributeError. All are fixed with regression tests. See CHANGELOG.md for the full list and behaviour changes.

Features

• DataFrame API: Familiar interface for working with tables — read_table, to_arrow, to_pandas, lazy, head, describe, count_rows, scan_batches
• Local Execution: Uses PyIceberg, PyArrow, and Polars for efficient local processing
• Catalog Support: REST catalogs (Dremio, Tabular, Polaris, …) with credential vending, plus PyIceberg's sql (SQLite/Postgres), memory, glue, hive, and dynamodb catalogs by pass-through
• CRUD Operations: Create, Read, Update, Delete tables and data
• Native upserts and transactions: ice.upsert(...) on PyIceberg's atomic Table.upsert, and with ice.transaction(tbl) as txn: for multi-operation atomic commits
• Predicate, projection and limit pushdown: the query builder pushes WHERE, SELECT and LIMIT into the Iceberg scan when it is sound to do so, and re-applies anything it cannot push locally so results stay correct
• Metadata tables: ice.inspect(tbl).snapshots() / .files() / .partitions() / .manifests() / .history() / .refs() as Polars frames
• Maintenance: Expire snapshots, remove orphan files (dry-run by default), compact data files (bin-pack, sort, approximate z-order)
• Data quality: constraint validation where nulls fail by default, wired into append_to_table(validators=[...]) as a write gate
• Typed: ships py.typed, so annotations are visible to downstream type checkers
• Structured logging: standard logging throughout the library; no print() to stdout
• Export: Export data to Parquet, CSV, and JSON

Documentation

Getting Started

• Creating Tables
• Reading Tables
• Updating Tables
• Deleting Tables
• CLI Usage
• Dependencies
• Environment Variables

Data Ingestion

• Native File Ingestion (CSV, JSON, Parquet, ORC, Avro)
• Optional File Ingestion (Excel, Delta, Google Sheets)
• Advanced File Ingestion (SQL, XML, SAS/SPSS)
• API Ingestion
• HuggingFace Ingestion
• HTML Ingestion
• Clipboard Ingestion
• Folder Ingestion
• Bulk Ingestion
• Incremental Ingestion

Querying & Processing

• Query Builder API
• SQL Support (DataFusion)
• Lazy Reading
• Distributed Processing (Ray)
• Async Operations
• Notebook Integration
• Scalable Updates

Table Management

• Namespace Management
• Schema Evolution
• Partition Management
• Branching & Tagging
• Views
• Catalog Operations
• Catalog Support Matrix
• Transactions & Upserts
• Metadata Tables

Maintenance & Quality

• Table Maintenance
• Native Maintenance
• Safe Compaction
• Streaming Auto-Compaction
• Data Quality
• Enhanced Data Quality
• Rollback & History

Advanced Features

• Visualization
• Incremental Processing
• Table Statistics
• Scalability Overview
• AI Agent
• MCP Server
• Pydantic Integration

Recipes

• ETL Pipeline
• SCD Type 2
• Data Quality Gate

Installation

pip install iceframe

For cloud storage support:

pip install "iceframe[aws]"# AWS S3
pip install "iceframe[gcs]"# Google Cloud Storage
pip install "iceframe[azure]"# Azure Data Lake Storage

Quick Start

1. Create a .env file with your catalog credentials (see .env.example):

ICEBERG_CATALOG_URI=https://catalog.dremio.cloud/api/icebergICEBERG_TOKEN=your_tokenICEBERG_WAREHOUSE=your_warehouseICEBERG_CATALOG_TYPE=rest

2. Use IceFrame in your code:

fromiceframeimportIceFrame, col, lit, load_catalog_config_from_envimportpolarsaspl# Initializeconfig=load_catalog_config_from_env()
ice=IceFrame(config)
# Create an EMPTY table (note: as of 0.12, create_table never writes data# even if you pass a DataFrame as the schema — use append_to_table afterwards)schema= {
"id": "long",
"name": "string",
"created_at": "timestamp",
}
ice.create_table("my_table", schema)
# Append datadata=pl.DataFrame({
"id": [1, 2],
"name": ["Alice", "Bob"],
"created_at": [pl.datetime(2024, 1, 1), pl.datetime(2024, 1, 2)],
})
ice.append_to_table("my_table", data)
# Read datadf=ice.read_table("my_table")
print(df)
# Query Builder API — col, lit, IceFrame, QueryBuilder all importable from# the package root as of 0.12.fromiceframe.functionsimportsumasice_sumdf= (ice.query("my_table")
.select("name", ice_sum(col("id")).alias("total_id"))
.group_by("name")
.execute())
print(df)

Filters: two dialects, named explicitly

filter_expr= historically accepted both an IceFrame Expression (pushed to Iceberg) and a string (Polars SQL, evaluated locally) — the same parameter name meaning two languages with opposite performance. As of 0.13.0 both are named:

# Pushed into the Iceberg scan: only matching files are read.ice.read_table("my_table", filter=col("age") >30)
# Evaluated locally after the scan, in Polars SQL.ice.read_table("my_table", filter_sql="age > 30")

filter_expr= still works and dispatches on type, but the explicit parameters are preferred.

Projection and limit pushdown

# `selected_fields=("id",)` and `limit=10` are pushed into the scan, so a# billion-row table is not materialised just to trim it.ice.query("events").select("id").limit(10).execute()

Pushdown is skipped when it would change the answer — an ORDER BY, a join, an aggregation, or a predicate that can't be fully pushed all force the limit to be applied after the fact.

Native upsert and transactions

# Atomic, incremental MERGE — only files containing matching rows are rewritten.ice.upsert("users", new_rows, join_cols=["id"])
# Schema change + append in a single commit.withice.transaction("events") astxn:
txn.set_properties({"owner": "data-eng"})
txn.append(arrow_table)

Metadata tables

ice.inspect("events").snapshots() # Polars DataFrameice.inspect("events").files()
ice.inspect("events").partitions()

Maintenance

# Compaction scoped by a filter rewrites ONLY matching rows. Before 0.13.0 this# replaced the entire table with the matching subset.ice.compact_data_files("events", filter_expr="event_date >= '2026-01-01'")
# Snapshot expiry, on PyIceberg's native maintenance API. Branch/tag heads are# always protected.ice.expire_snapshots("events", older_than_days=30, retain_last=5)
# Orphan-file cleanup is DRY-RUN BY DEFAULT: it deletes files permanently.candidates=ice.remove_orphan_files("events", older_than_days=7)
ice.remove_orphan_files("events", older_than_days=7, dry_run=False)

Data quality (nulls fail by default)

fromiceframe.qualityimportDataValidator# A null `age` FAILS "age > 0". Before 0.13.0 it passed, because Polars'# three-valued logic filtered null rows out of the violations frame.DataValidator().validate(df, ["age > 0"])
DataValidator().validate(df, ["age > 0"], null_policy="pass") # opt out# As a write gate:ice.append_to_table("users", df, validators=["age > 0"]) # raises ValidationError

Error handling

Every deliberate IceFrame error derives from IceFrameError, so infrastructure failures are distinguishable from bad input. For backwards compatibility the input-shaped errors are still ValueError subclasses:

fromiceframeimportIceFrameError, CatalogError, ValidationErrortry:
ice.append_to_table("users", df, validators=["age > 0"])
exceptValidationError:
... # your data was badexceptCatalogError:
... # the catalog was unreachable

Development

pip install -e ".[dev]"
pytest # runs offline against a local SQLite catalog
pytest --live # additionally runs the live REST-catalog tests
ruff check iceframe/ tests/

The core suite needs no credentials and no network: a session-scoped fixture builds a PyIceberg sql (SQLite) catalog with a file:// warehouse. Before 0.13.0 the read, write, query-builder, schema and stats tests only ran against the author's live Dremio catalog, so 61 of 179 tests skipped everywhere else.

Feature Comparison: IceFrame vs PyIceberg

IceFrame builds on top of PyIceberg, adding high-level abstractions and missing features.

Feature	PyIceberg (Native)	IceFrame (Enhanced)
Table CRUD	Low-level API	Simplified create_table, drop_table
Data Writing	Arrow/Pandas integration	Polars integration, Auto-schema inference
Branching	Basic support (WIP)	create_branch, fast_forward, WAP Pattern
Compaction	rewrite_data_files (limited)	bin_pack, sort strategies (Polars-based)
Views	Catalog-dependent	Unified ViewManager abstraction
Maintenance	expire_snapshots	GarbageCollector, Nativeremove_orphan_files
SQL Support	None	Fluent Query Builder (select, filter, join)
Ingestion	add_files	add_files wrapper + Incremental Ingestion recipes
Rollback	manage_snapshots	rollback_to_snapshot, rollback_to_timestamp
Async	None	AsyncIceFrame with its own bounded thread pool
Upsert / MERGE	Table.upsert	ice.upsert(...) wrapper + QueryBuilder.merge for column-level update rules
Transactions	Table.transaction()	with ice.transaction(tbl) as txn:
Metadata tables	Table.inspect.* (Arrow)	ice.inspect(tbl).* returning Polars frames
Data Quality	None	DataValidator constraints, null-failing by default, usable as a write gate
Agent surface	None	Read-only MCP server with row/byte caps, plus a multi-provider LLM agent

What IceFrame is not

Being straight about the limits:

• Merge-on-read delete writes are not supported. PyIceberg 0.11 has no public delete-file writer, so MoRWriter.write_position_deletes / write_equality_deletes raise UnsupportedOperationError. Deletes are copy-on-write. Reading tables that already contain delete files works fine.
• QueryBuilder.merge with column-level update rules reads the whole target table and overwrites it. Use ice.upsert(...) (native, incremental, atomic) whenever "replace matched rows wholesale" is what you want.
• Joins read each joined table in full. Only the driving table gets pushdown.
• Z-order is an approximation — a hierarchical sort, not a bit-interleaved Z-curve. The returned strategy says so.
• Window functions run locally in Polars, after the scan.