PyStore - Fast data store for Pandas timeseries data

PyStore is a simple (yet powerful) datastore for Pandas dataframes, and while it can store any Pandas object, it was designed with storing timeseries data in mind.

It's built on top of Pandas, Numpy, Dask, and Parquet (via pyarrow), to provide an easy to use datastore for Python developers that can easily query millions of rows per second per client.

New in 2025 Release (PR #77):

• MultiIndex Support - Store and retrieve DataFrames with Pandas MultiIndex
• Complex Data Types - Full support for Timedelta, Period, Interval, Categorical dtypes
• Timezone-Aware Operations - Proper handling of timezone data with UTC storage
• Async/Await Support - Non-blocking I/O operations for better performance
• Data Validation Framework - Extensible validation rules for data integrity
• Schema Evolution - Handle schema changes over time with flexible strategies
• Transaction Support - Atomic operations with rollback capabilities
• Performance Optimizations - Streaming operations and memory management

Performance Enhancements (Phase 3 Release):

• Streaming Operations - Memory-efficient append for datasets larger than RAM
• Batch Processing - 5-10x faster parallel read/write operations
• Intelligent Partitioning - Automatic time-based and size-based partitioning
• Memory Management - 70-90% memory reduction with monitoring and optimization
• Metadata Caching - 100x faster metadata access with TTL cache
• Query Optimization - Column selection and predicate pushdown at storage level

Performance improvements include:

• Append 1M rows: 3.75x faster, 90% less memory
• Batch operations: 6x faster for multiple items
• Column selection: 4x faster when reading subset of columns
• Filtered reads: 8x faster with predicate pushdown

Check out this blog post for the reasoning and philosophy behind PyStore, as well as a detailed tutorial with code examples.

Follow this PyStore tutorial in Jupyter notebook format.

Quickstart

Install PyStore

Install using pip:

pip install pystore --upgrade --no-cache-dir

Install using conda:

conda install -c ranaroussi pystore

INSTALLATION NOTE: If you don't have Snappy installed (compression/decompression library), you'll need to install it first.

Using PyStore

#!/usr/bin/env python# -*- coding: utf-8 -*-importpystoreimportyfinanceasyf# Set storage path (optional)# Defaults to `~/pystore` or `PYSTORE_PATH` environment variable (if set)pystore.set_path("~/pystore")
# List storespystore.list_stores()
# Connect to datastore (create it if not exist)store=pystore.store("mydatastore")
# List existing collectionsstore.list_collections()
# Access a collection (create it if not exist)collection=store.collection("NASDAQ")
# List items in collectioncollection.list_items()
# Load some data from yfinanceaapl=yf.download("AAPL", multi_level_index=False)
# Store the first 100 rows of the data in the collection under "AAPL"collection.write("AAPL", aapl[:100], metadata={"source": "yfinance"})
# Reading the item's dataitem=collection.item("AAPL")
data=item.data# <-- Dask dataframe (see dask.pydata.org)metadata=item.metadatadf=item.to_pandas()
# Append the rest of the rows to the "AAPL" itemcollection.append("AAPL", aapl[100:])
# Reading the item's dataitem=collection.item("AAPL")
data=item.datametadata=item.metadatadf=item.to_pandas()
# --- Query functionality ---# Query available symbols based on metadatacollection.list_items(some_key="some_value", other_key="other_value")
# --- Snapshot functionality ---# Snapshot a collection# (Point-in-time named reference for all current symbols in a collection)collection.create_snapshot("snapshot_name")
# List available snapshotscollection.list_snapshots()
# Get a version of a symbol given a snapshot namecollection.item("AAPL", snapshot="snapshot_name")
# Delete a collection snapshotcollection.delete_snapshot("snapshot_name")
# ...# Delete the item from the current versioncollection.delete_item("AAPL")
# Delete the collectionstore.delete_collection("NASDAQ")

Advanced Features

Async Operations:

importasynciofrompystoreimportasync_pystoreasyncdefasync_example():
asyncwithasync_pystore.store("mydatastore") asstore:
asyncwithstore.collection("NASDAQ") ascollection:
# Async writeawaitcollection.write("AAPL", df)
# Async readdf=awaitcollection.item("AAPL").to_pandas()
asyncio.run(async_example())

Data Validation:

frompystoreimportcreate_validator, ColumnExistsRule, RangeRule# Create a validatorvalidator=create_validator([
ColumnExistsRule(["Open", "High", "Low", "Close"]),
RangeRule("Close", min_value=0),
])
# Apply validator to collectioncollection.set_validator(validator)

Schema Evolution:

frompystoreimportSchemaEvolution, EvolutionStrategy# Enable schema evolutionevolution=collection.enable_schema_evolution(
"AAPL",
strategy=EvolutionStrategy.FLEXIBLE,
)
# Schema changes are handled automatically during appendcollection.append("AAPL", new_data_with_extra_columns)

Complex Data Types:

# DataFrames with Period, Interval, Categorical typesdf=pd.DataFrame({
"period": pd.period_range("2024-01", periods=12, freq="M"),
"interval": pd.IntervalIndex.from_tuples([(0, 1), (1, 2)]),
"category": pd.Categorical(["A", "B", "A"]),
"nested": [{"key": "value"}, [1, 2, 3], None],
})
collection.write("complex_data", df)

Performance Features:

# Streaming append for large datasetsdefdata_generator():
forchunkinpd.read_csv("huge_file.csv", chunksize=100000):
yieldchunkcollection.append_stream("large_data", data_generator())
# Batch operationsitems_to_write= {
"item1": df1,
"item2": df2,
"item3": df3,
}
collection.write_batch(items_to_write, parallel=True)
# Read multiple items efficientlyresults=collection.read_batch(["item1", "item2", "item3"])
# Memory-optimized readingfrompystore.memoryimportoptimize_dataframe_memory, read_in_chunks# Optimize DataFrame memory usagedf=collection.item("large_item").to_pandas()
df_optimized=optimize_dataframe_memory(df) # Up to 70% memory reduction# Read in chunks for processingforchunkinread_in_chunks(collection, "large_item", chunk_size=50000):
# Process chunk - automatically garbage collectedprocess(chunk)

Query Optimization:

# Column selection - read only what you needitem=collection.item("data")
df=item.to_pandas(columns=["price", "volume"]) # 4x faster for subset# Filter at storage leveldf=item.to_pandas(filters=[("price", ">", 100)]) # 8x faster

Using Dask schedulers

PyStore supports using Dask distributed.

To use a local Dask scheduler, add this to your code:

fromdask.distributedimportLocalClusterpystore.set_client(LocalCluster())

To use a distributed Dask scheduler, add this to your code:

pystore.set_client("tcp://xxx.xxx.xxx.xxx:xxxx")
pystore.set_path("/path/to/shared/volume/all/workers/can/access")

Concepts

PyStore provides namespaced collections of data. These collections allow bucketing data by source, user or some other metric (for example, frequency: End-Of-Day, Minute Bars, etc.). Each collection (or namespace) maps to directory containing partitioned parquet files for each item (e.g., symbol).

A good practice it to create collections that may look something like this:

• collection.EOD
• collection.ONEMINUTE

Requirements

• Python >= 3.8
• Pandas >= 2.0
• Numpy >= 1.20
• Dask >= 2023.1
• PyArrow >= 10.0 (Parquet engine)
• Snappy (Google's compression/decompression library)
• multitasking
• pytest-asyncio (for async testing)

PyStore was tested to work on *nix-like systems, including macOS.

Dependencies

PyStore utilizes Snappy, a fast and efficient compression/decompression library developed by Google. You'll need to install Snappy on your system before installing PyStore.

See the python-snappy GitHub repo for more information.

*nix Systems:

• APT: sudo apt-get install libsnappy-dev
• RPM: sudo yum install libsnappy-devel

macOS:

First, install Snappy's C library using Homebrew:

brew install snappy

Then, install Python's snappy using conda:

conda install python-snappy -c conda-forge

...or, using pip:

CPPFLAGS="-I/usr/local/include -L/usr/local/lib" pip install python-snappy

Windows:

Windows users should check out Snappy for Windows and this Stack Overflow post for help on installing Snappy and python-snappy.

Current Status

Core Features:

• Local filesystem support with Parquet storage
• Full Pandas DataFrame compatibility, including MultiIndex
• Snapshots for point-in-time data versioning
• Metadata support for data organization

Advanced Features (July 2025 Release):

• Complex data type serialization (Period, Interval, Categorical, nested objects)
• Timezone-aware datetime handling with UTC storage
• Async/await operations for non-blocking I/O
• Data validation framework with extensible rules
• Schema evolution for handling data structure changes
• Transaction support with rollback capabilities

Performance Features:

• Streaming operations for datasets larger than RAM
• Batch read/write with parallel processing
• Intelligent partitioning (time-based and size-based)
• Memory optimization with automatic type downcasting
• Metadata caching for faster access
• Query optimization with column selection and predicate pushdown

Known Limitations:

• MultiIndex append operations have limited support due to Dask limitations - while there's a workaround that converts MultiIndex to regular columns, it may not fully preserve the MultiIndex structure after append (test remains marked as expected failure)
• Some Parquet limitations with preserving exact index metadata

Future Plans:

• Amazon S3 support (via s3fs)
• Google Cloud Storage support (via gcsfs)
• Hadoop Distributed File System support (via hdfs3)

Acknowledgements

PyStore is hugely inspired by Man AHL's Arctic which uses MongoDB for storage and allows for versioning and other features. I highly recommend you check it out.

License

PyStore is licensed under the Apache License, Version 2.0. A copy of which is included in LICENSE.txt.

---

I'm very interested in your experience with PyStore. Please drop me a note with any feedback you have.

Contributions welcome!

• Ran Aroussi