SQD Data Processing Crates This repo contains Rust crates for blockchain data processing, indexing, and querying. They are used in various SQD components.
Crate Description sqd-primitivesBasic types, range types, and SID (sequence ID) primitives used throughout the system sqd-datasetDataset descriptions, table schemas, and schema options for defining data structures sqd-arrayArrow array operations including building, slicing, sorting, I/O, and chunking utilities sqd-bloom-filterBloom filter implementation for arrow buffers
Crate Description sqd-dataData models for various blockchain protocols. Used in the query engine and parquet writers sqd-data-coreCore data processing utilities: chunk building, table processing, serialization, and sorting algorithms sqd-data-clientHTTP client for fetching data from SQD data sources with streaming support sqd-data-sourceAbstractions for standard and mapped data sources
Crate Description sqd-queryQuery engine used by the Worker sqd-query-exampleExample demonstrating how to use the query engine with Parquet files sqd-storageRocksDB-based storage layer with key-value store and table management sqd-polarsPolars DataFrame integration for high-performance data analysis
Crate Description sqd-archiveArchive service for ingesting and storing data to S3 with layout management, progress tracking, and Prometheus metrics sqd-bdsBig Data Service (WIP) - Cassandra-based data storage sqd-hotblocksHotblocks database with portal-like API sqd-hotblocks-retainRetention coordinator that applies Hotblocks retain points based on scheduling status updates
The "docker" workflow should be triggered manually with the following inputs:
target — one of hotblocks, hotblocks-retain, or sqd-archive.tag — a 8-byte hash of the commit. This will be the published docker tag.platforms — platforms to build for. Using only linux/amd64 instead of default values can save a lot of building time.