sqlrs

sqlrs is an in-process sql query engine modeled off duckdb in Rust

• The goal of this project is to build a embedded in-process sql query engine for OLAP workloads.
• It leverages the power of Rust ecosystem and Apache Arrow.
• It achieved columnar-vectorized execution engine.
• It will support pipeline parallelism execution in the future.

🚧 The project is actively developing a new planner V2 that inspired by DuckDB, and will replace the planner v1 in the future.🚧

SQL demo

currently, the following SQL statements are supported, execute the commands into interactive mode to test them:

• make run: run sqlrs in planner_v1
• make run_v2: run sqlrs in planner_v2

-- supported in Roadmap 0.1 (planner_v1)select first_name from employee where last_name ='Hopkins';
-- supported in Roadmap 0.2 (planner_v1)selectsum(salary+1), count(salary), max(salary) from employee where id >1;
select state, count(state), sum(salary) from employee group by state;
-- load csv table
\load csv department ./tests/csv/department.csv
\load csv employee ./tests/csv/employee.csv-- show tables
\dt
-- supported in Roadmap 0.3 (planner_v1)select id from employee order by id desc offset 2limit1;
select*from employee left join state onemployee.state=state.state_codeandstate.state_name!='California State';
-- supported in Roadmap 0.4 (planner_v1)-- explain plan tree
\explain select a from t1;
-- Heuristic Optimizer that includes rules such as: Column pruning, Predicates pushdown, Limit pushdown etc.-- supported in Roadmap 0.5 (planner_v1)-- distinctselect distinct state from employee;
selectcount(distinct(b)) from t2;
-- aliasselect a as c1 from t1 order by c1 desclimit1;
selectt.afrom t1 t wheret.b>1order byt.adesclimit1;
-- uncorrelated scalar subqueryselect t.*from (select*from t1 where a >1) t wheret.b>7;
select a, (selectmax(b) from t1) max_b from t1;
-- supported in Roadmap 0.6 (planner_v2)-- create and insert table in memorycreatetablet1(v1 int, v2 int, v3 int);
createtablet2asselect*from read_csv('t2.csv');
insert into t1 values (0, 4, 1), (1, 5, 2);
select*from t1;
-- select only expressionsselect1, 2.3, '😇', true, null;
-- pragma commands
show tables;
describe t1;
-- previous SQL statementsselect v1+1as a from t1 where a >=2;
select v1 from t1 limit2 offset 1;
-- table functionsselect*from sqlrs_tables();
select*from sqlrs_columns();
select*from read_csv('t1.csv');
select*from read_csv('t1.csv', header=>true, delim=>',');
select*from't1.csv';
-- copy
copy t1 from't1.csv' ( DELIMITER '|', HEADER false);
-- date and intervalselectdate'1998-12-01'- interval '1' month;
select interval '1' year +date'1998-12-01';

Roadmap

High level description:

• Roadmap 0.1: Build a basic SQL query on CSV storage
• Roadmap 0.2: Support aggregation operators, e2e testing framework and interactive mode
• Roadmap 0.3: Support limit, order, and join operators
• Roadmap 0.4: Introduce a Heuristic Optimizer and common optimization rules
• Roadmap 0.5: Support distinct, alias, and uncorrelated scalar subquery
• Roadmap 0.6: New planner_v2 highly inspired by DuckDB

Please see Roadmap for more information of implementation steps

Deep Dive Series Blog (in Chinese)

• On my blog
• On zhihu