Uh oh!
There was an error while loading. Please reload this page.
Add TPCH-DS planning benchmark - #9907
Conversation
| MemTable::try_new(schema, vec![]).map(Arc::new).unwrap() | ||
| } | ||
| pub fn create_tpch_schemas() -> [(String, Schema); 8] { |
There was a problem hiding this comment.
I just moved this code into test_utils.
| // --- TPC-H --- | ||
| let tpch_ctx = register_defs(SessionContext::new(), tpch_schemas()); |
There was a problem hiding this comment.
Rather than using the same context for all tests, I made separate contexts for TPCH and TPC-DS
| use arrow::datatypes::{DataType, Field, Schema}; | ||
| /// Schemas for the TPCH tables | ||
| pub fn tpch_schemas() -> Vec<TableDef> { |
| // specific language governing permissions and limitations | ||
| // under the License. | ||
| use crate::TableDef; |
There was a problem hiding this comment.
This code was moved from tpcds_planning
| Ok(()) | ||
| } | ||
There was a problem hiding this comment.
This code was just moved to test_utils
| // --- TPC-DS --- | ||
| let tpcds_ctx = register_defs(SessionContext::new(), tpcds_schemas()); |
alamb
commented
Apr 2, 2024
FYI @andygrove and @matthewmturner -- I hope this also helps improve the planning performance of DataFusion |
comphead
left a comment
There was a problem hiding this comment.
how this benchmark can be started?
alamb
commented
Apr 2, 2024
It can be run with the existing sql_planner suite, like this for example: cargo bench --bench sql_plannerTo test with just the new tests, something like cargo bench --bench sql_planner -- tpcds |
alamb
commented
Apr 3, 2024
I have a (now not) secret plan to run these benchmarks against the 37.0.0 and report how much better planning time is for 38.0.0 with the work in #9595 from @haohuaijin / @matthewmturner and #9824 from you and some of the improvements I have cooking with @peter-toth / @jayzhan211 in #9637 |
alamb
commented
Apr 3, 2024
(thank you @comphead ) |
* Move TPCH schema definition * Move schema definitions into test_util * Add tpcds planning benchmark
Which issue does this PR close?
Part of faster planning #5637
Rationale for this change
As we work on improving planning speed (e.g #9637)
it would help to have more planner benchmarks to see how we are doing
What changes are included in this PR?
Are these changes tested?
I tested them manually
Are there any user-facing changes?