feat: Support tpch and tpch10 benchmark for csv format

[zhuqi-lucas]

Which issue does this PR close?

• Closes#16370

Rationale for this change

1. tpch data generate for csv format
2. Support tpch and tpch10 csv format

What changes are included in this PR?

Are these changes tested?

Yes:

./benchmarks/bench.sh data tpch***************************DataFusionBenchmarkRunner and DataGeneratorCOMMAND: data
BENCHMARK: tpch
DATA_DIR: /Users/zhuqi/arrow-datafusion/benchmarks/data
CARGO_COMMAND: cargo run --release
PREFER_HASH_JOIN: true
***************************Creating tpch dataset at ScaleFactor1in /Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1...
tbl files exist(/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/supplier.tbl exists).Expected answers exist (/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/answers/q1.out exists).parquet files exist (/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/supplier exists).
creating csv files using benchmark binary ...
Finished `release` profile [optimized] target(s) in 0.13s
Running `/Users/zhuqi/arrow-datafusion/target/release/tpch convert --input /Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1 --output /Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/csv --format csv`
Converting'/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/part.tbl' to csv files in directory '/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/csv/part'Conversion completed in28 ms
Converting'/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/supplier.tbl' to csv files in directory '/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/csv/supplier'Conversion completed in5 ms
Converting'/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/partsupp.tbl' to csv files in directory '/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/csv/partsupp'Conversion completed in43 ms
Converting'/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/customer.tbl' to csv files in directory '/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/csv/customer'Conversion completed in13 ms
Converting'/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/orders.tbl' to csv files in directory '/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/csv/orders'Conversion completed in129 ms
Converting'/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/lineitem.tbl' to csv files in directory '/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/csv/lineitem'Conversion completed in842 ms
Converting'/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/nation.tbl' to csv files in directory '/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/csv/nation'Conversion completed in0 ms
Converting'/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/region.tbl' to csv files in directory '/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1/csv/region'Conversion completed in0 ms
./benchmarks/bench.sh run tpch_csv***************************DataFusionBenchmarkScriptCOMMAND: run
BENCHMARK: tpch_csv
DATAFUSION_DIR: /Users/zhuqi/arrow-datafusion/benchmarks/..
BRANCH_NAME: issue_16370
DATA_DIR: /Users/zhuqi/arrow-datafusion/benchmarks/data
RESULTS_DIR: /Users/zhuqi/arrow-datafusion/benchmarks/results/issue_16370
CARGO_COMMAND: cargo run --release
PREFER_HASH_JOIN: true
***************************RESULTS_FILE: /Users/zhuqi/arrow-datafusion/benchmarks/results/issue_16370/tpch_sf1.jsonRunning tpch benchmark...
+ cargo run --release --bin tpch -- benchmark datafusion --iterations 5 --path /Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1 --prefer_hash_join true --format csv -o /Users/zhuqi/arrow-datafusion/benchmarks/results/issue_16370/tpch_sf1.jsonCompiling datafusion-benchmarks v48.0.0(/Users/zhuqi/arrow-datafusion/benchmarks)Finished `release` profile [optimized] target(s) in 4m 09s
Running `/Users/zhuqi/arrow-datafusion/target/release/tpch benchmark datafusion --iterations 5 --path /Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1 --prefer_hash_join true --format csv -o /Users/zhuqi/arrow-datafusion/benchmarks/results/issue_16370/tpch_sf1.json`
Running benchmarks with the following options:RunOpt{ query:None, common:CommonOpt{iterations:5,partitions:None,batch_size:None,mem_pool_type:"fair",memory_limit:None,sort_spill_reservation_bytes:None,debug:false}, path:"/Users/zhuqi/arrow-datafusion/benchmarks/data/tpch_sf1", file_format:"csv", mem_table:false, output_path:Some("/Users/zhuqi/arrow-datafusion/benchmarks/results/issue_16370/tpch_sf1.json"), disable_statistics:false, prefer_hash_join:true, sorted:false}Query1 iteration 0 took 180.5 ms and returned 4 rows
Query1 iteration 1 took 139.6 ms and returned 4 rows
Query1 iteration 2 took 140.9 ms and returned 4 rows
Query1 iteration 3 took 149.2 ms and returned 4 rows
Query1 iteration 4 took 150.5 ms and returned 4 rows
Query1 avg time:152.13 ms
Query2 iteration 0 took 107.7 ms and returned 100 rows
Query2 iteration 1 took 56.1 ms and returned 100 rows
Query2 iteration 2 took 58.8 ms and returned 100 rows
Query2 iteration 3 took 60.3 ms and returned 100 rows
Query2 iteration 4 took 59.4 ms and returned 100 rows
Query2 avg time:68.46 ms
Query3 iteration 0 took 215.4 ms and returned 10 rows
Query3 iteration 1 took 160.1 ms and returned 10 rows
Query3 iteration 2 took 159.1 ms and returned 10 rows
Query3 iteration 3 took 159.7 ms and returned 10 rows
Query3 iteration 4 took 145.5 ms and returned 10 rows
Query3 avg time:167.95 ms
Query4 iteration 0 took 136.1 ms and returned 5 rows
Query4 iteration 1 took 149.2 ms and returned 5 rows
Query4 iteration 2 took 150.4 ms and returned 5 rows
Query4 iteration 3 took 141.0 ms and returned 5 rows
Query4 iteration 4 took 146.6 ms and returned 5 rows
Query4 avg time:144.65 ms
Query5 iteration 0 took 157.7 ms and returned 5 rows
Query5 iteration 1 took 174.8 ms and returned 5 rows
Query5 iteration 2 took 161.4 ms and returned 5 rows
Query5 iteration 3 took 166.6 ms and returned 5 rows
Query5 iteration 4 took 160.1 ms and returned 5 rows
Query5 avg time:164.10 ms
Query6 iteration 0 took 107.9 ms and returned 1 rows
Query6 iteration 1 took 108.6 ms and returned 1 rows
Query6 iteration 2 took 104.4 ms and returned 1 rows
Query6 iteration 3 took 101.9 ms and returned 1 rows
Query6 iteration 4 took 105.6 ms and returned 1 rows
Query6 avg time:105.67 ms
Query7 iteration 0 took 185.4 ms and returned 4 rows
Query7 iteration 1 took 181.6 ms and returned 4 rows
Query7 iteration 2 took 195.2 ms and returned 4 rows
Query7 iteration 3 took 184.4 ms and returned 4 rows
Query7 iteration 4 took 190.6 ms and returned 4 rows
Query7 avg time:187.46 ms
Query8 iteration 0 took 173.1 ms and returned 2 rows
Query8 iteration 1 took 189.2 ms and returned 2 rows
Query8 iteration 2 took 195.0 ms and returned 2 rows
Query8 iteration 3 took 168.1 ms and returned 2 rows
Query8 iteration 4 took 178.3 ms and returned 2 rows
Query8 avg time:180.74 ms
Query9 iteration 0 took 197.3 ms and returned 175 rows
Query9 iteration 1 took 198.0 ms and returned 175 rows
Query9 iteration 2 took 209.2 ms and returned 175 rows
Query9 iteration 3 took 197.1 ms and returned 175 rows
Query9 iteration 4 took 198.7 ms and returned 175 rows
Query9 avg time:200.05 ms
Query10 iteration 0 took 151.8 ms and returned 20 rows
Query10 iteration 1 took 151.8 ms and returned 20 rows
Query10 iteration 2 took 145.3 ms and returned 20 rows
Query10 iteration 3 took 152.6 ms and returned 20 rows
Query10 iteration 4 took 146.4 ms and returned 20 rows
Query10 avg time:149.59 ms
Query11 iteration 0 took 38.7 ms and returned 1048 rows
Query11 iteration 1 took 38.2 ms and returned 1048 rows
Query11 iteration 2 took 36.3 ms and returned 1048 rows
Query11 iteration 3 took 38.1 ms and returned 1048 rows
Query11 iteration 4 took 37.6 ms and returned 1048 rows
Query11 avg time:37.76 ms
Query12 iteration 0 took 157.6 ms and returned 2 rows
Query12 iteration 1 took 160.4 ms and returned 2 rows
Query12 iteration 2 took 151.1 ms and returned 2 rows
Query12 iteration 3 took 154.6 ms and returned 2 rows
Query12 iteration 4 took 156.4 ms and returned 2 rows
Query12 avg time:156.03 ms
Query13 iteration 0 took 36.5 ms and returned 42 rows
Query13 iteration 1 took 42.3 ms and returned 42 rows
Query13 iteration 2 took 41.6 ms and returned 42 rows
Query13 iteration 3 took 40.7 ms and returned 42 rows
Query13 iteration 4 took 39.4 ms and returned 42 rows
Query13 avg time:40.10 ms
Query14 iteration 0 took 123.3 ms and returned 1 rows
Query14 iteration 1 took 113.5 ms and returned 1 rows
Query14 iteration 2 took 115.6 ms and returned 1 rows
Query14 iteration 3 took 114.0 ms and returned 1 rows
Query14 iteration 4 took 112.8 ms and returned 1 rows
Query14 avg time:115.81 ms
Query15 iteration 0 took 226.6 ms and returned 1 rows
Query15 iteration 1 took 228.9 ms and returned 1 rows
Query15 iteration 2 took 224.0 ms and returned 1 rows
Query15 iteration 3 took 215.3 ms and returned 1 rows
Query15 iteration 4 took 222.8 ms and returned 1 rows
Query15 avg time:223.50 ms
Query16 iteration 0 took 31.6 ms and returned 18314 rows
Query16 iteration 1 took 32.7 ms and returned 18314 rows
Query16 iteration 2 took 30.8 ms and returned 18314 rows
Query16 iteration 3 took 29.7 ms and returned 18314 rows
Query16 iteration 4 took 31.1 ms and returned 18314 rows
Query16 avg time:31.16 ms
Query17 iteration 0 took 232.8 ms and returned 1 rows
Query17 iteration 1 took 237.4 ms and returned 1 rows
Query17 iteration 2 took 233.0 ms and returned 1 rows
Query17 iteration 3 took 227.6 ms and returned 1 rows
Query17 iteration 4 took 224.8 ms and returned 1 rows
Query17 avg time:231.12 ms
Query18 iteration 0 took 293.1 ms and returned 57 rows
Query18 iteration 1 took 300.8 ms and returned 57 rows
Query18 iteration 2 took 288.7 ms and returned 57 rows
Query18 iteration 3 took 302.8 ms and returned 57 rows
Query18 iteration 4 took 271.0 ms and returned 57 rows
Query18 avg time:291.28 ms
Query19 iteration 0 took 125.4 ms and returned 1 rows
Query19 iteration 1 took 121.7 ms and returned 1 rows
Query19 iteration 2 took 118.0 ms and returned 1 rows
Query19 iteration 3 took 135.7 ms and returned 1 rows
Query19 iteration 4 took 117.1 ms and returned 1 rows
Query19 avg time:123.59 ms
Query20 iteration 0 took 140.2 ms and returned 186 rows
Query20 iteration 1 took 152.0 ms and returned 186 rows
Query20 iteration 2 took 145.3 ms and returned 186 rows
Query20 iteration 3 took 137.1 ms and returned 186 rows
Query20 iteration 4 took 138.7 ms and returned 186 rows
Query20 avg time:142.67 ms
Query21 iteration 0 took 416.1 ms and returned 100 rows
Query21 iteration 1 took 403.9 ms and returned 100 rows
Query21 iteration 2 took 416.0 ms and returned 100 rows
Query21 iteration 3 took 405.2 ms and returned 100 rows
Query21 iteration 4 took 417.6 ms and returned 100 rows
Query21 avg time:411.78 ms
Query22 iteration 0 took 38.1 ms and returned 7 rows
Query22 iteration 1 took 37.0 ms and returned 7 rows
Query22 iteration 2 took 36.4 ms and returned 7 rows
Query22 iteration 3 took 37.8 ms and returned 7 rows
Query22 iteration 4 took 36.5 ms and returned 7 rows
Query22 avg time:37.15 ms
+ set +x
Done

Are there any user-facing changes?

[2010YOUY01]

Tested locally and it LGTM, thanks!

[zhuqi-lucas]

Thank you @2010YOUY01 for review!

[alamb]

Nice -- thank you @zhuqi-lucas and @2010YOUY01