Document schema merging. by wiedld · Pull Request #17249 · apache/datafusion

wiedld · 2025-08-19T22:27:03Z

Which issue does this PR close?

Part of #12736.

Rationale for this change

Docs for how schema gets merge.
Note that we have 2 different definitions of schema "merging".

What changes are included in this PR?

Docs.

Are these changes tested?

N/A

Are there any user-facing changes?

Only docs.

…uction & modification (e.g. LP optimizers)

wiedld · 2025-08-19T22:33:14Z

+ /// - For qualified fields: both qualifier and field name must match
+ /// - For unqualified fields: only field name needs to match
+ ///
+ /// Note: the merging operation prefers the first `self` fields, and the second `other_schema` metadata.


Note that this definition of schema merging behaves a differently in precedence for fields (prefer self), vs metadata (prefer other).

wiedld · 2025-08-19T22:34:24Z

 /// Merges two optional `FieldMetadata` instances, overwriting any existing
- /// keys in `m` with keys from `n` if present
+ /// keys in `m` with keys from `n` if present.
+ ///
+ /// This function is commonly used in alias operations, particularly for literals
+ /// with metadata. When creating an alias expression, the metadata from the original
+ /// expression (such as a literal) is combined with any metadata specified on the alias.


This structure is intended for use with the aliasing of literals.
Since it has the same concept of field metadata merge, I felt it was useful to add these docs in the same PR.

wiedld · 2025-08-19T22:35:00Z

+///
+/// This function merges schemas from multiple logical plan inputs using [`DFSchema::merge`].
+/// Refer to that documentation for details on precedence and metadata handling.
 pub fn merge_schema(inputs: &[&LogicalPlan]) -> DFSchema {


This prefers us to the other docs:
https://github.com/apache/datafusion/pull/17249/files#r2286497514

Jefffrey

Some minor wording suggestions but overall looks good

alamb · 2025-08-21T18:36:31Z

I a running some benchmarks on this PR as a way to test my benchmark running script -- I don't expect that this PR changes the speed at all

alamb · 2025-08-21T18:37:48Z

🤖 ./gh_compare_branch.shBenchmark Script Running
Linux aal-dev 6.14.0-1014-gcp #15~24.04.1-Ubuntu SMP Fri Jul 25 23:26:08 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing 12736/doc-merge-schema (4494bf1) to 4bc0696diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb · 2025-08-21T19:18:42Z

🤖: Benchmark completed

Details

Comparing HEAD and 12736_doc-merge-schema
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ 12736_doc-merge-schema ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2022.01 ms │ 1897.66 ms │ +1.07x faster │
│ QQuery 1 │ 778.19 ms │ 768.65 ms │ no change │
│ QQuery 2 │ 1493.79 ms │ 1476.51 ms │ no change │
│ QQuery 3 │ 613.13 ms │ 594.20 ms │ no change │
│ QQuery 4 │ 1241.48 ms │ 1231.06 ms │ no change │
│ QQuery 5 │ 13523.95 ms │ 13430.39 ms │ no change │
│ QQuery 6 │ 2270.10 ms │ 2258.59 ms │ no change │
│ QQuery 7 │ 1651.40 ms │ 1626.77 ms │ no change │
└──────────────┴─────────────┴────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 23594.05ms │
│ Total Time (12736_doc-merge-schema) │ 23283.82ms │
│ Average Time (HEAD) │ 2949.26ms │
│ Average Time (12736_doc-merge-schema) │ 2910.48ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 7 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ 12736_doc-merge-schema ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.24 ms │ 2.19 ms │ no change │
│ QQuery 1 │ 28.12 ms │ 27.45 ms │ no change │
│ QQuery 2 │ 73.42 ms │ 75.28 ms │ no change │
│ QQuery 3 │ 88.42 ms │ 86.26 ms │ no change │
│ QQuery 4 │ 557.03 ms │ 586.36 ms │ 1.05x slower │
│ QQuery 5 │ 816.09 ms │ 808.15 ms │ no change │
│ QQuery 6 │ 2.16 ms │ 2.21 ms │ no change │
│ QQuery 7 │ 30.81 ms │ 31.54 ms │ no change │
│ QQuery 8 │ 768.57 ms │ 785.86 ms │ no change │
│ QQuery 9 │ 1056.84 ms │ 1076.10 ms │ no change │
│ QQuery 10 │ 218.93 ms │ 217.00 ms │ no change │
│ QQuery 11 │ 247.79 ms │ 248.24 ms │ no change │
│ QQuery 12 │ 791.71 ms │ 779.09 ms │ no change │
│ QQuery 13 │ 1132.47 ms │ 1123.44 ms │ no change │
│ QQuery 14 │ 733.89 ms │ 708.99 ms │ no change │
│ QQuery 15 │ 649.33 ms │ 669.14 ms │ no change │
│ QQuery 16 │ 1434.64 ms │ 1435.27 ms │ no change │
│ QQuery 17 │ 1433.91 ms │ 1428.75 ms │ no change │
│ QQuery 18 │ 2604.33 ms │ 2613.41 ms │ no change │
│ QQuery 19 │ 75.69 ms │ 77.43 ms │ no change │
│ QQuery 20 │ 1155.66 ms │ 1145.30 ms │ no change │
│ QQuery 21 │ 1288.68 ms │ 1316.12 ms │ no change │
│ QQuery 22 │ 2165.09 ms │ 2165.79 ms │ no change │
│ QQuery 23 │ 7328.76 ms │ 7372.55 ms │ no change │
│ QQuery 24 │ 212.86 ms │ 202.35 ms │ no change │
│ QQuery 25 │ 285.04 ms │ 282.05 ms │ no change │
│ QQuery 26 │ 201.48 ms │ 213.88 ms │ 1.06x slower │
│ QQuery 27 │ 1520.73 ms │ 1511.68 ms │ no change │
│ QQuery 28 │ 11652.05 ms │ 11606.96 ms │ no change │
│ QQuery 29 │ 509.54 ms │ 497.32 ms │ no change │
│ QQuery 30 │ 691.64 ms │ 718.52 ms │ no change │
│ QQuery 31 │ 697.49 ms │ 723.86 ms │ no change │
│ QQuery 32 │ 2170.73 ms │ 2193.99 ms │ no change │
│ QQuery 33 │ 2874.01 ms │ 2889.73 ms │ no change │
│ QQuery 34 │ 2906.96 ms │ 2935.44 ms │ no change │
│ QQuery 35 │ 1050.05 ms │ 1059.89 ms │ no change │
│ QQuery 36 │ 118.91 ms │ 117.97 ms │ no change │
│ QQuery 37 │ 51.54 ms │ 50.80 ms │ no change │
│ QQuery 38 │ 118.14 ms │ 117.00 ms │ no change │
│ QQuery 39 │ 191.68 ms │ 188.49 ms │ no change │
│ QQuery 40 │ 41.64 ms │ 40.99 ms │ no change │
│ QQuery 41 │ 40.25 ms │ 37.60 ms │ +1.07x faster │
│ QQuery 42 │ 31.20 ms │ 30.84 ms │ no change │
└──────────────┴─────────────┴────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 50050.51ms │
│ Total Time (12736_doc-merge-schema) │ 50201.31ms │
│ Average Time (HEAD) │ 1163.97ms │
│ Average Time (12736_doc-merge-schema) │ 1167.47ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 40 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ 12736_doc-merge-schema ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1 │ 94.79 ms │ 95.73 ms │ no change │
│ QQuery 2 │ 20.86 ms │ 20.88 ms │ no change │
│ QQuery 3 │ 32.07 ms │ 31.16 ms │ no change │
│ QQuery 4 │ 17.92 ms │ 17.86 ms │ no change │
│ QQuery 5 │ 49.51 ms │ 48.48 ms │ no change │
│ QQuery 6 │ 11.97 ms │ 11.88 ms │ no change │
│ QQuery 7 │ 84.93 ms │ 84.26 ms │ no change │
│ QQuery 8 │ 24.99 ms │ 23.87 ms │ no change │
│ QQuery 9 │ 53.41 ms │ 51.68 ms │ no change │
│ QQuery 10 │ 39.83 ms │ 38.97 ms │ no change │
│ QQuery 11 │ 37.74 ms │ 38.15 ms │ no change │
│ QQuery 12 │ 30.58 ms │ 29.90 ms │ no change │
│ QQuery 13 │ 26.01 ms │ 26.18 ms │ no change │
│ QQuery 14 │ 9.83 ms │ 9.61 ms │ no change │
│ QQuery 15 │ 18.86 ms │ 18.49 ms │ no change │
│ QQuery 16 │ 17.36 ms │ 17.22 ms │ no change │
│ QQuery 17 │ 90.11 ms │ 91.19 ms │ no change │
│ QQuery 18 │ 183.99 ms │ 176.98 ms │ no change │
│ QQuery 19 │ 23.68 ms │ 24.04 ms │ no change │
│ QQuery 20 │ 32.04 ms │ 31.81 ms │ no change │
│ QQuery 21 │ 132.98 ms │ 132.32 ms │ no change │
│ QQuery 22 │ 14.20 ms │ 13.81 ms │ no change │
└──────────────┴───────────┴────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1047.65ms │
│ Total Time (12736_doc-merge-schema) │ 1034.47ms │
│ Average Time (HEAD) │ 47.62ms │
│ Average Time (12736_doc-merge-schema) │ 47.02ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 22 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────┴───────────┘

alamb · 2025-08-21T19:37:00Z

🤖 ./gh_compare_branch.shBenchmark Script Running
Linux aal-dev 6.14.0-1014-gcp #15~24.04.1-Ubuntu SMP Fri Jul 25 23:26:08 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing 12736/doc-merge-schema (25c798a) to 4bc0696diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb · 2025-08-21T20:18:22Z

🤖: Benchmark completed

Details

Comparing HEAD and 12736_doc-merge-schema
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ 12736_doc-merge-schema ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │ 2670.92 ms │ 2603.84 ms │ no change │
│ QQuery 1 │ 1505.63 ms │ 1435.83 ms │ no change │
│ QQuery 2 │ 2718.72 ms │ 2714.18 ms │ no change │
│ QQuery 3 │ 1180.53 ms │ 1190.39 ms │ no change │
│ QQuery 4 │ 2231.94 ms │ 2231.46 ms │ no change │
│ QQuery 5 │ 27278.69 ms │ 27661.37 ms │ no change │
│ QQuery 6 │ 4161.13 ms │ 4135.29 ms │ no change │
│ QQuery 7 │ 3527.24 ms │ 3393.65 ms │ no change │
└──────────────┴─────────────┴────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 45274.80ms │
│ Total Time (12736_doc-merge-schema) │ 45366.02ms │
│ Average Time (HEAD) │ 5659.35ms │
│ Average Time (12736_doc-merge-schema) │ 5670.75ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 8 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ 12736_doc-merge-schema ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.18 ms │ 2.15 ms │ no change │
│ QQuery 1 │ 49.46 ms │ 48.04 ms │ no change │
│ QQuery 2 │ 132.94 ms │ 136.63 ms │ no change │
│ QQuery 3 │ 159.97 ms │ 156.46 ms │ no change │
│ QQuery 4 │ 1072.03 ms │ 991.56 ms │ +1.08x faster │
│ QQuery 5 │ 1561.55 ms │ 1477.74 ms │ +1.06x faster │
│ QQuery 6 │ 2.22 ms │ 2.14 ms │ no change │
│ QQuery 7 │ 54.77 ms │ 53.19 ms │ no change │
│ QQuery 8 │ 1470.38 ms │ 1402.71 ms │ no change │
│ QQuery 9 │ 1830.44 ms │ 1764.69 ms │ no change │
│ QQuery 10 │ 383.23 ms │ 382.20 ms │ no change │
│ QQuery 11 │ 435.94 ms │ 434.72 ms │ no change │
│ QQuery 12 │ 1369.31 ms │ 1342.90 ms │ no change │
│ QQuery 13 │ 2144.77 ms │ 2086.37 ms │ no change │
│ QQuery 14 │ 1279.43 ms │ 1254.94 ms │ no change │
│ QQuery 15 │ 1201.99 ms │ 1166.62 ms │ no change │
│ QQuery 16 │ 2585.63 ms │ 2609.20 ms │ no change │
│ QQuery 17 │ 2580.61 ms │ 2611.03 ms │ no change │
│ QQuery 18 │ 4744.51 ms │ 4898.22 ms │ no change │
│ QQuery 19 │ 124.25 ms │ 125.21 ms │ no change │
│ QQuery 20 │ 1935.82 ms │ 1948.13 ms │ no change │
│ QQuery 21 │ 2283.37 ms │ 2290.67 ms │ no change │
│ QQuery 22 │ 3930.27 ms │ 3914.54 ms │ no change │
│ QQuery 23 │ 14033.63 ms │ 14006.97 ms │ no change │
│ QQuery 24 │ 265.24 ms │ 256.38 ms │ no change │
│ QQuery 25 │ 510.80 ms │ 513.00 ms │ no change │
│ QQuery 26 │ 267.48 ms │ 265.96 ms │ no change │
│ QQuery 27 │ 2808.29 ms │ 2786.81 ms │ no change │
│ QQuery 28 │ 22932.67 ms │ 23018.51 ms │ no change │
│ QQuery 29 │ 980.96 ms │ 956.96 ms │ no change │
│ QQuery 30 │ 1356.65 ms │ 1300.39 ms │ no change │
│ QQuery 31 │ 1314.97 ms │ 1319.04 ms │ no change │
│ QQuery 32 │ 4188.63 ms │ 4670.85 ms │ 1.12x slower │
│ QQuery 33 │ 5375.34 ms │ 5617.15 ms │ no change │
│ QQuery 34 │ 5666.11 ms │ 5789.01 ms │ no change │
│ QQuery 35 │ 2003.03 ms │ 1966.78 ms │ no change │
│ QQuery 36 │ 120.42 ms │ 120.81 ms │ no change │
│ QQuery 37 │ 53.49 ms │ 52.20 ms │ no change │
│ QQuery 38 │ 118.98 ms │ 118.08 ms │ no change │
│ QQuery 39 │ 194.59 ms │ 192.92 ms │ no change │
│ QQuery 40 │ 44.94 ms │ 44.16 ms │ no change │
│ QQuery 41 │ 40.80 ms │ 39.87 ms │ no change │
│ QQuery 42 │ 31.16 ms │ 33.11 ms │ 1.06x slower │
└──────────────┴─────────────┴────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 93643.27ms │
│ Total Time (12736_doc-merge-schema) │ 94169.02ms │
│ Average Time (HEAD) │ 2177.75ms │
│ Average Time (12736_doc-merge-schema) │ 2189.98ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 39 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ 12736_doc-merge-schema ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1 │ 167.12 ms │ 169.50 ms │ no change │
│ QQuery 2 │ 26.76 ms │ 27.25 ms │ no change │
│ QQuery 3 │ 44.21 ms │ 43.97 ms │ no change │
│ QQuery 4 │ 26.70 ms │ 26.54 ms │ no change │
│ QQuery 5 │ 73.53 ms │ 73.30 ms │ no change │
│ QQuery 6 │ 19.98 ms │ 20.06 ms │ no change │
│ QQuery 7 │ 145.56 ms │ 139.22 ms │ no change │
│ QQuery 8 │ 32.55 ms │ 33.20 ms │ no change │
│ QQuery 9 │ 80.48 ms │ 81.26 ms │ no change │
│ QQuery 10 │ 56.22 ms │ 59.32 ms │ 1.06x slower │
│ QQuery 11 │ 41.00 ms │ 41.86 ms │ no change │
│ QQuery 12 │ 49.68 ms │ 50.24 ms │ no change │
│ QQuery 13 │ 44.29 ms │ 45.48 ms │ no change │
│ QQuery 14 │ 13.12 ms │ 13.40 ms │ no change │
│ QQuery 15 │ 23.88 ms │ 23.57 ms │ no change │
│ QQuery 16 │ 23.34 ms │ 23.80 ms │ no change │
│ QQuery 17 │ 142.40 ms │ 144.01 ms │ no change │
│ QQuery 18 │ 318.48 ms │ 319.45 ms │ no change │
│ QQuery 19 │ 36.32 ms │ 36.33 ms │ no change │
│ QQuery 20 │ 47.01 ms │ 47.42 ms │ no change │
│ QQuery 21 │ 215.75 ms │ 212.60 ms │ no change │
│ QQuery 22 │ 19.42 ms │ 19.80 ms │ no change │
└──────────────┴───────────┴────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1647.79ms │
│ Total Time (12736_doc-merge-schema) │ 1651.58ms │
│ Average Time (HEAD) │ 74.90ms │
│ Average Time (12736_doc-merge-schema) │ 75.07ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 21 │
│ Queries with Failure │ 0 │
└───────────────────────────────────────┴───────────┘

alamb · 2025-08-21T20:18:26Z

🤖 ./gh_compare_branch.shBenchmark Script Running
Linux aal-dev 6.14.0-1014-gcp #15~24.04.1-Ubuntu SMP Fri Jul 25 23:26:08 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing 12736/doc-merge-schema (25c798a) to 4bc0696diff using: tpch_topk
Results will be posted here when complete

alamb · 2025-08-21T20:18:28Z

🤖 ./gh_compare_branch_bench.shBenchmark Script Running
Linux aal-dev 6.14.0-1014-gcp #15~24.04.1-Ubuntu SMP Fri Jul 25 23:26:08 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing 12736/doc-merge-schema (25c798a) to 4bc0696diff
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=12736_doc-merge-schema
Results will be posted here when complete

alamb · 2025-08-21T21:16:27Z

🤖: Benchmark completed

Details

group 12736_doc-merge-schema main
----- ---------------------- ----
logical_aggregate_with_join 1.00 622.8±8.10µs ? ?/sec 1.00 624.4±3.33µs ? ?/sec
logical_select_all_from_1000 1.00 11.2±0.04ms ? ?/sec 1.00 11.2±0.08ms ? ?/sec
logical_select_one_from_700 1.00 411.4±6.38µs ? ?/sec 1.01 413.7±4.14µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 367.9±1.29µs ? ?/sec 1.01 370.7±1.80µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 354.2±2.19µs ? ?/sec 1.01 356.4±1.22µs ? ?/sec
physical_intersection 1.00 835.9±2.67µs ? ?/sec 1.00 837.0±5.92µs ? ?/sec
physical_join_consider_sort 1.00 1386.5±12.21µs ? ?/sec 1.00 1382.0±2.41µs ? ?/sec
physical_join_distinct 1.00 345.3±1.29µs ? ?/sec 1.00 345.8±1.30µs ? ?/sec
physical_many_self_joins 1.00 10.2±0.03ms ? ?/sec 1.00 10.2±0.03ms ? ?/sec
physical_plan_clickbench_all 1.00 189.2±1.89ms ? ?/sec 1.00 189.4±2.53ms ? ?/sec
physical_plan_clickbench_q1 1.01 2.5±0.03ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q10 1.01 3.5±0.10ms ? ?/sec 1.00 3.4±0.05ms ? ?/sec
physical_plan_clickbench_q11 1.00 3.6±0.06ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q12 1.00 3.8±0.06ms ? ?/sec 1.00 3.8±0.06ms ? ?/sec
physical_plan_clickbench_q13 1.00 3.4±0.03ms ? ?/sec 1.01 3.4±0.05ms ? ?/sec
physical_plan_clickbench_q14 1.01 3.6±0.05ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q15 1.00 3.5±0.12ms ? ?/sec 1.00 3.5±0.10ms ? ?/sec
physical_plan_clickbench_q16 1.00 3.3±0.03ms ? ?/sec 1.00 3.3±0.04ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.4±0.02ms ? ?/sec 1.00 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q18 1.01 2.9±0.03ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q19 1.00 3.8±0.06ms ? ?/sec 1.00 3.8±0.08ms ? ?/sec
physical_plan_clickbench_q2 1.02 3.0±0.02ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q20 1.00 2.6±0.02ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q21 1.02 3.0±0.09ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q22 1.00 3.6±0.03ms ? ?/sec 1.00 3.6±0.04ms ? ?/sec
physical_plan_clickbench_q23 1.02 3.9±0.05ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q24 1.01 4.4±0.04ms ? ?/sec 1.00 4.4±0.04ms ? ?/sec
physical_plan_clickbench_q25 1.01 3.2±0.02ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q26 1.02 3.0±0.03ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q27 1.02 3.2±0.05ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q28 1.00 3.9±0.04ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q29 1.01 4.6±0.09ms ? ?/sec 1.00 4.6±0.04ms ? ?/sec
physical_plan_clickbench_q3 1.01 2.9±0.03ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
physical_plan_clickbench_q30 1.02 13.3±0.11ms ? ?/sec 1.00 13.0±0.10ms ? ?/sec
physical_plan_clickbench_q31 1.01 3.9±0.03ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q32 1.01 3.9±0.05ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q33 1.02 3.4±0.04ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q34 1.01 3.1±0.03ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q35 1.00 3.2±0.03ms ? ?/sec 1.01 3.2±0.04ms ? ?/sec
physical_plan_clickbench_q36 1.00 3.9±0.03ms ? ?/sec 1.00 3.9±0.06ms ? ?/sec
physical_plan_clickbench_q37 1.02 4.0±0.06ms ? ?/sec 1.00 3.9±0.03ms ? ?/sec
physical_plan_clickbench_q38 1.00 3.9±0.03ms ? ?/sec 1.00 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q39 1.01 3.8±0.04ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q4 1.01 2.6±0.02ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q40 1.02 4.4±0.10ms ? ?/sec 1.00 4.4±0.06ms ? ?/sec
physical_plan_clickbench_q41 1.00 4.0±0.06ms ? ?/sec 1.00 3.9±0.07ms ? ?/sec
physical_plan_clickbench_q42 1.01 3.9±0.04ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q43 1.01 4.2±0.04ms ? ?/sec 1.00 4.1±0.04ms ? ?/sec
physical_plan_clickbench_q44 1.01 2.7±0.04ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_clickbench_q45 1.01 2.7±0.04ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q46 1.00 3.2±0.03ms ? ?/sec 1.00 3.2±0.05ms ? ?/sec
physical_plan_clickbench_q47 1.00 3.8±0.03ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q48 1.00 4.5±0.05ms ? ?/sec 1.01 4.5±0.09ms ? ?/sec
physical_plan_clickbench_q49 1.00 4.8±0.07ms ? ?/sec 1.00 4.8±0.06ms ? ?/sec
physical_plan_clickbench_q5 1.01 2.8±0.03ms ? ?/sec 1.00 2.8±0.04ms ? ?/sec
physical_plan_clickbench_q50 1.01 4.2±0.06ms ? ?/sec 1.00 4.2±0.05ms ? ?/sec
physical_plan_clickbench_q51 1.00 3.3±0.06ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q6 1.01 2.8±0.03ms ? ?/sec 1.00 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q7 1.02 2.6±0.09ms ? ?/sec 1.00 2.5±0.03ms ? ?/sec
physical_plan_clickbench_q8 1.01 3.5±0.03ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q9 1.01 3.3±0.05ms ? ?/sec 1.00 3.3±0.04ms ? ?/sec
physical_plan_tpcds_all 1.00 1038.9±7.37ms ? ?/sec 1.00 1040.1±7.70ms ? ?/sec
physical_plan_tpch_all 1.00 63.2±0.12ms ? ?/sec 1.00 63.2±0.26ms ? ?/sec
physical_plan_tpch_q1 1.00 2.0±0.01ms ? ?/sec 1.01 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.00 3.9±0.01ms ? ?/sec 1.00 3.9±0.01ms ? ?/sec
physical_plan_tpch_q11 1.00 3.4±0.02ms ? ?/sec 1.00 3.4±0.01ms ? ?/sec
physical_plan_tpch_q12 1.00 1832.3±29.21µs ? ?/sec 1.00 1832.2±8.61µs ? ?/sec
physical_plan_tpch_q13 1.00 1477.6±8.62µs ? ?/sec 1.00 1480.7±6.51µs ? ?/sec
physical_plan_tpch_q14 1.00 1982.7±8.62µs ? ?/sec 1.00 1982.3±6.25µs ? ?/sec
physical_plan_tpch_q16 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_tpch_q17 1.01 2.4±0.00ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_tpch_q18 1.01 2.7±0.00ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_tpch_q19 1.01 3.2±0.05ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
physical_plan_tpch_q2 1.00 5.7±0.02ms ? ?/sec 1.01 5.7±0.01ms ? ?/sec
physical_plan_tpch_q20 1.00 3.1±0.00ms ? ?/sec 1.01 3.2±0.08ms ? ?/sec
physical_plan_tpch_q21 1.00 4.1±0.01ms ? ?/sec 1.00 4.1±0.01ms ? ?/sec
physical_plan_tpch_q22 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_tpch_q3 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_tpch_q4 1.00 1520.5±7.24µs ? ?/sec 1.01 1534.6±8.20µs ? ?/sec
physical_plan_tpch_q5 1.00 3.2±0.00ms ? ?/sec 1.00 3.2±0.05ms ? ?/sec
physical_plan_tpch_q6 1.00 852.4±5.14µs ? ?/sec 1.01 861.7±6.26µs ? ?/sec
physical_plan_tpch_q7 1.00 4.4±0.05ms ? ?/sec 1.00 4.4±0.04ms ? ?/sec
physical_plan_tpch_q8 1.00 5.3±0.01ms ? ?/sec 1.00 5.3±0.02ms ? ?/sec
physical_plan_tpch_q9 1.00 4.2±0.03ms ? ?/sec 1.00 4.2±0.01ms ? ?/sec
physical_select_aggregates_from_200 1.00 16.7±0.04ms ? ?/sec 1.00 16.8±0.10ms ? ?/sec
physical_select_all_from_1000 1.00 24.6±0.10ms ? ?/sec 1.00 24.6±0.05ms ? ?/sec
physical_select_one_from_700 1.00 1065.2±4.71µs ? ?/sec 1.00 1064.4±10.93µs ? ?/sec
physical_sorted_union_orderby 1.00 41.4±0.20ms ? ?/sec 1.00 41.4±0.21ms ? ?/sec
physical_theta_join_consider_sort 1.01 1746.6±5.58µs ? ?/sec 1.00 1731.9±4.32µs ? ?/sec
physical_unnest_to_join 1.00 1306.9±4.94µs ? ?/sec 1.00 1301.7±11.85µs ? ?/sec
with_param_values_many_columns 1.00 142.4±1.11µs ? ?/sec 1.00 142.8±1.60µs ? ?/sec

alamb

Thank you @wiedld and @Jefffrey

wiedld added 3 commits August 19, 2025 15:21

chore: add docs explaining FieldMetadata::merge_options
5fe5d62

chore: document DFSchema::merge, which is used in logical plan constr…
35f89bc
…uction & modification (e.g. LP optimizers)

chore: merge_schema utils method
4494bf1

github-actionsBot added logical-expr Logical plan and expressions common Related to common crate labels Aug 19, 2025

wiedld commented Aug 19, 2025
View reviewed changes

wiedld mentioned this pull request Aug 19, 2025
Document Schema metadata expectations #12736
Open

wiedld marked this pull request as ready for review August 19, 2025 22:43

Jefffrey approved these changes Aug 20, 2025
View reviewed changes

Comment threaddatafusion/common/src/dfschema.rs Outdated
Comment threaddatafusion/common/src/dfschema.rs Outdated

chore: clarify wording
25c798a

This comment was marked as outdated.
Sign in to view

alamb approved these changes Aug 22, 2025
View reviewed changes

alamb merged commit 2f51d5f into apache:mainAug 22, 2025
27 checks passed

alamb added the documentation Improvements or additions to documentation label Aug 22, 2025

Uh oh!

Conversation

wiedld commented Aug 19, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

wiedldAug 19, 2025

Choose a reason for hiding this comment

Uh oh!

wiedldAug 19, 2025

Choose a reason for hiding this comment

Uh oh!

wiedldAug 19, 2025

Choose a reason for hiding this comment

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alamb commented Aug 21, 2025

Uh oh!

alamb commented Aug 21, 2025

Uh oh!

alamb commented Aug 21, 2025

Uh oh!

This comment was marked as outdated.

alamb commented Aug 21, 2025

Uh oh!

alamb commented Aug 21, 2025

Uh oh!

alamb commented Aug 21, 2025

Uh oh!

alamb commented Aug 21, 2025

Uh oh!

alamb commented Aug 21, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants