test: add advanced dictionary test

[Rich-T-kid]

Which issue does this PR close?

follow up PR for #23280

Rationale for this change

see comment #23280 (comment)

What changes are included in this PR?

Adds SQL logic tests to verify that GROUP BY on dictionary-encoded columns resolves on values rather than raw dictionary key integers. The tests use UNION ALL between independently-encoded subqueries (mixing Int32/Int16/Int8 key types and Utf8/LargeUtf8 value types)

Are these changes tested?

the test cover

• Correct grouping when the same value maps to different keys in different batches
• No false merging of distinct values that happen to share the same key number
• Null handling across batches with mismatched key assignments

Are there any user-facing changes?

no

[Rich-T-kid]

cc @kosiew

[kosiew]

@Rich-T-kid
Thanks for working on this. I think the regression test is headed in the right direction, but I don't think it currently exercises the case we want to protect against. In addition, I have one small suggestion to simplify the coverage.

[kosiew]

I don't think these new regression cases are exercising dictionary grouping at the aggregation boundary.

Because the UNION inputs use different dictionary key/value types, for example Dictionary(Int32, Utf8) unioned with Dictionary(Int16, LargeUtf8) here, the physical plan coerces both branches back to LargeUtf8 before UnionExec and AggregateExec: CAST(CAST(column1 AS Dictionary(...)) AS LargeUtf8).

That means the GROUP BY sees plain strings, so an implementation that incorrectly hashed dictionary key ids would still pass these tests.

Could you construct the inputs using the same dictionary type while varying the dictionary value order instead? Alternatively, if there's another approach, could you verify through the plan or test setup that AggregateExec is actually consuming separate dictionary arrays with different dictionaries?

[Rich-T-kid]

🤔 I didn't consider that datafusion would cast them before reaching the aggregation stage. Ive updated the test to use the same key/value types while varying the order.

[Rich-T-kid]

the test in 4ca86a1
arrow_cast builds the dictionary by encoding values in first-seen order. So:

┌──────────────┬────────┬────────┐
│ batch │ key 0 │ key 1 │
├──────────────┼────────┼────────┤
│ first_batch │ 'west' │ 'east' │
├──────────────┼────────┼────────┤
│ second_batch │ 'east' │ 'west' │
└──────────────┴────────┴────────┘

Key id 0 means 'west' in the first batch but 'east' in the second. A buggy implementation that groups by raw key id would put those together:

so west (group 0) would be seen 6 times while east (group 1) would be seen 2 times. this would be incorrect.
A correct implementation looks up the actual string value for each row and groups by that, producing 4 and 4 regardless of what key id was used in each batch.

[kosiew]

@Rich-T-kid

Thanks for the iteration.

Looks 👍 to me

[Rich-T-kid]

thanks @kosiew