Use arrow IPC Stream format for spill files

[davidhewitt]

Which issue does this PR close?

• ClosesExternalSorter Fails to Spill Dictionaries #4658

Rationale for this change

The IPC Stream format allows for dictionary replacement, unlike the IPC File format. As per #4658 (comment) the File format does not offer advantages for the spill use case.

What changes are included in this PR?

Replaced the functionality in spilled sorts to write the IPC Stream format, instead of the IPC File format.

Are these changes tested?

Covered by existing tests of spill, I adapted these where necessary.

Are there any user-facing changes?

The code change is internal only; if for some reason users were inspecting the contents of spilled files without using the datafusion APIs to read them, they will find the format has changed.

[xudong963]

Maybe it's better to add tests to cover your issue?

[davidhewitt]

Good point, I will add them probably tomorrow 👍

[comphead]

Wondering if this PR also addresses partially #14078

Another thing to mention we still lack some spilling test cases like #11541

[2010YOUY01]

Thank you for the nice work. I think it's ready to go after the regression test is added.

[davidhewitt]

Rebased on main & test pushed 👍

[davidhewitt]

I confirmed that this test fails on main with the error in #4658

[davidhewitt]

#14078 looks like a similar problem but I suspect that the IPC Stream format is not much different in cost to the file format. At least with the test added here if the file format is changed again in the future, dictionary arrays will not regress.

[comphead]

Thanks @davidhewitt I think this PR is good, please check the clippy

[comphead]

thanks @davidhewitt it is a really nice first contribution

[davidhewitt]

Thanks! clippy should be happy now... 🤞

[comphead]

Thanks @davidhewitt I let the PR some time for other approvers since this is a first contribution. I do not see any objections and planning to merge this PR, thanks again