GH-38558: [C++] Add support for null sort option per sort key

[Light-City]

Rationale for this change

support multi sortkey nulls first.

order by i nulls first, j, k nulls first;

The current null sorting only supports all sortkeys, not a certain sortkey, so NullPlacement is extended to the SortKey field. Since the underlying framework is very well written, when modifying this function, you only need to pass the null_placement of each SortKey in. That’s it.

What changes are included in this PR?

1.SortKey structure, NullPlacemnt transfer logic, sorting logic and Ording related, test related
2.Substriait related.
3.c_glib related.
4.SelectK related.
5.RankOptions related.

Are these changes tested?

yes, I changed the code inside vector_sort_test.cc and performed additional tests.

Are there any user-facing changes?

yes, pg database include null sorting of multiple sort keys.

• Closes: [C++][Compute] Fix null sorting of multiple sort keys #38558

[github-actions]

⚠️ GitHub issue #38558has been automatically assigned in GitHub to PR creator.

[Light-City]

Fixes #38633,
Fixes #38558

[Light-City]

This PR is ready, all tests have passed, and four of them failed seem to have nothing to do with this submission.
@kou@westonpace

[bkietz]

Please ensure that user code which still uses pc.select_k_unstable(arr, k=k, sort_keys=[("dummy", order)]) will not be broken by this change; IE most of these tests should not need to change.

[Light-City]

It seems that the default value cannot be set at present, CSortKey.null_placement=NullPlacement::AtEnd

[bkietz]

I mean that a tuple of name, order without null placement should still be accepted as a sort key. Currently IIUC passing this to _SortOptions._set_options would raise an exception.

[Light-City]

Yes, I understand your purpose. we hope that the original tuple can be processed correctly without throwing an exception.

[Light-City]

 cdef cppclass CSortKey" arrow::compute::SortKey":
CSortKey(CFieldRef target, CSortOrder order, CNullPlacement null_placement)
CFieldRef target
CSortOrder order
CNullPlacement null_placement

What I mean is that when we add a new fileld(here is null_placement) end to sortkey, this tuple is incompatible with the original one, that is, it seems that we cannot pass a default value here so that the original tuple can also be used.

our expected:

 cdef cppclass CSortKey" arrow::compute::SortKey":
CSortKey(CFieldRef target, CSortOrder order, CNullPlacement null_placement = AtEnd)
CFieldRef target
CSortOrder order
CNullPlacement null_placement

Then the following two situations can be supported

arr, k=k, sort_keys=[("dummy", order)])
arr, k=k, sort_keys=[("dummy", order, null_placement)])

[bkietz]

Removing this member is a breaking change, which should be explicitly called out in the PR description. Note that instead, we could keep this member (maybe deprecating it) as an optional default (see for example the same demotion applied to SetLookupOptions::skip_nulls in https://github.com/apache/arrow/pull/36739/files#diff-6bc7ecec6a4f7bcefc2511cde3bd809340ad0d94bb8f7cc5f4994063c798f2faR313 )

[Light-City]

When I wanted to set the deprecated field to a std::optional member, I found some conflicts while retaining the original construction, for example:

If the user calls SortOptions, use the following method:

auto null_placement = {SortKey{NullPlacement::AtEnd}, SortKey{}};
SortOptions(sort_keys, NullPlacement::AtStart)

SortOptions(std::vector<SortKey> sort_keys = {}, NullPlacement null_placement = NullPlacement::AtEnd) {
// Should I use SortKey's NullPlacement or NullPlacement::AtStart in the constructor?
}

Therefore, here I added the interface modification annotation and did not use std::optional.

[taepper]

What is the status on this?

[kou]

I close this in favor of #46926.