GH-35786: [C++] Add pairwise_diff function by js8544 · Pull Request #35787 · apache/arrow

js8544 · 2023-05-26T10:53:34Z

Rationale for this change

Add a pairwise_diff function similar to pandas' Series.Diff, the function computes the first order difference of an array.

What changes are included in this PR?

I followed these instructions. The function is implemented for numerical, temporal and decimal types. Chuck arrays are not yet supported.

Are these changes tested?

Yes. They are tested in vector_pairwise_test.cc and in python/pyarrow/tests/compute.py.

Are there any user-facing changes?

Yes, and docs are also updated in this PR.

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:53:56Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2023-05-26T10:54:20Z

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:54:23Z

⚠️ GitHub issue #35786has been automatically assigned in GitHub to PR creator.

js8544 · 2023-06-20T02:26:31Z

@lidavidm @pitrou @westonpace Hi guys, would any of you mind taking a look at this PR? It's been sitting around for almost a month. Thanks in advance!

lidavidm

Seems reasonable to me

js8544 · 2023-06-26T09:10:11Z

@pitrou @bkietz Would you mind having some extra look at this PR? Thanks!

bkietz

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

js8544 · 2023-06-28T09:15:06Z

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

Hi @bkietz, do you mean something like this:

ARROW_ASSIGN_OR_RAISE(auto subtract_func, registry->GetFunction("subtract"));
for (constauto& type : types) {
ARROW_ASSIGN_OR_RAISE(auto kernel,
subtract_func->DispatchExact({type, type}));
// reuse kernel's exec and signature
}

bkietz · 2023-06-28T11:47:33Z

@js8544 yes, exactly. I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

(sorry for the spurious close; bumped the wrong button)

js8544 · 2023-06-28T11:52:27Z

I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

That sounds better indeed! I'll try this now.

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

js8544 · 2023-06-28T14:53:41Z

@bkietz I've changed it to looping over subtract's kernels and wrapping their signature and kernel exec.

bkietz

This is looking great, thanks for refactoring!
Just a few nits

bkietz · 2023-06-28T16:58:02Z

+ auto offset = abs(periods);
+ offset = std::min(offset, input.length);
+ auto exec_length = input.length - offset;
+ // prepare bitmap
+ auto null_start = periods > 0 ? 0 : exec_length;
+ auto non_null_start = periods > 0 ? offset : 0;


Nit: could we call these two regions "computed" and "margin"? I think that'll be more obvious for the next maintainer, and it'd help if we move all the start/length calculation to this preamble too
Suggested change
auto offset = abs(periods);
offset = std::min(offset, input.length);
auto exec_length = input.length - offset;
// prepare bitmap
auto null_start = periods > 0 ? 0 : exec_length;
auto non_null_start = periods > 0 ? offset : 0;
// We only compute values in the region where the input-with-offset overlaps
// the original input. The margin where these do not overlap gets filled with null.
auto margin_length = std::min(abs(periods), input.length);
auto computed_length = input.length - margin_length;
auto margin_start = periods > 0 ? 0 : computed_length;
auto left_start = periods > 0 ? margin_length : 0;
auto right_start = periods > 0 ? 0 : margin_length;
// ...
// prepare bitmap

Updated. It's much clearer now. thanks for the suggestion!

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

bkietz

LGTM, thanks!

bkietz · 2023-06-29T12:41:39Z

macOs glib&python failures are network failures while communicating with brew https://github.com/apache/arrow/actions/runs/5408688637/jobs/9835499198?pr=35787

macOS c++ failure is a known issue #36329

ubuntu c++ failure is an s3 flake https://github.com/apache/arrow/actions/runs/5408688635/jobs/9828017525?pr=35787#step:7:4892

R test failure is a known issue #36346

conbench-apache-arrow · 2023-07-04T09:14:50Z

Conbench analyzed the 6 benchmark runs on commit 26c25d1b.

There were 4 benchmark results indicating a performance regression:

Commit Run on arm64-m6g-linux-compute at 2023-06-29 14:28:22Z
- params=1048576/100/min_time:1.000, source=cpp-micro, suite=arrow-compute-vector-partition-benchmark
Commit Run on ursa-thinkcentre-m75q at 2023-07-02 13:25:33Z
- params=1048576/10000, source=cpp-micro, suite=arrow-acero-aggregate-benchmark
and 2 more (see the report linked below)

The full Conbench report has more details.

js8544 · 2023-05-26T10:53:34Z

Rationale for this change

Add a pairwise_diff function similar to pandas' Series.Diff, the function computes the first order difference of an array.

What changes are included in this PR?

I followed these instructions. The function is implemented for numerical, temporal and decimal types. Chuck arrays are not yet supported.

Are these changes tested?

Yes. They are tested in vector_pairwise_test.cc and in python/pyarrow/tests/compute.py.

Are there any user-facing changes?

Yes, and docs are also updated in this PR.

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:53:56Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2023-05-26T10:54:20Z

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:54:23Z

⚠️ GitHub issue #35786has been automatically assigned in GitHub to PR creator.

js8544 · 2023-06-20T02:26:31Z

@lidavidm @pitrou @westonpace Hi guys, would any of you mind taking a look at this PR? It's been sitting around for almost a month. Thanks in advance!

lidavidm

Seems reasonable to me

js8544 · 2023-06-26T09:10:11Z

@pitrou @bkietz Would you mind having some extra look at this PR? Thanks!

bkietz

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

js8544 · 2023-06-28T09:15:06Z

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

Hi @bkietz, do you mean something like this:

ARROW_ASSIGN_OR_RAISE(auto subtract_func, registry->GetFunction("subtract"));
for (constauto& type : types) {
ARROW_ASSIGN_OR_RAISE(auto kernel,
subtract_func->DispatchExact({type, type}));
// reuse kernel's exec and signature
}

bkietz · 2023-06-28T11:47:33Z

@js8544 yes, exactly. I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

(sorry for the spurious close; bumped the wrong button)

js8544 · 2023-06-28T11:52:27Z

I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

That sounds better indeed! I'll try this now.

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

js8544 · 2023-06-28T14:53:41Z

@bkietz I've changed it to looping over subtract's kernels and wrapping their signature and kernel exec.

bkietz

This is looking great, thanks for refactoring!
Just a few nits

bkietz · 2023-06-28T16:58:02Z

+ auto offset = abs(periods);
+ offset = std::min(offset, input.length);
+ auto exec_length = input.length - offset;
+ // prepare bitmap
+ auto null_start = periods > 0 ? 0 : exec_length;
+ auto non_null_start = periods > 0 ? offset : 0;


Nit: could we call these two regions "computed" and "margin"? I think that'll be more obvious for the next maintainer, and it'd help if we move all the start/length calculation to this preamble too
Suggested change
auto offset = abs(periods);
offset = std::min(offset, input.length);
auto exec_length = input.length - offset;
// prepare bitmap
auto null_start = periods > 0 ? 0 : exec_length;
auto non_null_start = periods > 0 ? offset : 0;
// We only compute values in the region where the input-with-offset overlaps
// the original input. The margin where these do not overlap gets filled with null.
auto margin_length = std::min(abs(periods), input.length);
auto computed_length = input.length - margin_length;
auto margin_start = periods > 0 ? 0 : computed_length;
auto left_start = periods > 0 ? margin_length : 0;
auto right_start = periods > 0 ? 0 : margin_length;
// ...
// prepare bitmap

Updated. It's much clearer now. thanks for the suggestion!

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

bkietz

LGTM, thanks!

bkietz · 2023-06-29T12:41:39Z

macOs glib&python failures are network failures while communicating with brew https://github.com/apache/arrow/actions/runs/5408688637/jobs/9835499198?pr=35787

macOS c++ failure is a known issue #36329

ubuntu c++ failure is an s3 flake https://github.com/apache/arrow/actions/runs/5408688635/jobs/9828017525?pr=35787#step:7:4892

R test failure is a known issue #36346

conbench-apache-arrow · 2023-07-04T09:14:50Z

Conbench analyzed the 6 benchmark runs on commit 26c25d1b.

There were 4 benchmark results indicating a performance regression:

Commit Run on arm64-m6g-linux-compute at 2023-06-29 14:28:22Z
- params=1048576/100/min_time:1.000, source=cpp-micro, suite=arrow-compute-vector-partition-benchmark
Commit Run on ursa-thinkcentre-m75q at 2023-07-02 13:25:33Z
- params=1048576/10000, source=cpp-micro, suite=arrow-acero-aggregate-benchmark
and 2 more (see the report linked below)

The full Conbench report has more details.

js8544 · 2023-05-26T10:53:34Z

Rationale for this change

Add a pairwise_diff function similar to pandas' Series.Diff, the function computes the first order difference of an array.

What changes are included in this PR?

I followed these instructions. The function is implemented for numerical, temporal and decimal types. Chuck arrays are not yet supported.

Are these changes tested?

Yes. They are tested in vector_pairwise_test.cc and in python/pyarrow/tests/compute.py.

Are there any user-facing changes?

Yes, and docs are also updated in this PR.

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:53:56Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2023-05-26T10:54:20Z

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:54:23Z

⚠️ GitHub issue #35786has been automatically assigned in GitHub to PR creator.

js8544 · 2023-06-20T02:26:31Z

@lidavidm @pitrou @westonpace Hi guys, would any of you mind taking a look at this PR? It's been sitting around for almost a month. Thanks in advance!

lidavidm

Seems reasonable to me

js8544 · 2023-06-26T09:10:11Z

@pitrou @bkietz Would you mind having some extra look at this PR? Thanks!

bkietz

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

js8544 · 2023-06-28T09:15:06Z

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

Hi @bkietz, do you mean something like this:

ARROW_ASSIGN_OR_RAISE(auto subtract_func, registry->GetFunction("subtract"));
for (constauto& type : types) {
ARROW_ASSIGN_OR_RAISE(auto kernel,
subtract_func->DispatchExact({type, type}));
// reuse kernel's exec and signature
}

bkietz · 2023-06-28T11:47:33Z

@js8544 yes, exactly. I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

(sorry for the spurious close; bumped the wrong button)

js8544 · 2023-06-28T11:52:27Z

I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

That sounds better indeed! I'll try this now.

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

js8544 · 2023-06-28T14:53:41Z

@bkietz I've changed it to looping over subtract's kernels and wrapping their signature and kernel exec.

bkietz

This is looking great, thanks for refactoring!
Just a few nits

bkietz · 2023-06-28T16:58:02Z

+ auto offset = abs(periods);
+ offset = std::min(offset, input.length);
+ auto exec_length = input.length - offset;
+ // prepare bitmap
+ auto null_start = periods > 0 ? 0 : exec_length;
+ auto non_null_start = periods > 0 ? offset : 0;


Nit: could we call these two regions "computed" and "margin"? I think that'll be more obvious for the next maintainer, and it'd help if we move all the start/length calculation to this preamble too
Suggested change
auto offset = abs(periods);
offset = std::min(offset, input.length);
auto exec_length = input.length - offset;
// prepare bitmap
auto null_start = periods > 0 ? 0 : exec_length;
auto non_null_start = periods > 0 ? offset : 0;
// We only compute values in the region where the input-with-offset overlaps
// the original input. The margin where these do not overlap gets filled with null.
auto margin_length = std::min(abs(periods), input.length);
auto computed_length = input.length - margin_length;
auto margin_start = periods > 0 ? 0 : computed_length;
auto left_start = periods > 0 ? margin_length : 0;
auto right_start = periods > 0 ? 0 : margin_length;
// ...
// prepare bitmap

Updated. It's much clearer now. thanks for the suggestion!

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

bkietz

LGTM, thanks!

bkietz · 2023-06-29T12:41:39Z

macOs glib&python failures are network failures while communicating with brew https://github.com/apache/arrow/actions/runs/5408688637/jobs/9835499198?pr=35787

macOS c++ failure is a known issue #36329

ubuntu c++ failure is an s3 flake https://github.com/apache/arrow/actions/runs/5408688635/jobs/9828017525?pr=35787#step:7:4892

R test failure is a known issue #36346

conbench-apache-arrow · 2023-07-04T09:14:50Z

Conbench analyzed the 6 benchmark runs on commit 26c25d1b.

There were 4 benchmark results indicating a performance regression:

Commit Run on arm64-m6g-linux-compute at 2023-06-29 14:28:22Z
- params=1048576/100/min_time:1.000, source=cpp-micro, suite=arrow-compute-vector-partition-benchmark
Commit Run on ursa-thinkcentre-m75q at 2023-07-02 13:25:33Z
- params=1048576/10000, source=cpp-micro, suite=arrow-acero-aggregate-benchmark
and 2 more (see the report linked below)

The full Conbench report has more details.

js8544 · 2023-05-26T10:53:34Z

Rationale for this change

Add a pairwise_diff function similar to pandas' Series.Diff, the function computes the first order difference of an array.

What changes are included in this PR?

I followed these instructions. The function is implemented for numerical, temporal and decimal types. Chuck arrays are not yet supported.

Are these changes tested?

Yes. They are tested in vector_pairwise_test.cc and in python/pyarrow/tests/compute.py.

Are there any user-facing changes?

Yes, and docs are also updated in this PR.

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:53:56Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2023-05-26T10:54:20Z

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:54:23Z

⚠️ GitHub issue #35786has been automatically assigned in GitHub to PR creator.

js8544 · 2023-06-20T02:26:31Z

@lidavidm @pitrou @westonpace Hi guys, would any of you mind taking a look at this PR? It's been sitting around for almost a month. Thanks in advance!

lidavidm

Seems reasonable to me

js8544 · 2023-06-26T09:10:11Z

@pitrou @bkietz Would you mind having some extra look at this PR? Thanks!

bkietz

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

js8544 · 2023-06-28T09:15:06Z

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

Hi @bkietz, do you mean something like this:

ARROW_ASSIGN_OR_RAISE(auto subtract_func, registry->GetFunction("subtract"));
for (constauto& type : types) {
ARROW_ASSIGN_OR_RAISE(auto kernel,
subtract_func->DispatchExact({type, type}));
// reuse kernel's exec and signature
}

bkietz · 2023-06-28T11:47:33Z

@js8544 yes, exactly. I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

(sorry for the spurious close; bumped the wrong button)

js8544 · 2023-06-28T11:52:27Z

I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

That sounds better indeed! I'll try this now.

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

js8544 · 2023-06-28T14:53:41Z

@bkietz I've changed it to looping over subtract's kernels and wrapping their signature and kernel exec.

bkietz

This is looking great, thanks for refactoring!
Just a few nits

bkietz · 2023-06-28T16:58:02Z

+ auto offset = abs(periods);
+ offset = std::min(offset, input.length);
+ auto exec_length = input.length - offset;
+ // prepare bitmap
+ auto null_start = periods > 0 ? 0 : exec_length;
+ auto non_null_start = periods > 0 ? offset : 0;


Nit: could we call these two regions "computed" and "margin"? I think that'll be more obvious for the next maintainer, and it'd help if we move all the start/length calculation to this preamble too
Suggested change
auto offset = abs(periods);
offset = std::min(offset, input.length);
auto exec_length = input.length - offset;
// prepare bitmap
auto null_start = periods > 0 ? 0 : exec_length;
auto non_null_start = periods > 0 ? offset : 0;
// We only compute values in the region where the input-with-offset overlaps
// the original input. The margin where these do not overlap gets filled with null.
auto margin_length = std::min(abs(periods), input.length);
auto computed_length = input.length - margin_length;
auto margin_start = periods > 0 ? 0 : computed_length;
auto left_start = periods > 0 ? margin_length : 0;
auto right_start = periods > 0 ? 0 : margin_length;
// ...
// prepare bitmap

Updated. It's much clearer now. thanks for the suggestion!

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

bkietz

LGTM, thanks!

bkietz · 2023-06-29T12:41:39Z

macOs glib&python failures are network failures while communicating with brew https://github.com/apache/arrow/actions/runs/5408688637/jobs/9835499198?pr=35787

macOS c++ failure is a known issue #36329

ubuntu c++ failure is an s3 flake https://github.com/apache/arrow/actions/runs/5408688635/jobs/9828017525?pr=35787#step:7:4892

R test failure is a known issue #36346

conbench-apache-arrow · 2023-07-04T09:14:50Z

Conbench analyzed the 6 benchmark runs on commit 26c25d1b.

There were 4 benchmark results indicating a performance regression:

Commit Run on arm64-m6g-linux-compute at 2023-06-29 14:28:22Z
- params=1048576/100/min_time:1.000, source=cpp-micro, suite=arrow-compute-vector-partition-benchmark
Commit Run on ursa-thinkcentre-m75q at 2023-07-02 13:25:33Z
- params=1048576/10000, source=cpp-micro, suite=arrow-acero-aggregate-benchmark
and 2 more (see the report linked below)

The full Conbench report has more details.

js8544 · 2023-05-26T10:53:34Z

Rationale for this change

Add a pairwise_diff function similar to pandas' Series.Diff, the function computes the first order difference of an array.

What changes are included in this PR?

I followed these instructions. The function is implemented for numerical, temporal and decimal types. Chuck arrays are not yet supported.

Are these changes tested?

Yes. They are tested in vector_pairwise_test.cc and in python/pyarrow/tests/compute.py.

Are there any user-facing changes?

Yes, and docs are also updated in this PR.

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:53:56Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2023-05-26T10:54:20Z

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:54:23Z

⚠️ GitHub issue #35786has been automatically assigned in GitHub to PR creator.

js8544 · 2023-06-20T02:26:31Z

@lidavidm @pitrou @westonpace Hi guys, would any of you mind taking a look at this PR? It's been sitting around for almost a month. Thanks in advance!

lidavidm

Seems reasonable to me

js8544 · 2023-06-26T09:10:11Z

@pitrou @bkietz Would you mind having some extra look at this PR? Thanks!

bkietz

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

js8544 · 2023-06-28T09:15:06Z

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

Hi @bkietz, do you mean something like this:

ARROW_ASSIGN_OR_RAISE(auto subtract_func, registry->GetFunction("subtract"));
for (constauto& type : types) {
ARROW_ASSIGN_OR_RAISE(auto kernel,
subtract_func->DispatchExact({type, type}));
// reuse kernel's exec and signature
}

bkietz · 2023-06-28T11:47:33Z

@js8544 yes, exactly. I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

(sorry for the spurious close; bumped the wrong button)

js8544 · 2023-06-28T11:52:27Z

I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

That sounds better indeed! I'll try this now.

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

js8544 · 2023-06-28T14:53:41Z

@bkietz I've changed it to looping over subtract's kernels and wrapping their signature and kernel exec.

bkietz

This is looking great, thanks for refactoring!
Just a few nits

bkietz · 2023-06-28T16:58:02Z

+ auto offset = abs(periods);
+ offset = std::min(offset, input.length);
+ auto exec_length = input.length - offset;
+ // prepare bitmap
+ auto null_start = periods > 0 ? 0 : exec_length;
+ auto non_null_start = periods > 0 ? offset : 0;


Nit: could we call these two regions "computed" and "margin"? I think that'll be more obvious for the next maintainer, and it'd help if we move all the start/length calculation to this preamble too
Suggested change
auto offset = abs(periods);
offset = std::min(offset, input.length);
auto exec_length = input.length - offset;
// prepare bitmap
auto null_start = periods > 0 ? 0 : exec_length;
auto non_null_start = periods > 0 ? offset : 0;
// We only compute values in the region where the input-with-offset overlaps
// the original input. The margin where these do not overlap gets filled with null.
auto margin_length = std::min(abs(periods), input.length);
auto computed_length = input.length - margin_length;
auto margin_start = periods > 0 ? 0 : computed_length;
auto left_start = periods > 0 ? margin_length : 0;
auto right_start = periods > 0 ? 0 : margin_length;
// ...
// prepare bitmap

Updated. It's much clearer now. thanks for the suggestion!

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

bkietz

LGTM, thanks!

bkietz · 2023-06-29T12:41:39Z

macOs glib&python failures are network failures while communicating with brew https://github.com/apache/arrow/actions/runs/5408688637/jobs/9835499198?pr=35787

macOS c++ failure is a known issue #36329

ubuntu c++ failure is an s3 flake https://github.com/apache/arrow/actions/runs/5408688635/jobs/9828017525?pr=35787#step:7:4892

R test failure is a known issue #36346

conbench-apache-arrow · 2023-07-04T09:14:50Z

Conbench analyzed the 6 benchmark runs on commit 26c25d1b.

There were 4 benchmark results indicating a performance regression:

Commit Run on arm64-m6g-linux-compute at 2023-06-29 14:28:22Z
- params=1048576/100/min_time:1.000, source=cpp-micro, suite=arrow-compute-vector-partition-benchmark
Commit Run on ursa-thinkcentre-m75q at 2023-07-02 13:25:33Z
- params=1048576/10000, source=cpp-micro, suite=arrow-acero-aggregate-benchmark
and 2 more (see the report linked below)

The full Conbench report has more details.

js8544 · 2023-05-26T10:53:34Z

Rationale for this change

Add a pairwise_diff function similar to pandas' Series.Diff, the function computes the first order difference of an array.

What changes are included in this PR?

I followed these instructions. The function is implemented for numerical, temporal and decimal types. Chuck arrays are not yet supported.

Are these changes tested?

Yes. They are tested in vector_pairwise_test.cc and in python/pyarrow/tests/compute.py.

Are there any user-facing changes?

Yes, and docs are also updated in this PR.

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:53:56Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2023-05-26T10:54:20Z

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:54:23Z

⚠️ GitHub issue #35786has been automatically assigned in GitHub to PR creator.

js8544 · 2023-06-20T02:26:31Z

@lidavidm @pitrou @westonpace Hi guys, would any of you mind taking a look at this PR? It's been sitting around for almost a month. Thanks in advance!

lidavidm

Seems reasonable to me

js8544 · 2023-06-26T09:10:11Z

@pitrou @bkietz Would you mind having some extra look at this PR? Thanks!

bkietz

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

js8544 · 2023-06-28T09:15:06Z

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

Hi @bkietz, do you mean something like this:

ARROW_ASSIGN_OR_RAISE(auto subtract_func, registry->GetFunction("subtract"));
for (constauto& type : types) {
ARROW_ASSIGN_OR_RAISE(auto kernel,
subtract_func->DispatchExact({type, type}));
// reuse kernel's exec and signature
}

bkietz · 2023-06-28T11:47:33Z

@js8544 yes, exactly. I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

(sorry for the spurious close; bumped the wrong button)

js8544 · 2023-06-28T11:52:27Z

I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

That sounds better indeed! I'll try this now.

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

js8544 · 2023-06-28T14:53:41Z

@bkietz I've changed it to looping over subtract's kernels and wrapping their signature and kernel exec.

bkietz

This is looking great, thanks for refactoring!
Just a few nits

bkietz · 2023-06-28T16:58:02Z

+ auto offset = abs(periods);
+ offset = std::min(offset, input.length);
+ auto exec_length = input.length - offset;
+ // prepare bitmap
+ auto null_start = periods > 0 ? 0 : exec_length;
+ auto non_null_start = periods > 0 ? offset : 0;


Nit: could we call these two regions "computed" and "margin"? I think that'll be more obvious for the next maintainer, and it'd help if we move all the start/length calculation to this preamble too
Suggested change
auto offset = abs(periods);
offset = std::min(offset, input.length);
auto exec_length = input.length - offset;
// prepare bitmap
auto null_start = periods > 0 ? 0 : exec_length;
auto non_null_start = periods > 0 ? offset : 0;
// We only compute values in the region where the input-with-offset overlaps
// the original input. The margin where these do not overlap gets filled with null.
auto margin_length = std::min(abs(periods), input.length);
auto computed_length = input.length - margin_length;
auto margin_start = periods > 0 ? 0 : computed_length;
auto left_start = periods > 0 ? margin_length : 0;
auto right_start = periods > 0 ? 0 : margin_length;
// ...
// prepare bitmap

Updated. It's much clearer now. thanks for the suggestion!

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

bkietz

LGTM, thanks!

bkietz · 2023-06-29T12:41:39Z

macOs glib&python failures are network failures while communicating with brew https://github.com/apache/arrow/actions/runs/5408688637/jobs/9835499198?pr=35787

macOS c++ failure is a known issue #36329

ubuntu c++ failure is an s3 flake https://github.com/apache/arrow/actions/runs/5408688635/jobs/9828017525?pr=35787#step:7:4892

R test failure is a known issue #36346

conbench-apache-arrow · 2023-07-04T09:14:50Z

Conbench analyzed the 6 benchmark runs on commit 26c25d1b.

There were 4 benchmark results indicating a performance regression:

Commit Run on arm64-m6g-linux-compute at 2023-06-29 14:28:22Z
- params=1048576/100/min_time:1.000, source=cpp-micro, suite=arrow-compute-vector-partition-benchmark
Commit Run on ursa-thinkcentre-m75q at 2023-07-02 13:25:33Z
- params=1048576/10000, source=cpp-micro, suite=arrow-acero-aggregate-benchmark
and 2 more (see the report linked below)

The full Conbench report has more details.

js8544 · 2023-05-26T10:53:34Z

Rationale for this change

Add a pairwise_diff function similar to pandas' Series.Diff, the function computes the first order difference of an array.

What changes are included in this PR?

I followed these instructions. The function is implemented for numerical, temporal and decimal types. Chuck arrays are not yet supported.

Are these changes tested?

Yes. They are tested in vector_pairwise_test.cc and in python/pyarrow/tests/compute.py.

Are there any user-facing changes?

Yes, and docs are also updated in this PR.

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:53:56Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2023-05-26T10:54:20Z

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:54:23Z

⚠️ GitHub issue #35786has been automatically assigned in GitHub to PR creator.

js8544 · 2023-06-20T02:26:31Z

@lidavidm @pitrou @westonpace Hi guys, would any of you mind taking a look at this PR? It's been sitting around for almost a month. Thanks in advance!

lidavidm

Seems reasonable to me

js8544 · 2023-06-26T09:10:11Z

@pitrou @bkietz Would you mind having some extra look at this PR? Thanks!

bkietz

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

js8544 · 2023-06-28T09:15:06Z

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

Hi @bkietz, do you mean something like this:

ARROW_ASSIGN_OR_RAISE(auto subtract_func, registry->GetFunction("subtract"));
for (constauto& type : types) {
ARROW_ASSIGN_OR_RAISE(auto kernel,
subtract_func->DispatchExact({type, type}));
// reuse kernel's exec and signature
}

bkietz · 2023-06-28T11:47:33Z

@js8544 yes, exactly. I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

(sorry for the spurious close; bumped the wrong button)

js8544 · 2023-06-28T11:52:27Z

I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

That sounds better indeed! I'll try this now.

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

js8544 · 2023-06-28T14:53:41Z

@bkietz I've changed it to looping over subtract's kernels and wrapping their signature and kernel exec.

bkietz

This is looking great, thanks for refactoring!
Just a few nits

bkietz · 2023-06-28T16:58:02Z

+ auto offset = abs(periods);
+ offset = std::min(offset, input.length);
+ auto exec_length = input.length - offset;
+ // prepare bitmap
+ auto null_start = periods > 0 ? 0 : exec_length;
+ auto non_null_start = periods > 0 ? offset : 0;


Nit: could we call these two regions "computed" and "margin"? I think that'll be more obvious for the next maintainer, and it'd help if we move all the start/length calculation to this preamble too
Suggested change
auto offset = abs(periods);
offset = std::min(offset, input.length);
auto exec_length = input.length - offset;
// prepare bitmap
auto null_start = periods > 0 ? 0 : exec_length;
auto non_null_start = periods > 0 ? offset : 0;
// We only compute values in the region where the input-with-offset overlaps
// the original input. The margin where these do not overlap gets filled with null.
auto margin_length = std::min(abs(periods), input.length);
auto computed_length = input.length - margin_length;
auto margin_start = periods > 0 ? 0 : computed_length;
auto left_start = periods > 0 ? margin_length : 0;
auto right_start = periods > 0 ? 0 : margin_length;
// ...
// prepare bitmap

Updated. It's much clearer now. thanks for the suggestion!

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

bkietz

LGTM, thanks!

bkietz · 2023-06-29T12:41:39Z

macOs glib&python failures are network failures while communicating with brew https://github.com/apache/arrow/actions/runs/5408688637/jobs/9835499198?pr=35787

macOS c++ failure is a known issue #36329

ubuntu c++ failure is an s3 flake https://github.com/apache/arrow/actions/runs/5408688635/jobs/9828017525?pr=35787#step:7:4892

R test failure is a known issue #36346

conbench-apache-arrow · 2023-07-04T09:14:50Z

Conbench analyzed the 6 benchmark runs on commit 26c25d1b.

There were 4 benchmark results indicating a performance regression:

Commit Run on arm64-m6g-linux-compute at 2023-06-29 14:28:22Z
- params=1048576/100/min_time:1.000, source=cpp-micro, suite=arrow-compute-vector-partition-benchmark
Commit Run on ursa-thinkcentre-m75q at 2023-07-02 13:25:33Z
- params=1048576/10000, source=cpp-micro, suite=arrow-acero-aggregate-benchmark
and 2 more (see the report linked below)

The full Conbench report has more details.

js8544 · 2023-05-26T10:53:34Z

Rationale for this change

Add a pairwise_diff function similar to pandas' Series.Diff, the function computes the first order difference of an array.

What changes are included in this PR?

I followed these instructions. The function is implemented for numerical, temporal and decimal types. Chuck arrays are not yet supported.

Are these changes tested?

Yes. They are tested in vector_pairwise_test.cc and in python/pyarrow/tests/compute.py.

Are there any user-facing changes?

Yes, and docs are also updated in this PR.

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:53:56Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2023-05-26T10:54:20Z

Closes: [C++] Add pairwise_diff function #35786

github-actions · 2023-05-26T10:54:23Z

⚠️ GitHub issue #35786has been automatically assigned in GitHub to PR creator.

js8544 · 2023-06-20T02:26:31Z

@lidavidm @pitrou @westonpace Hi guys, would any of you mind taking a look at this PR? It's been sitting around for almost a month. Thanks in advance!

lidavidm

Seems reasonable to me

js8544 · 2023-06-26T09:10:11Z

@pitrou @bkietz Would you mind having some extra look at this PR? Thanks!

bkietz

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

js8544 · 2023-06-28T09:15:06Z

This mostly looks good to me. However, in light of the strong relationship between this Function and subtract I'd prefer to see more reuse of that function's existing logic. For example when retrieving the output type for pairwise_diff($in_type), couldn't we just as easily call subtract's DispatchExact with ($in_type, $in_type)? And instead of referencing Subtract's Ops directly, couldn't we use the kernel retrieved from subtract's DispatchExact- just passing the relevant slices of the input as the arguments to the subtract kernel? I think this would have negligible impact on performance and would greatly reduce the future maintenance burden for this function, since any new types added to subtract will then automatically be supported by pairwise_diff.

Hi @bkietz, do you mean something like this:

ARROW_ASSIGN_OR_RAISE(auto subtract_func, registry->GetFunction("subtract"));
for (constauto& type : types) {
ARROW_ASSIGN_OR_RAISE(auto kernel,
subtract_func->DispatchExact({type, type}));
// reuse kernel's exec and signature
}

bkietz · 2023-06-28T11:47:33Z

@js8544 yes, exactly. I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

(sorry for the spurious close; bumped the wrong button)

js8544 · 2023-06-28T11:52:27Z

I think we might be able to go a step further and loop over subtract's kernels directly, without the need to list input types explicitly and go through DispatchExact.

That sounds better indeed! I'll try this now.

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

js8544 · 2023-06-28T14:53:41Z

@bkietz I've changed it to looping over subtract's kernels and wrapping their signature and kernel exec.

bkietz

This is looking great, thanks for refactoring!
Just a few nits

bkietz · 2023-06-28T16:58:02Z

+ auto offset = abs(periods);
+ offset = std::min(offset, input.length);
+ auto exec_length = input.length - offset;
+ // prepare bitmap
+ auto null_start = periods > 0 ? 0 : exec_length;
+ auto non_null_start = periods > 0 ? offset : 0;


Nit: could we call these two regions "computed" and "margin"? I think that'll be more obvious for the next maintainer, and it'd help if we move all the start/length calculation to this preamble too
Suggested change
auto offset = abs(periods);
offset = std::min(offset, input.length);
auto exec_length = input.length - offset;
// prepare bitmap
auto null_start = periods > 0 ? 0 : exec_length;
auto non_null_start = periods > 0 ? offset : 0;
// We only compute values in the region where the input-with-offset overlaps
// the original input. The margin where these do not overlap gets filled with null.
auto margin_length = std::min(abs(periods), input.length);
auto computed_length = input.length - margin_length;
auto margin_start = periods > 0 ? 0 : computed_length;
auto left_start = periods > 0 ? margin_length : 0;
auto right_start = periods > 0 ? 0 : margin_length;
// ...
// prepare bitmap

Updated. It's much clearer now. thanks for the suggestion!

Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

bkietz

LGTM, thanks!

bkietz · 2023-06-29T12:41:39Z

macOs glib&python failures are network failures while communicating with brew https://github.com/apache/arrow/actions/runs/5408688637/jobs/9835499198?pr=35787

macOS c++ failure is a known issue #36329

ubuntu c++ failure is an s3 flake https://github.com/apache/arrow/actions/runs/5408688635/jobs/9828017525?pr=35787#step:7:4892

R test failure is a known issue #36346

conbench-apache-arrow · 2023-07-04T09:14:50Z

Conbench analyzed the 6 benchmark runs on commit 26c25d1b.

There were 4 benchmark results indicating a performance regression:

Commit Run on arm64-m6g-linux-compute at 2023-06-29 14:28:22Z
- params=1048576/100/min_time:1.000, source=cpp-micro, suite=arrow-compute-vector-partition-benchmark
Commit Run on ursa-thinkcentre-m75q at 2023-07-02 13:25:33Z
- params=1048576/10000, source=cpp-micro, suite=arrow-acero-aggregate-benchmark
and 2 more (see the report linked below)

The full Conbench report has more details.

js8544 requested review from AlenkaF and westonpace as code owners May 26, 2023 10:53

github-actionsBot added Component: C++ Component: Documentation Component: Python labels May 26, 2023

js8544 changed the title ~~Add pairwise diff function~~GH-35786: [C++] Add pairwise_diff functionMay 26, 2023

github-actionsBot added the awaiting review Awaiting review label May 26, 2023

js8544 mentioned this pull request Jun 10, 2023
[C++] Implement cumulative product, max, and min compute functions #32190
Closed

lidavidm approved these changes Jun 22, 2023
View reviewed changes

github-actionsBot added awaiting merge Awaiting merge and removed awaiting review Awaiting review labels Jun 22, 2023

js8544 added 12 commits June 24, 2023 12:58

Add pairwise diff function
c77075e

make linter happy
789bc9e

fix doc
885e78c

fix warning
1f8d8c5

refactor to two different functions for better extending to other ops
4a61a8f

fix doc
6741ba2

use subtract kernel exec
fabd43a

lint
c757748

use template for resolver
dadd720

pr feedback
3d8179a

save exec by value
6da2dc2

update var name
d16d034

bkietz requested changes Jun 26, 2023
View reviewed changes

Comment threadcpp/src/arrow/compute/api_vector.h Outdated
Comment threadcpp/src/arrow/compute/kernels/vector_pairwise.cc Outdated
Comment threaddocs/source/cpp/compute.rst Outdated

github-actionsBot removed the awaiting merge Awaiting merge label Jun 26, 2023

js8544 force-pushed the jinshang/diff_kernel branch from 3f037b6 to 6741ba2Compare June 28, 2023 06:52

github-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jun 28, 2023

bkietz closed this Jun 28, 2023

bkietz reopened this Jun 28, 2023

js8544and others added 3 commits June 28, 2023 22:48

Update docs/source/cpp/compute.rst
a6ad857
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

Update cpp/src/arrow/compute/api_vector.h
fe641f0
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

update doc
17464af

js8544 requested a review from bkietz June 28, 2023 14:53

js8544 commented Jun 28, 2023
View reviewed changes

Comment threadcpp/src/arrow/compute/kernel.h Outdated

js8544 commented Jun 28, 2023
View reviewed changes

Comment threadcpp/src/arrow/compute/kernel.h

bkietz requested changes Jun 28, 2023
View reviewed changes

github-actionsBot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Jun 28, 2023

js8544and others added 2 commits June 29, 2023 11:19

Update docs/source/cpp/compute.rst
d335071
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>

lint
c51f659

js8544 requested a review from bkietz June 29, 2023 03:21

bkietz approved these changes Jun 29, 2023
View reviewed changes

github-actionsBot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Jun 29, 2023

bkietz merged commit 26c25d1 into apache:mainJun 29, 2023

bkietz removed the awaiting merge Awaiting merge label Jun 29, 2023