ARROW-9967: [Python] Add compute module docs + expose more option classes

[arw2019]

#8163 exposes pyarrow.compute kernels and generates their docstrings. This PR adds documentation for the module in the User Guide and the Python API reference.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9967

[arw2019]

Noticed that this was missing on the c++ page

[pitrou]

Ah, thank you! However, can you leave the table alphabetically-sorted?

[arw2019]

Actually my bad, you already had it there. Reverted

[jorisvandenbossche]

@arw2019 thanks a lot for working on this! (and sorry for the overlap with #8163)

[jorisvandenbossche]

this is not really a transform?

[arw2019]

Yeah that's fair. Would Structural Transforms be a good place for it?

[jorisvandenbossche]

this array is not used? (maybe show pc.equals(a, b) ?)

[arw2019]

@jorisvandenbossche@pitrou

AFAIK we can't have methods named and and or in Python. For now I went with and_ and or_ here, for consistency with and_kleene and or_kleene, but for example numpy uses np.logical_and and np.logical_or

[pitrou]

We already have a function named "and":

>>>frompyarrowimportcompute>>>getattr(compute, "and")
<functionpyarrow.compute.and(left, right, *, memory_pool=None)>

So you should just fix compute.py so that it's exported as and_.

[arw2019]

Right. Fixed now

[pitrou]

Will this render the docstrings?

[arw2019]

I think so. It's what's done on the other pages:
https://github.com/apache/arrow/blob/master/docs/source/python/api/datatypes.rst

I'm having trouble building the docs locally (related to ARROW-10018 I think) so haven't been able to check

[jorisvandenbossche]

This won't render the docstring here, but it will create a separate page for each of those functions and link to that from this table.

(the separate pages might be a bit overkill since the docstrings here are often not that informative yet, but it's indeed how we do it for other functions as well)

[arw2019]

Question: in 952649d I'm attempting to expose PartitionNthOptions but it errors at compilation. What is (are) my mistake(s)?

This is needed to make the partition_nth_indices Python wrapper usable.

[pitrou]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly. Also, it is better to use __cinit__ here so that the constructor never gets skipped, so for example (untested):

def__cinit__(self, int64_t pivot):
self.partition_nth_options.reset(new CPartitionNthOptions(pivot))

[arw2019]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly.

Thanks @pitrou! That was the problem

[arw2019]

This is ready for re-review.

I believe that I've addressed the feedback from previous reviews. I've also now exposed all the option classes so that all the kernels listed in the docs are usable from Python.

[arw2019]

@jorisvandenbossche Will this work? I'm not 100% sure as I haven't yet compiled the docs to explicitly check

[pitrou]

I'm not sure why unique_ptr is used in some of the other classes, but it shouldn't be necessary. See MinMaxOptions for an example without.

[arw2019]

In PartitionNthOptions we need it because it doesn't have a null constructor so it can't be allocated on the stack. That's not the case for VarianceOptions, though, so I'll take a look

[arw2019]

I've rewritten VarianceOptions without unique_ptr.

I think that with the other three options classes exposed in this PR (SetLookupOptions, PartitionNthOptions and StrptimeOptions) it's necessary to use unique_ptr or, alternatively, we would need to define default constructors

[kszucs]

@github-actions crossbow submit test-ubuntu-18.04-docs

[github-actions]

Revision: 73272bf38f2d8406b60a6ebaa20c5f4e58dff075

Submitted crossbow builds: ursa-labs/crossbow @ actions-615

Task	Status
test-ubuntu-18.04-docs

[pitrou]

+1. Will merge after a couple changes. Thank you @arw2019 !

[pitrou]

CI failures are unrelated.

[arw2019]

Thanks @pitrou@jorisvandenbossche for reviewing!

[arw2019]

#8163 exposes pyarrow.compute kernels and generates their docstrings. This PR adds documentation for the module in the User Guide and the Python API reference.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9967

[arw2019]

Noticed that this was missing on the c++ page

[pitrou]

Ah, thank you! However, can you leave the table alphabetically-sorted?

[arw2019]

Actually my bad, you already had it there. Reverted

[jorisvandenbossche]

@arw2019 thanks a lot for working on this! (and sorry for the overlap with #8163)

[jorisvandenbossche]

this is not really a transform?

[arw2019]

Yeah that's fair. Would Structural Transforms be a good place for it?

[jorisvandenbossche]

this array is not used? (maybe show pc.equals(a, b) ?)

[arw2019]

@jorisvandenbossche@pitrou

AFAIK we can't have methods named and and or in Python. For now I went with and_ and or_ here, for consistency with and_kleene and or_kleene, but for example numpy uses np.logical_and and np.logical_or

[pitrou]

We already have a function named "and":

>>>frompyarrowimportcompute>>>getattr(compute, "and")
<functionpyarrow.compute.and(left, right, *, memory_pool=None)>

So you should just fix compute.py so that it's exported as and_.

[arw2019]

Right. Fixed now

[pitrou]

Will this render the docstrings?

[arw2019]

I think so. It's what's done on the other pages:
https://github.com/apache/arrow/blob/master/docs/source/python/api/datatypes.rst

I'm having trouble building the docs locally (related to ARROW-10018 I think) so haven't been able to check

[jorisvandenbossche]

This won't render the docstring here, but it will create a separate page for each of those functions and link to that from this table.

(the separate pages might be a bit overkill since the docstrings here are often not that informative yet, but it's indeed how we do it for other functions as well)

[arw2019]

Question: in 952649d I'm attempting to expose PartitionNthOptions but it errors at compilation. What is (are) my mistake(s)?

This is needed to make the partition_nth_indices Python wrapper usable.

[pitrou]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly. Also, it is better to use __cinit__ here so that the constructor never gets skipped, so for example (untested):

def__cinit__(self, int64_t pivot):
self.partition_nth_options.reset(new CPartitionNthOptions(pivot))

[arw2019]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly.

Thanks @pitrou! That was the problem

[arw2019]

This is ready for re-review.

I believe that I've addressed the feedback from previous reviews. I've also now exposed all the option classes so that all the kernels listed in the docs are usable from Python.

[arw2019]

@jorisvandenbossche Will this work? I'm not 100% sure as I haven't yet compiled the docs to explicitly check

[pitrou]

I'm not sure why unique_ptr is used in some of the other classes, but it shouldn't be necessary. See MinMaxOptions for an example without.

[arw2019]

In PartitionNthOptions we need it because it doesn't have a null constructor so it can't be allocated on the stack. That's not the case for VarianceOptions, though, so I'll take a look

[arw2019]

I've rewritten VarianceOptions without unique_ptr.

I think that with the other three options classes exposed in this PR (SetLookupOptions, PartitionNthOptions and StrptimeOptions) it's necessary to use unique_ptr or, alternatively, we would need to define default constructors

[kszucs]

@github-actions crossbow submit test-ubuntu-18.04-docs

[github-actions]

Revision: 73272bf38f2d8406b60a6ebaa20c5f4e58dff075

Submitted crossbow builds: ursa-labs/crossbow @ actions-615

Task	Status
test-ubuntu-18.04-docs

[pitrou]

+1. Will merge after a couple changes. Thank you @arw2019 !

[pitrou]

CI failures are unrelated.

[arw2019]

Thanks @pitrou@jorisvandenbossche for reviewing!

[arw2019]

#8163 exposes pyarrow.compute kernels and generates their docstrings. This PR adds documentation for the module in the User Guide and the Python API reference.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9967

[arw2019]

Noticed that this was missing on the c++ page

[pitrou]

Ah, thank you! However, can you leave the table alphabetically-sorted?

[arw2019]

Actually my bad, you already had it there. Reverted

[jorisvandenbossche]

@arw2019 thanks a lot for working on this! (and sorry for the overlap with #8163)

[jorisvandenbossche]

this is not really a transform?

[arw2019]

Yeah that's fair. Would Structural Transforms be a good place for it?

[jorisvandenbossche]

this array is not used? (maybe show pc.equals(a, b) ?)

[arw2019]

@jorisvandenbossche@pitrou

AFAIK we can't have methods named and and or in Python. For now I went with and_ and or_ here, for consistency with and_kleene and or_kleene, but for example numpy uses np.logical_and and np.logical_or

[pitrou]

We already have a function named "and":

>>>frompyarrowimportcompute>>>getattr(compute, "and")
<functionpyarrow.compute.and(left, right, *, memory_pool=None)>

So you should just fix compute.py so that it's exported as and_.

[arw2019]

Right. Fixed now

[pitrou]

Will this render the docstrings?

[arw2019]

I think so. It's what's done on the other pages:
https://github.com/apache/arrow/blob/master/docs/source/python/api/datatypes.rst

I'm having trouble building the docs locally (related to ARROW-10018 I think) so haven't been able to check

[jorisvandenbossche]

This won't render the docstring here, but it will create a separate page for each of those functions and link to that from this table.

(the separate pages might be a bit overkill since the docstrings here are often not that informative yet, but it's indeed how we do it for other functions as well)

[arw2019]

Question: in 952649d I'm attempting to expose PartitionNthOptions but it errors at compilation. What is (are) my mistake(s)?

This is needed to make the partition_nth_indices Python wrapper usable.

[pitrou]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly. Also, it is better to use __cinit__ here so that the constructor never gets skipped, so for example (untested):

def__cinit__(self, int64_t pivot):
self.partition_nth_options.reset(new CPartitionNthOptions(pivot))

[arw2019]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly.

Thanks @pitrou! That was the problem

[arw2019]

This is ready for re-review.

I believe that I've addressed the feedback from previous reviews. I've also now exposed all the option classes so that all the kernels listed in the docs are usable from Python.

[arw2019]

@jorisvandenbossche Will this work? I'm not 100% sure as I haven't yet compiled the docs to explicitly check

[pitrou]

I'm not sure why unique_ptr is used in some of the other classes, but it shouldn't be necessary. See MinMaxOptions for an example without.

[arw2019]

In PartitionNthOptions we need it because it doesn't have a null constructor so it can't be allocated on the stack. That's not the case for VarianceOptions, though, so I'll take a look

[arw2019]

I've rewritten VarianceOptions without unique_ptr.

I think that with the other three options classes exposed in this PR (SetLookupOptions, PartitionNthOptions and StrptimeOptions) it's necessary to use unique_ptr or, alternatively, we would need to define default constructors

[kszucs]

@github-actions crossbow submit test-ubuntu-18.04-docs

[github-actions]

Revision: 73272bf38f2d8406b60a6ebaa20c5f4e58dff075

Submitted crossbow builds: ursa-labs/crossbow @ actions-615

Task	Status
test-ubuntu-18.04-docs

[pitrou]

+1. Will merge after a couple changes. Thank you @arw2019 !

[pitrou]

CI failures are unrelated.

[arw2019]

Thanks @pitrou@jorisvandenbossche for reviewing!

[arw2019]

#8163 exposes pyarrow.compute kernels and generates their docstrings. This PR adds documentation for the module in the User Guide and the Python API reference.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9967

[arw2019]

Noticed that this was missing on the c++ page

[pitrou]

Ah, thank you! However, can you leave the table alphabetically-sorted?

[arw2019]

Actually my bad, you already had it there. Reverted

[jorisvandenbossche]

@arw2019 thanks a lot for working on this! (and sorry for the overlap with #8163)

[jorisvandenbossche]

this is not really a transform?

[arw2019]

Yeah that's fair. Would Structural Transforms be a good place for it?

[jorisvandenbossche]

this array is not used? (maybe show pc.equals(a, b) ?)

[arw2019]

@jorisvandenbossche@pitrou

AFAIK we can't have methods named and and or in Python. For now I went with and_ and or_ here, for consistency with and_kleene and or_kleene, but for example numpy uses np.logical_and and np.logical_or

[pitrou]

We already have a function named "and":

>>>frompyarrowimportcompute>>>getattr(compute, "and")
<functionpyarrow.compute.and(left, right, *, memory_pool=None)>

So you should just fix compute.py so that it's exported as and_.

[arw2019]

Right. Fixed now

[pitrou]

Will this render the docstrings?

[arw2019]

I think so. It's what's done on the other pages:
https://github.com/apache/arrow/blob/master/docs/source/python/api/datatypes.rst

I'm having trouble building the docs locally (related to ARROW-10018 I think) so haven't been able to check

[jorisvandenbossche]

This won't render the docstring here, but it will create a separate page for each of those functions and link to that from this table.

(the separate pages might be a bit overkill since the docstrings here are often not that informative yet, but it's indeed how we do it for other functions as well)

[arw2019]

Question: in 952649d I'm attempting to expose PartitionNthOptions but it errors at compilation. What is (are) my mistake(s)?

This is needed to make the partition_nth_indices Python wrapper usable.

[pitrou]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly. Also, it is better to use __cinit__ here so that the constructor never gets skipped, so for example (untested):

def__cinit__(self, int64_t pivot):
self.partition_nth_options.reset(new CPartitionNthOptions(pivot))

[arw2019]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly.

Thanks @pitrou! That was the problem

[arw2019]

This is ready for re-review.

I believe that I've addressed the feedback from previous reviews. I've also now exposed all the option classes so that all the kernels listed in the docs are usable from Python.

[arw2019]

@jorisvandenbossche Will this work? I'm not 100% sure as I haven't yet compiled the docs to explicitly check

[pitrou]

I'm not sure why unique_ptr is used in some of the other classes, but it shouldn't be necessary. See MinMaxOptions for an example without.

[arw2019]

In PartitionNthOptions we need it because it doesn't have a null constructor so it can't be allocated on the stack. That's not the case for VarianceOptions, though, so I'll take a look

[arw2019]

I've rewritten VarianceOptions without unique_ptr.

I think that with the other three options classes exposed in this PR (SetLookupOptions, PartitionNthOptions and StrptimeOptions) it's necessary to use unique_ptr or, alternatively, we would need to define default constructors

[kszucs]

@github-actions crossbow submit test-ubuntu-18.04-docs

[github-actions]

Revision: 73272bf38f2d8406b60a6ebaa20c5f4e58dff075

Submitted crossbow builds: ursa-labs/crossbow @ actions-615

Task	Status
test-ubuntu-18.04-docs

[pitrou]

+1. Will merge after a couple changes. Thank you @arw2019 !

[pitrou]

CI failures are unrelated.

[arw2019]

Thanks @pitrou@jorisvandenbossche for reviewing!

[arw2019]

#8163 exposes pyarrow.compute kernels and generates their docstrings. This PR adds documentation for the module in the User Guide and the Python API reference.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9967

[arw2019]

Noticed that this was missing on the c++ page

[pitrou]

Ah, thank you! However, can you leave the table alphabetically-sorted?

[arw2019]

Actually my bad, you already had it there. Reverted

[jorisvandenbossche]

@arw2019 thanks a lot for working on this! (and sorry for the overlap with #8163)

[jorisvandenbossche]

this is not really a transform?

[arw2019]

Yeah that's fair. Would Structural Transforms be a good place for it?

[jorisvandenbossche]

this array is not used? (maybe show pc.equals(a, b) ?)

[arw2019]

@jorisvandenbossche@pitrou

AFAIK we can't have methods named and and or in Python. For now I went with and_ and or_ here, for consistency with and_kleene and or_kleene, but for example numpy uses np.logical_and and np.logical_or

[pitrou]

We already have a function named "and":

>>>frompyarrowimportcompute>>>getattr(compute, "and")
<functionpyarrow.compute.and(left, right, *, memory_pool=None)>

So you should just fix compute.py so that it's exported as and_.

[arw2019]

Right. Fixed now

[pitrou]

Will this render the docstrings?

[arw2019]

I think so. It's what's done on the other pages:
https://github.com/apache/arrow/blob/master/docs/source/python/api/datatypes.rst

I'm having trouble building the docs locally (related to ARROW-10018 I think) so haven't been able to check

[jorisvandenbossche]

This won't render the docstring here, but it will create a separate page for each of those functions and link to that from this table.

(the separate pages might be a bit overkill since the docstrings here are often not that informative yet, but it's indeed how we do it for other functions as well)

[arw2019]

Question: in 952649d I'm attempting to expose PartitionNthOptions but it errors at compilation. What is (are) my mistake(s)?

This is needed to make the partition_nth_indices Python wrapper usable.

[pitrou]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly. Also, it is better to use __cinit__ here so that the constructor never gets skipped, so for example (untested):

def__cinit__(self, int64_t pivot):
self.partition_nth_options.reset(new CPartitionNthOptions(pivot))

[arw2019]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly.

Thanks @pitrou! That was the problem

[arw2019]

This is ready for re-review.

I believe that I've addressed the feedback from previous reviews. I've also now exposed all the option classes so that all the kernels listed in the docs are usable from Python.

[arw2019]

@jorisvandenbossche Will this work? I'm not 100% sure as I haven't yet compiled the docs to explicitly check

[pitrou]

I'm not sure why unique_ptr is used in some of the other classes, but it shouldn't be necessary. See MinMaxOptions for an example without.

[arw2019]

In PartitionNthOptions we need it because it doesn't have a null constructor so it can't be allocated on the stack. That's not the case for VarianceOptions, though, so I'll take a look

[arw2019]

I've rewritten VarianceOptions without unique_ptr.

I think that with the other three options classes exposed in this PR (SetLookupOptions, PartitionNthOptions and StrptimeOptions) it's necessary to use unique_ptr or, alternatively, we would need to define default constructors

[kszucs]

@github-actions crossbow submit test-ubuntu-18.04-docs

[github-actions]

Revision: 73272bf38f2d8406b60a6ebaa20c5f4e58dff075

Submitted crossbow builds: ursa-labs/crossbow @ actions-615

Task	Status
test-ubuntu-18.04-docs

[pitrou]

+1. Will merge after a couple changes. Thank you @arw2019 !

[pitrou]

CI failures are unrelated.

[arw2019]

Thanks @pitrou@jorisvandenbossche for reviewing!

[arw2019]

#8163 exposes pyarrow.compute kernels and generates their docstrings. This PR adds documentation for the module in the User Guide and the Python API reference.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9967

[arw2019]

Noticed that this was missing on the c++ page

[pitrou]

Ah, thank you! However, can you leave the table alphabetically-sorted?

[arw2019]

Actually my bad, you already had it there. Reverted

[jorisvandenbossche]

@arw2019 thanks a lot for working on this! (and sorry for the overlap with #8163)

[jorisvandenbossche]

this is not really a transform?

[arw2019]

Yeah that's fair. Would Structural Transforms be a good place for it?

[jorisvandenbossche]

this array is not used? (maybe show pc.equals(a, b) ?)

[arw2019]

@jorisvandenbossche@pitrou

AFAIK we can't have methods named and and or in Python. For now I went with and_ and or_ here, for consistency with and_kleene and or_kleene, but for example numpy uses np.logical_and and np.logical_or

[pitrou]

We already have a function named "and":

>>>frompyarrowimportcompute>>>getattr(compute, "and")
<functionpyarrow.compute.and(left, right, *, memory_pool=None)>

So you should just fix compute.py so that it's exported as and_.

[arw2019]

Right. Fixed now

[pitrou]

Will this render the docstrings?

[arw2019]

I think so. It's what's done on the other pages:
https://github.com/apache/arrow/blob/master/docs/source/python/api/datatypes.rst

I'm having trouble building the docs locally (related to ARROW-10018 I think) so haven't been able to check

[jorisvandenbossche]

This won't render the docstring here, but it will create a separate page for each of those functions and link to that from this table.

(the separate pages might be a bit overkill since the docstrings here are often not that informative yet, but it's indeed how we do it for other functions as well)

[arw2019]

Question: in 952649d I'm attempting to expose PartitionNthOptions but it errors at compilation. What is (are) my mistake(s)?

This is needed to make the partition_nth_indices Python wrapper usable.

[pitrou]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly. Also, it is better to use __cinit__ here so that the constructor never gets skipped, so for example (untested):

def__cinit__(self, int64_t pivot):
self.partition_nth_options.reset(new CPartitionNthOptions(pivot))

[arw2019]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly.

Thanks @pitrou! That was the problem

[arw2019]

This is ready for re-review.

I believe that I've addressed the feedback from previous reviews. I've also now exposed all the option classes so that all the kernels listed in the docs are usable from Python.

[arw2019]

@jorisvandenbossche Will this work? I'm not 100% sure as I haven't yet compiled the docs to explicitly check

[pitrou]

I'm not sure why unique_ptr is used in some of the other classes, but it shouldn't be necessary. See MinMaxOptions for an example without.

[arw2019]

In PartitionNthOptions we need it because it doesn't have a null constructor so it can't be allocated on the stack. That's not the case for VarianceOptions, though, so I'll take a look

[arw2019]

I've rewritten VarianceOptions without unique_ptr.

I think that with the other three options classes exposed in this PR (SetLookupOptions, PartitionNthOptions and StrptimeOptions) it's necessary to use unique_ptr or, alternatively, we would need to define default constructors

[kszucs]

@github-actions crossbow submit test-ubuntu-18.04-docs

[github-actions]

Revision: 73272bf38f2d8406b60a6ebaa20c5f4e58dff075

Submitted crossbow builds: ursa-labs/crossbow @ actions-615

Task	Status
test-ubuntu-18.04-docs

[pitrou]

+1. Will merge after a couple changes. Thank you @arw2019 !

[pitrou]

CI failures are unrelated.

[arw2019]

Thanks @pitrou@jorisvandenbossche for reviewing!

[arw2019]

#8163 exposes pyarrow.compute kernels and generates their docstrings. This PR adds documentation for the module in the User Guide and the Python API reference.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9967

[arw2019]

Noticed that this was missing on the c++ page

[pitrou]

Ah, thank you! However, can you leave the table alphabetically-sorted?

[arw2019]

Actually my bad, you already had it there. Reverted

[jorisvandenbossche]

@arw2019 thanks a lot for working on this! (and sorry for the overlap with #8163)

[jorisvandenbossche]

this is not really a transform?

[arw2019]

Yeah that's fair. Would Structural Transforms be a good place for it?

[jorisvandenbossche]

this array is not used? (maybe show pc.equals(a, b) ?)

[arw2019]

@jorisvandenbossche@pitrou

AFAIK we can't have methods named and and or in Python. For now I went with and_ and or_ here, for consistency with and_kleene and or_kleene, but for example numpy uses np.logical_and and np.logical_or

[pitrou]

We already have a function named "and":

>>>frompyarrowimportcompute>>>getattr(compute, "and")
<functionpyarrow.compute.and(left, right, *, memory_pool=None)>

So you should just fix compute.py so that it's exported as and_.

[arw2019]

Right. Fixed now

[pitrou]

Will this render the docstrings?

[arw2019]

I think so. It's what's done on the other pages:
https://github.com/apache/arrow/blob/master/docs/source/python/api/datatypes.rst

I'm having trouble building the docs locally (related to ARROW-10018 I think) so haven't been able to check

[jorisvandenbossche]

This won't render the docstring here, but it will create a separate page for each of those functions and link to that from this table.

(the separate pages might be a bit overkill since the docstrings here are often not that informative yet, but it's indeed how we do it for other functions as well)

[arw2019]

Question: in 952649d I'm attempting to expose PartitionNthOptions but it errors at compilation. What is (are) my mistake(s)?

This is needed to make the partition_nth_indices Python wrapper usable.

[pitrou]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly. Also, it is better to use __cinit__ here so that the constructor never gets skipped, so for example (untested):

def__cinit__(self, int64_t pivot):
self.partition_nth_options.reset(new CPartitionNthOptions(pivot))

[arw2019]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly.

Thanks @pitrou! That was the problem

[arw2019]

This is ready for re-review.

I believe that I've addressed the feedback from previous reviews. I've also now exposed all the option classes so that all the kernels listed in the docs are usable from Python.

[arw2019]

@jorisvandenbossche Will this work? I'm not 100% sure as I haven't yet compiled the docs to explicitly check

[pitrou]

I'm not sure why unique_ptr is used in some of the other classes, but it shouldn't be necessary. See MinMaxOptions for an example without.

[arw2019]

In PartitionNthOptions we need it because it doesn't have a null constructor so it can't be allocated on the stack. That's not the case for VarianceOptions, though, so I'll take a look

[arw2019]

I've rewritten VarianceOptions without unique_ptr.

I think that with the other three options classes exposed in this PR (SetLookupOptions, PartitionNthOptions and StrptimeOptions) it's necessary to use unique_ptr or, alternatively, we would need to define default constructors

[kszucs]

@github-actions crossbow submit test-ubuntu-18.04-docs

[github-actions]

Revision: 73272bf38f2d8406b60a6ebaa20c5f4e58dff075

Submitted crossbow builds: ursa-labs/crossbow @ actions-615

Task	Status
test-ubuntu-18.04-docs

[pitrou]

+1. Will merge after a couple changes. Thank you @arw2019 !

[pitrou]

CI failures are unrelated.

[arw2019]

Thanks @pitrou@jorisvandenbossche for reviewing!

[arw2019]

#8163 exposes pyarrow.compute kernels and generates their docstrings. This PR adds documentation for the module in the User Guide and the Python API reference.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9967

[arw2019]

Noticed that this was missing on the c++ page

[pitrou]

Ah, thank you! However, can you leave the table alphabetically-sorted?

[arw2019]

Actually my bad, you already had it there. Reverted

[jorisvandenbossche]

@arw2019 thanks a lot for working on this! (and sorry for the overlap with #8163)

[jorisvandenbossche]

this is not really a transform?

[arw2019]

Yeah that's fair. Would Structural Transforms be a good place for it?

[jorisvandenbossche]

this array is not used? (maybe show pc.equals(a, b) ?)

[arw2019]

@jorisvandenbossche@pitrou

AFAIK we can't have methods named and and or in Python. For now I went with and_ and or_ here, for consistency with and_kleene and or_kleene, but for example numpy uses np.logical_and and np.logical_or

[pitrou]

We already have a function named "and":

>>>frompyarrowimportcompute>>>getattr(compute, "and")
<functionpyarrow.compute.and(left, right, *, memory_pool=None)>

So you should just fix compute.py so that it's exported as and_.

[arw2019]

Right. Fixed now

[pitrou]

Will this render the docstrings?

[arw2019]

I think so. It's what's done on the other pages:
https://github.com/apache/arrow/blob/master/docs/source/python/api/datatypes.rst

I'm having trouble building the docs locally (related to ARROW-10018 I think) so haven't been able to check

[jorisvandenbossche]

This won't render the docstring here, but it will create a separate page for each of those functions and link to that from this table.

(the separate pages might be a bit overkill since the docstrings here are often not that informative yet, but it's indeed how we do it for other functions as well)

[arw2019]

Question: in 952649d I'm attempting to expose PartitionNthOptions but it errors at compilation. What is (are) my mistake(s)?

This is needed to make the partition_nth_indices Python wrapper usable.

[pitrou]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly. Also, it is better to use __cinit__ here so that the constructor never gets skipped, so for example (untested):

def__cinit__(self, int64_t pivot):
self.partition_nth_options.reset(new CPartitionNthOptions(pivot))

[arw2019]

Cython generally doesn't convert implicitly between arbitrary Python objects and C/C++ types, you have to type your code explicitly.

Thanks @pitrou! That was the problem

[arw2019]

This is ready for re-review.

I believe that I've addressed the feedback from previous reviews. I've also now exposed all the option classes so that all the kernels listed in the docs are usable from Python.

[arw2019]

@jorisvandenbossche Will this work? I'm not 100% sure as I haven't yet compiled the docs to explicitly check

[pitrou]

I'm not sure why unique_ptr is used in some of the other classes, but it shouldn't be necessary. See MinMaxOptions for an example without.

[arw2019]

In PartitionNthOptions we need it because it doesn't have a null constructor so it can't be allocated on the stack. That's not the case for VarianceOptions, though, so I'll take a look

[arw2019]

I've rewritten VarianceOptions without unique_ptr.

I think that with the other three options classes exposed in this PR (SetLookupOptions, PartitionNthOptions and StrptimeOptions) it's necessary to use unique_ptr or, alternatively, we would need to define default constructors

[kszucs]

@github-actions crossbow submit test-ubuntu-18.04-docs

[github-actions]

Revision: 73272bf38f2d8406b60a6ebaa20c5f4e58dff075

Submitted crossbow builds: ursa-labs/crossbow @ actions-615

Task	Status
test-ubuntu-18.04-docs

[pitrou]

+1. Will merge after a couple changes. Thank you @arw2019 !

[pitrou]

CI failures are unrelated.

[arw2019]

Thanks @pitrou@jorisvandenbossche for reviewing!