PR: Add allow_copy flag to interchange protocol

[steff456]

This PR adds a flag to throw an exception if the export cannot be zero-copy. (e.g. for pandas, possible due to block manager where rows are contiguous and columns are not) .

• Add allow_zero_copy flag to the DataFrame class.
• Propagate the flag to the buffer and raise a RuntimeError when it is true
• Fix test_noncontiguous_columns
• Make update in the requirements doc

[rgommers]

To summarize the options for naming this keyword that were suggested yesterday:

• allow_copy (your current choice)
• allow_memory_copy
• no_copy_only

I like the positive allow_* better than the negative no_*. Happy with either or allow_copy or allow_memory_copy.

Regarding the default value (now True):

• @maartenbreddels preferred the "performance safety" of False
• @jorisvandenbossche pointed out that currently strings in Pandas (implemented via object dtype) will always require a copy, but that that in the next Pandas release there will be an Arrow-based variable length string option which can be zero-copy.

I believe that there will be enough cases that will always require a copy (object-strings, columns backed by non-contiguous arrays, non-columnar data storage like for Koalas and Ibis) that using True as the default would be best. And then the very performance-sensitive (= usually more advanced) users can set it to False.

[rgommers]

Make update in the requirements doc

Adding the following to the requirements docs:

We'll also list some things that were discussed but are not requirements:
...
4. Support for strided storage in buffers.
_Rationale: this is supported by a subset of dataframes only, mainly those
that use NumPy arrays. In many real-world use cases, strided arrays will
force a copy at some point, so requiring contiguous memory layout (and hence
an extra copy at the moment `__dataframe__` is used) is considered a good
trade-off for reduced implementation complexity._

[rgommers]

Did some more testing and review, and found some issues. Also this PR is not against main, so I'll close it. Resubmitted as gh-51.

[rgommers]

Rephrasing this a bit:

 ``allow_copy`` is a keyword that defines whether or not the library is
allowed to make a copy of the data. This can for example happen if a
library supports strided buffers (those would need a copy, because this
protocol specifies contiguous buffers).

[rgommers]

"False and a copy is needed"

[rgommers]

This removes the actual check for striding (not x.strides == (x.dtype.itemsize,):), that needs to be put back.

[rgommers]

Logic:

ifnotx.strides== (x.dtype.itemsize,):
# The protocol does not support strided buffers, so a copy is# necessary. If that's not allowed, we need to raise an exception.ifallow_copy:
x=x.copy()
else:
raiseRuntimeError("Exports cannot be zero-copy in the case ""of a non-contiguous buffer")

[rgommers]

And then adding a test case shows there's a problem - we don't hold on to the memory.

[steff456]

This PR adds a flag to throw an exception if the export cannot be zero-copy. (e.g. for pandas, possible due to block manager where rows are contiguous and columns are not) .

• Add allow_zero_copy flag to the DataFrame class.
• Propagate the flag to the buffer and raise a RuntimeError when it is true
• Fix test_noncontiguous_columns
• Make update in the requirements doc

[rgommers]

To summarize the options for naming this keyword that were suggested yesterday:

• allow_copy (your current choice)
• allow_memory_copy
• no_copy_only

I like the positive allow_* better than the negative no_*. Happy with either or allow_copy or allow_memory_copy.

Regarding the default value (now True):

• @maartenbreddels preferred the "performance safety" of False
• @jorisvandenbossche pointed out that currently strings in Pandas (implemented via object dtype) will always require a copy, but that that in the next Pandas release there will be an Arrow-based variable length string option which can be zero-copy.

I believe that there will be enough cases that will always require a copy (object-strings, columns backed by non-contiguous arrays, non-columnar data storage like for Koalas and Ibis) that using True as the default would be best. And then the very performance-sensitive (= usually more advanced) users can set it to False.

[rgommers]

Make update in the requirements doc

Adding the following to the requirements docs:

We'll also list some things that were discussed but are not requirements:
...
4. Support for strided storage in buffers.
_Rationale: this is supported by a subset of dataframes only, mainly those
that use NumPy arrays. In many real-world use cases, strided arrays will
force a copy at some point, so requiring contiguous memory layout (and hence
an extra copy at the moment `__dataframe__` is used) is considered a good
trade-off for reduced implementation complexity._

[rgommers]

Did some more testing and review, and found some issues. Also this PR is not against main, so I'll close it. Resubmitted as gh-51.

[rgommers]

Rephrasing this a bit:

 ``allow_copy`` is a keyword that defines whether or not the library is
allowed to make a copy of the data. This can for example happen if a
library supports strided buffers (those would need a copy, because this
protocol specifies contiguous buffers).

[rgommers]

"False and a copy is needed"

[rgommers]

This removes the actual check for striding (not x.strides == (x.dtype.itemsize,):), that needs to be put back.

[rgommers]

Logic:

ifnotx.strides== (x.dtype.itemsize,):
# The protocol does not support strided buffers, so a copy is# necessary. If that's not allowed, we need to raise an exception.ifallow_copy:
x=x.copy()
else:
raiseRuntimeError("Exports cannot be zero-copy in the case ""of a non-contiguous buffer")

[rgommers]

And then adding a test case shows there's a problem - we don't hold on to the memory.

[steff456]

This PR adds a flag to throw an exception if the export cannot be zero-copy. (e.g. for pandas, possible due to block manager where rows are contiguous and columns are not) .

• Add allow_zero_copy flag to the DataFrame class.
• Propagate the flag to the buffer and raise a RuntimeError when it is true
• Fix test_noncontiguous_columns
• Make update in the requirements doc

[rgommers]

To summarize the options for naming this keyword that were suggested yesterday:

• allow_copy (your current choice)
• allow_memory_copy
• no_copy_only

I like the positive allow_* better than the negative no_*. Happy with either or allow_copy or allow_memory_copy.

Regarding the default value (now True):

• @maartenbreddels preferred the "performance safety" of False
• @jorisvandenbossche pointed out that currently strings in Pandas (implemented via object dtype) will always require a copy, but that that in the next Pandas release there will be an Arrow-based variable length string option which can be zero-copy.

I believe that there will be enough cases that will always require a copy (object-strings, columns backed by non-contiguous arrays, non-columnar data storage like for Koalas and Ibis) that using True as the default would be best. And then the very performance-sensitive (= usually more advanced) users can set it to False.

[rgommers]

Make update in the requirements doc

Adding the following to the requirements docs:

We'll also list some things that were discussed but are not requirements:
...
4. Support for strided storage in buffers.
_Rationale: this is supported by a subset of dataframes only, mainly those
that use NumPy arrays. In many real-world use cases, strided arrays will
force a copy at some point, so requiring contiguous memory layout (and hence
an extra copy at the moment `__dataframe__` is used) is considered a good
trade-off for reduced implementation complexity._

[rgommers]

Did some more testing and review, and found some issues. Also this PR is not against main, so I'll close it. Resubmitted as gh-51.

[rgommers]

Rephrasing this a bit:

 ``allow_copy`` is a keyword that defines whether or not the library is
allowed to make a copy of the data. This can for example happen if a
library supports strided buffers (those would need a copy, because this
protocol specifies contiguous buffers).

[rgommers]

"False and a copy is needed"

[rgommers]

This removes the actual check for striding (not x.strides == (x.dtype.itemsize,):), that needs to be put back.

[rgommers]

Logic:

ifnotx.strides== (x.dtype.itemsize,):
# The protocol does not support strided buffers, so a copy is# necessary. If that's not allowed, we need to raise an exception.ifallow_copy:
x=x.copy()
else:
raiseRuntimeError("Exports cannot be zero-copy in the case ""of a non-contiguous buffer")

[rgommers]

And then adding a test case shows there's a problem - we don't hold on to the memory.

[steff456]

This PR adds a flag to throw an exception if the export cannot be zero-copy. (e.g. for pandas, possible due to block manager where rows are contiguous and columns are not) .

• Add allow_zero_copy flag to the DataFrame class.
• Propagate the flag to the buffer and raise a RuntimeError when it is true
• Fix test_noncontiguous_columns
• Make update in the requirements doc

[rgommers]

To summarize the options for naming this keyword that were suggested yesterday:

• allow_copy (your current choice)
• allow_memory_copy
• no_copy_only

I like the positive allow_* better than the negative no_*. Happy with either or allow_copy or allow_memory_copy.

Regarding the default value (now True):

• @maartenbreddels preferred the "performance safety" of False
• @jorisvandenbossche pointed out that currently strings in Pandas (implemented via object dtype) will always require a copy, but that that in the next Pandas release there will be an Arrow-based variable length string option which can be zero-copy.

I believe that there will be enough cases that will always require a copy (object-strings, columns backed by non-contiguous arrays, non-columnar data storage like for Koalas and Ibis) that using True as the default would be best. And then the very performance-sensitive (= usually more advanced) users can set it to False.

[rgommers]

Make update in the requirements doc

Adding the following to the requirements docs:

We'll also list some things that were discussed but are not requirements:
...
4. Support for strided storage in buffers.
_Rationale: this is supported by a subset of dataframes only, mainly those
that use NumPy arrays. In many real-world use cases, strided arrays will
force a copy at some point, so requiring contiguous memory layout (and hence
an extra copy at the moment `__dataframe__` is used) is considered a good
trade-off for reduced implementation complexity._

[rgommers]

Did some more testing and review, and found some issues. Also this PR is not against main, so I'll close it. Resubmitted as gh-51.

[rgommers]

Rephrasing this a bit:

 ``allow_copy`` is a keyword that defines whether or not the library is
allowed to make a copy of the data. This can for example happen if a
library supports strided buffers (those would need a copy, because this
protocol specifies contiguous buffers).

[rgommers]

"False and a copy is needed"

[rgommers]

This removes the actual check for striding (not x.strides == (x.dtype.itemsize,):), that needs to be put back.

[rgommers]

Logic:

ifnotx.strides== (x.dtype.itemsize,):
# The protocol does not support strided buffers, so a copy is# necessary. If that's not allowed, we need to raise an exception.ifallow_copy:
x=x.copy()
else:
raiseRuntimeError("Exports cannot be zero-copy in the case ""of a non-contiguous buffer")

[rgommers]

And then adding a test case shows there's a problem - we don't hold on to the memory.

[steff456]

This PR adds a flag to throw an exception if the export cannot be zero-copy. (e.g. for pandas, possible due to block manager where rows are contiguous and columns are not) .

• Add allow_zero_copy flag to the DataFrame class.
• Propagate the flag to the buffer and raise a RuntimeError when it is true
• Fix test_noncontiguous_columns
• Make update in the requirements doc

[rgommers]

To summarize the options for naming this keyword that were suggested yesterday:

• allow_copy (your current choice)
• allow_memory_copy
• no_copy_only

I like the positive allow_* better than the negative no_*. Happy with either or allow_copy or allow_memory_copy.

Regarding the default value (now True):

• @maartenbreddels preferred the "performance safety" of False
• @jorisvandenbossche pointed out that currently strings in Pandas (implemented via object dtype) will always require a copy, but that that in the next Pandas release there will be an Arrow-based variable length string option which can be zero-copy.

I believe that there will be enough cases that will always require a copy (object-strings, columns backed by non-contiguous arrays, non-columnar data storage like for Koalas and Ibis) that using True as the default would be best. And then the very performance-sensitive (= usually more advanced) users can set it to False.

[rgommers]

Make update in the requirements doc

Adding the following to the requirements docs:

We'll also list some things that were discussed but are not requirements:
...
4. Support for strided storage in buffers.
_Rationale: this is supported by a subset of dataframes only, mainly those
that use NumPy arrays. In many real-world use cases, strided arrays will
force a copy at some point, so requiring contiguous memory layout (and hence
an extra copy at the moment `__dataframe__` is used) is considered a good
trade-off for reduced implementation complexity._

[rgommers]

Did some more testing and review, and found some issues. Also this PR is not against main, so I'll close it. Resubmitted as gh-51.

[rgommers]

Rephrasing this a bit:

 ``allow_copy`` is a keyword that defines whether or not the library is
allowed to make a copy of the data. This can for example happen if a
library supports strided buffers (those would need a copy, because this
protocol specifies contiguous buffers).

[rgommers]

"False and a copy is needed"

[rgommers]

This removes the actual check for striding (not x.strides == (x.dtype.itemsize,):), that needs to be put back.

[rgommers]

Logic:

ifnotx.strides== (x.dtype.itemsize,):
# The protocol does not support strided buffers, so a copy is# necessary. If that's not allowed, we need to raise an exception.ifallow_copy:
x=x.copy()
else:
raiseRuntimeError("Exports cannot be zero-copy in the case ""of a non-contiguous buffer")

[rgommers]

And then adding a test case shows there's a problem - we don't hold on to the memory.

[steff456]

This PR adds a flag to throw an exception if the export cannot be zero-copy. (e.g. for pandas, possible due to block manager where rows are contiguous and columns are not) .

• Add allow_zero_copy flag to the DataFrame class.
• Propagate the flag to the buffer and raise a RuntimeError when it is true
• Fix test_noncontiguous_columns
• Make update in the requirements doc

[rgommers]

To summarize the options for naming this keyword that were suggested yesterday:

• allow_copy (your current choice)
• allow_memory_copy
• no_copy_only

I like the positive allow_* better than the negative no_*. Happy with either or allow_copy or allow_memory_copy.

Regarding the default value (now True):

• @maartenbreddels preferred the "performance safety" of False
• @jorisvandenbossche pointed out that currently strings in Pandas (implemented via object dtype) will always require a copy, but that that in the next Pandas release there will be an Arrow-based variable length string option which can be zero-copy.

I believe that there will be enough cases that will always require a copy (object-strings, columns backed by non-contiguous arrays, non-columnar data storage like for Koalas and Ibis) that using True as the default would be best. And then the very performance-sensitive (= usually more advanced) users can set it to False.

[rgommers]

Make update in the requirements doc

Adding the following to the requirements docs:

We'll also list some things that were discussed but are not requirements:
...
4. Support for strided storage in buffers.
_Rationale: this is supported by a subset of dataframes only, mainly those
that use NumPy arrays. In many real-world use cases, strided arrays will
force a copy at some point, so requiring contiguous memory layout (and hence
an extra copy at the moment `__dataframe__` is used) is considered a good
trade-off for reduced implementation complexity._

[rgommers]

Did some more testing and review, and found some issues. Also this PR is not against main, so I'll close it. Resubmitted as gh-51.

[rgommers]

Rephrasing this a bit:

 ``allow_copy`` is a keyword that defines whether or not the library is
allowed to make a copy of the data. This can for example happen if a
library supports strided buffers (those would need a copy, because this
protocol specifies contiguous buffers).

[rgommers]

"False and a copy is needed"

[rgommers]

This removes the actual check for striding (not x.strides == (x.dtype.itemsize,):), that needs to be put back.

[rgommers]

Logic:

ifnotx.strides== (x.dtype.itemsize,):
# The protocol does not support strided buffers, so a copy is# necessary. If that's not allowed, we need to raise an exception.ifallow_copy:
x=x.copy()
else:
raiseRuntimeError("Exports cannot be zero-copy in the case ""of a non-contiguous buffer")

[rgommers]

And then adding a test case shows there's a problem - we don't hold on to the memory.

[steff456]

This PR adds a flag to throw an exception if the export cannot be zero-copy. (e.g. for pandas, possible due to block manager where rows are contiguous and columns are not) .

• Add allow_zero_copy flag to the DataFrame class.
• Propagate the flag to the buffer and raise a RuntimeError when it is true
• Fix test_noncontiguous_columns
• Make update in the requirements doc

[rgommers]

To summarize the options for naming this keyword that were suggested yesterday:

• allow_copy (your current choice)
• allow_memory_copy
• no_copy_only

I like the positive allow_* better than the negative no_*. Happy with either or allow_copy or allow_memory_copy.

Regarding the default value (now True):

• @maartenbreddels preferred the "performance safety" of False
• @jorisvandenbossche pointed out that currently strings in Pandas (implemented via object dtype) will always require a copy, but that that in the next Pandas release there will be an Arrow-based variable length string option which can be zero-copy.

I believe that there will be enough cases that will always require a copy (object-strings, columns backed by non-contiguous arrays, non-columnar data storage like for Koalas and Ibis) that using True as the default would be best. And then the very performance-sensitive (= usually more advanced) users can set it to False.

[rgommers]

Make update in the requirements doc

Adding the following to the requirements docs:

We'll also list some things that were discussed but are not requirements:
...
4. Support for strided storage in buffers.
_Rationale: this is supported by a subset of dataframes only, mainly those
that use NumPy arrays. In many real-world use cases, strided arrays will
force a copy at some point, so requiring contiguous memory layout (and hence
an extra copy at the moment `__dataframe__` is used) is considered a good
trade-off for reduced implementation complexity._

[rgommers]

Did some more testing and review, and found some issues. Also this PR is not against main, so I'll close it. Resubmitted as gh-51.

[rgommers]

Rephrasing this a bit:

 ``allow_copy`` is a keyword that defines whether or not the library is
allowed to make a copy of the data. This can for example happen if a
library supports strided buffers (those would need a copy, because this
protocol specifies contiguous buffers).

[rgommers]

"False and a copy is needed"

[rgommers]

This removes the actual check for striding (not x.strides == (x.dtype.itemsize,):), that needs to be put back.

[rgommers]

Logic:

ifnotx.strides== (x.dtype.itemsize,):
# The protocol does not support strided buffers, so a copy is# necessary. If that's not allowed, we need to raise an exception.ifallow_copy:
x=x.copy()
else:
raiseRuntimeError("Exports cannot be zero-copy in the case ""of a non-contiguous buffer")

[rgommers]

And then adding a test case shows there's a problem - we don't hold on to the memory.

[steff456]

This PR adds a flag to throw an exception if the export cannot be zero-copy. (e.g. for pandas, possible due to block manager where rows are contiguous and columns are not) .

• Add allow_zero_copy flag to the DataFrame class.
• Propagate the flag to the buffer and raise a RuntimeError when it is true
• Fix test_noncontiguous_columns
• Make update in the requirements doc

[rgommers]

To summarize the options for naming this keyword that were suggested yesterday:

• allow_copy (your current choice)
• allow_memory_copy
• no_copy_only

I like the positive allow_* better than the negative no_*. Happy with either or allow_copy or allow_memory_copy.

Regarding the default value (now True):

• @maartenbreddels preferred the "performance safety" of False
• @jorisvandenbossche pointed out that currently strings in Pandas (implemented via object dtype) will always require a copy, but that that in the next Pandas release there will be an Arrow-based variable length string option which can be zero-copy.

I believe that there will be enough cases that will always require a copy (object-strings, columns backed by non-contiguous arrays, non-columnar data storage like for Koalas and Ibis) that using True as the default would be best. And then the very performance-sensitive (= usually more advanced) users can set it to False.

[rgommers]

Make update in the requirements doc

Adding the following to the requirements docs:

We'll also list some things that were discussed but are not requirements:
...
4. Support for strided storage in buffers.
_Rationale: this is supported by a subset of dataframes only, mainly those
that use NumPy arrays. In many real-world use cases, strided arrays will
force a copy at some point, so requiring contiguous memory layout (and hence
an extra copy at the moment `__dataframe__` is used) is considered a good
trade-off for reduced implementation complexity._

[rgommers]

Did some more testing and review, and found some issues. Also this PR is not against main, so I'll close it. Resubmitted as gh-51.

[rgommers]

Rephrasing this a bit:

 ``allow_copy`` is a keyword that defines whether or not the library is
allowed to make a copy of the data. This can for example happen if a
library supports strided buffers (those would need a copy, because this
protocol specifies contiguous buffers).

[rgommers]

"False and a copy is needed"

[rgommers]

This removes the actual check for striding (not x.strides == (x.dtype.itemsize,):), that needs to be put back.

[rgommers]

Logic:

ifnotx.strides== (x.dtype.itemsize,):
# The protocol does not support strided buffers, so a copy is# necessary. If that's not allowed, we need to raise an exception.ifallow_copy:
x=x.copy()
else:
raiseRuntimeError("Exports cannot be zero-copy in the case ""of a non-contiguous buffer")

[rgommers]

And then adding a test case shows there's a problem - we don't hold on to the memory.