How to consume a single buffer & connection to array interchange

[rgommers]

For dataframe interchange, the smallest building block is a "buffer" (see gh-35, gh-38) - a block of memory. Interpreting that is nontrivial, especially if the goal is to build an interchange protocol in Python. That's why DLPack, buffer protocol, __array_interface__, __cuda_array_interface__, __array__ and __arrow_array__ all exist, and are still complicated.

For what a buffer is, currently it's only a data pointer (ptr) and a size (bufsize) which together describe a contiguous block of memory, plus a device attribute (__dlpack_device__) and optionally DLPack support (__dlpack__). One open question is:

• Should a buffer support strides? This helps describe actual memory layout, e.g. Pandas can have strided columns (see Add a prototype of the dataframe interchange protocol #38 (comment))

The other, larger question is how to make buffers nice to deal with for implementers of the protocol. The current Pandas prototype shows the issue:

def convert_column_to_ndarray(col : ColumnObject) -> np.ndarray:
 """
 """
 if col.offset != 0:
 raise NotImplementedError("column.offset > 0 not handled yet")

 if col.describe_null not in (0, 1):
 raise NotImplementedError("Null values represented as masks or "
 "sentinel values not handled yet")

 # Handle the dtype
 _dtype = col.dtype
 kind = _dtype[0]
 bitwidth = _dtype[1]
 if _dtype[0] not in (0, 1, 2, 20):
 raise RuntimeError("Not a boolean, integer or floating-point dtype")

 _ints = {8: np.int8, 16: np.int16, 32: np.int32, 64: np.int64}
 _uints = {8: np.uint8, 16: np.uint16, 32: np.uint32, 64: np.uint64}
 _floats = {32: np.float32, 64: np.float64}
 _np_dtypes = {0: _ints, 1: _uints, 2: _floats, 20: {8: bool}}
 column_dtype = _np_dtypes[kind][bitwidth]

 # No DLPack yet, so need to construct a new ndarray from the data pointer
 # and size in the buffer plus the dtype on the column
 _buffer = col.get_data_buffer()
 ctypes_type = np.ctypeslib.as_ctypes_type(column_dtype)
 data_pointer = ctypes.cast(_buffer.ptr, ctypes.POINTER(ctypes_type))

 # NOTE: `x` does not own its memory, so the caller of this function must
 # either make a copy or hold on to a reference of the column or
 # buffer! (not done yet, this is pretty awful ...)
 x = np.ctypeslib.as_array(data_pointer,
 shape=(_buffer.bufsize // (bitwidth//8),))

 return x

From #38 (review) (@kkraus14 & @rgommers):

In __cuda_array_interface__ we've generally stated that holding a reference to the producing object must guarantee the lifetime of the memory and that has worked relatively well.

Yes that works and I've thought about it. The trouble is where to hold the reference. You really need one reference per buffer, not just store a reference to the whole exchange dataframe object (buffers can end up elsewhere outside the new pandas dataframe here). And given that a buffer just has a raw pointer plus a size, there's nothing to hold on to. I don't think there's a sane pure Python solution.

__cuda_array_interface__ is directly attached to the object you need to hold on to, which is not the case for this Buffer.

I'd argue this is a place where we should really align with the array interchange protocol though as the same problem is being solved there.

Yep, for numerical data types the solution can simply be: hurry up with implementing __dlpack__, and the problem goes away. The dtypes that DLPack does not support are more of an issue.

From #38 (comment) (@jorisvandenbossche):

I personally think it would be useful to keep those existing interface methods (or array, or arrow_array). For people that are using those interface, that will be easier to interface with the interchange protocol than manually converting the buffers.

Alternative/extension to the current design

We could change the plain memory description + __dlpack__ to:

1. Implementations MUST support a memory description with ptr, bufsize, and device
2. Implementations MAY support buffers in their native format (e.g. add a native enum attribute, and if both producer and consumer happen to use that native format, they can call the corresponding protocol - __arrow_array__ or __array__)
3. Implementations MAY support any exchange protocol (DLPack, __cuda_array_interface__, buffer protocol, __array_interface__).

(1) is required for any implementation to be able to talk to any other implementation, but also the most clunky to support because it needs to solve the "who owns this memory and how do you prevent it from being freed" all over again. What is needed there is

The advantage of (2) and (3) are that they have the most hairy issue already solved, and will likely be faster.

And the MUST/MAY should address @kkraus14's concern that people will just standardize on the lowest common denominator (numpy).

What is missing for dealing with memory buffers

A summary of why this is hard is:

1. Underlying implementations are not compatible. E.g., NumPy doesn't support variable length strings or bit masks, Arrow does not support strided arrays or byte masks.
2. DLPack is the only protocol with device support, but it does not support all dtypes that are needed.

So what we are aiming for (ambitiously) is:

• Something flexible enough to be a superset of NumPy and Arrow, with full device support.
• In pure Python

The "holding a reference to the producing object must guarantee the lifetime of the memory and that has worked relatively well" seems necessary for supporting the raw memory description. This probably means that (a) the Buffer object should include the right Python object to keep a reference to (for Pandas that would typically be a 1-D numpy array), and (b) there must be some machinery to keep this reference alive (TBD what that looks like, likely not pure Python) in the implementation.

[kkraus14]

Yes that works and I've thought about it. The trouble is where to hold the reference. You really need one reference per buffer, not just store a reference to the whole exchange dataframe object (buffers can end up elsewhere outside the new pandas dataframe here). And given that a buffer just has a raw pointer plus a size, there's nothing to hold on to. I don't think there's a sane pure Python solution.

The way we handle this in cudf is we have a cudf.Buffer class which is non-owning. It has a ._owner attribute which holds a reference to an object that is responsible for managing the lifetime of the actual memory. For our owning memory, we have our rmm (RAPIDS Memory Manager) library, and it's associated class of rmm.DeviceBuffer which uses a C++ smart pointer to a C++ class under the hood to control memory lifetime. Regardless, we haven't had any issues with the cudf.Buffer class and using ._owner to hold a reference to an object that guarantees the lifetime of the memory referenced by the Buffer instance.

__cuda_array_interface__ is directly attached to the object you need to hold on to, which is not the case for this Buffer.

Actually in our case it directly is. We often hold a reference to a cupy array or numba device array or pytorch tensor as the underlying owner of memory underneath a Buffer. We also directly implement __cuda_array_interface__ in rmm.DeviceBuffer (https://github.com/rapidsai/rmm/blob/branch-0.19/python/rmm/_lib/device_buffer.pyx#L115-L124) and cudf.Buffer (https://github.com/rapidsai/cudf/blob/branch-0.19/python/cudf/cudf/core/buffer.py#L75-L84) so that buffers can be directly consumed by the other array libraries and then viewed as a different type.

I.E. we do this in a few places with string columns, where we use __cuda_array_interface__ to share the offsets buffer into something like CuPy and then view it as int32 to do some logical operations on it.

Yep, for numerical data types the solution can simply be: hurry up with implementing __dlpack__, and the problem goes away. The dtypes that DLPack does not support are more of an issue.

Yes, for numerical and other columns that can be represented by a single buffer it would be nice to support __dlpack__ directly, but what I'm thinking for a more general column interchange protocol is that it would be composed of a dlpack usage per buffer. I.E. for a string column with a nullmask buffer, a character buffer, and an offsets buffer, we could imagine exchanging the 3 buffers using 3 dlpack instances. The main problem I can envision in using this approach is that certain libraries may have columns be the memory owning entity as opposed to direct buffers, which could then make it difficult for them to get out to separate reference counted buffer objects for controlling the memory lifetime separately.

[rgommers]

Regardless, we haven't had any issues with the cudf.Buffer class and using ._owner to hold a reference to an object that guarantees the lifetime of the memory referenced by the Buffer instance.

That makes sense. This attribute:

 owner : object, optional
 Python object to which the lifetime of the memory
 allocation is tied. If provided, a reference to this
 object is kept in this Buffer.

is also needed in this protocol - except it can't be optional.

We often hold a reference to a cupy array or numba device array or pytorch tensor as the underlying owner of memory underneath a Buffer.

That looks similar (the principle, not the code) to how numpy deals with __array_interface__. In particular (stripping out all null checks etc, see multiarray/ctors.c):

PyArray_FromInterface(PyObject *origin)

 /* Get data buffer from interface specification */
 attr = _PyDict_GetItemStringWithError(iface, "data");

 /* Case for data access through pointer */
 if (attr && PyTuple_Check(attr)) {
 PyObject *dataptr;

 dataptr = PyTuple_GET_ITEM(attr, 0);
 if (PyLong_Check(dataptr)) {
 data = PyLong_AsVoidPtr(dataptr);

 base = origin;

 ret = (PyArrayObject *)PyArray_NewFromDescrAndBase(
 &PyArray_Type, dtype,
 n, dims, NULL, data,
 dataflags, NULL, base);

Here origin is equivalent to the owner in cuDF, and a reference to it is stored in the base attribute of a numpy array.

[jorisvandenbossche]

Alternative/extension to the current design

Those 3 bullet points of (possibly) supported APIs, is this for the Buffer level? Because it's probably also useful to have that on the Column level as well.

E.g., NumPy doesn't support variable length strings or bit masks, Arrow does not support strided arrays or byte masks.

Might not be that relevant, but: Arrow does support byte masks in some way, I.e. as boolean arrays (it just can't use it as nulls to compute anything with it, but you can store it). You may not be able to use byte masks in a plain Arrow array, but at the same time I don't think you can use byte masks in numpy's __array_interface__ either? So also for numpy you need to treat this as a separate array/buffer, so the same can be done for Arrow (eg in Arrow it could be a StructArray with a values field and a mask field)

[kkraus14]

You may not be able to use byte masks in a plain Arrow array, but at the same time I don't think you can use byte masks in numpy's __array_interface__ either? So also for numpy you need to treat this as a separate array/buffer, so that same can be done for Arrow (eg in Arrow it could be a StructArray with a values field and a mask field)

In __array_interface__ and __cuda_array_interface__ there's a mask attribute that is another object that exposes __array_interface__ or __cuda_array_interface__ respectively that allows for a byte mask.

[jorisvandenbossche]

Ah, cool, I didn't know that.
Now, in practice it seems that numpy doesn't use that? (a numpy masked array doesn't return it, and if you create a numpy array from an object returning a dict with a "mask" key, nothing happens with it)

[rgommers]

Now, in practice it seems that numpy doesn't use that? (a numpy masked array doesn't return it, and if you create a numpy array from an object returning a dict with a "mask" key, nothing happens with it)

Indeed, the mask attribute has been in the docs since they were imported into the numpy repo in 2008, but it was never used AFAICT. There was once a plan for a proper masked array implementation in numpy core, rather than the bolted on bit in numpy.ma I believe.

[rgommers]

We could change the plain memory description + __dlpack__ to:

1. Implementations MUST support a memory description with `ptr`, `bufsize`, and device

2. Implementations MAY support buffers in their native format (e.g. add a `native` enum attribute, and if both producer and consumer happen to use that native format, they can call the corresponding protocol - `__arrow_array__` or `__array__`)

3. Implementations MAY support any exchange protocol (DLPack, `__cuda_array_interface__`, buffer protocol, `__array_interface__`).

Summarizing some of the main comments:

1. It may make sense to have these protocols on the column level too. that would make life easier for the consumer. That would have minor differences in execution semantics (memory may get released at different times), but it would help the majority of use cases.
2. Using only DLPack at the buffer level seems attractive. But, it doesn't support all dtypes, so may complicate things.
3. A reference implementation will be useful; at least at the buffer level most libraries will want/need to do the same thing, and it's error-prone code.

Action for me: create two implementations, a minimal and a maximal one, so it's easier to compare.

[kkraus14]

• Using only DLPack at the buffer level seems attractive. But, it doesn't support all dtypes, so may complicate things.

What dtypes does it not support that it needs to at the buffer level? I thought we had previously decided that buffers were untyped since they could be interpreted in different ways? Even if they're typed, I would assume they're int or uint types which dlpack supports.

[jorisvandenbossche]

Using only DLPack at the buffer level seems attractive. But, it doesn't support all dtypes, so may complicate things.

But buffers don't have a dtype?

[rgommers]

Oh right, resolution of that was "due to variable length strings, buffers must be untyped (dtype info lives on the column not the buffer)". There's still an inconsistency in that __dlpack__ does contain the dtype, so what we then should do is create some convention (e.g. always use intxx dtypes at the buffer level, with the bit width corresponding to that of the dtype at the column level).

[kkraus14]

And this goes back to the strides discussion, that if buffers are untyped, does it really make sense to have strides on them versus controlling that on the Column level? I.E. take something like a theoretical uint128 array, where it would have a data buffer. If I knew the max value of the array was 10, then I could have a typecast operation to basically any uint or int type be a zero-memory op and instead just set a new stride. I would argue this makes a lot more sense to handle at the Column level as opposed to the Buffer level.

[jorisvandenbossche]

Now we are considering going all in on DLPack at the buffer level, a potential alternative to consider would be using the actual Arrow C Data Interface instead? (at the column level)

When describing/discussing the requirements in #35, an argument against the Arrow C Data Interface is that we were looking for a Python __dataframe__ interface rather than a C-level interface. But __dataframe__ itself would still be the same python interface as we have been discussing, but the actual interchange of column (chunks) could still use the Arrow C interface (since we consider DLPack at this level, it seems we are fine with a non-pure Python exchange interface).

In the end, for primitive arrays, DLPack and Arrow C Data interface are basically the same (it's almost the same C struct, in arrow there are only some additional null pointers for child/dictionary arrays that are not needed for a primitive array; eg implementing conversion to/from numpy for primitive arrays wouldn't be harder compared to DLPack). But, Arrow C Data interface natively supports the more complex nested/multi-buffer cases we also want to support (eg variable length strings, categoricals).

What's missing for using the Arrow C Data interface is:

• an official Python interface to access the pointers (there are currently semi-private _export_to_c/_import_from_c, but those could be made official in a method like __arrow_c_data__, similar as __dlpack__)
• support for devices (as discussed before)

Both those points have been discussed / worked out for DLPack in context of the Array API (I think? didn't fully follow this), so transferring what has been learned from that to add the same capabilities to the Arrow C data interface could be a nice addition to Arrow and would IMO be a great outcome of the consortium efforts.

[kkraus14]

I think it was decided that strided data would be supported which the Arrow C data interface doesn't support as well which we'd need to propose adding.

Another key point is with the Arrow C data interface is that there isn't a way to control lifetime of individual buffers versus the column. I.E. for a string column say I use the __arrow_c_data__ protocol to go between libraries and then go down to the offsets buffer with both the original string column and the interchanged string column both going out of scope. I need a way to guarantee the memory for the offsets buffer stays alive, and the only way to do that is to make sure the release callback isn't called in this situation.

[jorisvandenbossche]

I think it was decided that strided data would be supported which the Arrow C data interface doesn't support as well which we'd need to propose adding.

I looked back at the last meeting notes, and there are some notes about how to implement it (should the strides live at buffer or column level? etc), and not much about reasons to support it (but maybe we didn't capture everything in the notes).
At #38 (comment) there is a bit of discussion about it, and a reason is the current pandas internals that would not be zero-copy in certain cases (for DataFrames constructed from a 2D numpy array, and that has not yet been copied / operated on after construction). Personally I don't find that a blocking issue for not supporting strides.

Another key point is with the Arrow C data interface is that there isn't a way to control lifetime of individual buffers versus the column ... and the only way to do that is to make sure the release callback isn't called in this situation.

And is this problematic to ensure? (honest question, this is out of my comfort zone)

[kkraus14]

And is this problematic to ensure? (honest question, this is out of my comfort zone)

I would say it adds some complexity but is manageable, but moreso on devices like GPUs with much more limited memory available, every extra byte counts where we'd want to as aggressively release buffers / free memory as possible.