Skip to content

Add specification for the __binsparse__ protocol - #912

Draft
hameerabbasi wants to merge 7 commits into
data-apis:mainfrom
hameerabbasi:binsparse-protocol
Draft

Add specification for the __binsparse__ protocol#912
hameerabbasi wants to merge 7 commits into
data-apis:mainfrom
hameerabbasi:binsparse-protocol

Conversation

@hameerabbasi

@hameerabbasihameerabbasi commented Mar 10, 2025

Copy link
Copy Markdown
Contributor

This pull request adds the specification for the binsparse protocol (closes#840).

@willow-ahrens@BenBrock from the binsparse team.
@mtsokol@ivirshup for scipy.sparse
@leofang for cupyx.sparse
@pearu for torch.sparse
@jakevdp for JAX/TensorFlow

Introduction

The binsparse protocol is meant to be a specification for on-disk storage of ND sparse arrays. It requires just two things from a back-end implementing it:

a. A way to store 1D and 2D (dense) arrays (we have this via DLPack)
b. A way to parse and interpret JSON (we have this via the json module)

Psuedocode implementation

Here's a psuedocode example using two libraries, xp1 and xp2, both supporting sparse arrays:

# In library code:xp2_sparray=xp2.from_binsparse(xp1_sparray, ...)
# Orxp2_sparray=xp2.asarray(xp1_sparray, ...)
# This psuedocode impl is common between `xp1` and `xp2`deffrom_binsparse(x: object, /, *, device: device|None=None, copy: bool|None=None) ->array:
binsparse_descr=getattr(x, "__binsparse_descriptor__", None)
binsparse_impl=getattr(x, "__binsparse__", None)
ifbinsparse_implisNoneorbinsparse_descrisNone:
raiseTypeError(...)
binsparse_descriptor=binsparse_descr()
# Will raise an error if the format/descriptor is unsupported.sparse_type=_type_from_binsparse_descriptor(binsparse_descriptor)
constituent_arrays=binsparse_impl()
my_constituent_arrays= {
k: from_dlpack(arr, device=device, copy=copy) fork, arrinconstituent_arrays.items()
}
returnsparse_type.from_strided_arrays(my_constituent_arrays, shape=...)

Compare this to the following example converting SciPy COO arrays to PyData/Sparse:

importsparseimportscipy.sparseasspsimportnumpyasnpsparse_array=sparse.COO(np.stack(sps_array.coords), sps_array.data, shape=sps_array.shape)

Parallel implementation in sparse: pydata/sparse#764
Parallel implementation in SciPy: scipy/scipy#22553

@hameerabbasi
hameerabbasiforce-pushed the binsparse-protocol branch 2 times, most recently from f5d1642 to f01c98dCompareMarch 10, 2025 10:50
Comment threadspec/draft/design_topics/data_interchange.rst Outdated
Comment threadspec/draft/design_topics/data_interchange.rst Outdated
Comment threadspec/draft/extensions/sparse_interchange.rst Outdated
Comment threadsrc/array_api_stubs/_draft/array_object.py Outdated
Comment threadsrc/array_api_stubs/_draft/array_object.py Outdated
Comment threadsrc/array_api_stubs/_draft/creation_functions.py Outdated
Comment threadsrc/array_api_stubs/_draft/creation_functions.py Outdated
Comment threadsrc/array_api_stubs/_draft/creation_functions.py Outdated
Comment threadsrc/array_api_stubs/_draft/creation_functions.py Outdated
Comment threadsrc/array_api_stubs/_draft/creation_functions.py Outdated
@kgrytekgryte added API extension Adds new functions or objects to the API. topic: DLPack DLPack. labels Apr 3, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

API extensionAdds new functions or objects to the API.topic: DLPackDLPack.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RFC: In-memory sparse array interchange

2 participants

@hameerabbasi@kgryte