Python bindings for tatami

Overview

The mattress package implements Python bindings to the tatami C++ library for matrix representations. Downstream packages can use mattress to develop C++ extensions that are interoperable with many different matrix classes, e.g., dense, sparse, delayed or file-backed. mattress is inspired by the beachmat Bioconductor package, which does the same thing for R packages.

Instructions

mattress is published to PyPI, so installation is simple:

pip install mattress

mattress is intended for Python package developers writing C++ extensions that operate on matrices. The aim is to allow package C++ code to accept all types of matrix representations without requiring re-compilation of the associated code. To achive this:

1. Add mattress.includes() and assorthead.includes() to the compiler's include path. This can be done through include_dirs= of the Extension() definition in setup.py or by adding a target_include_directories() in CMake, depending on the build system.
2. Call mattress.initialize() on a Python matrix object to wrap it in a tatami-compatible C++ representation. This returns an InitializedMatrix with a ptr property that contains a pointer to the C++ matrix.
3. Pass ptr to C++ code as a uintptr_t referencing a tatami::Matrix, which can be interrogated as described in the tatami documentation.

So, for example, the C++ code in our downstream package might look like the code below:

#include"mattress.h"intdo_something(uintptr_t ptr) {
constauto& mat_ptr = mattress::cast(ptr)->ptr;
// Do something with the tatami interface.return1;
}
// Assuming we're using pybind11, but any framework that can accept a uintptr_t is fine.PYBIND11_MODULE(lib_downstream, m) {
m.def("do_something", &do_something);
}

Which can then be called from Python:

from . importlib_downstreamaslibfrommattressimportinitializedefdo_something(x):
tmat=initialize(x)
returnlib.do_something(tmat.ptr)

Check out the included header for more definitions.

Supported matrices

Dense numpy matrices of varying numeric type:

importnumpyasnpfrommattressimportinitializex=np.random.rand(1000, 100)
init=initialize(x)
ix= (x*100).astype(np.uint16)
init2=initialize(ix)

Compressed sparse matrices from scipy with varying index/data types:

fromscipyimportsparseasspfrommattressimportinitializexc=sp.random(100, 20, format="csc")
init=initialize(xc)
xr=sp.random(100, 20, format="csc", dtype=np.uint8)
init2=initialize(xr)

Delayed arrays from the delayedarray package:

fromdelayedarrayimportDelayedArrayfromscipyimportsparseasspfrommattressimportinitializeimportnumpyxd=DelayedArray(sp.random(100, 20, format="csc"))
xd=numpy.log1p(xd*5)
init=initialize(xd)

Sparse arrays from delayedarray are also supported:

importdelayedarrayfromnumpyimportfloat64, int32frommattressimportinitializesa=delayedarray.SparseNdarray((50, 20), None, dtype=float64, index_dtype=int32)
init=initialize(sa)

See below to extend initialize() to custom matrix representations.

Utility methods

The InitializedMatrix instance returned by initialize() provides a few Python-visible methods for querying the C++ matrix.

init.nrow() //numberofrowsinit.column(1) //contentsofcolumn1init.sparse() //whetherthematrixissparse.

It also has a few methods for computing common statistics:

init.row_sums()
init.column_variances(num_threads=2)
grouping= [i%3foriinrange(init.ncol())]
init.row_medians_by_group(grouping)
init.row_nan_counts()
init.column_ranges()

These are mostly intended for non-intensive work or testing/debugging. It is expected that any serious computation should be performed by iterating over the matrix in C++.

Operating on an existing pointer

If we already have a InitializedMatrix, we can easily apply additional operations by wrapping it in the relevant delayedarray layers and calling initialize() afterwards. For example, if we want to add a scalar, we might do:

fromdelayedarrayimportDelayedArrayfrommattressimportinitializeimportnumpyx=numpy.random.rand(1000, 10)
init=initialize(x)
wrapped=DelayedArray(init) +1init2=initialize(wrapped)

This is more efficient as it re-uses the InitializedMatrix already generated from x. It is also more convenient as we don't have to carry around x to generate init2.

Extending to custom matrices

Developers can extend mattress to custom matrix classes by registering new methods with the initialize() generic. This should return a InitializedMatrix object containing a uintptr_t cast from a pointer to a tatami::Matrix (see the included header). Once this is done, all calls to initialize() will be able to handle matrices of the newly registered types.

from . importlib_downstreamaslibimportmattress@mattress.initialize.registerdef_initialize_my_custom_matrix(x: MyCustomMatrix):
data=x.some_internal_datareturnmattress.InitializedMatrix(lib.initialize_custom(data))

If the initialized tatami::Matrix contains references to Python-managed data, e.g., in NumPy arrays, we must ensure that the data is not garbage-collected during the lifetime of the tatami::Matrix. This is achieved by storing a reference to the data in the original member of the mattress::BoundMatrix.