Is it possible to share Rcpp object/pointer between R and Python?

[gregorgorjanc]

I am developing R access to the tskit C API (a library for working with massive genomic datasets by mapping how we relate to each other through our ancestors in the past - https://tskit.dev/). tskit has a very good C API that is further enhanced via a Python API. Both are very complete, well tested, and documented, but the Python API is more popular. I am well aware of reticulate to call Python API from R, but that doesn't suit all our needs(calling the tskit C API within loops in R/Rcpp, hence my effort to bring tskit C API to R/Rcpp).

I have developed an R package tskitr at https://github.com/HighlanderLab/tskitr which includes the tskit's C code and have 1) wrote some Rcpp functions that call the C API and 2) developed library build with instructions for other packages to use tskit C API via Rcpp in their work (the reason why I have built the package in the first place).

While this all works very well thanks to Rcpp, I don't want to duplicate the work done on the Python API. Most users will still be encouraged to analyse the objects using Python API, possibly via reticulate.

While we can build tskit objects on the R side (using Rcpp), export them to disk, load into Python and analyse in standalone or reticulate Python, I wonder if we could share Rcpp object/pointer between R and reticulate Python sessions? Doing a bit of LLMing, I surmise that the answer is no due to ABI incompatibility, but I thought I would ask real human experts! In the meanwhile I have written ts_r_to_py() and ts_py_to_r() that use disk-write-read to transfer between R and reticulate Python. tskit objects are complex so not sure I can figure out r_to_py()/py_to_r() wrappers and I only have a pointer in R session anyway - looking at the C++ code of the reticulate package I get lost very quickly.

Thanks!

[eddelbuettel]

From the top of my head, we should be able to send a pointer a struct around. That is what Arrow does after. But I do not have a working example nor can I think of one -- so what you do by serializing to disk seems like a fair approach. Alternatives would be to write to (in memory?) data bases like duckdb, or (if you remain on Linux) rely on shared memory to access a common object.

All these are great questions and would be really good to have a studied reference example.

[kevinushey]

I'm spitballing, but does tskit provide any APIs for creating a Python object from an existing C pointer to a tskit object? If so, I think you could safely share those pointer addresses between R / Rcpp and Python.

A somewhat bare version of what I'm imagining is the "advanced" Armadillo constructors, whereby you can create an Armadillo matrix from a pointer to auxiliary memory / data. https://arma.sourceforge.net/docs.html#adv_constructors_mat

[eddelbuettel]

Yup. It is a somewhat simpler example as Armadillo 'merely' reuses length * sizeof(x) for either an int or double vector. But the idea of the abstraction is the same: if we share the same C/C++ headers between Python and R so that sizeof(struct foo) is the same than we can instantiate a new object off a pointer. The other thought I had was that instead of serializing to disk as you do you could also look into one of the usual serialization-on-wire examples such as Protocol Buffers, MsgPack, ...

[gregorgorjanc]

I'm spitballing, but does tskit provide any APIs for creating a Python object from an existing C pointer to a tskit object? If so, I think you could safely share those pointer addresses between R / Rcpp and Python.

It's my understanding that Python API is wrapped around the C API, though I don't understand how this is done (I am not a Python developer). Will talk to tskit devs and come back.

[Enchufa2]

As Dirk said, this is exactly what Arrow does, so the answer is yes. You only need to ensure that both sides know the data structure, and then you pass the pointer around. r_to_py and py_to_r are S3 generics, so you just need to write your own methods for your data type. You can see what arrow does e.g. for arrow:::r_to_py.Array and arrow:::py_to_r.pyarrow.lib.Array.

And now that we are at it, I was taking a look and I see this. You are doing too much work here. You can just define a new XPtr type that knows the C function it needs to call to free the object it wraps. This is e.g. what units does here.

[gregorgorjanc]

Thanks @Enchufa2, this gives me plenty of study “pointers”!

[eddelbuettel]

Shared pointer is definitely one way to do this, and it is done by quite a few packages. Maybe search github.com/cran/ for XPtr or EXTPTRSXP or R_ExternalPtr. I really would like to construct a 'simple example' around this. Anybody else game for that?

[gregorgorjanc]

Here I am compiling pointers to what arrow does:

• r_to_py.Array() is defined here and looks like this:

r_to_py.Array <- function(x, convert = FALSE) {
schema_ptr <- allocate_arrow_schema()
array_ptr <- allocate_arrow_array()
on.exit({
delete_arrow_schema(schema_ptr)
delete_arrow_array(array_ptr)
})
# Import with convert = FALSE so that `_import_from_c` returns a Python object
pa <- reticulate::import("pyarrow", convert = FALSE)
x$export_to_c(array_ptr, schema_ptr)
out <- pa$Array$`_import_from_c`(
pyarrow_compatible_pointer(array_ptr),
pyarrow_compatible_pointer(schema_ptr)
)
# But set the convert attribute on the return object to the requested value
assign("convert", convert, out)
out
}

• allocate_arrow_array() (and delete_arrow_array()) look like:

R
# https://github.com/apache/arrow/blob/0ef674bc038282d2249a8e179f19a743554a4145/r/R/arrowExports.R#L259
allocate_arrow_array <- function() {
.Call(`_arrow_allocate_arrow_array`)
}
# https://github.com/apache/arrow/blob/0ef674bc038282d2249a8e179f19a743554a4145/r/R/arrowExports.R#L263
delete_arrow_array <- function(ptr) {
invisible(.Call(`_arrow_delete_arrow_array`, ptr))
}
C++
# https://github.com/apache/arrow/blob/0ef674bc038282d2249a8e179f19a743554a4145/r/src/bridge.cpp#L61
// [[arrow::export]]
arrow::r::Pointer<struct ArrowArray> allocate_arrow_array() { return {}; }
# https://github.com/apache/arrow/blob/0ef674bc038282d2249a8e179f19a743554a4145/r/src/bridge.cpp#L63C1-L64C86
// [[arrow::export]]
void delete_arrow_array(arrow::r::Pointer<struct ArrowArray> ptr) { ptr.finalize(); }

• x$export_to_c(array_ptr, schema_ptr) from here is a method on Array defined as an R6 class and it looks like:

...
export_to_c = function(array_ptr, schema_ptr) ExportArray(self, array_ptr, schema_ptr)
...

• ExportArray() from here looks like:

// [[arrow::export]]
void ExportArray(const std::shared_ptr<arrow::Array>& array,
arrow::r::Pointer<struct ArrowArray> array_ptr,
arrow::r::Pointer<struct ArrowSchema> schema_ptr) {
StopIfNotOk(arrow::ExportArray(*array, array_ptr, schema_ptr));
}

• pa$Array$_import_from_c from here looks like:

@staticmethod
def _import_from_c(in_ptr, type):
"""
Import Array from a C ArrowArray struct, given its pointer
and the imported array type.
Parameters
----------
in_ptr: int
The raw pointer to a C ArrowArray struct.
type: DataType or int
Either a DataType object, or the raw pointer to a C ArrowSchema
struct.
This is a low-level function intended for expert users.
"""
cdef:
void* c_ptr = _as_c_pointer(in_ptr)
void* c_type_ptr
shared_ptr[CArray] c_array
c_type = pyarrow_unwrap_data_type(type)
if c_type == nullptr:
# Not a DataType object, perhaps a raw ArrowSchema pointer
c_type_ptr = _as_c_pointer(type)
with nogil:
c_array = GetResultValue(ImportArray(
<ArrowArray*> c_ptr, <ArrowSchema*> c_type_ptr))
else:
with nogil:
c_array = GetResultValue(ImportArray(<ArrowArray*> c_ptr,
c_type))
return pyarrow_wrap_array(c_array)

• pyarrow_compatible_pointer(array_ptr) from here looks like:

pyarrow_compatible_pointer <- function(ptr) {
# GH-39933: Workaround because there is no built-in way to send a
# 64-bit integer to Python from an R object
py <- reticulate::import_builtins(convert = FALSE)
addr <- external_pointer_addr_character(ptr)
py$int(addr)
}

• external_pointer_addr_character() from here looks like:

// [[arrow::export]]
std::string external_pointer_addr_character(SEXP external_pointer) {
void* ptr_void = R_ExternalPtrAddr(external_pointer);
uint64_t ptr_int64 = reinterpret_cast<uintptr_t>(ptr_void);
return std::to_string(ptr_int64);
}

• py_to_r.pyarrow.lib.Array() from here looks like:

py_to_r.pyarrow.lib.Array <- function(x, ...) {
schema_ptr <- allocate_arrow_schema()
array_ptr <- allocate_arrow_array()
on.exit({
delete_arrow_schema(schema_ptr)
delete_arrow_array(array_ptr)
})
x$`_export_to_c`(
pyarrow_compatible_pointer(array_ptr),
pyarrow_compatible_pointer(schema_ptr)
)
Array$import_from_c(array_ptr, schema_ptr)
}

• _export_to_c() from here looks like:

 def _export_to_c(self, out_ptr, out_schema_ptr=0):
"""
Export to a C ArrowArray struct, given its pointer.
If a C ArrowSchema struct pointer is also given, the array type
is exported to it at the same time.
Parameters
----------
out_ptr: int
The raw pointer to a C ArrowArray struct.
out_schema_ptr: int (optional)
The raw pointer to a C ArrowSchema struct.
Be careful: if you don't pass the ArrowArray struct to a consumer,
array memory will leak. This is a low-level function intended for
expert users.
"""
cdef:
void* c_ptr = _as_c_pointer(out_ptr)
void* c_schema_ptr = _as_c_pointer(out_schema_ptr,
allow_null=True)
with nogil:
check_status(ExportArray(deref(self.sp_array),
<ArrowArray*> c_ptr,
<ArrowSchema*> c_schema_ptr))

• And registering S3 methods

.onLoad <- function(...) {
for (cl in c("Array", "ChunkedArray")) {
s3_register("reticulate::py_to_r", paste0("pyarrow.lib.", cl))
s3_register("reticulate::r_to_py", cl)
}
}

[Enchufa2]

As you can see they're not doing much, despite the amount of boilerplate seems to suggest. They just pass a pointer around and read it into the proper struct at the other side. The problem is to pass a 64-bit pointer when we have 32-bit integers in R. So they just convert it to a string, then convert that string back to int with Python.

[gregorgorjanc]

For completness adding here link to onether tab I had openned on "Integrating PyArrow with R", but the above code "summary" is the gist of how arrow goes about sending pointers between R and Python.

@Enchufa2 I agree that the code does not look complicated, but the sheer amount of boilerplate is a bit off-putting;)

[eddelbuettel]

@gregorgorjanc This topic, as mentioned, is important, and I would like to if possible illustrate it.

You started by saying that you wanted to with data structures from tskit. I presume those would 'classic' C or C++ structs, and not necessarily Arrow data structures (which generally vectors, or groups of vectors as in a data.frame)?

If so, then working with an external pointer (XPtr to Rcpp) and a PyCapsule on the Python side may work. There is for example [this PR to reticulate](rstudio/reticulate#674 supporting just that. Now, I personally do too little Python to be able to help on other side but maybe someone else can fill in that gap?

[eddelbuettel]

@gregorgorjanc And for completeness, if a standard data.frame is good then here is an example where I use nanoarrow to wrap an existing data.frame I have in R -- I use mtcars; I think penguins stumbles over NA values using penguins also works -- and pass it to rpolars (as a check) and to the Python Polars package via reticulate.

#!/usr/bin/env Rscript
suppressMessages({
## reticulate uses ~/.cache/R/reticulate/uv## with the virtualenv in ~/.virtualenv/r-reticulate
Sys.unsetenv("RETICULATE_PYTHON")
library(reticulate)
library(polars)
})
## we add the rownames as an explicit column heremtcars_df<-data.frame(model=rownames(mtcars), mtcars)
## we use a nanoarrow 'RecordBatch'narb<-nanoarrow::as_nanoarrow_array(mtcars_df)
viaR<-function(narb) {
pdf<-polars::as_polars_df(narb)
print(pdf)
invisible(NULL)
}
## local helper to unpack pyCapsulecapsule_prep<-function(na) {
## pointer addresses for array and schema in 'long long int' formatarrchr<-nanoarrow::nanoarrow_pointer_addr_chr(na)
schchr<-nanoarrow::nanoarrow_pointer_addr_chr(nanoarrow::infer_nanoarrow_schema(na))
## Import with convert = FALSE so that `_import_from_c` returns a Python objectpy<-reticulate::import_builtins(convert=FALSE)
pa<-reticulate::import("pyarrow", convert=FALSE)
out<-pa$Array$`_import_from_c`(py$int(reticulate::r_to_py(arrchr)),
py$int(reticulate::r_to_py(schchr)))
out
}
viaPython<-function(narb) {
py_require("polars")
## now import Polarspo<- import("polars")
## local helper function until `nanoarrow` catches upna2<- capsule_prep(narb)
## use it to import the Table into a Polars DataFramedf<-po$from_dataframe(na2)
print(df)
invisible(NULL)
}
viaR(narb) # works
viaPython(narb) # also works

This still needs one 'hand shake' via the capsule_prep() helper. IIRC the corresponding functionality should be baked into the next nanoarrow release too. I have a few examples like this as I prefer to work on this on the C++ side via nanoarrow, without having to compile or link with the (heavier) Arrow C++ libraries, and can still pass Arrow objects around.

For completeness the output is

$ Rscript mtcars_polars.R
shape: (32, 12)
┌───────────────────┬──────┬─────┬───────┬───┬─────┬─────┬──────┬──────┐
│ model ┆ mpg ┆ cyl ┆ disp ┆ … ┆ vs ┆ am ┆ gear ┆ carb │
│ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ f64 ┆ f64 ┆ f64 ┆ ┆ f64 ┆ f64 ┆ f64 ┆ f64 │
╞═══════════════════╪══════╪═════╪═══════╪═══╪═════╪═════╪══════╪══════╡
│ Mazda RX4 ┆ 21.0 ┆ 6.0 ┆ 160.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 4.0 ┆ 4.0 │
│ Mazda RX4 Wag ┆ 21.0 ┆ 6.0 ┆ 160.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 4.0 ┆ 4.0 │
│ Datsun 710 ┆ 22.8 ┆ 4.0 ┆ 108.0 ┆ … ┆ 1.0 ┆ 1.0 ┆ 4.0 ┆ 1.0 │
│ Hornet 4 Drive ┆ 21.4 ┆ 6.0 ┆ 258.0 ┆ … ┆ 1.0 ┆ 0.0 ┆ 3.0 ┆ 1.0 │
│ Hornet Sportabout ┆ 18.7 ┆ 8.0 ┆ 360.0 ┆ … ┆ 0.0 ┆ 0.0 ┆ 3.0 ┆ 2.0 │
│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │
│ Lotus Europa ┆ 30.4 ┆ 4.0 ┆ 95.1 ┆ … ┆ 1.0 ┆ 1.0 ┆ 5.0 ┆ 2.0 │
│ Ford Pantera L ┆ 15.8 ┆ 8.0 ┆ 351.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0 ┆ 4.0 │
│ Ferrari Dino ┆ 19.7 ┆ 6.0 ┆ 145.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0 ┆ 6.0 │
│ Maserati Bora ┆ 15.0 ┆ 8.0 ┆ 301.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0 ┆ 8.0 │
│ Volvo 142E ┆ 21.4 ┆ 4.0 ┆ 121.0 ┆ … ┆ 1.0 ┆ 1.0 ┆ 4.0 ┆ 2.0 │
└───────────────────┴──────┴─────┴───────┴───┴─────┴─────┴──────┴──────┘
shape: (32, 12)
┌───────────────────┬──────┬─────┬───────┬───┬─────┬─────┬──────┬──────┐
│ model ┆ mpg ┆ cyl ┆ disp ┆ … ┆ vs ┆ am ┆ gear ┆ carb │
│ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ f64 ┆ f64 ┆ f64 ┆ ┆ f64 ┆ f64 ┆ f64 ┆ f64 │
╞═══════════════════╪══════╪═════╪═══════╪═══╪═════╪═════╪══════╪══════╡
│ Mazda RX4 ┆ 21.0 ┆ 6.0 ┆ 160.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 4.0 ┆ 4.0 │
│ Mazda RX4 Wag ┆ 21.0 ┆ 6.0 ┆ 160.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 4.0 ┆ 4.0 │
│ Datsun 710 ┆ 22.8 ┆ 4.0 ┆ 108.0 ┆ … ┆ 1.0 ┆ 1.0 ┆ 4.0 ┆ 1.0 │
│ Hornet 4 Drive ┆ 21.4 ┆ 6.0 ┆ 258.0 ┆ … ┆ 1.0 ┆ 0.0 ┆ 3.0 ┆ 1.0 │
│ Hornet Sportabout ┆ 18.7 ┆ 8.0 ┆ 360.0 ┆ … ┆ 0.0 ┆ 0.0 ┆ 3.0 ┆ 2.0 │
│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │
│ Lotus Europa ┆ 30.4 ┆ 4.0 ┆ 95.1 ┆ … ┆ 1.0 ┆ 1.0 ┆ 5.0 ┆ 2.0 │
│ Ford Pantera L ┆ 15.8 ┆ 8.0 ┆ 351.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0 ┆ 4.0 │
│ Ferrari Dino ┆ 19.7 ┆ 6.0 ┆ 145.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0 ┆ 6.0 │
│ Maserati Bora ┆ 15.0 ┆ 8.0 ┆ 301.0 ┆ … ┆ 0.0 ┆ 1.0 ┆ 5.0 ┆ 8.0 │
│ Volvo 142E ┆ 21.4 ┆ 4.0 ┆ 121.0 ┆ … ┆ 1.0 ┆ 1.0 ┆ 4.0 ┆ 2.0 │
└───────────────────┴──────┴─────┴───────┴───┴─────┴─────┴──────┴──────┘
$ 

[gregorgorjanc]

Indeed my use case is with structs so the stopwatch example below is more relevant than arrow!

[eddelbuettel]

Ah. And I just learned in the nanoarrow thread that helped me with all this (see here) that if the Python variant of nanoarrow is installed we no longer need the capsule_prep(). One does however need the pre-release version of R's nanoarrow but this should get updated at CRAN shortly.

[eddelbuettel]

@gregorgorjanc I have a complete, working and (at least I think so) quite interesting demo using external pointers. I wrote a minimal Python package, and I think it mirrors your situation.

I was looking around for a small and simple enough struct or class in C++ that I already wrapped on the R side. And I found a gem I had hidden away in package RcppSpdlog -- upstream has a very neat and very minimal stopwatch class. This is similar to the tic() and toc() genre ... but even simpler, and now shareable with Python. For reference, the original class is here, my derived more minimal one in RcppSpdlog is here.

Back to our use case. A nice feature is that all it took in Python for this particular example was to add a) one new constructor taking a string encoding the memory address of the object from this shared C++ class and b) exposing the class (and of course this new constructor) via pybind11. So this may well fit your case as you have access to the Python bindings of your tskit. The string takes the address of the external pointer we already have in R.

I will clean this up a little and make the Python package a repo, and will then blog or write a bit more. That may take me a day or two. In the meantime, a demo (from R) follows.

> library(reticulate)
> use_virtualenv("/opt/venv/stopwatch")
>cc<- import("stopwatch") # access the Python object from R>>## quick demo just from the R side>sw<-RcppSpdlog::get_stopwatch() # we use a simple C++ struct as example> Sys.sleep(1.234) # imagine doing some code here> print(sw) # stopwatch shows elapsed time1.399661>>xptr::is_xptr(sw) # from R, a stopwatch is an external pointer (plus two S3 methods)
[1] TRUE>xptr::xptr_address(sw) # access where the external pointer points to, format is "0x...."
[1] "0x5bde23be23b0">>sw2<-xptr::new_xptr(xptr::xptr_address(sw)) # cloned (!!) but that is unclassed, so an external pointer> attr(sw2, "class") <- c("stopwatch", "externalptr")# class it .. and use it!> print(sw2) # so `xptr` allows us close and use19.477163>>sw3<-cc$Stopwatch( xptr::xptr_address(sw) ) # and so does the Python object _with an added string ctor_> print(sw3$elapsed())
datetime.timedelta(seconds=24, microseconds=69720) # different output _format_ as we hit the Python formatter>> print(sw) # and it still works on the R side36.453815>

We can also run this as one script (and I reduced the sleep to 0.5s):

$ Rscript example.R
0.500946 [1] TRUE
[1] "0x5711cd7e48f0"
0.502682 datetime.timedelta(microseconds=503092)
0.503710 $ 

which shows the marginal 'cost' accruing from running the R (and Python) interpreter and crossing back and forth. Total cost including the standard output printing (and of course ignoring the demo 'sleep' of half a second) is 3.7 milliseconds.

[eddelbuettel]

So ... not sure if @gregorgorjanc is still followoing along but we now have new repositories:

• chronometre-py: a proper Python (mock) package with a C++ class, unit tests, the works
• chronometre-r: a demo R package instantiating an object and using it in Python and R

The actual object comes from another package, is created in R and via external pointer helpers accessed. In particular, we access its memory location as a string, and pass that string via reticulate to the sibbling Python package which instantiates its own pointer to the same memory. Et voilà we have an object shared between both, and everybody lives happily ever after.

The repo README.md files have the demos, it is essentially the same as above.

[gregorgorjanc]

This is odd. I tried on another machine I have not used for this before and I get the below working - it must be some of your setting? reticulate is using uv. Maybe you should remove the cache - see ?py_require (at the end of the help page is section on this, in nutshell do unlink(tools::R_user_dir("reticulate", "cache"), recursive = TRUE)).

> mkdir chronometre
> cd chronometre
> gh repo clone eddelbuettel/chronometre-py
Cloning into 'chronometre-py'...
remote: Enumerating objects: 86, done.
remote: Counting objects: 100% (86/86), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 86 (delta 32), reused 70 (delta 16), pack-reused 0 (from 0)
Receiving objects: 100% (86/86), 16.69 KiB | 8.35 MiB/s, done.
Resolving deltas: 100% (32/32), done.
> gh repo clone eddelbuettel/chronometre-r
Cloning into 'chronometre-r'...
remote: Enumerating objects: 62, done.
remote: Counting objects: 100% (62/62), done.
remote: Compressing objects: 100% (39/39), done.
remote: Total 62 (delta 25), reused 50 (delta 15), pack-reused 0 (from 0)
Receiving objects: 100% (62/62), 11.78 KiB | 11.78 MiB/s, done.
Resolving deltas: 100% (25/25), done.
> R
R version 4.5.0 Patched (2025-05-01 r88186) -- "How About a Twenty-Six"
Copyright (C) 2025 The R Foundation for Statistical Computing
Platform: aarch64-apple-darwin20
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> install.packages(pkg = c("reticulate", "devtools", "RcppSpdlog", "xptr"))
Installing packages into ‘/Users/.../Library/R/arm64/4.5/library’
(as ‘lib’ is unspecified)
--- Please select a CRAN mirror for use in this session ---
trying URL 'https://cloud.r-project.org/bin/macosx/big-sur-arm64/contrib/4.5/reticulate_1.44.1.tgz'
trying URL 'https://cloud.r-project.org/bin/macosx/big-sur-arm64/contrib/4.5/devtools_2.4.6.tgz'
trying URL 'https://cloud.r-project.org/bin/macosx/big-sur-arm64/contrib/4.5/RcppSpdlog_0.0.26.tgz'
trying URL 'https://cloud.r-project.org/bin/macosx/big-sur-arm64/contrib/4.5/xptr_1.2.0.tgz'
The downloaded binary packages are in
/var/folders/d2/brkpzf554_n71tww54fqt3b401_bcv/T//Rtmpf8n8pi/downloaded_packages
> > py_pkg_dir <- normalizePath("chronometre-py")
> dir(py_pkg_dir)
[1] "main.py" "pyproject.toml" "pytest.ini" "README.md" [5] "requirements.txt" "setup.py" "src" "tests" [9] "uv.lock" > py_pkg_str <- paste0("chronometre@", py_pkg_dir)
> py_pkg_str
[1] "chronometre@/Users/.../chronometre/chronometre-py"
> reticulate::py_require(packages = py_pkg_str)
> setwd("chronometre-r")
> devtools::install()
── R CMD build ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔ checking for file ‘/Users/.../chronometre/chronometre-r/DESCRIPTION’ ...
─ preparing ‘chronometre’:
✔ checking DESCRIPTION meta-information ...
─ installing the package to process help pages
─ saving partial Rd database (822ms)
─ checking for LF line-endings in source and make files and shell scripts
─ checking for empty or unneeded directories
─ building ‘chronometre_0.0.1.tar.gz’
Warning: invalid uid value replaced by that for user 'nobody'
Warning: invalid gid value replaced by that for user 'nobody'
Running /Library/Frameworks/R.framework/Resources/bin/R CMD INSTALL \
/var/folders/d2/brkpzf554_n71tww54fqt3b401_bcv/T//Rtmpf8n8pi/chronometre_0.0.1.tar.gz \
--install-tests * installing to library ‘/Users/.../Library/R/arm64/4.5/library’
* installing *source* package ‘chronometre’ ...
** this is package ‘chronometre’ version ‘0.0.1’
** using staged installation
** demo
** inst
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (chronometre)
> ch <- reticulate::import("chronometre")
Built chronometre @ file:///Users/.../chronometre/chronometre-py
Installed 2 packages in 48ms

[eddelbuettel]

reticulate listens to a few environment variables to govern behaviour; do you have any set? I seem to set one in an example or two; these days I generally start these example within an activated venv. I think I didn't this morning. Could revisit this. (Also looks like I once set RETICULATE_PYTHON but no longer do.) I should also try all this in a clean Docker environment.

[gregorgorjanc]

I haven't set any env vars for reticulate.

[eddelbuettel]

Let's rest this here. Ubuntu 25.10, everything current, no need to re-install as packages present so briefer session after new temp dir, cloning and starting a fresh R -- gets the same fail as before:

>py_pkg_dir<- normalizePath("chronometre-py")
> dir(py_pkg_dir)
[1] "main.py""pyproject.toml""pytest.ini""README.md""requirements.txt""setup.py""src""tests""uv.lock">py_pkg_str<- paste0("chronometre@", py_pkg_dir)
>py_pkg_str
[1] "chronometre@/tmp/r/chronometre/chronometre-py">reticulate::py_require(packages=py_pkg_str)
>ch<-reticulate::import("chronometre")
Errorin py_module_import(module, convert=convert) :ModuleNotFoundError:Nomodulenamed'chronometre'Run`reticulate::py_last_error()`fordetails.>

Anyway, your issue is addressed, your new package is in the pretest queue (and maybe feel free to cite this discussion in it), my example works for me if I explicitly set a venv first as I do in my (extended) demo/chronometre.R that ships with the package so all good from here.

PS And I found an editing oversight in the demo -- an 'unconditional' venv setup was left in there even though replaced by the 'conditional' code preceding it. The demo now works for me in at least three ways

• called directory from the repository from shell i.e. via Rscript demo/chronometre.R
• called as demo from the installed package i.e. via demo("chronometre", package="chronometre", ask=FALSE)
• called as demo from the installed package in a rocker/r2u container first enhanced via apt install python3-venv python3-pip python3-dev (as also suggested by reticulate

Those three ways in which I document the package, and they work. I will prepare a 0.0.2 correcting the small demo oversight. It is good we found that!

PPS I had also forgotten that the CI setup (which is kept very simple relying on my r-ci setup) also a full-blown complete run with Python via reticulate -- serving as another clearly reproducible and reproducing example.

[gregorgorjanc]

Yes, I submitted RcppTskit to CRAN a few days ago, but atm I am still using the disk serialisation. I need to engage with upstream devs on implementing the shared pointer(s) via pybind as you have demonstrated in the example - that might take some time ...