memshare: Shared memory multithreading in R via C++17, with zero-copy access through ALTREP data structures.

---

Overview

memshare enables multicore computation in R without redundant memory copies. Large vectors, matrices, or lists are stored once in shared memory and exposed to R processes as ALTREP views. This allows workers in a PSOCK cluster to operate on the same physical data while avoiding serialization overhead.

Key features:

• Shared memory allocation via C++17 back-end (shm_open on Unix, MapViewOfFile on Windows).
• ALTREP wrappers so R sees shared objects as native vectors or matrices.
• High-level parallels to parallel::parApply and parallel::parLapply:
  • memApply() — apply a function row/column-wise over a matrix in shared memory.
  • memLapply() — apply a function element-wise over a list in shared memory.
• Automatic cleanup of shared objects when all views are released or when the package is unloaded.
• Tested on Linux, macOS, and Windows.

---

Installation

From CRAN:

install.packages("memshare")

From GitHub (development version):

remotes::install_github("mthrun/memshare")

System requirements: R ≥ 4.0 and a C++17 compiler.

Quick start

Example 1: Parallel correlations with a matrix

library(memshare)
library(parallel)
set.seed(1)
n=10000p=2000X=matrix(rnorm(n*p), n, p)
y= rnorm(n)
res= memApply(
X=X, MARGIN=2,
FUN=function(v, y) cor(v, y),
VARS=list(y=y)
)
str(res)

Example 2: List operations

library(memshare)
library(parallel)
list_length=1000matrix_dim=100ListV= lapply(
1:list_length,
function(i) matrix(rnorm(matrix_dim*matrix_dim),
nrow=matrix_dim, ncol=matrix_dim))
y= rnorm(matrix_dim)
namespace="ID123"res=memshare::memLapply(ListV, function(el, y) {
el%*%y
}, NAMESPACE=namespace, VARS=list(y=y), MAX.CORES=1)

Each element el of ListV is multiplied by y in parallel. The list resides once in shared memory.

Concepts

• Pages: memory regions owned by the current R session that loaded the package.

• Views: ALTREP wrappers exposing shared memory variables (read/write capable).

• Namespaces: string identifiers defining a shared memory context across sessions.

When the package is detached, all handles and associated shared memory pages are released, unless another R process still holds references.

Shared-memory lifecycle

memshare exposes explicit lifecycle functions so you can control when data is placed in shared memory and when it is freed.

registerVariables(namespace, variableList)

Allocate shared memory and copy R objects (matrices or vectors, or lists for memLapply) into it.

• namespace: character(1). Identifier of the shared memory context shared across processes.
• variableList: a named list of objects to register. Names become the keys under which you can later retrieve views.

Example

library(memshare)
ns<-"my_namespace"X<-matrix(rnorm(1e4), 100, 100)
y<- rnorm(100)
registerVariables(ns, list(X=X, y=y))
# Now X and y live once in shared memory and can be accessed from other R processes

releaseVariables(namespace, variableNames)

Delete variables from the shared memory space. Shared regions are only removed when no active views remain.

• namespace: character(1) used above.
• variableNames: character vector of variable names to free.

Example

# After all workers have released their views:
releaseVariables(ns, c("X", "y"))

Working with views: retrieveViews() and releaseViews()

To avoid duplication, workers attach to shared memory by views:

• retrieveViews(namespace, c("X","y")) returns ALTREP-backed objects that behave like ordinary R matrices/vectors.
• Always call releaseViews(namespace, ...) when finished so that the backing memory can be reclaimed.

Example (worker-side)

vlist<- retrieveViews(ns, c("X","y"))
# use vlist$X and vlist$y
releaseViews(ns, c("X","y"))

Tip: memApply() and memLapply() manage views for you automatically, but the low-level API above is useful for custom workflows.

Manual

The full manual for users or developers is available here: Package documentation

References

[Thrun/Märte, 2026] Thrun, M. C., & Märte, J.: Memshare: Memory Sharing for Multicore Computation in R with an Application to Feature Selection by Mutual Information using PDE, R Journal, Vol. 17(4), pp. 306 - 322, doi 10.32614/RJ-2025-043, 2026.

[Thrun et al., 2020] Thrun, M.C., Gehlert, T., & Ultsch, A.: Analyzing the Fine Structure of Distributions, PLOS ONE, 15(10), e0238835, 2020.

[Ultsch, 2005] Ultsch, A.: Pareto Density Estimation: A Density Estimation for Knowledge Discovery, Proceedings of the 28th Annual Conference of the German Classification Society, Springer, 2005