stringfish

stringfish is a framework for string and sequence operations using the ALTREP system (introduced in R 3.5) as a way to represent R objects using custom memory layout.

This package has two primary goals:

• Provide R users a way to speed up common string operations compared to base R (see benchmarks below)
• Create a common ALTREP framework that can be used by other packages with full interoperability

stringfish currently provides two ALTREP backends with the same semantics: sf_vec, a simple vector of string objects, and slice_store, which stores strings within large contiguous blocks of memory. They make different storage tradeoffs, but the same stringfish operations work across both.

For text data, stringfish is intentionally UTF-8-centric outside of explicit byte mode, so conversions, comparisons, and ALTREP views stay consistent across normal R vectors and both backends.

Installation

install.packages("stringfish", type="source", configure.args="--with-simd=AVX2")

Benchmark

The simplest way to show the utility of the ALTREP framework is through a quick benchmark comparing stringfish and base R.

On favorable workloads, some functions in stringfish can be more than an order of magnitude faster than vectorized base R operations, and built-in multithreading can widen that gap further. On large text datasets, this can turn minutes of computation into seconds.

Currently implemented functions

A list of implemented stringfish functions and analogous base R functions:

• sf_iconv (iconv)
• sf_nchar (nchar)
• sf_substr (substr)
• sf_paste (paste0)
• sf_collapse (paste0)
• sf_readLines (readLines)
• sf_writeLines (writeLines)
• sf_grepl (grepl)
• sf_gsub (gsub)
• sf_toupper (toupper)
• sf_tolower (tolower)
• sf_starts (startsWith)
• sf_ends (endsWith)
• sf_trim (trimws)
• sf_split (strsplit)
• sf_match (match for strings only)
• sf_compare/sf_equals (==, ALTREP-aware semantic string equality)
• sf_concat/sfc (c)

Utility functions:

• sf_vector_create – creates a new empty sf_vec-backed stringfish vector
• sf_vector – backwards-compatible alias for sf_vector_create
• slice_store_create – creates a new empty slice_store-backed stringfish vector
• slice_store_create_with_size – creates a slice_store-backed stringfish vector with an explicit initial slice size
• sf_assign – assign strings into a stringfish vector in place (like x[i] <- "mystring")
• convert_to_sf_vector – converts a character vector to a stringfish vector
• convert_to_slice_store – converts a character vector to a stringfish slice store
• get_string_type – determines string type (whether ALTREP or normal)
• materialize – converts any ALTREP object into a normal R object
• random_strings – creates random strings as either a stringfish or normal R vector
• string_identical – compares strings either semantically or exactly across encodings

In addition, many R operations in base R and other packages are already ALTREP-aware (i.e. they don’t cause materialization). Functions that subset or index into string vectors generally do not materialize.

• sample
• head
• tail
• [ – e.g. x[20:30]
• various tidyverse filters and operations
• Etc.

stringfish functions are not intended to exactly replicate their base R analogues. One difference is that subject parameters are always the first argument, which is easier to use with pipes. E.g., gsub(pattern, replacement, subject) becomes sf_gsub(subject, pattern, replacement).

Extensibility

stringfish as a framework is intended to be easily extensible. Stringfish vectors can be worked into Rcpp scripts or even into other packages. The example below creates an sf_vec-backed output because it is simple and direct, but the same indexing semantics work across both backends.

Below is a detailed Rcpp script that creates a function to alternate upper and lower case of strings.

// [[Rcpp::depends(stringfish)]]#include<Rcpp.h>#include"sf_external.h"usingnamespaceRcpp;
// [[Rcpp::export]]SEXPsf_alternate_case(SEXPx) {
// Iterate through a character vector using the RStringIndexer class// If the input vector x is a stringfish character vector it will do so without materializationRStringIndexerr(x);
size_tlen=r.size();
// Create an output stringfish vector// Like all R objects, it must be protected from garbage collectionSEXPoutput=PROTECT(sf_vector_create(len));
// Obtain a reference to the underlying output datasf_vec_data&output_data=sf_vec_data_ref(output);
// You can use range based for loop via an iterator class that returns RStringIndexer::rstring_info e// rstring info is a struct containing const char * ptr, int len, and an encoding flag// ptr should be treated as a byte pointer plus length, not as a null-terminated C string// a NA string is represented by a nullptr// Alternatively, access the data via the function r.getCharLenCE(i)size_ti=0;
for(auto e : r) {
// check if string is NA and go to next if it isif(e.ptr==nullptr) {
i++; // increment output indexcontinue;
}
// Create a temporary output string and process the results.// This example intentionally toggles ASCII letters only.std::stringtemp(e.len, '\0');
boolcase_switch= false;
for(intj=0; j<e.len; j++) {
if((e.ptr[j] >= 65) && (e.ptr[j] <= 90)) { // char j is upper caseif((case_switch= !case_switch)) { // check if we should convert to lower casetemp[j] =e.ptr[j] +32;
continue;
}
} elseif((e.ptr[j] >= 97) && (e.ptr[j] <= 122)) { // char j is lower caseif(!(case_switch= !case_switch)) { // check if we should convert to upper casetemp[j] =e.ptr[j] -32;
continue;
}
} elseif(e.ptr[j] ==32) {
case_switch= false;
}
temp[j] =e.ptr[j];
}
// Create a new vector element sfstring and insert the processed string into the stringfish vector// sfstring has three constructors, 1) taking a std::string and encoding, // 2) a char pointer and encoding, or 3) a CHARSXP object (e.g. sfstring(NA_STRING))output_data[i] =sfstring(temp, e.enc);
i++; // increment output index
}
// Finally, call unprotect and return resultUNPROTECT(1);
returnoutput;
}

Example function call:

sf_alternate_case("hello world") [1] "hElLo wOrLd"