novelshare

Are you looking to reproduce our ACL 2026 article experiments? See the corresponding branch.

Novelshare is a library that allows to share annotations of a copyrighted corpus, provided the user of the corpus has a (possibly slightly different version) of the copyrighted data.

Installation

Currently, novelshare is not on PyPi, but you can install it directly from GitHub with pip install 'git+https://github.com/CompNet/novelshare'.

Library user guide

Hashing your corpus

fromnovelshare.hashimporthash_tokens# assuming my_tokens is a list of tokens, and my_annotations is a list# of single or multiple annotations (one or more annotations per# token)my_tokens, my_annotations=load_my_corpus()
# hash tokens with the desired hash length (2 is a solid default# value)hashed_tokens=hash_tokens(my_tokens, hash_len=2)
withopen("hashed_corpus.conll", "w") asf:
fortoken, annotationsinzip(hashed_tokens, my_annotations):
f.write(f"{token}{annotations}\n")

Aligned Annotations of a Shared Hashed Corpus

Aligning tokens is done using the novelshare.align.align_tokens function:

fromnovelshare.alignimportalign_tokens# let's suppose you wish to align your own tokens with a hashed corpus# 1. load your own tokensmy_tokens=load_my_tokens()
# 2. load the hashed tokenshashed_tokens=load_hashed_tokens()
annotations=load_annotations()
# each token has one or more annotationsassertlen(hashed_tokens) ==len(annotations)
# you can align your tokens to annotations using align_tokens!aligned_tokens=align_tokens(hashed_tokens, my_tokens, hash_len=2)
assertlen(aligned_tokens) ==len(annotations)

The more the user tokens differ from the source tokens, the more errors will occur in the alignment process. It is possible to use additional alignment plugins to improve performance. Here are some examples:

fromnovelshare.alignimport (
make_plugin_propagate,
make_plugin_mlm,
make_plugin_retokenize,
make_plugin_case,
)
# Option #1: ligthweight but effective, using the propagate plugin alonealigned=align_tokens(
hashed_tokens,
my_tokens,
hash_len=2,
alignnment_plugins=[make_plugin_propagate()],
)
# Option #2: heavier but more powerful, using a sequence of pluginsaligned=align_tokens(
hashed_tokens,
my_tokens,
hash_len=2,
alignment_plugins=[
make_plugin_propagate(),
make_plugin_case(),
make_plugin_retokenize(max_token_len=16, max_splits_nb=8),
],
)
# Option #2: heaviest but the most powerful, using masked language# modeling to end the sequence of pluginsaligned=align_tokens(
hashed_tokens,
my_tokens,
hash_len=2,
alignment_plugins=[
make_plugin_propagate(),
make_plugin_case(),
make_plugin_retokenize(max_token_len=16, max_splits_nb=8),
# if you have a GPU, you can pass device="cuda" for GPU# accelerated inference.make_plugin_mlm("answerdotai/ModernBERT-base", window=32)
],
)

Adding alignment plugins is, however, also increasing runtime. To reduce runtime, it is possible to take advantage of the fact that a dataset might be chunked. This can happen, for example, in the case of a book divided into chapters. Since novelshare uses difflib to align sequences which is O(n^2), it is usually noticeably faster to align chapters separately rather than aligning the whole document at once. The drawback is that one needs aligned chapters. novelshare support this usecase out of the box, as the align_tokens function can take a list of tokens or a list of chunks:

fromtypingimportAnyfromnovelshare.alignimportalign_tokenshashed_chapters: list[list[str]] =load_chapters()
annotations: list[list[Any]] =load_annotations()
my_chapters: list[list[str]] =load_my_chapters()
# align_tokens supports list of chunks out of the box!aligned=align_tokens(hashed_chapters, my_chapters, hash_len=2)

Development setup

uv (preferred)

Clone the repository and use uv:

uv sync

By default this install everything for and the cuda version of PyTorch. We have several extras to configure your installation. For development dependencies:

uv sync --dev

For GPU acceleration, depending on your GPU, you can use:

# if you have a cuda 12.8 compatible GPU
uv sync --extra cu128
# if you have a rocm 6.3 compatible GPU
uv sync --extra rocm63

The cpu extra also exists for the cpu version of torch.

guix

We provide a reproducible environment with guix:

guix time-machine -C channels.scm -- shell -C -m manifest.scm

Tests

We use pytest for testing, so you can run tests with python -m pytest tests.

Citation

If you use novelshare in your research, please cite:

@InProceedings{Amalvy2026,
author = {Amalvy, A. and Labatut, V. and Bost, X. and Huang, H.-H.},
title = {Overcoming Copyright Barriers in Corpus Distribution Through Non-Reversible Hashing},
year = 2026,
booktitle = {64th Annual Meeting of the Association for Computational Linguistics (to appear)}, }