subtext-codec

Steganographic data encoding in natural language via arithmetic coding against an LLM

subtext-codec hides arbitrary binary data inside seemingly normal LLM-generated text. The payload drives an arithmetic coder whose probability model is the language model's own next-token distribution, so the text it produces is distributed exactly as ordinary sampling would be while secretly encoding bytes. With the same model, tokenizer, prompt prefix and parameters, the process is fully reversible.

How it works

The payload drives an arithmetic decoder whose probability model is the language model itself. Emitting a token the model assigns probability p consumes -log2(p) payload bits, so a confident step carries almost nothing and an uncertain one carries a lot.

Two properties follow, and they are the entire point of the design:

The generated text is distributed exactly as ordinary sampling at the chosen temperature. Nothing is ever forced off-distribution.
Capacity per token equals the distribution's entropy, which is the information-theoretic ceiling for text that stays indistinguishable from sampling.

Encoding

The payload is framed with a length and a CRC32, then read as a bitstream.
At each step the model's logits are clipped to top_k, filtered to tokens that are safe to emit (see below), and turned into an integer frequency table at temperature.
The arithmetic decoder consumes payload bits and selects a token from that table.
Generation stops as soon as the emitted tokens pin down every payload bit.

Decoding

Start from the same prompt prefix and model, and rebuild the same table at each step.
Run the arithmetic encoder over the observed tokens, which reproduces the bitstream.
Stop once the declared length has been recovered, then check the CRC.

Decoding needs the text, the prompt prefix, the same model and tokenizer, and the codec parameters -- all of which the key file carries. No terminator token is required, so text after the message is simply ignored.

Why the encoder checks itself

The coder re-encodes each symbol as it emits it and refuses to return a message whose bits it cannot reproduce. That is not belt-and-braces: feeding the arithmetic decoder a zero-filled tail past the end of the payload parks its value register exactly on the interval midpoint, where it underflows forever and the payload's last two bits are never emitted. The result is a message that looks fine and silently fails to decode. The check is what turns that class of bug into a loud failure.

Why the safety filter exists

The encoder chooses token ids, but the decoder is handed text. Nothing guarantees that writing tokens out and reading them back returns the same ids: a byte-level BPE vocabulary will merge an emitted token into its neighbours, so 'an' written after 'Kanz' reads back as 'anz' and every subsequent step decodes against the wrong prefix.

So a candidate is only usable if appending it leaves the surrounding text re-tokenizing unchanged. Both sides apply that same filter to the same prefix, so they stay in step, and the encoder verifies the finished text tokenizes back to exactly what it built before handing it over. A message that cannot satisfy this raises at encode time rather than becoming an artifact that silently fails to decode.

Features

Entropy-optimal capacity -- each token carries -log2(p) bits, the theoretical maximum for output that still looks like sampling
Output distributed exactly as the model's own sampling, so no per-token statistical tell
Temperature as a principled capacity dial, replacing rank truncation
Tokenizer-stable candidate selection, verified before any message is returned
Self-inverse check at encode time -- the coder re-encodes as it goes and refuses to emit a message it cannot read back
Framed payload with a CRC32, so a wrong key, prompt or model is detected rather than yielding plausible wrong bytes
Optional zlib compression (--compress), which shortens the cover text and makes the payload bits more uniform
Automatic rolling context window, so a payload can exceed the model's context window with no repeated prompt and no tuning
Incremental KV-cached stepping, so cost is linear in message length rather than quadratic
Deterministic throughout; verified bit-identical on CPU and CUDA across fp32/bf16/fp16
Hugging Face Transformers backend -- works with most causal LMs

Installation

From PyPI (or via uv):

uv pip install subtext-codec
# or: pip install subtext-codec

From source:

git clone https://github.com/shevisj/subtext-codec
cd subtext-codec
uv venv --python 3.13
uv pip install -r requirements.txt
uv pip install -e .

torch and transformers are the only runtime dependencies. The published package sets lower bounds only, so it installs alongside whatever torch build you already have; requirements.txt pins an exact, CPU-only development environment and is the one to use when reproducing a decode.

Usage

Run subtext-codec --version to print the installed version. The CLI otherwise exposes encode and decode. Shared flags:

--key -- path to the codec key file (required)
--model-name-or-path -- HuggingFace model id or local path; optional if stored in the key
--prompt-prefix -- prefix used for both encode and decode; taken from the key if present
--device -- e.g. cpu or cuda (falls back to the key, then cpu)
--torch-dtype -- weight dtype (auto, fp16, bf16, fp32)
--max-context-length -- cap on sequence length; defaults to the model's own limit
--seed -- deterministic seed (default: 0)
--quiet -- suppress the progress line on stderr

Encode bytes into text

subtext-codec encode \
--model-name-or-path gpt2 \
--prompt-prefix "Once upon a time, " \
--input-bytes secret.txt \
--output-text message.txt \
--key key.json \
--temperature 1.5

Encode-only flags:

--temperature -- the capacity dial (default: 1.5). Higher packs more payload per token and shortens the cover text, at the cost of more erratic prose. See Choosing a temperature.
--top-k -- candidates considered per step (default: 64). This bounds the stability filter's work; it is not a capacity dial. Must be at least 2.
--compress -- zlib-compress the payload before encoding. Shortens the cover text and makes the payload bits more uniform. Recorded in the key, so decoding reverses it automatically. See Compression.
--max-new-tokens -- fail rather than generate more than this many tokens
--no-store-model -- keep the model id out of the key
--verify -- decode the result before writing it, confirming the round trip end to end

Payloads too long for one context window are handled automatically; see Large payloads.

Unless --quiet, encoding prints a one-line summary to stderr -- payload size, cover tokens, bits per token and mean surprisal (and how many times the context rolled) -- so you can see the coder is running at the distribution's entropy.

The output text is just the generated story, with no metadata header. key.json records temperature, top_k, the prompt prefix, the device, the dtype, (unless --no-store-model) the model id, and -- when compression or the rolling window is used -- a compression or window field and a "version": "v2" marker.

If the path given to --key already exists, its values are reused as defaults and any CLI overrides are written back, so a second message needs only the key:

subtext-codec encode --key key.json --input-bytes secret.bin --output-text message.txt

Decode text back into bytes

subtext-codec decode \
--input-text message.txt \
--key key.json \
--output-bytes decoded.bin

Parameters come from the key unless overridden for the run. Decoding never modifies the key file -- the one artifact a message cannot be recovered without is not rewritten by a read operation.

Decoding stops as soon as the payload's declared length has been recovered, so trailing text is harmless. Text before the prompt prefix is skipped too. If the message was altered, or the key, prompt or model do not match, the CRC32 catches it rather than returning plausible wrong bytes.

Sample artifacts

samples/secret.txt is a payload to play with. Build the rest of the fixture from it:

python samples/regenerate.py

That encodes it with gpt2 on CPU -- a ~500MB download, no GPU or gated checkpoint needed -- and writes message.txt, key.json and decoded.txt alongside it, verifying the round trip before writing anything. Encoding is deterministic, so it reproduces the same message every time.

Then decode it back the normal way:

subtext-codec decode \
--input-text samples/message.txt \
--key samples/key.json \
--output-bytes /tmp/decoded.txt
diff samples/secret.txt /tmp/decoded.txt &&echo"exact match"

(Before 1.0 the fixture was checked in, but it was a v2 message against a gated 16GB Llama checkpoint -- unreadable by this version and unrunnable by most people. Generating it locally is both smaller and honest.)

Programmatic usage

See the demo notebook.

Choosing a temperature

temperature is the capacity dial. Raising it flattens the distribution, so each token carries more payload and the cover text gets shorter -- but the text becomes more erratic, exactly as sampling at that temperature would. top_k is not a capacity dial here; it only bounds how much work the stability filter does.

Measured on the 445-byte samples/secret.txt, Qwen2.5-7B (bf16) on an RTX 5090, top_k=64:

Temperature	Cover tokens	Bits/token	Character
1.0	2914	1.22	best prose, but far too long to be plausible
1.5	894	3.98	the knee -- reads like a real, rambling review
2.0	709	5.02	densest, but visibly erratic and prone to invention

Mean surprisal of the emitted tokens matches capacity at every setting (1.23 / 4.02 / 5.07 bits), which is the confirmation that the coder is running at the distribution's entropy rather than leaving capacity on the table.

For reference, the rank-coding scheme this replaced in 1.0 managed 4.35 bits/token at its best setting and 1.57 at its most conservative, and its output was never model-distributed. Arithmetic coding at temperature 1.5 matches its best capacity, and at 2.0 exceeds it, while staying on-distribution throughout.

A bigger model does not buy more capacity. A stronger model concentrates probability mass, which lowers entropy and therefore bits per token. It buys better prose at a given temperature, not shorter text. Keep payloads to a few hundred bytes, and pick a prompt whose natural continuation is long-form so the required length is not conspicuous.

Compression

--compress zlib-compresses the payload before encoding. It does two useful things:

Shortens the cover text. Fewer payload bytes means fewer tokens to carry them.
Makes the bits more uniform, which is exactly what the indistinguishability argument assumes. Raw ASCII is biased; compressed (or encrypted) data is not.

subtext-codec encode --key key.json --input-bytes secret.txt --output-text message.txt --compress

Compression is recorded in the key, so decode reverses it automatically -- you do not pass --compress when decoding. A compressed message uses the v2 wire format (see Compatibility). The whole payload is framed with a length and CRC32 before compression, so a key that misdescribes compression is caught rather than returning the raw compressed bytes.

Compression rarely helps on data that is already high-entropy (encrypted blobs, media). It helps most on text and structured data.

Large payloads

A payload can outgrow the model's context window, and that is handled automatically -- there is nothing to configure. When the context fills, the codec re-prefixes the most recent half of the tokens and keeps generating, a rolling window that slides forward as the message grows. The text stays one continuous passage: the prompt appears once, at the start, and never repeats.

# no special flags: a long payload just works
subtext-codec encode --key key.json --input-bytes big.bin --output-text message.txt

The decoder resets at the same points, so it recovers the message exactly. The window size it used rides in the key (a window field), so decoding needs nothing extra. A message that rolls uses the v2 wire format; one that fits a single window stays v1.

Two things worth knowing:

The window follows the model's context limit (or --max-context-length, if you set a smaller one). Because the size is recorded in the key, encode and decode agree automatically -- but if you decode with a different explicit --max-context-length, the key's value still wins.
Coherence is over the window, not the whole text. Past the first reset the model conditions on a sliding window of recent tokens rather than the entire history, so very long messages read like natural long-form drift rather than a single tightly-planned document. See the caveat in Limitations.

Encryption

The codec is not a cipher -- the key is metadata, not a secret, and anyone with it and the model can read the message. For confidentiality, encrypt the payload yourself before encoding. Encryption also gives you the uniform bits the indistinguishability argument wants, so you do not additionally need --compress (encrypted data will not compress anyway).

# encrypt, then hide; the codec never sees the plaintext
age -r age1... secret.txt | subtext-codec encode --key key.json --input-bytes - --output-text message.txt
# recover, then decrypt
subtext-codec decode --key key.json --input-text message.txt --output-bytes - | age -d -i key.txt > secret.txt

Any tool works -- age, gpg, openssl enc. Keeping encryption out of the codec is deliberate: this is a research prototype, and rolling in a cipher would imply a security property it does not provide.

Determinism and devices

The codec reduces to one requirement: encode and decode must derive the same frequency table at every step, down to the last integer. Everything after the forward pass runs on CPU no matter where the model lives: the logits are cast to float32 for the stable sort and the candidate filter, then to float64 for the softmax and quantization. So the only device-sensitive component is the logits themselves. Encode and decode also drive the model identically (prefill the prompt, then one token at a time), so they hit the same kernels at the same shapes.

This has been measured, not just reasoned about. On an RTX 5090 (Blackwell, sm_120, torch 2.13 + CUDA 13), repeated passes over the same tokens produce bit-identical logits in fp32, bf16 and fp16, and all three round-trip on GPU. GPU is as reliable as CPU.

What matters is that the environment matches between encode and decode:

Factor	Effect
Different dtype (bf16 ↔ fp32)	Breaks. The key records the dtype; keep it.
TF32 on vs off	Pinned off by `set_deterministic`, so it no longer varies.
Different attention backend (SDPA / FlashAttention / eager)	Can break across machines. Same environment, same choice.
Different torch / transformers version	Can break; pin them (`requirements.txt`) if a message must survive an upgrade.
Different device (`cuda` ↔ `cpu`)	Unreliable, but not automatically fatal -- see below.

Crossing devices is not supported, though it is not guaranteed to fail: gpt2 in fp32 does decode on CPU from a CUDA encode, because the two happen to agree bit for bit. Larger models and bf16 will not be so lucky. Do not rely on it. What is guaranteed is that a mismatch is caught rather than returning plausible wrong bytes -- that is the payload framing's job.

set_deterministic also sets CUBLAS_WORKSPACE_CONFIG=:4096:8 (via package import, before torch loads) and disables cuDNN autotuning. It holds fp32 matmuls at full precision, which costs some speed on Ampere and later; a deliberate trade, since a mismatch costs the entire message.

bf16 and fp16 work but leave less margin -- half the mantissa means adjacent logits sit closer together. fp32 is safer when you can afford it. Exact ties are fine in any precision: the sort is stable, so ties resolve by token id.

To confirm on your own hardware before trusting a message to it:

uv run pytest tests/test_determinism.py -v

Those parametrize over every device and dtype present, so on a GPU box they additionally check bit-identical logits on CUDA, matching bands, and fp32/bf16/fp16 round trips.

Compatibility

There are two wire formats:

v1 -- a single segment framing length || crc32 || data, exactly as 1.0 wrote it. A plain message that fits one context window (no compression, no rolling) is still written as v1, so it stays byte-for-byte readable by 1.0.
v2 -- added in 1.1 for compression and the rolling window. The key records "version": "v2" and, as needed, a compression field and a window field. Compressed payloads frame a length and CRC over the original data so a wrong compression flag is caught; rolled payloads carry the window so the decoder resets at the same points. A 1.0 reader rejects a v2 key outright rather than mis-reading it, which is why the new features carry a new version.

1.1 reads both. It writes v2 only when you compress or when the payload is long enough to roll the context; everything else stays v1.

Every pre-1.0 format is gone along with the code that decoded them: they used rank coding, which corrupted roughly a third of its own messages and never produced model-distributed output. Pre-1.0 also numbered a format v1, but that one carried base/top_p and no temperature, so the collision is detectable -- such a key fails with an explanation rather than a generic error. To read a message encoded before 1.0, install subtext-codec~=0.2.0; to keep it, re-encode the payload with this version.

Limitations

Brittle to edits: changing a single token of the output breaks decoding. This is an encoding scheme, not an error-correcting one.
Model-dependent: requires the exact same weights, tokenizer and dtype. Decoding a bf16 encode in fp32 will not work.
Not confidential on its own: the key is metadata, not a cipher. Encrypt the payload before encoding it -- both for secrecy and because the indistinguishability argument assumes uniform payload bits.
Context length: handled automatically by the rolling window, but past the first reset the model conditions on recent tokens rather than the whole history. Coherence is local to the window, and a discriminator with the full text could in principle notice the long-range structure "resetting" -- subtle for a reasonable window, but a real difference from full-context sampling, on top of the temperature caveat below.
Capacity: the distribution's entropy, typically 1-5 bits per token depending on temperature. Expect a few hundred tokens per hundred bytes.
Length is the remaining tell: the per-token statistics are right, but a 900-token hotel review is still an odd thing to find. Pick a prompt whose natural continuation is long-form.
"Indistinguishable" is relative to the temperature you chose: output at temperature=1.5 is exactly temperature-1.5 sampling, which is only unremarkable to someone who has no expectation about the setting. An adversary who knows you would have sampled at 1.0 can distinguish it in aggregate. Lower temperature closes that gap and lengthens the text.

This project is a research prototype, not a secure or production steganography system.

Testing

uv run pytest # everything
uv run pytest -m "not slow"# skip tests that download a checkpoint

The slow tests use sshleifer/tiny-gpt2 and skip themselves if the Hub is unreachable.

License

MIT

subtext-codec

Steganographic data encoding in natural language via arithmetic coding against an LLM

subtext-codec hides arbitrary binary data inside seemingly normal LLM-generated text. The payload drives an arithmetic coder whose probability model is the language model's own next-token distribution, so the text it produces is distributed exactly as ordinary sampling would be while secretly encoding bytes. With the same model, tokenizer, prompt prefix and parameters, the process is fully reversible.

How it works

The payload drives an arithmetic decoder whose probability model is the language model itself. Emitting a token the model assigns probability p consumes -log2(p) payload bits, so a confident step carries almost nothing and an uncertain one carries a lot.

Two properties follow, and they are the entire point of the design:

The generated text is distributed exactly as ordinary sampling at the chosen temperature. Nothing is ever forced off-distribution.
Capacity per token equals the distribution's entropy, which is the information-theoretic ceiling for text that stays indistinguishable from sampling.

Encoding

The payload is framed with a length and a CRC32, then read as a bitstream.
At each step the model's logits are clipped to top_k, filtered to tokens that are safe to emit (see below), and turned into an integer frequency table at temperature.
The arithmetic decoder consumes payload bits and selects a token from that table.
Generation stops as soon as the emitted tokens pin down every payload bit.

Decoding

Start from the same prompt prefix and model, and rebuild the same table at each step.
Run the arithmetic encoder over the observed tokens, which reproduces the bitstream.
Stop once the declared length has been recovered, then check the CRC.

Decoding needs the text, the prompt prefix, the same model and tokenizer, and the codec parameters -- all of which the key file carries. No terminator token is required, so text after the message is simply ignored.

Why the encoder checks itself

The coder re-encodes each symbol as it emits it and refuses to return a message whose bits it cannot reproduce. That is not belt-and-braces: feeding the arithmetic decoder a zero-filled tail past the end of the payload parks its value register exactly on the interval midpoint, where it underflows forever and the payload's last two bits are never emitted. The result is a message that looks fine and silently fails to decode. The check is what turns that class of bug into a loud failure.

Why the safety filter exists

The encoder chooses token ids, but the decoder is handed text. Nothing guarantees that writing tokens out and reading them back returns the same ids: a byte-level BPE vocabulary will merge an emitted token into its neighbours, so 'an' written after 'Kanz' reads back as 'anz' and every subsequent step decodes against the wrong prefix.

So a candidate is only usable if appending it leaves the surrounding text re-tokenizing unchanged. Both sides apply that same filter to the same prefix, so they stay in step, and the encoder verifies the finished text tokenizes back to exactly what it built before handing it over. A message that cannot satisfy this raises at encode time rather than becoming an artifact that silently fails to decode.

Features

Entropy-optimal capacity -- each token carries -log2(p) bits, the theoretical maximum for output that still looks like sampling
Output distributed exactly as the model's own sampling, so no per-token statistical tell
Temperature as a principled capacity dial, replacing rank truncation
Tokenizer-stable candidate selection, verified before any message is returned
Self-inverse check at encode time -- the coder re-encodes as it goes and refuses to emit a message it cannot read back
Framed payload with a CRC32, so a wrong key, prompt or model is detected rather than yielding plausible wrong bytes
Optional zlib compression (--compress), which shortens the cover text and makes the payload bits more uniform
Automatic rolling context window, so a payload can exceed the model's context window with no repeated prompt and no tuning
Incremental KV-cached stepping, so cost is linear in message length rather than quadratic
Deterministic throughout; verified bit-identical on CPU and CUDA across fp32/bf16/fp16
Hugging Face Transformers backend -- works with most causal LMs

Installation

From PyPI (or via uv):

uv pip install subtext-codec
# or: pip install subtext-codec

From source:

git clone https://github.com/shevisj/subtext-codec
cd subtext-codec
uv venv --python 3.13
uv pip install -r requirements.txt
uv pip install -e .

torch and transformers are the only runtime dependencies. The published package sets lower bounds only, so it installs alongside whatever torch build you already have; requirements.txt pins an exact, CPU-only development environment and is the one to use when reproducing a decode.

Usage

Run subtext-codec --version to print the installed version. The CLI otherwise exposes encode and decode. Shared flags:

--key -- path to the codec key file (required)
--model-name-or-path -- HuggingFace model id or local path; optional if stored in the key
--prompt-prefix -- prefix used for both encode and decode; taken from the key if present
--device -- e.g. cpu or cuda (falls back to the key, then cpu)
--torch-dtype -- weight dtype (auto, fp16, bf16, fp32)
--max-context-length -- cap on sequence length; defaults to the model's own limit
--seed -- deterministic seed (default: 0)
--quiet -- suppress the progress line on stderr

Encode bytes into text

subtext-codec encode \
--model-name-or-path gpt2 \
--prompt-prefix "Once upon a time, " \
--input-bytes secret.txt \
--output-text message.txt \
--key key.json \
--temperature 1.5

Encode-only flags:

--temperature -- the capacity dial (default: 1.5). Higher packs more payload per token and shortens the cover text, at the cost of more erratic prose. See Choosing a temperature.
--top-k -- candidates considered per step (default: 64). This bounds the stability filter's work; it is not a capacity dial. Must be at least 2.
--compress -- zlib-compress the payload before encoding. Shortens the cover text and makes the payload bits more uniform. Recorded in the key, so decoding reverses it automatically. See Compression.
--max-new-tokens -- fail rather than generate more than this many tokens
--no-store-model -- keep the model id out of the key
--verify -- decode the result before writing it, confirming the round trip end to end

Payloads too long for one context window are handled automatically; see Large payloads.

Unless --quiet, encoding prints a one-line summary to stderr -- payload size, cover tokens, bits per token and mean surprisal (and how many times the context rolled) -- so you can see the coder is running at the distribution's entropy.

The output text is just the generated story, with no metadata header. key.json records temperature, top_k, the prompt prefix, the device, the dtype, (unless --no-store-model) the model id, and -- when compression or the rolling window is used -- a compression or window field and a "version": "v2" marker.

If the path given to --key already exists, its values are reused as defaults and any CLI overrides are written back, so a second message needs only the key:

subtext-codec encode --key key.json --input-bytes secret.bin --output-text message.txt

Decode text back into bytes

subtext-codec decode \
--input-text message.txt \
--key key.json \
--output-bytes decoded.bin

Parameters come from the key unless overridden for the run. Decoding never modifies the key file -- the one artifact a message cannot be recovered without is not rewritten by a read operation.

Decoding stops as soon as the payload's declared length has been recovered, so trailing text is harmless. Text before the prompt prefix is skipped too. If the message was altered, or the key, prompt or model do not match, the CRC32 catches it rather than returning plausible wrong bytes.

Sample artifacts

samples/secret.txt is a payload to play with. Build the rest of the fixture from it:

python samples/regenerate.py

That encodes it with gpt2 on CPU -- a ~500MB download, no GPU or gated checkpoint needed -- and writes message.txt, key.json and decoded.txt alongside it, verifying the round trip before writing anything. Encoding is deterministic, so it reproduces the same message every time.

Then decode it back the normal way:

subtext-codec decode \
--input-text samples/message.txt \
--key samples/key.json \
--output-bytes /tmp/decoded.txt
diff samples/secret.txt /tmp/decoded.txt &&echo"exact match"

(Before 1.0 the fixture was checked in, but it was a v2 message against a gated 16GB Llama checkpoint -- unreadable by this version and unrunnable by most people. Generating it locally is both smaller and honest.)

Programmatic usage

See the demo notebook.

Choosing a temperature

temperature is the capacity dial. Raising it flattens the distribution, so each token carries more payload and the cover text gets shorter -- but the text becomes more erratic, exactly as sampling at that temperature would. top_k is not a capacity dial here; it only bounds how much work the stability filter does.

Measured on the 445-byte samples/secret.txt, Qwen2.5-7B (bf16) on an RTX 5090, top_k=64:

Temperature	Cover tokens	Bits/token	Character
1.0	2914	1.22	best prose, but far too long to be plausible
1.5	894	3.98	the knee -- reads like a real, rambling review
2.0	709	5.02	densest, but visibly erratic and prone to invention

Mean surprisal of the emitted tokens matches capacity at every setting (1.23 / 4.02 / 5.07 bits), which is the confirmation that the coder is running at the distribution's entropy rather than leaving capacity on the table.

For reference, the rank-coding scheme this replaced in 1.0 managed 4.35 bits/token at its best setting and 1.57 at its most conservative, and its output was never model-distributed. Arithmetic coding at temperature 1.5 matches its best capacity, and at 2.0 exceeds it, while staying on-distribution throughout.

A bigger model does not buy more capacity. A stronger model concentrates probability mass, which lowers entropy and therefore bits per token. It buys better prose at a given temperature, not shorter text. Keep payloads to a few hundred bytes, and pick a prompt whose natural continuation is long-form so the required length is not conspicuous.

Compression

--compress zlib-compresses the payload before encoding. It does two useful things:

Shortens the cover text. Fewer payload bytes means fewer tokens to carry them.
Makes the bits more uniform, which is exactly what the indistinguishability argument assumes. Raw ASCII is biased; compressed (or encrypted) data is not.

subtext-codec encode --key key.json --input-bytes secret.txt --output-text message.txt --compress

Compression is recorded in the key, so decode reverses it automatically -- you do not pass --compress when decoding. A compressed message uses the v2 wire format (see Compatibility). The whole payload is framed with a length and CRC32 before compression, so a key that misdescribes compression is caught rather than returning the raw compressed bytes.

Compression rarely helps on data that is already high-entropy (encrypted blobs, media). It helps most on text and structured data.

Large payloads

A payload can outgrow the model's context window, and that is handled automatically -- there is nothing to configure. When the context fills, the codec re-prefixes the most recent half of the tokens and keeps generating, a rolling window that slides forward as the message grows. The text stays one continuous passage: the prompt appears once, at the start, and never repeats.

# no special flags: a long payload just works
subtext-codec encode --key key.json --input-bytes big.bin --output-text message.txt

The decoder resets at the same points, so it recovers the message exactly. The window size it used rides in the key (a window field), so decoding needs nothing extra. A message that rolls uses the v2 wire format; one that fits a single window stays v1.

Two things worth knowing:

The window follows the model's context limit (or --max-context-length, if you set a smaller one). Because the size is recorded in the key, encode and decode agree automatically -- but if you decode with a different explicit --max-context-length, the key's value still wins.
Coherence is over the window, not the whole text. Past the first reset the model conditions on a sliding window of recent tokens rather than the entire history, so very long messages read like natural long-form drift rather than a single tightly-planned document. See the caveat in Limitations.

Encryption

The codec is not a cipher -- the key is metadata, not a secret, and anyone with it and the model can read the message. For confidentiality, encrypt the payload yourself before encoding. Encryption also gives you the uniform bits the indistinguishability argument wants, so you do not additionally need --compress (encrypted data will not compress anyway).

# encrypt, then hide; the codec never sees the plaintext
age -r age1... secret.txt | subtext-codec encode --key key.json --input-bytes - --output-text message.txt
# recover, then decrypt
subtext-codec decode --key key.json --input-text message.txt --output-bytes - | age -d -i key.txt > secret.txt

Any tool works -- age, gpg, openssl enc. Keeping encryption out of the codec is deliberate: this is a research prototype, and rolling in a cipher would imply a security property it does not provide.

Determinism and devices

The codec reduces to one requirement: encode and decode must derive the same frequency table at every step, down to the last integer. Everything after the forward pass runs on CPU no matter where the model lives: the logits are cast to float32 for the stable sort and the candidate filter, then to float64 for the softmax and quantization. So the only device-sensitive component is the logits themselves. Encode and decode also drive the model identically (prefill the prompt, then one token at a time), so they hit the same kernels at the same shapes.

This has been measured, not just reasoned about. On an RTX 5090 (Blackwell, sm_120, torch 2.13 + CUDA 13), repeated passes over the same tokens produce bit-identical logits in fp32, bf16 and fp16, and all three round-trip on GPU. GPU is as reliable as CPU.

What matters is that the environment matches between encode and decode:

Factor	Effect
Different dtype (bf16 ↔ fp32)	Breaks. The key records the dtype; keep it.
TF32 on vs off	Pinned off by `set_deterministic`, so it no longer varies.
Different attention backend (SDPA / FlashAttention / eager)	Can break across machines. Same environment, same choice.
Different torch / transformers version	Can break; pin them (`requirements.txt`) if a message must survive an upgrade.
Different device (`cuda` ↔ `cpu`)	Unreliable, but not automatically fatal -- see below.

Crossing devices is not supported, though it is not guaranteed to fail: gpt2 in fp32 does decode on CPU from a CUDA encode, because the two happen to agree bit for bit. Larger models and bf16 will not be so lucky. Do not rely on it. What is guaranteed is that a mismatch is caught rather than returning plausible wrong bytes -- that is the payload framing's job.

set_deterministic also sets CUBLAS_WORKSPACE_CONFIG=:4096:8 (via package import, before torch loads) and disables cuDNN autotuning. It holds fp32 matmuls at full precision, which costs some speed on Ampere and later; a deliberate trade, since a mismatch costs the entire message.

bf16 and fp16 work but leave less margin -- half the mantissa means adjacent logits sit closer together. fp32 is safer when you can afford it. Exact ties are fine in any precision: the sort is stable, so ties resolve by token id.

To confirm on your own hardware before trusting a message to it:

uv run pytest tests/test_determinism.py -v

Those parametrize over every device and dtype present, so on a GPU box they additionally check bit-identical logits on CUDA, matching bands, and fp32/bf16/fp16 round trips.

Compatibility

There are two wire formats:

v1 -- a single segment framing length || crc32 || data, exactly as 1.0 wrote it. A plain message that fits one context window (no compression, no rolling) is still written as v1, so it stays byte-for-byte readable by 1.0.
v2 -- added in 1.1 for compression and the rolling window. The key records "version": "v2" and, as needed, a compression field and a window field. Compressed payloads frame a length and CRC over the original data so a wrong compression flag is caught; rolled payloads carry the window so the decoder resets at the same points. A 1.0 reader rejects a v2 key outright rather than mis-reading it, which is why the new features carry a new version.

1.1 reads both. It writes v2 only when you compress or when the payload is long enough to roll the context; everything else stays v1.

Every pre-1.0 format is gone along with the code that decoded them: they used rank coding, which corrupted roughly a third of its own messages and never produced model-distributed output. Pre-1.0 also numbered a format v1, but that one carried base/top_p and no temperature, so the collision is detectable -- such a key fails with an explanation rather than a generic error. To read a message encoded before 1.0, install subtext-codec~=0.2.0; to keep it, re-encode the payload with this version.

Limitations

Brittle to edits: changing a single token of the output breaks decoding. This is an encoding scheme, not an error-correcting one.
Model-dependent: requires the exact same weights, tokenizer and dtype. Decoding a bf16 encode in fp32 will not work.
Not confidential on its own: the key is metadata, not a cipher. Encrypt the payload before encoding it -- both for secrecy and because the indistinguishability argument assumes uniform payload bits.
Context length: handled automatically by the rolling window, but past the first reset the model conditions on recent tokens rather than the whole history. Coherence is local to the window, and a discriminator with the full text could in principle notice the long-range structure "resetting" -- subtle for a reasonable window, but a real difference from full-context sampling, on top of the temperature caveat below.
Capacity: the distribution's entropy, typically 1-5 bits per token depending on temperature. Expect a few hundred tokens per hundred bytes.
Length is the remaining tell: the per-token statistics are right, but a 900-token hotel review is still an odd thing to find. Pick a prompt whose natural continuation is long-form.
"Indistinguishable" is relative to the temperature you chose: output at temperature=1.5 is exactly temperature-1.5 sampling, which is only unremarkable to someone who has no expectation about the setting. An adversary who knows you would have sampled at 1.0 can distinguish it in aggregate. Lower temperature closes that gap and lengthens the text.

This project is a research prototype, not a secure or production steganography system.

Testing

uv run pytest # everything
uv run pytest -m "not slow"# skip tests that download a checkpoint

The slow tests use sshleifer/tiny-gpt2 and skip themselves if the Hub is unreachable.

License

MIT

subtext-codec

Steganographic data encoding in natural language via arithmetic coding against an LLM

subtext-codec hides arbitrary binary data inside seemingly normal LLM-generated text. The payload drives an arithmetic coder whose probability model is the language model's own next-token distribution, so the text it produces is distributed exactly as ordinary sampling would be while secretly encoding bytes. With the same model, tokenizer, prompt prefix and parameters, the process is fully reversible.

How it works

The payload drives an arithmetic decoder whose probability model is the language model itself. Emitting a token the model assigns probability p consumes -log2(p) payload bits, so a confident step carries almost nothing and an uncertain one carries a lot.

Two properties follow, and they are the entire point of the design:

The generated text is distributed exactly as ordinary sampling at the chosen temperature. Nothing is ever forced off-distribution.
Capacity per token equals the distribution's entropy, which is the information-theoretic ceiling for text that stays indistinguishable from sampling.

Encoding

The payload is framed with a length and a CRC32, then read as a bitstream.
At each step the model's logits are clipped to top_k, filtered to tokens that are safe to emit (see below), and turned into an integer frequency table at temperature.
The arithmetic decoder consumes payload bits and selects a token from that table.
Generation stops as soon as the emitted tokens pin down every payload bit.

Decoding

Start from the same prompt prefix and model, and rebuild the same table at each step.
Run the arithmetic encoder over the observed tokens, which reproduces the bitstream.
Stop once the declared length has been recovered, then check the CRC.

Decoding needs the text, the prompt prefix, the same model and tokenizer, and the codec parameters -- all of which the key file carries. No terminator token is required, so text after the message is simply ignored.

Why the encoder checks itself

The coder re-encodes each symbol as it emits it and refuses to return a message whose bits it cannot reproduce. That is not belt-and-braces: feeding the arithmetic decoder a zero-filled tail past the end of the payload parks its value register exactly on the interval midpoint, where it underflows forever and the payload's last two bits are never emitted. The result is a message that looks fine and silently fails to decode. The check is what turns that class of bug into a loud failure.

Why the safety filter exists

The encoder chooses token ids, but the decoder is handed text. Nothing guarantees that writing tokens out and reading them back returns the same ids: a byte-level BPE vocabulary will merge an emitted token into its neighbours, so 'an' written after 'Kanz' reads back as 'anz' and every subsequent step decodes against the wrong prefix.

So a candidate is only usable if appending it leaves the surrounding text re-tokenizing unchanged. Both sides apply that same filter to the same prefix, so they stay in step, and the encoder verifies the finished text tokenizes back to exactly what it built before handing it over. A message that cannot satisfy this raises at encode time rather than becoming an artifact that silently fails to decode.

Features

Entropy-optimal capacity -- each token carries -log2(p) bits, the theoretical maximum for output that still looks like sampling
Output distributed exactly as the model's own sampling, so no per-token statistical tell
Temperature as a principled capacity dial, replacing rank truncation
Tokenizer-stable candidate selection, verified before any message is returned
Self-inverse check at encode time -- the coder re-encodes as it goes and refuses to emit a message it cannot read back
Framed payload with a CRC32, so a wrong key, prompt or model is detected rather than yielding plausible wrong bytes
Optional zlib compression (--compress), which shortens the cover text and makes the payload bits more uniform
Automatic rolling context window, so a payload can exceed the model's context window with no repeated prompt and no tuning
Incremental KV-cached stepping, so cost is linear in message length rather than quadratic
Deterministic throughout; verified bit-identical on CPU and CUDA across fp32/bf16/fp16
Hugging Face Transformers backend -- works with most causal LMs

Installation

From PyPI (or via uv):

uv pip install subtext-codec
# or: pip install subtext-codec

From source:

git clone https://github.com/shevisj/subtext-codec
cd subtext-codec
uv venv --python 3.13
uv pip install -r requirements.txt
uv pip install -e .

torch and transformers are the only runtime dependencies. The published package sets lower bounds only, so it installs alongside whatever torch build you already have; requirements.txt pins an exact, CPU-only development environment and is the one to use when reproducing a decode.

Usage

Run subtext-codec --version to print the installed version. The CLI otherwise exposes encode and decode. Shared flags:

--key -- path to the codec key file (required)
--model-name-or-path -- HuggingFace model id or local path; optional if stored in the key
--prompt-prefix -- prefix used for both encode and decode; taken from the key if present
--device -- e.g. cpu or cuda (falls back to the key, then cpu)
--torch-dtype -- weight dtype (auto, fp16, bf16, fp32)
--max-context-length -- cap on sequence length; defaults to the model's own limit
--seed -- deterministic seed (default: 0)
--quiet -- suppress the progress line on stderr

Encode bytes into text

subtext-codec encode \
--model-name-or-path gpt2 \
--prompt-prefix "Once upon a time, " \
--input-bytes secret.txt \
--output-text message.txt \
--key key.json \
--temperature 1.5

Encode-only flags:

--temperature -- the capacity dial (default: 1.5). Higher packs more payload per token and shortens the cover text, at the cost of more erratic prose. See Choosing a temperature.
--top-k -- candidates considered per step (default: 64). This bounds the stability filter's work; it is not a capacity dial. Must be at least 2.
--compress -- zlib-compress the payload before encoding. Shortens the cover text and makes the payload bits more uniform. Recorded in the key, so decoding reverses it automatically. See Compression.
--max-new-tokens -- fail rather than generate more than this many tokens
--no-store-model -- keep the model id out of the key
--verify -- decode the result before writing it, confirming the round trip end to end

Payloads too long for one context window are handled automatically; see Large payloads.

Unless --quiet, encoding prints a one-line summary to stderr -- payload size, cover tokens, bits per token and mean surprisal (and how many times the context rolled) -- so you can see the coder is running at the distribution's entropy.

The output text is just the generated story, with no metadata header. key.json records temperature, top_k, the prompt prefix, the device, the dtype, (unless --no-store-model) the model id, and -- when compression or the rolling window is used -- a compression or window field and a "version": "v2" marker.

If the path given to --key already exists, its values are reused as defaults and any CLI overrides are written back, so a second message needs only the key:

subtext-codec encode --key key.json --input-bytes secret.bin --output-text message.txt

Decode text back into bytes

subtext-codec decode \
--input-text message.txt \
--key key.json \
--output-bytes decoded.bin

Parameters come from the key unless overridden for the run. Decoding never modifies the key file -- the one artifact a message cannot be recovered without is not rewritten by a read operation.

Decoding stops as soon as the payload's declared length has been recovered, so trailing text is harmless. Text before the prompt prefix is skipped too. If the message was altered, or the key, prompt or model do not match, the CRC32 catches it rather than returning plausible wrong bytes.

Sample artifacts

samples/secret.txt is a payload to play with. Build the rest of the fixture from it:

python samples/regenerate.py

That encodes it with gpt2 on CPU -- a ~500MB download, no GPU or gated checkpoint needed -- and writes message.txt, key.json and decoded.txt alongside it, verifying the round trip before writing anything. Encoding is deterministic, so it reproduces the same message every time.

Then decode it back the normal way:

subtext-codec decode \
--input-text samples/message.txt \
--key samples/key.json \
--output-bytes /tmp/decoded.txt
diff samples/secret.txt /tmp/decoded.txt &&echo"exact match"

(Before 1.0 the fixture was checked in, but it was a v2 message against a gated 16GB Llama checkpoint -- unreadable by this version and unrunnable by most people. Generating it locally is both smaller and honest.)

Programmatic usage

See the demo notebook.

Choosing a temperature

temperature is the capacity dial. Raising it flattens the distribution, so each token carries more payload and the cover text gets shorter -- but the text becomes more erratic, exactly as sampling at that temperature would. top_k is not a capacity dial here; it only bounds how much work the stability filter does.

Measured on the 445-byte samples/secret.txt, Qwen2.5-7B (bf16) on an RTX 5090, top_k=64:

Temperature	Cover tokens	Bits/token	Character
1.0	2914	1.22	best prose, but far too long to be plausible
1.5	894	3.98	the knee -- reads like a real, rambling review
2.0	709	5.02	densest, but visibly erratic and prone to invention

Mean surprisal of the emitted tokens matches capacity at every setting (1.23 / 4.02 / 5.07 bits), which is the confirmation that the coder is running at the distribution's entropy rather than leaving capacity on the table.

For reference, the rank-coding scheme this replaced in 1.0 managed 4.35 bits/token at its best setting and 1.57 at its most conservative, and its output was never model-distributed. Arithmetic coding at temperature 1.5 matches its best capacity, and at 2.0 exceeds it, while staying on-distribution throughout.

A bigger model does not buy more capacity. A stronger model concentrates probability mass, which lowers entropy and therefore bits per token. It buys better prose at a given temperature, not shorter text. Keep payloads to a few hundred bytes, and pick a prompt whose natural continuation is long-form so the required length is not conspicuous.

Compression

--compress zlib-compresses the payload before encoding. It does two useful things:

Shortens the cover text. Fewer payload bytes means fewer tokens to carry them.
Makes the bits more uniform, which is exactly what the indistinguishability argument assumes. Raw ASCII is biased; compressed (or encrypted) data is not.

subtext-codec encode --key key.json --input-bytes secret.txt --output-text message.txt --compress

Compression is recorded in the key, so decode reverses it automatically -- you do not pass --compress when decoding. A compressed message uses the v2 wire format (see Compatibility). The whole payload is framed with a length and CRC32 before compression, so a key that misdescribes compression is caught rather than returning the raw compressed bytes.

Compression rarely helps on data that is already high-entropy (encrypted blobs, media). It helps most on text and structured data.

Large payloads

A payload can outgrow the model's context window, and that is handled automatically -- there is nothing to configure. When the context fills, the codec re-prefixes the most recent half of the tokens and keeps generating, a rolling window that slides forward as the message grows. The text stays one continuous passage: the prompt appears once, at the start, and never repeats.

# no special flags: a long payload just works
subtext-codec encode --key key.json --input-bytes big.bin --output-text message.txt

The decoder resets at the same points, so it recovers the message exactly. The window size it used rides in the key (a window field), so decoding needs nothing extra. A message that rolls uses the v2 wire format; one that fits a single window stays v1.

Two things worth knowing:

The window follows the model's context limit (or --max-context-length, if you set a smaller one). Because the size is recorded in the key, encode and decode agree automatically -- but if you decode with a different explicit --max-context-length, the key's value still wins.
Coherence is over the window, not the whole text. Past the first reset the model conditions on a sliding window of recent tokens rather than the entire history, so very long messages read like natural long-form drift rather than a single tightly-planned document. See the caveat in Limitations.

Encryption

The codec is not a cipher -- the key is metadata, not a secret, and anyone with it and the model can read the message. For confidentiality, encrypt the payload yourself before encoding. Encryption also gives you the uniform bits the indistinguishability argument wants, so you do not additionally need --compress (encrypted data will not compress anyway).

# encrypt, then hide; the codec never sees the plaintext
age -r age1... secret.txt | subtext-codec encode --key key.json --input-bytes - --output-text message.txt
# recover, then decrypt
subtext-codec decode --key key.json --input-text message.txt --output-bytes - | age -d -i key.txt > secret.txt

Any tool works -- age, gpg, openssl enc. Keeping encryption out of the codec is deliberate: this is a research prototype, and rolling in a cipher would imply a security property it does not provide.

Determinism and devices

The codec reduces to one requirement: encode and decode must derive the same frequency table at every step, down to the last integer. Everything after the forward pass runs on CPU no matter where the model lives: the logits are cast to float32 for the stable sort and the candidate filter, then to float64 for the softmax and quantization. So the only device-sensitive component is the logits themselves. Encode and decode also drive the model identically (prefill the prompt, then one token at a time), so they hit the same kernels at the same shapes.

This has been measured, not just reasoned about. On an RTX 5090 (Blackwell, sm_120, torch 2.13 + CUDA 13), repeated passes over the same tokens produce bit-identical logits in fp32, bf16 and fp16, and all three round-trip on GPU. GPU is as reliable as CPU.

What matters is that the environment matches between encode and decode:

Factor	Effect
Different dtype (bf16 ↔ fp32)	Breaks. The key records the dtype; keep it.
TF32 on vs off	Pinned off by `set_deterministic`, so it no longer varies.
Different attention backend (SDPA / FlashAttention / eager)	Can break across machines. Same environment, same choice.
Different torch / transformers version	Can break; pin them (`requirements.txt`) if a message must survive an upgrade.
Different device (`cuda` ↔ `cpu`)	Unreliable, but not automatically fatal -- see below.

Crossing devices is not supported, though it is not guaranteed to fail: gpt2 in fp32 does decode on CPU from a CUDA encode, because the two happen to agree bit for bit. Larger models and bf16 will not be so lucky. Do not rely on it. What is guaranteed is that a mismatch is caught rather than returning plausible wrong bytes -- that is the payload framing's job.

set_deterministic also sets CUBLAS_WORKSPACE_CONFIG=:4096:8 (via package import, before torch loads) and disables cuDNN autotuning. It holds fp32 matmuls at full precision, which costs some speed on Ampere and later; a deliberate trade, since a mismatch costs the entire message.

bf16 and fp16 work but leave less margin -- half the mantissa means adjacent logits sit closer together. fp32 is safer when you can afford it. Exact ties are fine in any precision: the sort is stable, so ties resolve by token id.

To confirm on your own hardware before trusting a message to it:

uv run pytest tests/test_determinism.py -v

Those parametrize over every device and dtype present, so on a GPU box they additionally check bit-identical logits on CUDA, matching bands, and fp32/bf16/fp16 round trips.

Compatibility

There are two wire formats:

v1 -- a single segment framing length || crc32 || data, exactly as 1.0 wrote it. A plain message that fits one context window (no compression, no rolling) is still written as v1, so it stays byte-for-byte readable by 1.0.
v2 -- added in 1.1 for compression and the rolling window. The key records "version": "v2" and, as needed, a compression field and a window field. Compressed payloads frame a length and CRC over the original data so a wrong compression flag is caught; rolled payloads carry the window so the decoder resets at the same points. A 1.0 reader rejects a v2 key outright rather than mis-reading it, which is why the new features carry a new version.

1.1 reads both. It writes v2 only when you compress or when the payload is long enough to roll the context; everything else stays v1.

Every pre-1.0 format is gone along with the code that decoded them: they used rank coding, which corrupted roughly a third of its own messages and never produced model-distributed output. Pre-1.0 also numbered a format v1, but that one carried base/top_p and no temperature, so the collision is detectable -- such a key fails with an explanation rather than a generic error. To read a message encoded before 1.0, install subtext-codec~=0.2.0; to keep it, re-encode the payload with this version.

Limitations

Brittle to edits: changing a single token of the output breaks decoding. This is an encoding scheme, not an error-correcting one.
Model-dependent: requires the exact same weights, tokenizer and dtype. Decoding a bf16 encode in fp32 will not work.
Not confidential on its own: the key is metadata, not a cipher. Encrypt the payload before encoding it -- both for secrecy and because the indistinguishability argument assumes uniform payload bits.
Context length: handled automatically by the rolling window, but past the first reset the model conditions on recent tokens rather than the whole history. Coherence is local to the window, and a discriminator with the full text could in principle notice the long-range structure "resetting" -- subtle for a reasonable window, but a real difference from full-context sampling, on top of the temperature caveat below.
Capacity: the distribution's entropy, typically 1-5 bits per token depending on temperature. Expect a few hundred tokens per hundred bytes.
Length is the remaining tell: the per-token statistics are right, but a 900-token hotel review is still an odd thing to find. Pick a prompt whose natural continuation is long-form.
"Indistinguishable" is relative to the temperature you chose: output at temperature=1.5 is exactly temperature-1.5 sampling, which is only unremarkable to someone who has no expectation about the setting. An adversary who knows you would have sampled at 1.0 can distinguish it in aggregate. Lower temperature closes that gap and lengthens the text.

This project is a research prototype, not a secure or production steganography system.

Testing

uv run pytest # everything
uv run pytest -m "not slow"# skip tests that download a checkpoint

The slow tests use sshleifer/tiny-gpt2 and skip themselves if the Hub is unreachable.

License

MIT

subtext-codec

Steganographic data encoding in natural language via arithmetic coding against an LLM

subtext-codec hides arbitrary binary data inside seemingly normal LLM-generated text. The payload drives an arithmetic coder whose probability model is the language model's own next-token distribution, so the text it produces is distributed exactly as ordinary sampling would be while secretly encoding bytes. With the same model, tokenizer, prompt prefix and parameters, the process is fully reversible.

How it works

The payload drives an arithmetic decoder whose probability model is the language model itself. Emitting a token the model assigns probability p consumes -log2(p) payload bits, so a confident step carries almost nothing and an uncertain one carries a lot.

Two properties follow, and they are the entire point of the design:

The generated text is distributed exactly as ordinary sampling at the chosen temperature. Nothing is ever forced off-distribution.
Capacity per token equals the distribution's entropy, which is the information-theoretic ceiling for text that stays indistinguishable from sampling.

Encoding

The payload is framed with a length and a CRC32, then read as a bitstream.
At each step the model's logits are clipped to top_k, filtered to tokens that are safe to emit (see below), and turned into an integer frequency table at temperature.
The arithmetic decoder consumes payload bits and selects a token from that table.
Generation stops as soon as the emitted tokens pin down every payload bit.

Decoding

Start from the same prompt prefix and model, and rebuild the same table at each step.
Run the arithmetic encoder over the observed tokens, which reproduces the bitstream.
Stop once the declared length has been recovered, then check the CRC.

Decoding needs the text, the prompt prefix, the same model and tokenizer, and the codec parameters -- all of which the key file carries. No terminator token is required, so text after the message is simply ignored.

Why the encoder checks itself

The coder re-encodes each symbol as it emits it and refuses to return a message whose bits it cannot reproduce. That is not belt-and-braces: feeding the arithmetic decoder a zero-filled tail past the end of the payload parks its value register exactly on the interval midpoint, where it underflows forever and the payload's last two bits are never emitted. The result is a message that looks fine and silently fails to decode. The check is what turns that class of bug into a loud failure.

Why the safety filter exists

The encoder chooses token ids, but the decoder is handed text. Nothing guarantees that writing tokens out and reading them back returns the same ids: a byte-level BPE vocabulary will merge an emitted token into its neighbours, so 'an' written after 'Kanz' reads back as 'anz' and every subsequent step decodes against the wrong prefix.

So a candidate is only usable if appending it leaves the surrounding text re-tokenizing unchanged. Both sides apply that same filter to the same prefix, so they stay in step, and the encoder verifies the finished text tokenizes back to exactly what it built before handing it over. A message that cannot satisfy this raises at encode time rather than becoming an artifact that silently fails to decode.

Features

Entropy-optimal capacity -- each token carries -log2(p) bits, the theoretical maximum for output that still looks like sampling
Output distributed exactly as the model's own sampling, so no per-token statistical tell
Temperature as a principled capacity dial, replacing rank truncation
Tokenizer-stable candidate selection, verified before any message is returned
Self-inverse check at encode time -- the coder re-encodes as it goes and refuses to emit a message it cannot read back
Framed payload with a CRC32, so a wrong key, prompt or model is detected rather than yielding plausible wrong bytes
Optional zlib compression (--compress), which shortens the cover text and makes the payload bits more uniform
Automatic rolling context window, so a payload can exceed the model's context window with no repeated prompt and no tuning
Incremental KV-cached stepping, so cost is linear in message length rather than quadratic
Deterministic throughout; verified bit-identical on CPU and CUDA across fp32/bf16/fp16
Hugging Face Transformers backend -- works with most causal LMs

Installation

From PyPI (or via uv):

uv pip install subtext-codec
# or: pip install subtext-codec

From source:

git clone https://github.com/shevisj/subtext-codec
cd subtext-codec
uv venv --python 3.13
uv pip install -r requirements.txt
uv pip install -e .

torch and transformers are the only runtime dependencies. The published package sets lower bounds only, so it installs alongside whatever torch build you already have; requirements.txt pins an exact, CPU-only development environment and is the one to use when reproducing a decode.

Usage

Run subtext-codec --version to print the installed version. The CLI otherwise exposes encode and decode. Shared flags:

--key -- path to the codec key file (required)
--model-name-or-path -- HuggingFace model id or local path; optional if stored in the key
--prompt-prefix -- prefix used for both encode and decode; taken from the key if present
--device -- e.g. cpu or cuda (falls back to the key, then cpu)
--torch-dtype -- weight dtype (auto, fp16, bf16, fp32)
--max-context-length -- cap on sequence length; defaults to the model's own limit
--seed -- deterministic seed (default: 0)
--quiet -- suppress the progress line on stderr

Encode bytes into text

subtext-codec encode \
--model-name-or-path gpt2 \
--prompt-prefix "Once upon a time, " \
--input-bytes secret.txt \
--output-text message.txt \
--key key.json \
--temperature 1.5

Encode-only flags:

--temperature -- the capacity dial (default: 1.5). Higher packs more payload per token and shortens the cover text, at the cost of more erratic prose. See Choosing a temperature.
--top-k -- candidates considered per step (default: 64). This bounds the stability filter's work; it is not a capacity dial. Must be at least 2.
--compress -- zlib-compress the payload before encoding. Shortens the cover text and makes the payload bits more uniform. Recorded in the key, so decoding reverses it automatically. See Compression.
--max-new-tokens -- fail rather than generate more than this many tokens
--no-store-model -- keep the model id out of the key
--verify -- decode the result before writing it, confirming the round trip end to end

Payloads too long for one context window are handled automatically; see Large payloads.

Unless --quiet, encoding prints a one-line summary to stderr -- payload size, cover tokens, bits per token and mean surprisal (and how many times the context rolled) -- so you can see the coder is running at the distribution's entropy.

The output text is just the generated story, with no metadata header. key.json records temperature, top_k, the prompt prefix, the device, the dtype, (unless --no-store-model) the model id, and -- when compression or the rolling window is used -- a compression or window field and a "version": "v2" marker.

If the path given to --key already exists, its values are reused as defaults and any CLI overrides are written back, so a second message needs only the key:

subtext-codec encode --key key.json --input-bytes secret.bin --output-text message.txt

Decode text back into bytes

subtext-codec decode \
--input-text message.txt \
--key key.json \
--output-bytes decoded.bin

Parameters come from the key unless overridden for the run. Decoding never modifies the key file -- the one artifact a message cannot be recovered without is not rewritten by a read operation.

Decoding stops as soon as the payload's declared length has been recovered, so trailing text is harmless. Text before the prompt prefix is skipped too. If the message was altered, or the key, prompt or model do not match, the CRC32 catches it rather than returning plausible wrong bytes.

Sample artifacts

samples/secret.txt is a payload to play with. Build the rest of the fixture from it:

python samples/regenerate.py

That encodes it with gpt2 on CPU -- a ~500MB download, no GPU or gated checkpoint needed -- and writes message.txt, key.json and decoded.txt alongside it, verifying the round trip before writing anything. Encoding is deterministic, so it reproduces the same message every time.

Then decode it back the normal way:

subtext-codec decode \
--input-text samples/message.txt \
--key samples/key.json \
--output-bytes /tmp/decoded.txt
diff samples/secret.txt /tmp/decoded.txt &&echo"exact match"

(Before 1.0 the fixture was checked in, but it was a v2 message against a gated 16GB Llama checkpoint -- unreadable by this version and unrunnable by most people. Generating it locally is both smaller and honest.)

Programmatic usage

See the demo notebook.

Choosing a temperature

temperature is the capacity dial. Raising it flattens the distribution, so each token carries more payload and the cover text gets shorter -- but the text becomes more erratic, exactly as sampling at that temperature would. top_k is not a capacity dial here; it only bounds how much work the stability filter does.

Measured on the 445-byte samples/secret.txt, Qwen2.5-7B (bf16) on an RTX 5090, top_k=64:

Temperature	Cover tokens	Bits/token	Character
1.0	2914	1.22	best prose, but far too long to be plausible
1.5	894	3.98	the knee -- reads like a real, rambling review
2.0	709	5.02	densest, but visibly erratic and prone to invention

Mean surprisal of the emitted tokens matches capacity at every setting (1.23 / 4.02 / 5.07 bits), which is the confirmation that the coder is running at the distribution's entropy rather than leaving capacity on the table.

For reference, the rank-coding scheme this replaced in 1.0 managed 4.35 bits/token at its best setting and 1.57 at its most conservative, and its output was never model-distributed. Arithmetic coding at temperature 1.5 matches its best capacity, and at 2.0 exceeds it, while staying on-distribution throughout.

A bigger model does not buy more capacity. A stronger model concentrates probability mass, which lowers entropy and therefore bits per token. It buys better prose at a given temperature, not shorter text. Keep payloads to a few hundred bytes, and pick a prompt whose natural continuation is long-form so the required length is not conspicuous.

Compression

--compress zlib-compresses the payload before encoding. It does two useful things:

Shortens the cover text. Fewer payload bytes means fewer tokens to carry them.
Makes the bits more uniform, which is exactly what the indistinguishability argument assumes. Raw ASCII is biased; compressed (or encrypted) data is not.

subtext-codec encode --key key.json --input-bytes secret.txt --output-text message.txt --compress

Compression is recorded in the key, so decode reverses it automatically -- you do not pass --compress when decoding. A compressed message uses the v2 wire format (see Compatibility). The whole payload is framed with a length and CRC32 before compression, so a key that misdescribes compression is caught rather than returning the raw compressed bytes.

Compression rarely helps on data that is already high-entropy (encrypted blobs, media). It helps most on text and structured data.

Large payloads

A payload can outgrow the model's context window, and that is handled automatically -- there is nothing to configure. When the context fills, the codec re-prefixes the most recent half of the tokens and keeps generating, a rolling window that slides forward as the message grows. The text stays one continuous passage: the prompt appears once, at the start, and never repeats.

# no special flags: a long payload just works
subtext-codec encode --key key.json --input-bytes big.bin --output-text message.txt

The decoder resets at the same points, so it recovers the message exactly. The window size it used rides in the key (a window field), so decoding needs nothing extra. A message that rolls uses the v2 wire format; one that fits a single window stays v1.

Two things worth knowing:

The window follows the model's context limit (or --max-context-length, if you set a smaller one). Because the size is recorded in the key, encode and decode agree automatically -- but if you decode with a different explicit --max-context-length, the key's value still wins.
Coherence is over the window, not the whole text. Past the first reset the model conditions on a sliding window of recent tokens rather than the entire history, so very long messages read like natural long-form drift rather than a single tightly-planned document. See the caveat in Limitations.

Encryption

The codec is not a cipher -- the key is metadata, not a secret, and anyone with it and the model can read the message. For confidentiality, encrypt the payload yourself before encoding. Encryption also gives you the uniform bits the indistinguishability argument wants, so you do not additionally need --compress (encrypted data will not compress anyway).

# encrypt, then hide; the codec never sees the plaintext
age -r age1... secret.txt | subtext-codec encode --key key.json --input-bytes - --output-text message.txt
# recover, then decrypt
subtext-codec decode --key key.json --input-text message.txt --output-bytes - | age -d -i key.txt > secret.txt

Any tool works -- age, gpg, openssl enc. Keeping encryption out of the codec is deliberate: this is a research prototype, and rolling in a cipher would imply a security property it does not provide.

Determinism and devices

The codec reduces to one requirement: encode and decode must derive the same frequency table at every step, down to the last integer. Everything after the forward pass runs on CPU no matter where the model lives: the logits are cast to float32 for the stable sort and the candidate filter, then to float64 for the softmax and quantization. So the only device-sensitive component is the logits themselves. Encode and decode also drive the model identically (prefill the prompt, then one token at a time), so they hit the same kernels at the same shapes.

This has been measured, not just reasoned about. On an RTX 5090 (Blackwell, sm_120, torch 2.13 + CUDA 13), repeated passes over the same tokens produce bit-identical logits in fp32, bf16 and fp16, and all three round-trip on GPU. GPU is as reliable as CPU.

What matters is that the environment matches between encode and decode:

Factor	Effect
Different dtype (bf16 ↔ fp32)	Breaks. The key records the dtype; keep it.
TF32 on vs off	Pinned off by `set_deterministic`, so it no longer varies.
Different attention backend (SDPA / FlashAttention / eager)	Can break across machines. Same environment, same choice.
Different torch / transformers version	Can break; pin them (`requirements.txt`) if a message must survive an upgrade.
Different device (`cuda` ↔ `cpu`)	Unreliable, but not automatically fatal -- see below.

Crossing devices is not supported, though it is not guaranteed to fail: gpt2 in fp32 does decode on CPU from a CUDA encode, because the two happen to agree bit for bit. Larger models and bf16 will not be so lucky. Do not rely on it. What is guaranteed is that a mismatch is caught rather than returning plausible wrong bytes -- that is the payload framing's job.

set_deterministic also sets CUBLAS_WORKSPACE_CONFIG=:4096:8 (via package import, before torch loads) and disables cuDNN autotuning. It holds fp32 matmuls at full precision, which costs some speed on Ampere and later; a deliberate trade, since a mismatch costs the entire message.

bf16 and fp16 work but leave less margin -- half the mantissa means adjacent logits sit closer together. fp32 is safer when you can afford it. Exact ties are fine in any precision: the sort is stable, so ties resolve by token id.

To confirm on your own hardware before trusting a message to it:

uv run pytest tests/test_determinism.py -v

Those parametrize over every device and dtype present, so on a GPU box they additionally check bit-identical logits on CUDA, matching bands, and fp32/bf16/fp16 round trips.

Compatibility

There are two wire formats:

v1 -- a single segment framing length || crc32 || data, exactly as 1.0 wrote it. A plain message that fits one context window (no compression, no rolling) is still written as v1, so it stays byte-for-byte readable by 1.0.
v2 -- added in 1.1 for compression and the rolling window. The key records "version": "v2" and, as needed, a compression field and a window field. Compressed payloads frame a length and CRC over the original data so a wrong compression flag is caught; rolled payloads carry the window so the decoder resets at the same points. A 1.0 reader rejects a v2 key outright rather than mis-reading it, which is why the new features carry a new version.

1.1 reads both. It writes v2 only when you compress or when the payload is long enough to roll the context; everything else stays v1.

Every pre-1.0 format is gone along with the code that decoded them: they used rank coding, which corrupted roughly a third of its own messages and never produced model-distributed output. Pre-1.0 also numbered a format v1, but that one carried base/top_p and no temperature, so the collision is detectable -- such a key fails with an explanation rather than a generic error. To read a message encoded before 1.0, install subtext-codec~=0.2.0; to keep it, re-encode the payload with this version.

Limitations

Brittle to edits: changing a single token of the output breaks decoding. This is an encoding scheme, not an error-correcting one.
Model-dependent: requires the exact same weights, tokenizer and dtype. Decoding a bf16 encode in fp32 will not work.
Not confidential on its own: the key is metadata, not a cipher. Encrypt the payload before encoding it -- both for secrecy and because the indistinguishability argument assumes uniform payload bits.
Context length: handled automatically by the rolling window, but past the first reset the model conditions on recent tokens rather than the whole history. Coherence is local to the window, and a discriminator with the full text could in principle notice the long-range structure "resetting" -- subtle for a reasonable window, but a real difference from full-context sampling, on top of the temperature caveat below.
Capacity: the distribution's entropy, typically 1-5 bits per token depending on temperature. Expect a few hundred tokens per hundred bytes.
Length is the remaining tell: the per-token statistics are right, but a 900-token hotel review is still an odd thing to find. Pick a prompt whose natural continuation is long-form.
"Indistinguishable" is relative to the temperature you chose: output at temperature=1.5 is exactly temperature-1.5 sampling, which is only unremarkable to someone who has no expectation about the setting. An adversary who knows you would have sampled at 1.0 can distinguish it in aggregate. Lower temperature closes that gap and lengthens the text.

This project is a research prototype, not a secure or production steganography system.

Testing

uv run pytest # everything
uv run pytest -m "not slow"# skip tests that download a checkpoint

The slow tests use sshleifer/tiny-gpt2 and skip themselves if the Hub is unreachable.

License

MIT

subtext-codec

Steganographic data encoding in natural language via arithmetic coding against an LLM

subtext-codec hides arbitrary binary data inside seemingly normal LLM-generated text. The payload drives an arithmetic coder whose probability model is the language model's own next-token distribution, so the text it produces is distributed exactly as ordinary sampling would be while secretly encoding bytes. With the same model, tokenizer, prompt prefix and parameters, the process is fully reversible.

How it works

The payload drives an arithmetic decoder whose probability model is the language model itself. Emitting a token the model assigns probability p consumes -log2(p) payload bits, so a confident step carries almost nothing and an uncertain one carries a lot.

Two properties follow, and they are the entire point of the design:

The generated text is distributed exactly as ordinary sampling at the chosen temperature. Nothing is ever forced off-distribution.
Capacity per token equals the distribution's entropy, which is the information-theoretic ceiling for text that stays indistinguishable from sampling.

Encoding

The payload is framed with a length and a CRC32, then read as a bitstream.
At each step the model's logits are clipped to top_k, filtered to tokens that are safe to emit (see below), and turned into an integer frequency table at temperature.
The arithmetic decoder consumes payload bits and selects a token from that table.
Generation stops as soon as the emitted tokens pin down every payload bit.

Decoding

Start from the same prompt prefix and model, and rebuild the same table at each step.
Run the arithmetic encoder over the observed tokens, which reproduces the bitstream.
Stop once the declared length has been recovered, then check the CRC.

Decoding needs the text, the prompt prefix, the same model and tokenizer, and the codec parameters -- all of which the key file carries. No terminator token is required, so text after the message is simply ignored.

Why the encoder checks itself

The coder re-encodes each symbol as it emits it and refuses to return a message whose bits it cannot reproduce. That is not belt-and-braces: feeding the arithmetic decoder a zero-filled tail past the end of the payload parks its value register exactly on the interval midpoint, where it underflows forever and the payload's last two bits are never emitted. The result is a message that looks fine and silently fails to decode. The check is what turns that class of bug into a loud failure.

Why the safety filter exists

The encoder chooses token ids, but the decoder is handed text. Nothing guarantees that writing tokens out and reading them back returns the same ids: a byte-level BPE vocabulary will merge an emitted token into its neighbours, so 'an' written after 'Kanz' reads back as 'anz' and every subsequent step decodes against the wrong prefix.

So a candidate is only usable if appending it leaves the surrounding text re-tokenizing unchanged. Both sides apply that same filter to the same prefix, so they stay in step, and the encoder verifies the finished text tokenizes back to exactly what it built before handing it over. A message that cannot satisfy this raises at encode time rather than becoming an artifact that silently fails to decode.

Features

Entropy-optimal capacity -- each token carries -log2(p) bits, the theoretical maximum for output that still looks like sampling
Output distributed exactly as the model's own sampling, so no per-token statistical tell
Temperature as a principled capacity dial, replacing rank truncation
Tokenizer-stable candidate selection, verified before any message is returned
Self-inverse check at encode time -- the coder re-encodes as it goes and refuses to emit a message it cannot read back
Framed payload with a CRC32, so a wrong key, prompt or model is detected rather than yielding plausible wrong bytes
Optional zlib compression (--compress), which shortens the cover text and makes the payload bits more uniform
Automatic rolling context window, so a payload can exceed the model's context window with no repeated prompt and no tuning
Incremental KV-cached stepping, so cost is linear in message length rather than quadratic
Deterministic throughout; verified bit-identical on CPU and CUDA across fp32/bf16/fp16
Hugging Face Transformers backend -- works with most causal LMs

Installation

From PyPI (or via uv):

uv pip install subtext-codec
# or: pip install subtext-codec

From source:

git clone https://github.com/shevisj/subtext-codec
cd subtext-codec
uv venv --python 3.13
uv pip install -r requirements.txt
uv pip install -e .

torch and transformers are the only runtime dependencies. The published package sets lower bounds only, so it installs alongside whatever torch build you already have; requirements.txt pins an exact, CPU-only development environment and is the one to use when reproducing a decode.

Usage

Run subtext-codec --version to print the installed version. The CLI otherwise exposes encode and decode. Shared flags:

--key -- path to the codec key file (required)
--model-name-or-path -- HuggingFace model id or local path; optional if stored in the key
--prompt-prefix -- prefix used for both encode and decode; taken from the key if present
--device -- e.g. cpu or cuda (falls back to the key, then cpu)
--torch-dtype -- weight dtype (auto, fp16, bf16, fp32)
--max-context-length -- cap on sequence length; defaults to the model's own limit
--seed -- deterministic seed (default: 0)
--quiet -- suppress the progress line on stderr

Encode bytes into text

subtext-codec encode \
--model-name-or-path gpt2 \
--prompt-prefix "Once upon a time, " \
--input-bytes secret.txt \
--output-text message.txt \
--key key.json \
--temperature 1.5

Encode-only flags:

--temperature -- the capacity dial (default: 1.5). Higher packs more payload per token and shortens the cover text, at the cost of more erratic prose. See Choosing a temperature.
--top-k -- candidates considered per step (default: 64). This bounds the stability filter's work; it is not a capacity dial. Must be at least 2.
--compress -- zlib-compress the payload before encoding. Shortens the cover text and makes the payload bits more uniform. Recorded in the key, so decoding reverses it automatically. See Compression.
--max-new-tokens -- fail rather than generate more than this many tokens
--no-store-model -- keep the model id out of the key
--verify -- decode the result before writing it, confirming the round trip end to end

Payloads too long for one context window are handled automatically; see Large payloads.

Unless --quiet, encoding prints a one-line summary to stderr -- payload size, cover tokens, bits per token and mean surprisal (and how many times the context rolled) -- so you can see the coder is running at the distribution's entropy.

The output text is just the generated story, with no metadata header. key.json records temperature, top_k, the prompt prefix, the device, the dtype, (unless --no-store-model) the model id, and -- when compression or the rolling window is used -- a compression or window field and a "version": "v2" marker.

If the path given to --key already exists, its values are reused as defaults and any CLI overrides are written back, so a second message needs only the key:

subtext-codec encode --key key.json --input-bytes secret.bin --output-text message.txt

Decode text back into bytes

subtext-codec decode \
--input-text message.txt \
--key key.json \
--output-bytes decoded.bin

Parameters come from the key unless overridden for the run. Decoding never modifies the key file -- the one artifact a message cannot be recovered without is not rewritten by a read operation.

Decoding stops as soon as the payload's declared length has been recovered, so trailing text is harmless. Text before the prompt prefix is skipped too. If the message was altered, or the key, prompt or model do not match, the CRC32 catches it rather than returning plausible wrong bytes.

Sample artifacts

samples/secret.txt is a payload to play with. Build the rest of the fixture from it:

python samples/regenerate.py

That encodes it with gpt2 on CPU -- a ~500MB download, no GPU or gated checkpoint needed -- and writes message.txt, key.json and decoded.txt alongside it, verifying the round trip before writing anything. Encoding is deterministic, so it reproduces the same message every time.

Then decode it back the normal way:

subtext-codec decode \
--input-text samples/message.txt \
--key samples/key.json \
--output-bytes /tmp/decoded.txt
diff samples/secret.txt /tmp/decoded.txt &&echo"exact match"

(Before 1.0 the fixture was checked in, but it was a v2 message against a gated 16GB Llama checkpoint -- unreadable by this version and unrunnable by most people. Generating it locally is both smaller and honest.)

Programmatic usage

See the demo notebook.

Choosing a temperature

temperature is the capacity dial. Raising it flattens the distribution, so each token carries more payload and the cover text gets shorter -- but the text becomes more erratic, exactly as sampling at that temperature would. top_k is not a capacity dial here; it only bounds how much work the stability filter does.

Measured on the 445-byte samples/secret.txt, Qwen2.5-7B (bf16) on an RTX 5090, top_k=64:

Temperature	Cover tokens	Bits/token	Character
1.0	2914	1.22	best prose, but far too long to be plausible
1.5	894	3.98	the knee -- reads like a real, rambling review
2.0	709	5.02	densest, but visibly erratic and prone to invention

Mean surprisal of the emitted tokens matches capacity at every setting (1.23 / 4.02 / 5.07 bits), which is the confirmation that the coder is running at the distribution's entropy rather than leaving capacity on the table.

For reference, the rank-coding scheme this replaced in 1.0 managed 4.35 bits/token at its best setting and 1.57 at its most conservative, and its output was never model-distributed. Arithmetic coding at temperature 1.5 matches its best capacity, and at 2.0 exceeds it, while staying on-distribution throughout.

A bigger model does not buy more capacity. A stronger model concentrates probability mass, which lowers entropy and therefore bits per token. It buys better prose at a given temperature, not shorter text. Keep payloads to a few hundred bytes, and pick a prompt whose natural continuation is long-form so the required length is not conspicuous.

Compression

--compress zlib-compresses the payload before encoding. It does two useful things:

Shortens the cover text. Fewer payload bytes means fewer tokens to carry them.
Makes the bits more uniform, which is exactly what the indistinguishability argument assumes. Raw ASCII is biased; compressed (or encrypted) data is not.

subtext-codec encode --key key.json --input-bytes secret.txt --output-text message.txt --compress

Compression is recorded in the key, so decode reverses it automatically -- you do not pass --compress when decoding. A compressed message uses the v2 wire format (see Compatibility). The whole payload is framed with a length and CRC32 before compression, so a key that misdescribes compression is caught rather than returning the raw compressed bytes.

Compression rarely helps on data that is already high-entropy (encrypted blobs, media). It helps most on text and structured data.

Large payloads

A payload can outgrow the model's context window, and that is handled automatically -- there is nothing to configure. When the context fills, the codec re-prefixes the most recent half of the tokens and keeps generating, a rolling window that slides forward as the message grows. The text stays one continuous passage: the prompt appears once, at the start, and never repeats.

# no special flags: a long payload just works
subtext-codec encode --key key.json --input-bytes big.bin --output-text message.txt

The decoder resets at the same points, so it recovers the message exactly. The window size it used rides in the key (a window field), so decoding needs nothing extra. A message that rolls uses the v2 wire format; one that fits a single window stays v1.

Two things worth knowing:

The window follows the model's context limit (or --max-context-length, if you set a smaller one). Because the size is recorded in the key, encode and decode agree automatically -- but if you decode with a different explicit --max-context-length, the key's value still wins.
Coherence is over the window, not the whole text. Past the first reset the model conditions on a sliding window of recent tokens rather than the entire history, so very long messages read like natural long-form drift rather than a single tightly-planned document. See the caveat in Limitations.

Encryption

The codec is not a cipher -- the key is metadata, not a secret, and anyone with it and the model can read the message. For confidentiality, encrypt the payload yourself before encoding. Encryption also gives you the uniform bits the indistinguishability argument wants, so you do not additionally need --compress (encrypted data will not compress anyway).

# encrypt, then hide; the codec never sees the plaintext
age -r age1... secret.txt | subtext-codec encode --key key.json --input-bytes - --output-text message.txt
# recover, then decrypt
subtext-codec decode --key key.json --input-text message.txt --output-bytes - | age -d -i key.txt > secret.txt

Any tool works -- age, gpg, openssl enc. Keeping encryption out of the codec is deliberate: this is a research prototype, and rolling in a cipher would imply a security property it does not provide.

Determinism and devices

The codec reduces to one requirement: encode and decode must derive the same frequency table at every step, down to the last integer. Everything after the forward pass runs on CPU no matter where the model lives: the logits are cast to float32 for the stable sort and the candidate filter, then to float64 for the softmax and quantization. So the only device-sensitive component is the logits themselves. Encode and decode also drive the model identically (prefill the prompt, then one token at a time), so they hit the same kernels at the same shapes.

This has been measured, not just reasoned about. On an RTX 5090 (Blackwell, sm_120, torch 2.13 + CUDA 13), repeated passes over the same tokens produce bit-identical logits in fp32, bf16 and fp16, and all three round-trip on GPU. GPU is as reliable as CPU.

What matters is that the environment matches between encode and decode:

Factor	Effect
Different dtype (bf16 ↔ fp32)	Breaks. The key records the dtype; keep it.
TF32 on vs off	Pinned off by `set_deterministic`, so it no longer varies.
Different attention backend (SDPA / FlashAttention / eager)	Can break across machines. Same environment, same choice.
Different torch / transformers version	Can break; pin them (`requirements.txt`) if a message must survive an upgrade.
Different device (`cuda` ↔ `cpu`)	Unreliable, but not automatically fatal -- see below.

Crossing devices is not supported, though it is not guaranteed to fail: gpt2 in fp32 does decode on CPU from a CUDA encode, because the two happen to agree bit for bit. Larger models and bf16 will not be so lucky. Do not rely on it. What is guaranteed is that a mismatch is caught rather than returning plausible wrong bytes -- that is the payload framing's job.

set_deterministic also sets CUBLAS_WORKSPACE_CONFIG=:4096:8 (via package import, before torch loads) and disables cuDNN autotuning. It holds fp32 matmuls at full precision, which costs some speed on Ampere and later; a deliberate trade, since a mismatch costs the entire message.

bf16 and fp16 work but leave less margin -- half the mantissa means adjacent logits sit closer together. fp32 is safer when you can afford it. Exact ties are fine in any precision: the sort is stable, so ties resolve by token id.

To confirm on your own hardware before trusting a message to it:

uv run pytest tests/test_determinism.py -v

Those parametrize over every device and dtype present, so on a GPU box they additionally check bit-identical logits on CUDA, matching bands, and fp32/bf16/fp16 round trips.

Compatibility

There are two wire formats:

v1 -- a single segment framing length || crc32 || data, exactly as 1.0 wrote it. A plain message that fits one context window (no compression, no rolling) is still written as v1, so it stays byte-for-byte readable by 1.0.
v2 -- added in 1.1 for compression and the rolling window. The key records "version": "v2" and, as needed, a compression field and a window field. Compressed payloads frame a length and CRC over the original data so a wrong compression flag is caught; rolled payloads carry the window so the decoder resets at the same points. A 1.0 reader rejects a v2 key outright rather than mis-reading it, which is why the new features carry a new version.

1.1 reads both. It writes v2 only when you compress or when the payload is long enough to roll the context; everything else stays v1.

Every pre-1.0 format is gone along with the code that decoded them: they used rank coding, which corrupted roughly a third of its own messages and never produced model-distributed output. Pre-1.0 also numbered a format v1, but that one carried base/top_p and no temperature, so the collision is detectable -- such a key fails with an explanation rather than a generic error. To read a message encoded before 1.0, install subtext-codec~=0.2.0; to keep it, re-encode the payload with this version.

Limitations

Brittle to edits: changing a single token of the output breaks decoding. This is an encoding scheme, not an error-correcting one.
Model-dependent: requires the exact same weights, tokenizer and dtype. Decoding a bf16 encode in fp32 will not work.
Not confidential on its own: the key is metadata, not a cipher. Encrypt the payload before encoding it -- both for secrecy and because the indistinguishability argument assumes uniform payload bits.
Context length: handled automatically by the rolling window, but past the first reset the model conditions on recent tokens rather than the whole history. Coherence is local to the window, and a discriminator with the full text could in principle notice the long-range structure "resetting" -- subtle for a reasonable window, but a real difference from full-context sampling, on top of the temperature caveat below.
Capacity: the distribution's entropy, typically 1-5 bits per token depending on temperature. Expect a few hundred tokens per hundred bytes.
Length is the remaining tell: the per-token statistics are right, but a 900-token hotel review is still an odd thing to find. Pick a prompt whose natural continuation is long-form.
"Indistinguishable" is relative to the temperature you chose: output at temperature=1.5 is exactly temperature-1.5 sampling, which is only unremarkable to someone who has no expectation about the setting. An adversary who knows you would have sampled at 1.0 can distinguish it in aggregate. Lower temperature closes that gap and lengthens the text.

This project is a research prototype, not a secure or production steganography system.

Testing

uv run pytest # everything
uv run pytest -m "not slow"# skip tests that download a checkpoint

The slow tests use sshleifer/tiny-gpt2 and skip themselves if the Hub is unreachable.

License

MIT

subtext-codec

Steganographic data encoding in natural language via arithmetic coding against an LLM

subtext-codec hides arbitrary binary data inside seemingly normal LLM-generated text. The payload drives an arithmetic coder whose probability model is the language model's own next-token distribution, so the text it produces is distributed exactly as ordinary sampling would be while secretly encoding bytes. With the same model, tokenizer, prompt prefix and parameters, the process is fully reversible.

How it works

The payload drives an arithmetic decoder whose probability model is the language model itself. Emitting a token the model assigns probability p consumes -log2(p) payload bits, so a confident step carries almost nothing and an uncertain one carries a lot.

Two properties follow, and they are the entire point of the design:

The generated text is distributed exactly as ordinary sampling at the chosen temperature. Nothing is ever forced off-distribution.
Capacity per token equals the distribution's entropy, which is the information-theoretic ceiling for text that stays indistinguishable from sampling.

Encoding

The payload is framed with a length and a CRC32, then read as a bitstream.
At each step the model's logits are clipped to top_k, filtered to tokens that are safe to emit (see below), and turned into an integer frequency table at temperature.
The arithmetic decoder consumes payload bits and selects a token from that table.
Generation stops as soon as the emitted tokens pin down every payload bit.

Decoding

Start from the same prompt prefix and model, and rebuild the same table at each step.
Run the arithmetic encoder over the observed tokens, which reproduces the bitstream.
Stop once the declared length has been recovered, then check the CRC.

Decoding needs the text, the prompt prefix, the same model and tokenizer, and the codec parameters -- all of which the key file carries. No terminator token is required, so text after the message is simply ignored.

Why the encoder checks itself

The coder re-encodes each symbol as it emits it and refuses to return a message whose bits it cannot reproduce. That is not belt-and-braces: feeding the arithmetic decoder a zero-filled tail past the end of the payload parks its value register exactly on the interval midpoint, where it underflows forever and the payload's last two bits are never emitted. The result is a message that looks fine and silently fails to decode. The check is what turns that class of bug into a loud failure.

Why the safety filter exists

The encoder chooses token ids, but the decoder is handed text. Nothing guarantees that writing tokens out and reading them back returns the same ids: a byte-level BPE vocabulary will merge an emitted token into its neighbours, so 'an' written after 'Kanz' reads back as 'anz' and every subsequent step decodes against the wrong prefix.

So a candidate is only usable if appending it leaves the surrounding text re-tokenizing unchanged. Both sides apply that same filter to the same prefix, so they stay in step, and the encoder verifies the finished text tokenizes back to exactly what it built before handing it over. A message that cannot satisfy this raises at encode time rather than becoming an artifact that silently fails to decode.

Features

Entropy-optimal capacity -- each token carries -log2(p) bits, the theoretical maximum for output that still looks like sampling
Output distributed exactly as the model's own sampling, so no per-token statistical tell
Temperature as a principled capacity dial, replacing rank truncation
Tokenizer-stable candidate selection, verified before any message is returned
Self-inverse check at encode time -- the coder re-encodes as it goes and refuses to emit a message it cannot read back
Framed payload with a CRC32, so a wrong key, prompt or model is detected rather than yielding plausible wrong bytes
Optional zlib compression (--compress), which shortens the cover text and makes the payload bits more uniform
Automatic rolling context window, so a payload can exceed the model's context window with no repeated prompt and no tuning
Incremental KV-cached stepping, so cost is linear in message length rather than quadratic
Deterministic throughout; verified bit-identical on CPU and CUDA across fp32/bf16/fp16
Hugging Face Transformers backend -- works with most causal LMs

Installation

From PyPI (or via uv):

uv pip install subtext-codec
# or: pip install subtext-codec

From source:

git clone https://github.com/shevisj/subtext-codec
cd subtext-codec
uv venv --python 3.13
uv pip install -r requirements.txt
uv pip install -e .

torch and transformers are the only runtime dependencies. The published package sets lower bounds only, so it installs alongside whatever torch build you already have; requirements.txt pins an exact, CPU-only development environment and is the one to use when reproducing a decode.

Usage

Run subtext-codec --version to print the installed version. The CLI otherwise exposes encode and decode. Shared flags:

--key -- path to the codec key file (required)
--model-name-or-path -- HuggingFace model id or local path; optional if stored in the key
--prompt-prefix -- prefix used for both encode and decode; taken from the key if present
--device -- e.g. cpu or cuda (falls back to the key, then cpu)
--torch-dtype -- weight dtype (auto, fp16, bf16, fp32)
--max-context-length -- cap on sequence length; defaults to the model's own limit
--seed -- deterministic seed (default: 0)
--quiet -- suppress the progress line on stderr

Encode bytes into text

subtext-codec encode \
--model-name-or-path gpt2 \
--prompt-prefix "Once upon a time, " \
--input-bytes secret.txt \
--output-text message.txt \
--key key.json \
--temperature 1.5

Encode-only flags:

--temperature -- the capacity dial (default: 1.5). Higher packs more payload per token and shortens the cover text, at the cost of more erratic prose. See Choosing a temperature.
--top-k -- candidates considered per step (default: 64). This bounds the stability filter's work; it is not a capacity dial. Must be at least 2.
--compress -- zlib-compress the payload before encoding. Shortens the cover text and makes the payload bits more uniform. Recorded in the key, so decoding reverses it automatically. See Compression.
--max-new-tokens -- fail rather than generate more than this many tokens
--no-store-model -- keep the model id out of the key
--verify -- decode the result before writing it, confirming the round trip end to end

Payloads too long for one context window are handled automatically; see Large payloads.

Unless --quiet, encoding prints a one-line summary to stderr -- payload size, cover tokens, bits per token and mean surprisal (and how many times the context rolled) -- so you can see the coder is running at the distribution's entropy.

The output text is just the generated story, with no metadata header. key.json records temperature, top_k, the prompt prefix, the device, the dtype, (unless --no-store-model) the model id, and -- when compression or the rolling window is used -- a compression or window field and a "version": "v2" marker.

If the path given to --key already exists, its values are reused as defaults and any CLI overrides are written back, so a second message needs only the key:

subtext-codec encode --key key.json --input-bytes secret.bin --output-text message.txt

Decode text back into bytes

subtext-codec decode \
--input-text message.txt \
--key key.json \
--output-bytes decoded.bin

Parameters come from the key unless overridden for the run. Decoding never modifies the key file -- the one artifact a message cannot be recovered without is not rewritten by a read operation.

Decoding stops as soon as the payload's declared length has been recovered, so trailing text is harmless. Text before the prompt prefix is skipped too. If the message was altered, or the key, prompt or model do not match, the CRC32 catches it rather than returning plausible wrong bytes.

Sample artifacts

samples/secret.txt is a payload to play with. Build the rest of the fixture from it:

python samples/regenerate.py

That encodes it with gpt2 on CPU -- a ~500MB download, no GPU or gated checkpoint needed -- and writes message.txt, key.json and decoded.txt alongside it, verifying the round trip before writing anything. Encoding is deterministic, so it reproduces the same message every time.

Then decode it back the normal way:

subtext-codec decode \
--input-text samples/message.txt \
--key samples/key.json \
--output-bytes /tmp/decoded.txt
diff samples/secret.txt /tmp/decoded.txt &&echo"exact match"

(Before 1.0 the fixture was checked in, but it was a v2 message against a gated 16GB Llama checkpoint -- unreadable by this version and unrunnable by most people. Generating it locally is both smaller and honest.)

Programmatic usage

See the demo notebook.

Choosing a temperature

temperature is the capacity dial. Raising it flattens the distribution, so each token carries more payload and the cover text gets shorter -- but the text becomes more erratic, exactly as sampling at that temperature would. top_k is not a capacity dial here; it only bounds how much work the stability filter does.

Measured on the 445-byte samples/secret.txt, Qwen2.5-7B (bf16) on an RTX 5090, top_k=64:

Temperature	Cover tokens	Bits/token	Character
1.0	2914	1.22	best prose, but far too long to be plausible
1.5	894	3.98	the knee -- reads like a real, rambling review
2.0	709	5.02	densest, but visibly erratic and prone to invention

Mean surprisal of the emitted tokens matches capacity at every setting (1.23 / 4.02 / 5.07 bits), which is the confirmation that the coder is running at the distribution's entropy rather than leaving capacity on the table.

For reference, the rank-coding scheme this replaced in 1.0 managed 4.35 bits/token at its best setting and 1.57 at its most conservative, and its output was never model-distributed. Arithmetic coding at temperature 1.5 matches its best capacity, and at 2.0 exceeds it, while staying on-distribution throughout.

A bigger model does not buy more capacity. A stronger model concentrates probability mass, which lowers entropy and therefore bits per token. It buys better prose at a given temperature, not shorter text. Keep payloads to a few hundred bytes, and pick a prompt whose natural continuation is long-form so the required length is not conspicuous.

Compression

--compress zlib-compresses the payload before encoding. It does two useful things:

Shortens the cover text. Fewer payload bytes means fewer tokens to carry them.
Makes the bits more uniform, which is exactly what the indistinguishability argument assumes. Raw ASCII is biased; compressed (or encrypted) data is not.

subtext-codec encode --key key.json --input-bytes secret.txt --output-text message.txt --compress

Compression is recorded in the key, so decode reverses it automatically -- you do not pass --compress when decoding. A compressed message uses the v2 wire format (see Compatibility). The whole payload is framed with a length and CRC32 before compression, so a key that misdescribes compression is caught rather than returning the raw compressed bytes.

Compression rarely helps on data that is already high-entropy (encrypted blobs, media). It helps most on text and structured data.

Large payloads

A payload can outgrow the model's context window, and that is handled automatically -- there is nothing to configure. When the context fills, the codec re-prefixes the most recent half of the tokens and keeps generating, a rolling window that slides forward as the message grows. The text stays one continuous passage: the prompt appears once, at the start, and never repeats.

# no special flags: a long payload just works
subtext-codec encode --key key.json --input-bytes big.bin --output-text message.txt

The decoder resets at the same points, so it recovers the message exactly. The window size it used rides in the key (a window field), so decoding needs nothing extra. A message that rolls uses the v2 wire format; one that fits a single window stays v1.

Two things worth knowing:

The window follows the model's context limit (or --max-context-length, if you set a smaller one). Because the size is recorded in the key, encode and decode agree automatically -- but if you decode with a different explicit --max-context-length, the key's value still wins.
Coherence is over the window, not the whole text. Past the first reset the model conditions on a sliding window of recent tokens rather than the entire history, so very long messages read like natural long-form drift rather than a single tightly-planned document. See the caveat in Limitations.

Encryption

The codec is not a cipher -- the key is metadata, not a secret, and anyone with it and the model can read the message. For confidentiality, encrypt the payload yourself before encoding. Encryption also gives you the uniform bits the indistinguishability argument wants, so you do not additionally need --compress (encrypted data will not compress anyway).

# encrypt, then hide; the codec never sees the plaintext
age -r age1... secret.txt | subtext-codec encode --key key.json --input-bytes - --output-text message.txt
# recover, then decrypt
subtext-codec decode --key key.json --input-text message.txt --output-bytes - | age -d -i key.txt > secret.txt

Any tool works -- age, gpg, openssl enc. Keeping encryption out of the codec is deliberate: this is a research prototype, and rolling in a cipher would imply a security property it does not provide.

Determinism and devices

The codec reduces to one requirement: encode and decode must derive the same frequency table at every step, down to the last integer. Everything after the forward pass runs on CPU no matter where the model lives: the logits are cast to float32 for the stable sort and the candidate filter, then to float64 for the softmax and quantization. So the only device-sensitive component is the logits themselves. Encode and decode also drive the model identically (prefill the prompt, then one token at a time), so they hit the same kernels at the same shapes.

This has been measured, not just reasoned about. On an RTX 5090 (Blackwell, sm_120, torch 2.13 + CUDA 13), repeated passes over the same tokens produce bit-identical logits in fp32, bf16 and fp16, and all three round-trip on GPU. GPU is as reliable as CPU.

What matters is that the environment matches between encode and decode:

Factor	Effect
Different dtype (bf16 ↔ fp32)	Breaks. The key records the dtype; keep it.
TF32 on vs off	Pinned off by `set_deterministic`, so it no longer varies.
Different attention backend (SDPA / FlashAttention / eager)	Can break across machines. Same environment, same choice.
Different torch / transformers version	Can break; pin them (`requirements.txt`) if a message must survive an upgrade.
Different device (`cuda` ↔ `cpu`)	Unreliable, but not automatically fatal -- see below.

Crossing devices is not supported, though it is not guaranteed to fail: gpt2 in fp32 does decode on CPU from a CUDA encode, because the two happen to agree bit for bit. Larger models and bf16 will not be so lucky. Do not rely on it. What is guaranteed is that a mismatch is caught rather than returning plausible wrong bytes -- that is the payload framing's job.

set_deterministic also sets CUBLAS_WORKSPACE_CONFIG=:4096:8 (via package import, before torch loads) and disables cuDNN autotuning. It holds fp32 matmuls at full precision, which costs some speed on Ampere and later; a deliberate trade, since a mismatch costs the entire message.

bf16 and fp16 work but leave less margin -- half the mantissa means adjacent logits sit closer together. fp32 is safer when you can afford it. Exact ties are fine in any precision: the sort is stable, so ties resolve by token id.

To confirm on your own hardware before trusting a message to it:

uv run pytest tests/test_determinism.py -v

Those parametrize over every device and dtype present, so on a GPU box they additionally check bit-identical logits on CUDA, matching bands, and fp32/bf16/fp16 round trips.

Compatibility

There are two wire formats:

v1 -- a single segment framing length || crc32 || data, exactly as 1.0 wrote it. A plain message that fits one context window (no compression, no rolling) is still written as v1, so it stays byte-for-byte readable by 1.0.
v2 -- added in 1.1 for compression and the rolling window. The key records "version": "v2" and, as needed, a compression field and a window field. Compressed payloads frame a length and CRC over the original data so a wrong compression flag is caught; rolled payloads carry the window so the decoder resets at the same points. A 1.0 reader rejects a v2 key outright rather than mis-reading it, which is why the new features carry a new version.

1.1 reads both. It writes v2 only when you compress or when the payload is long enough to roll the context; everything else stays v1.

Every pre-1.0 format is gone along with the code that decoded them: they used rank coding, which corrupted roughly a third of its own messages and never produced model-distributed output. Pre-1.0 also numbered a format v1, but that one carried base/top_p and no temperature, so the collision is detectable -- such a key fails with an explanation rather than a generic error. To read a message encoded before 1.0, install subtext-codec~=0.2.0; to keep it, re-encode the payload with this version.

Limitations

Brittle to edits: changing a single token of the output breaks decoding. This is an encoding scheme, not an error-correcting one.
Model-dependent: requires the exact same weights, tokenizer and dtype. Decoding a bf16 encode in fp32 will not work.
Not confidential on its own: the key is metadata, not a cipher. Encrypt the payload before encoding it -- both for secrecy and because the indistinguishability argument assumes uniform payload bits.
Context length: handled automatically by the rolling window, but past the first reset the model conditions on recent tokens rather than the whole history. Coherence is local to the window, and a discriminator with the full text could in principle notice the long-range structure "resetting" -- subtle for a reasonable window, but a real difference from full-context sampling, on top of the temperature caveat below.
Capacity: the distribution's entropy, typically 1-5 bits per token depending on temperature. Expect a few hundred tokens per hundred bytes.
Length is the remaining tell: the per-token statistics are right, but a 900-token hotel review is still an odd thing to find. Pick a prompt whose natural continuation is long-form.
"Indistinguishable" is relative to the temperature you chose: output at temperature=1.5 is exactly temperature-1.5 sampling, which is only unremarkable to someone who has no expectation about the setting. An adversary who knows you would have sampled at 1.0 can distinguish it in aggregate. Lower temperature closes that gap and lengthens the text.

This project is a research prototype, not a secure or production steganography system.

Testing

uv run pytest # everything
uv run pytest -m "not slow"# skip tests that download a checkpoint

The slow tests use sshleifer/tiny-gpt2 and skip themselves if the Hub is unreachable.

License

MIT

subtext-codec

Steganographic data encoding in natural language via arithmetic coding against an LLM

subtext-codec hides arbitrary binary data inside seemingly normal LLM-generated text. The payload drives an arithmetic coder whose probability model is the language model's own next-token distribution, so the text it produces is distributed exactly as ordinary sampling would be while secretly encoding bytes. With the same model, tokenizer, prompt prefix and parameters, the process is fully reversible.

How it works

The payload drives an arithmetic decoder whose probability model is the language model itself. Emitting a token the model assigns probability p consumes -log2(p) payload bits, so a confident step carries almost nothing and an uncertain one carries a lot.

Two properties follow, and they are the entire point of the design:

The generated text is distributed exactly as ordinary sampling at the chosen temperature. Nothing is ever forced off-distribution.
Capacity per token equals the distribution's entropy, which is the information-theoretic ceiling for text that stays indistinguishable from sampling.

Encoding

The payload is framed with a length and a CRC32, then read as a bitstream.
At each step the model's logits are clipped to top_k, filtered to tokens that are safe to emit (see below), and turned into an integer frequency table at temperature.
The arithmetic decoder consumes payload bits and selects a token from that table.
Generation stops as soon as the emitted tokens pin down every payload bit.

Decoding

Start from the same prompt prefix and model, and rebuild the same table at each step.
Run the arithmetic encoder over the observed tokens, which reproduces the bitstream.
Stop once the declared length has been recovered, then check the CRC.

Decoding needs the text, the prompt prefix, the same model and tokenizer, and the codec parameters -- all of which the key file carries. No terminator token is required, so text after the message is simply ignored.

Why the encoder checks itself

The coder re-encodes each symbol as it emits it and refuses to return a message whose bits it cannot reproduce. That is not belt-and-braces: feeding the arithmetic decoder a zero-filled tail past the end of the payload parks its value register exactly on the interval midpoint, where it underflows forever and the payload's last two bits are never emitted. The result is a message that looks fine and silently fails to decode. The check is what turns that class of bug into a loud failure.

Why the safety filter exists

The encoder chooses token ids, but the decoder is handed text. Nothing guarantees that writing tokens out and reading them back returns the same ids: a byte-level BPE vocabulary will merge an emitted token into its neighbours, so 'an' written after 'Kanz' reads back as 'anz' and every subsequent step decodes against the wrong prefix.

So a candidate is only usable if appending it leaves the surrounding text re-tokenizing unchanged. Both sides apply that same filter to the same prefix, so they stay in step, and the encoder verifies the finished text tokenizes back to exactly what it built before handing it over. A message that cannot satisfy this raises at encode time rather than becoming an artifact that silently fails to decode.

Features

Entropy-optimal capacity -- each token carries -log2(p) bits, the theoretical maximum for output that still looks like sampling
Output distributed exactly as the model's own sampling, so no per-token statistical tell
Temperature as a principled capacity dial, replacing rank truncation
Tokenizer-stable candidate selection, verified before any message is returned
Self-inverse check at encode time -- the coder re-encodes as it goes and refuses to emit a message it cannot read back
Framed payload with a CRC32, so a wrong key, prompt or model is detected rather than yielding plausible wrong bytes
Optional zlib compression (--compress), which shortens the cover text and makes the payload bits more uniform
Automatic rolling context window, so a payload can exceed the model's context window with no repeated prompt and no tuning
Incremental KV-cached stepping, so cost is linear in message length rather than quadratic
Deterministic throughout; verified bit-identical on CPU and CUDA across fp32/bf16/fp16
Hugging Face Transformers backend -- works with most causal LMs

Installation

From PyPI (or via uv):

uv pip install subtext-codec
# or: pip install subtext-codec

From source:

git clone https://github.com/shevisj/subtext-codec
cd subtext-codec
uv venv --python 3.13
uv pip install -r requirements.txt
uv pip install -e .

torch and transformers are the only runtime dependencies. The published package sets lower bounds only, so it installs alongside whatever torch build you already have; requirements.txt pins an exact, CPU-only development environment and is the one to use when reproducing a decode.

Usage

Run subtext-codec --version to print the installed version. The CLI otherwise exposes encode and decode. Shared flags:

--key -- path to the codec key file (required)
--model-name-or-path -- HuggingFace model id or local path; optional if stored in the key
--prompt-prefix -- prefix used for both encode and decode; taken from the key if present
--device -- e.g. cpu or cuda (falls back to the key, then cpu)
--torch-dtype -- weight dtype (auto, fp16, bf16, fp32)
--max-context-length -- cap on sequence length; defaults to the model's own limit
--seed -- deterministic seed (default: 0)
--quiet -- suppress the progress line on stderr

Encode bytes into text

subtext-codec encode \
--model-name-or-path gpt2 \
--prompt-prefix "Once upon a time, " \
--input-bytes secret.txt \
--output-text message.txt \
--key key.json \
--temperature 1.5

Encode-only flags:

--temperature -- the capacity dial (default: 1.5). Higher packs more payload per token and shortens the cover text, at the cost of more erratic prose. See Choosing a temperature.
--top-k -- candidates considered per step (default: 64). This bounds the stability filter's work; it is not a capacity dial. Must be at least 2.
--compress -- zlib-compress the payload before encoding. Shortens the cover text and makes the payload bits more uniform. Recorded in the key, so decoding reverses it automatically. See Compression.
--max-new-tokens -- fail rather than generate more than this many tokens
--no-store-model -- keep the model id out of the key
--verify -- decode the result before writing it, confirming the round trip end to end

Payloads too long for one context window are handled automatically; see Large payloads.

Unless --quiet, encoding prints a one-line summary to stderr -- payload size, cover tokens, bits per token and mean surprisal (and how many times the context rolled) -- so you can see the coder is running at the distribution's entropy.

The output text is just the generated story, with no metadata header. key.json records temperature, top_k, the prompt prefix, the device, the dtype, (unless --no-store-model) the model id, and -- when compression or the rolling window is used -- a compression or window field and a "version": "v2" marker.

If the path given to --key already exists, its values are reused as defaults and any CLI overrides are written back, so a second message needs only the key:

subtext-codec encode --key key.json --input-bytes secret.bin --output-text message.txt

Decode text back into bytes

subtext-codec decode \
--input-text message.txt \
--key key.json \
--output-bytes decoded.bin

Parameters come from the key unless overridden for the run. Decoding never modifies the key file -- the one artifact a message cannot be recovered without is not rewritten by a read operation.

Decoding stops as soon as the payload's declared length has been recovered, so trailing text is harmless. Text before the prompt prefix is skipped too. If the message was altered, or the key, prompt or model do not match, the CRC32 catches it rather than returning plausible wrong bytes.

Sample artifacts

samples/secret.txt is a payload to play with. Build the rest of the fixture from it:

python samples/regenerate.py

That encodes it with gpt2 on CPU -- a ~500MB download, no GPU or gated checkpoint needed -- and writes message.txt, key.json and decoded.txt alongside it, verifying the round trip before writing anything. Encoding is deterministic, so it reproduces the same message every time.

Then decode it back the normal way:

subtext-codec decode \
--input-text samples/message.txt \
--key samples/key.json \
--output-bytes /tmp/decoded.txt
diff samples/secret.txt /tmp/decoded.txt &&echo"exact match"

(Before 1.0 the fixture was checked in, but it was a v2 message against a gated 16GB Llama checkpoint -- unreadable by this version and unrunnable by most people. Generating it locally is both smaller and honest.)

Programmatic usage

See the demo notebook.

Choosing a temperature

temperature is the capacity dial. Raising it flattens the distribution, so each token carries more payload and the cover text gets shorter -- but the text becomes more erratic, exactly as sampling at that temperature would. top_k is not a capacity dial here; it only bounds how much work the stability filter does.

Measured on the 445-byte samples/secret.txt, Qwen2.5-7B (bf16) on an RTX 5090, top_k=64:

Temperature	Cover tokens	Bits/token	Character
1.0	2914	1.22	best prose, but far too long to be plausible
1.5	894	3.98	the knee -- reads like a real, rambling review
2.0	709	5.02	densest, but visibly erratic and prone to invention

Mean surprisal of the emitted tokens matches capacity at every setting (1.23 / 4.02 / 5.07 bits), which is the confirmation that the coder is running at the distribution's entropy rather than leaving capacity on the table.

For reference, the rank-coding scheme this replaced in 1.0 managed 4.35 bits/token at its best setting and 1.57 at its most conservative, and its output was never model-distributed. Arithmetic coding at temperature 1.5 matches its best capacity, and at 2.0 exceeds it, while staying on-distribution throughout.

A bigger model does not buy more capacity. A stronger model concentrates probability mass, which lowers entropy and therefore bits per token. It buys better prose at a given temperature, not shorter text. Keep payloads to a few hundred bytes, and pick a prompt whose natural continuation is long-form so the required length is not conspicuous.

Compression

--compress zlib-compresses the payload before encoding. It does two useful things:

Shortens the cover text. Fewer payload bytes means fewer tokens to carry them.
Makes the bits more uniform, which is exactly what the indistinguishability argument assumes. Raw ASCII is biased; compressed (or encrypted) data is not.

subtext-codec encode --key key.json --input-bytes secret.txt --output-text message.txt --compress

Compression is recorded in the key, so decode reverses it automatically -- you do not pass --compress when decoding. A compressed message uses the v2 wire format (see Compatibility). The whole payload is framed with a length and CRC32 before compression, so a key that misdescribes compression is caught rather than returning the raw compressed bytes.

Compression rarely helps on data that is already high-entropy (encrypted blobs, media). It helps most on text and structured data.

Large payloads

A payload can outgrow the model's context window, and that is handled automatically -- there is nothing to configure. When the context fills, the codec re-prefixes the most recent half of the tokens and keeps generating, a rolling window that slides forward as the message grows. The text stays one continuous passage: the prompt appears once, at the start, and never repeats.

# no special flags: a long payload just works
subtext-codec encode --key key.json --input-bytes big.bin --output-text message.txt

The decoder resets at the same points, so it recovers the message exactly. The window size it used rides in the key (a window field), so decoding needs nothing extra. A message that rolls uses the v2 wire format; one that fits a single window stays v1.

Two things worth knowing:

The window follows the model's context limit (or --max-context-length, if you set a smaller one). Because the size is recorded in the key, encode and decode agree automatically -- but if you decode with a different explicit --max-context-length, the key's value still wins.
Coherence is over the window, not the whole text. Past the first reset the model conditions on a sliding window of recent tokens rather than the entire history, so very long messages read like natural long-form drift rather than a single tightly-planned document. See the caveat in Limitations.

Encryption

The codec is not a cipher -- the key is metadata, not a secret, and anyone with it and the model can read the message. For confidentiality, encrypt the payload yourself before encoding. Encryption also gives you the uniform bits the indistinguishability argument wants, so you do not additionally need --compress (encrypted data will not compress anyway).

# encrypt, then hide; the codec never sees the plaintext
age -r age1... secret.txt | subtext-codec encode --key key.json --input-bytes - --output-text message.txt
# recover, then decrypt
subtext-codec decode --key key.json --input-text message.txt --output-bytes - | age -d -i key.txt > secret.txt

Any tool works -- age, gpg, openssl enc. Keeping encryption out of the codec is deliberate: this is a research prototype, and rolling in a cipher would imply a security property it does not provide.

Determinism and devices

The codec reduces to one requirement: encode and decode must derive the same frequency table at every step, down to the last integer. Everything after the forward pass runs on CPU no matter where the model lives: the logits are cast to float32 for the stable sort and the candidate filter, then to float64 for the softmax and quantization. So the only device-sensitive component is the logits themselves. Encode and decode also drive the model identically (prefill the prompt, then one token at a time), so they hit the same kernels at the same shapes.

This has been measured, not just reasoned about. On an RTX 5090 (Blackwell, sm_120, torch 2.13 + CUDA 13), repeated passes over the same tokens produce bit-identical logits in fp32, bf16 and fp16, and all three round-trip on GPU. GPU is as reliable as CPU.

What matters is that the environment matches between encode and decode:

Factor	Effect
Different dtype (bf16 ↔ fp32)	Breaks. The key records the dtype; keep it.
TF32 on vs off	Pinned off by `set_deterministic`, so it no longer varies.
Different attention backend (SDPA / FlashAttention / eager)	Can break across machines. Same environment, same choice.
Different torch / transformers version	Can break; pin them (`requirements.txt`) if a message must survive an upgrade.
Different device (`cuda` ↔ `cpu`)	Unreliable, but not automatically fatal -- see below.

Crossing devices is not supported, though it is not guaranteed to fail: gpt2 in fp32 does decode on CPU from a CUDA encode, because the two happen to agree bit for bit. Larger models and bf16 will not be so lucky. Do not rely on it. What is guaranteed is that a mismatch is caught rather than returning plausible wrong bytes -- that is the payload framing's job.

set_deterministic also sets CUBLAS_WORKSPACE_CONFIG=:4096:8 (via package import, before torch loads) and disables cuDNN autotuning. It holds fp32 matmuls at full precision, which costs some speed on Ampere and later; a deliberate trade, since a mismatch costs the entire message.

bf16 and fp16 work but leave less margin -- half the mantissa means adjacent logits sit closer together. fp32 is safer when you can afford it. Exact ties are fine in any precision: the sort is stable, so ties resolve by token id.

To confirm on your own hardware before trusting a message to it:

uv run pytest tests/test_determinism.py -v

Those parametrize over every device and dtype present, so on a GPU box they additionally check bit-identical logits on CUDA, matching bands, and fp32/bf16/fp16 round trips.

Compatibility

There are two wire formats:

v1 -- a single segment framing length || crc32 || data, exactly as 1.0 wrote it. A plain message that fits one context window (no compression, no rolling) is still written as v1, so it stays byte-for-byte readable by 1.0.
v2 -- added in 1.1 for compression and the rolling window. The key records "version": "v2" and, as needed, a compression field and a window field. Compressed payloads frame a length and CRC over the original data so a wrong compression flag is caught; rolled payloads carry the window so the decoder resets at the same points. A 1.0 reader rejects a v2 key outright rather than mis-reading it, which is why the new features carry a new version.

1.1 reads both. It writes v2 only when you compress or when the payload is long enough to roll the context; everything else stays v1.

Every pre-1.0 format is gone along with the code that decoded them: they used rank coding, which corrupted roughly a third of its own messages and never produced model-distributed output. Pre-1.0 also numbered a format v1, but that one carried base/top_p and no temperature, so the collision is detectable -- such a key fails with an explanation rather than a generic error. To read a message encoded before 1.0, install subtext-codec~=0.2.0; to keep it, re-encode the payload with this version.

Limitations

Brittle to edits: changing a single token of the output breaks decoding. This is an encoding scheme, not an error-correcting one.
Model-dependent: requires the exact same weights, tokenizer and dtype. Decoding a bf16 encode in fp32 will not work.
Not confidential on its own: the key is metadata, not a cipher. Encrypt the payload before encoding it -- both for secrecy and because the indistinguishability argument assumes uniform payload bits.
Context length: handled automatically by the rolling window, but past the first reset the model conditions on recent tokens rather than the whole history. Coherence is local to the window, and a discriminator with the full text could in principle notice the long-range structure "resetting" -- subtle for a reasonable window, but a real difference from full-context sampling, on top of the temperature caveat below.
Capacity: the distribution's entropy, typically 1-5 bits per token depending on temperature. Expect a few hundred tokens per hundred bytes.
Length is the remaining tell: the per-token statistics are right, but a 900-token hotel review is still an odd thing to find. Pick a prompt whose natural continuation is long-form.
"Indistinguishable" is relative to the temperature you chose: output at temperature=1.5 is exactly temperature-1.5 sampling, which is only unremarkable to someone who has no expectation about the setting. An adversary who knows you would have sampled at 1.0 can distinguish it in aggregate. Lower temperature closes that gap and lengthens the text.

This project is a research prototype, not a secure or production steganography system.

Testing

uv run pytest # everything
uv run pytest -m "not slow"# skip tests that download a checkpoint

The slow tests use sshleifer/tiny-gpt2 and skip themselves if the Hub is unreachable.

License

MIT

subtext-codec

Steganographic data encoding in natural language via arithmetic coding against an LLM

subtext-codec hides arbitrary binary data inside seemingly normal LLM-generated text. The payload drives an arithmetic coder whose probability model is the language model's own next-token distribution, so the text it produces is distributed exactly as ordinary sampling would be while secretly encoding bytes. With the same model, tokenizer, prompt prefix and parameters, the process is fully reversible.

How it works

The payload drives an arithmetic decoder whose probability model is the language model itself. Emitting a token the model assigns probability p consumes -log2(p) payload bits, so a confident step carries almost nothing and an uncertain one carries a lot.

Two properties follow, and they are the entire point of the design:

The generated text is distributed exactly as ordinary sampling at the chosen temperature. Nothing is ever forced off-distribution.
Capacity per token equals the distribution's entropy, which is the information-theoretic ceiling for text that stays indistinguishable from sampling.

Encoding

The payload is framed with a length and a CRC32, then read as a bitstream.
At each step the model's logits are clipped to top_k, filtered to tokens that are safe to emit (see below), and turned into an integer frequency table at temperature.
The arithmetic decoder consumes payload bits and selects a token from that table.
Generation stops as soon as the emitted tokens pin down every payload bit.

Decoding

Start from the same prompt prefix and model, and rebuild the same table at each step.
Run the arithmetic encoder over the observed tokens, which reproduces the bitstream.
Stop once the declared length has been recovered, then check the CRC.

Decoding needs the text, the prompt prefix, the same model and tokenizer, and the codec parameters -- all of which the key file carries. No terminator token is required, so text after the message is simply ignored.

Why the encoder checks itself

The coder re-encodes each symbol as it emits it and refuses to return a message whose bits it cannot reproduce. That is not belt-and-braces: feeding the arithmetic decoder a zero-filled tail past the end of the payload parks its value register exactly on the interval midpoint, where it underflows forever and the payload's last two bits are never emitted. The result is a message that looks fine and silently fails to decode. The check is what turns that class of bug into a loud failure.

Why the safety filter exists

The encoder chooses token ids, but the decoder is handed text. Nothing guarantees that writing tokens out and reading them back returns the same ids: a byte-level BPE vocabulary will merge an emitted token into its neighbours, so 'an' written after 'Kanz' reads back as 'anz' and every subsequent step decodes against the wrong prefix.

So a candidate is only usable if appending it leaves the surrounding text re-tokenizing unchanged. Both sides apply that same filter to the same prefix, so they stay in step, and the encoder verifies the finished text tokenizes back to exactly what it built before handing it over. A message that cannot satisfy this raises at encode time rather than becoming an artifact that silently fails to decode.

Features

Entropy-optimal capacity -- each token carries -log2(p) bits, the theoretical maximum for output that still looks like sampling
Output distributed exactly as the model's own sampling, so no per-token statistical tell
Temperature as a principled capacity dial, replacing rank truncation
Tokenizer-stable candidate selection, verified before any message is returned
Self-inverse check at encode time -- the coder re-encodes as it goes and refuses to emit a message it cannot read back
Framed payload with a CRC32, so a wrong key, prompt or model is detected rather than yielding plausible wrong bytes
Optional zlib compression (--compress), which shortens the cover text and makes the payload bits more uniform
Automatic rolling context window, so a payload can exceed the model's context window with no repeated prompt and no tuning
Incremental KV-cached stepping, so cost is linear in message length rather than quadratic
Deterministic throughout; verified bit-identical on CPU and CUDA across fp32/bf16/fp16
Hugging Face Transformers backend -- works with most causal LMs

Installation

From PyPI (or via uv):

uv pip install subtext-codec
# or: pip install subtext-codec

From source:

git clone https://github.com/shevisj/subtext-codec
cd subtext-codec
uv venv --python 3.13
uv pip install -r requirements.txt
uv pip install -e .

torch and transformers are the only runtime dependencies. The published package sets lower bounds only, so it installs alongside whatever torch build you already have; requirements.txt pins an exact, CPU-only development environment and is the one to use when reproducing a decode.

Usage

Run subtext-codec --version to print the installed version. The CLI otherwise exposes encode and decode. Shared flags:

--key -- path to the codec key file (required)
--model-name-or-path -- HuggingFace model id or local path; optional if stored in the key
--prompt-prefix -- prefix used for both encode and decode; taken from the key if present
--device -- e.g. cpu or cuda (falls back to the key, then cpu)
--torch-dtype -- weight dtype (auto, fp16, bf16, fp32)
--max-context-length -- cap on sequence length; defaults to the model's own limit
--seed -- deterministic seed (default: 0)
--quiet -- suppress the progress line on stderr

Encode bytes into text

subtext-codec encode \
--model-name-or-path gpt2 \
--prompt-prefix "Once upon a time, " \
--input-bytes secret.txt \
--output-text message.txt \
--key key.json \
--temperature 1.5

Encode-only flags:

--temperature -- the capacity dial (default: 1.5). Higher packs more payload per token and shortens the cover text, at the cost of more erratic prose. See Choosing a temperature.
--top-k -- candidates considered per step (default: 64). This bounds the stability filter's work; it is not a capacity dial. Must be at least 2.
--compress -- zlib-compress the payload before encoding. Shortens the cover text and makes the payload bits more uniform. Recorded in the key, so decoding reverses it automatically. See Compression.
--max-new-tokens -- fail rather than generate more than this many tokens
--no-store-model -- keep the model id out of the key
--verify -- decode the result before writing it, confirming the round trip end to end

Payloads too long for one context window are handled automatically; see Large payloads.

Unless --quiet, encoding prints a one-line summary to stderr -- payload size, cover tokens, bits per token and mean surprisal (and how many times the context rolled) -- so you can see the coder is running at the distribution's entropy.

The output text is just the generated story, with no metadata header. key.json records temperature, top_k, the prompt prefix, the device, the dtype, (unless --no-store-model) the model id, and -- when compression or the rolling window is used -- a compression or window field and a "version": "v2" marker.

If the path given to --key already exists, its values are reused as defaults and any CLI overrides are written back, so a second message needs only the key:

subtext-codec encode --key key.json --input-bytes secret.bin --output-text message.txt

Decode text back into bytes

subtext-codec decode \
--input-text message.txt \
--key key.json \
--output-bytes decoded.bin

Parameters come from the key unless overridden for the run. Decoding never modifies the key file -- the one artifact a message cannot be recovered without is not rewritten by a read operation.

Decoding stops as soon as the payload's declared length has been recovered, so trailing text is harmless. Text before the prompt prefix is skipped too. If the message was altered, or the key, prompt or model do not match, the CRC32 catches it rather than returning plausible wrong bytes.

Sample artifacts

samples/secret.txt is a payload to play with. Build the rest of the fixture from it:

python samples/regenerate.py

That encodes it with gpt2 on CPU -- a ~500MB download, no GPU or gated checkpoint needed -- and writes message.txt, key.json and decoded.txt alongside it, verifying the round trip before writing anything. Encoding is deterministic, so it reproduces the same message every time.

Then decode it back the normal way:

subtext-codec decode \
--input-text samples/message.txt \
--key samples/key.json \
--output-bytes /tmp/decoded.txt
diff samples/secret.txt /tmp/decoded.txt &&echo"exact match"

(Before 1.0 the fixture was checked in, but it was a v2 message against a gated 16GB Llama checkpoint -- unreadable by this version and unrunnable by most people. Generating it locally is both smaller and honest.)

Programmatic usage

See the demo notebook.

Choosing a temperature

temperature is the capacity dial. Raising it flattens the distribution, so each token carries more payload and the cover text gets shorter -- but the text becomes more erratic, exactly as sampling at that temperature would. top_k is not a capacity dial here; it only bounds how much work the stability filter does.

Measured on the 445-byte samples/secret.txt, Qwen2.5-7B (bf16) on an RTX 5090, top_k=64:

Temperature	Cover tokens	Bits/token	Character
1.0	2914	1.22	best prose, but far too long to be plausible
1.5	894	3.98	the knee -- reads like a real, rambling review
2.0	709	5.02	densest, but visibly erratic and prone to invention

Mean surprisal of the emitted tokens matches capacity at every setting (1.23 / 4.02 / 5.07 bits), which is the confirmation that the coder is running at the distribution's entropy rather than leaving capacity on the table.

For reference, the rank-coding scheme this replaced in 1.0 managed 4.35 bits/token at its best setting and 1.57 at its most conservative, and its output was never model-distributed. Arithmetic coding at temperature 1.5 matches its best capacity, and at 2.0 exceeds it, while staying on-distribution throughout.

A bigger model does not buy more capacity. A stronger model concentrates probability mass, which lowers entropy and therefore bits per token. It buys better prose at a given temperature, not shorter text. Keep payloads to a few hundred bytes, and pick a prompt whose natural continuation is long-form so the required length is not conspicuous.

Compression

--compress zlib-compresses the payload before encoding. It does two useful things:

Shortens the cover text. Fewer payload bytes means fewer tokens to carry them.
Makes the bits more uniform, which is exactly what the indistinguishability argument assumes. Raw ASCII is biased; compressed (or encrypted) data is not.

subtext-codec encode --key key.json --input-bytes secret.txt --output-text message.txt --compress

Compression is recorded in the key, so decode reverses it automatically -- you do not pass --compress when decoding. A compressed message uses the v2 wire format (see Compatibility). The whole payload is framed with a length and CRC32 before compression, so a key that misdescribes compression is caught rather than returning the raw compressed bytes.

Compression rarely helps on data that is already high-entropy (encrypted blobs, media). It helps most on text and structured data.

Large payloads

A payload can outgrow the model's context window, and that is handled automatically -- there is nothing to configure. When the context fills, the codec re-prefixes the most recent half of the tokens and keeps generating, a rolling window that slides forward as the message grows. The text stays one continuous passage: the prompt appears once, at the start, and never repeats.

# no special flags: a long payload just works
subtext-codec encode --key key.json --input-bytes big.bin --output-text message.txt

The decoder resets at the same points, so it recovers the message exactly. The window size it used rides in the key (a window field), so decoding needs nothing extra. A message that rolls uses the v2 wire format; one that fits a single window stays v1.

Two things worth knowing:

The window follows the model's context limit (or --max-context-length, if you set a smaller one). Because the size is recorded in the key, encode and decode agree automatically -- but if you decode with a different explicit --max-context-length, the key's value still wins.
Coherence is over the window, not the whole text. Past the first reset the model conditions on a sliding window of recent tokens rather than the entire history, so very long messages read like natural long-form drift rather than a single tightly-planned document. See the caveat in Limitations.

Encryption

The codec is not a cipher -- the key is metadata, not a secret, and anyone with it and the model can read the message. For confidentiality, encrypt the payload yourself before encoding. Encryption also gives you the uniform bits the indistinguishability argument wants, so you do not additionally need --compress (encrypted data will not compress anyway).

# encrypt, then hide; the codec never sees the plaintext
age -r age1... secret.txt | subtext-codec encode --key key.json --input-bytes - --output-text message.txt
# recover, then decrypt
subtext-codec decode --key key.json --input-text message.txt --output-bytes - | age -d -i key.txt > secret.txt

Any tool works -- age, gpg, openssl enc. Keeping encryption out of the codec is deliberate: this is a research prototype, and rolling in a cipher would imply a security property it does not provide.

Determinism and devices

The codec reduces to one requirement: encode and decode must derive the same frequency table at every step, down to the last integer. Everything after the forward pass runs on CPU no matter where the model lives: the logits are cast to float32 for the stable sort and the candidate filter, then to float64 for the softmax and quantization. So the only device-sensitive component is the logits themselves. Encode and decode also drive the model identically (prefill the prompt, then one token at a time), so they hit the same kernels at the same shapes.

This has been measured, not just reasoned about. On an RTX 5090 (Blackwell, sm_120, torch 2.13 + CUDA 13), repeated passes over the same tokens produce bit-identical logits in fp32, bf16 and fp16, and all three round-trip on GPU. GPU is as reliable as CPU.

What matters is that the environment matches between encode and decode:

Factor	Effect
Different dtype (bf16 ↔ fp32)	Breaks. The key records the dtype; keep it.
TF32 on vs off	Pinned off by `set_deterministic`, so it no longer varies.
Different attention backend (SDPA / FlashAttention / eager)	Can break across machines. Same environment, same choice.
Different torch / transformers version	Can break; pin them (`requirements.txt`) if a message must survive an upgrade.
Different device (`cuda` ↔ `cpu`)	Unreliable, but not automatically fatal -- see below.

Crossing devices is not supported, though it is not guaranteed to fail: gpt2 in fp32 does decode on CPU from a CUDA encode, because the two happen to agree bit for bit. Larger models and bf16 will not be so lucky. Do not rely on it. What is guaranteed is that a mismatch is caught rather than returning plausible wrong bytes -- that is the payload framing's job.

set_deterministic also sets CUBLAS_WORKSPACE_CONFIG=:4096:8 (via package import, before torch loads) and disables cuDNN autotuning. It holds fp32 matmuls at full precision, which costs some speed on Ampere and later; a deliberate trade, since a mismatch costs the entire message.

bf16 and fp16 work but leave less margin -- half the mantissa means adjacent logits sit closer together. fp32 is safer when you can afford it. Exact ties are fine in any precision: the sort is stable, so ties resolve by token id.

To confirm on your own hardware before trusting a message to it:

uv run pytest tests/test_determinism.py -v

Those parametrize over every device and dtype present, so on a GPU box they additionally check bit-identical logits on CUDA, matching bands, and fp32/bf16/fp16 round trips.

Compatibility

There are two wire formats:

v1 -- a single segment framing length || crc32 || data, exactly as 1.0 wrote it. A plain message that fits one context window (no compression, no rolling) is still written as v1, so it stays byte-for-byte readable by 1.0.
v2 -- added in 1.1 for compression and the rolling window. The key records "version": "v2" and, as needed, a compression field and a window field. Compressed payloads frame a length and CRC over the original data so a wrong compression flag is caught; rolled payloads carry the window so the decoder resets at the same points. A 1.0 reader rejects a v2 key outright rather than mis-reading it, which is why the new features carry a new version.

1.1 reads both. It writes v2 only when you compress or when the payload is long enough to roll the context; everything else stays v1.

Every pre-1.0 format is gone along with the code that decoded them: they used rank coding, which corrupted roughly a third of its own messages and never produced model-distributed output. Pre-1.0 also numbered a format v1, but that one carried base/top_p and no temperature, so the collision is detectable -- such a key fails with an explanation rather than a generic error. To read a message encoded before 1.0, install subtext-codec~=0.2.0; to keep it, re-encode the payload with this version.

Limitations

Brittle to edits: changing a single token of the output breaks decoding. This is an encoding scheme, not an error-correcting one.
Model-dependent: requires the exact same weights, tokenizer and dtype. Decoding a bf16 encode in fp32 will not work.
Not confidential on its own: the key is metadata, not a cipher. Encrypt the payload before encoding it -- both for secrecy and because the indistinguishability argument assumes uniform payload bits.
Context length: handled automatically by the rolling window, but past the first reset the model conditions on recent tokens rather than the whole history. Coherence is local to the window, and a discriminator with the full text could in principle notice the long-range structure "resetting" -- subtle for a reasonable window, but a real difference from full-context sampling, on top of the temperature caveat below.
Capacity: the distribution's entropy, typically 1-5 bits per token depending on temperature. Expect a few hundred tokens per hundred bytes.
Length is the remaining tell: the per-token statistics are right, but a 900-token hotel review is still an odd thing to find. Pick a prompt whose natural continuation is long-form.
"Indistinguishable" is relative to the temperature you chose: output at temperature=1.5 is exactly temperature-1.5 sampling, which is only unremarkable to someone who has no expectation about the setting. An adversary who knows you would have sampled at 1.0 can distinguish it in aggregate. Lower temperature closes that gap and lengthens the text.

This project is a research prototype, not a secure or production steganography system.

Testing

uv run pytest # everything
uv run pytest -m "not slow"# skip tests that download a checkpoint

The slow tests use sshleifer/tiny-gpt2 and skip themselves if the Hub is unreachable.

License

MIT

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
samples		samples
subtext_codec		subtext_codec
tests		tests
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
demo.ipynb		demo.ipynb
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
samples		samples
subtext_codec		subtext_codec
tests		tests
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
demo.ipynb		demo.ipynb
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
samples		samples
subtext_codec		subtext_codec
tests		tests
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
demo.ipynb		demo.ipynb
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
samples		samples
subtext_codec		subtext_codec
tests		tests
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
demo.ipynb		demo.ipynb
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
samples		samples
subtext_codec		subtext_codec
tests		tests
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
demo.ipynb		demo.ipynb
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

subtext-codec

Steganographic data encoding in natural language via arithmetic coding against an LLM

How it works

Encoding

Decoding

Why the encoder checks itself

Why the safety filter exists

Features

Installation

Usage

Encode bytes into text

Decode text back into bytes

Sample artifacts

Programmatic usage

Choosing a temperature

Compression

Large payloads

Encryption

Determinism and devices

Compatibility

Limitations

Testing

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

subtext-codec

Steganographic data encoding in natural language via arithmetic coding against an LLM

How it works

Encoding

Decoding

Why the encoder checks itself

Why the safety filter exists

Features

Installation

Usage

Encode bytes into text

Decode text back into bytes

Sample artifacts

Programmatic usage

Choosing a temperature

Compression

Large payloads

Encryption

Determinism and devices

Compatibility

Limitations

Testing

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

subtext-codec

Steganographic data encoding in natural language via arithmetic coding against an LLM