Skip to content

Check what the caller supplied Vm returned before stripping the 0x prefix - #52

Merged
thedavidmeister merged 3 commits into
mainfrom
2026-08-16-libhexstring-vm-output-check
Aug 16, 2026
Merged

Check what the caller supplied Vm returned before stripping the 0x prefix#52
thedavidmeister merged 3 commits into
mainfrom
2026-08-16-libhexstring-vm-output-check

Conversation

@thedavidmeister

@thedavidmeisterthedavidmeister commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Closes#43.

The decision: check it

The issue frames the real tension — Vm is foundry's, and a library re-checking
its own cheatcode's output is unusual — and leaves the call open. This PR adds
the check. The reasoning, because the counter-argument is a good one:

The thing being trusted is a parameter, not a cheatcode.bytesToHex(Vm vm, bytes memory data) does not call the cheatcode address; it calls whatever
address the caller cast to Vm. Vm(anyAddress) type checks. So the string the
pointer arithmetic operates on is, by the shape of the function signature, an
untrusted input — the same category as data. "Do not re-check foundry" would be
the right answer if the library reached for VM_ADDRESS itself, and the honest
alternative fix is exactly that (see below); it is not the right answer for a
value that arrives through the parameter list.

The failure is silent, and what it corrupts is source that compiles. The
"" and "Z" rows of the issue's table produce a length word near 2**256,
which is loud in the sense that anything downstream will run out of gas. The
"abcdef" row is the one that matters: two characters of real data are dropped
and the result is spliced into hex"..." in generated Solidity, giving a
bytes constant that compiles, commits, and holds the wrong value. For a
repository whose entire product is generated source, a wrong constant that
compiles is the worst available failure class, and a revert is strictly better.

There is no counterweight.bytesToHex is internal and pure, reached
only from build scripts and tests, and it already makes an external call. Two
comparisons cost nothing that anyone is measuring.

The property is already asserted — in the wrong place.#37 landed
testBytesToHexIsVmToStringWithoutPrefix, which asserts over 2048 fuzz runs that
vm.toString(data) is 2 * data.length + 2 characters beginning 0x. The repo
already agrees this property is load bearing. But that assertion only runs in
this repo's CI; a downstream consumer runs the generator, not this suite. Moving
the same predicate into the function is what makes it hold at every consumer's
build. The check here is deliberately the same predicate, not a weaker one.

What changed

src/lib/LibHexString.sol:

uint256 expectedLength = data.length*2+2;
bytesmemory hexBytes =bytes(hexString);
if (hexBytes.length!= expectedLength || hexBytes[0] !=bytes1("0") || hexBytes[1] !=bytes1("x")) {
revertUnexpectedHexString(hexString, expectedLength);
}

bytes(hexString) is a free cast on the same pointer, so nothing is copied. The
length equality short circuits ahead of the two index reads, so those are always
in bounds. The docstring now states the revert; the error is a typed
UnexpectedHexString(string hexString, uint256 expectedLength) carrying the
offending string rather than a message to match on.

Also the /// that #43 notes at the end — the continuation line of the in body
comment inside the assembly block used the NatSpec marker where // is meant.

Why the length is an equality and not >= 2

>= 2 is the minimal fix for the underflow and it is not enough: it leaves the
"abcdef" row, which is the row the issue calls the one that matters, and it
leaves a Vm returning "0xaa" for hex"aabb" emitting a constant holding half
the data. The property the function actually depends on is "the return is
toString(bytes)'s defined output", and its structural half — length and prefix —
is checkable in two comparisons. The remaining half, that the nibbles are the
right nibbles, is not checkable without reimplementing the encoder, so it is out.
That is where the line falls, and it is drawn at a property rather than at the
list of inputs the issue happened to enumerate.

The alternative that was rejected

The other way to close this is to stop taking Vm as a parameter and use
Vm(VM_ADDRESS) internally, which removes the untrusted input rather than
checking it. That is a strictly better answer to the narrow question and a much
worse change: Vm is threaded as a parameter through all 11 public functions of
LibCodeGen and both of LibFs, so it is an API break for every consumer, and
it forecloses the vm.prank-style setups a consumer might legitimately want. Not
in this PR's scope, and not obviously wanted at all.

The check is on the shape of the returned string, not on the identity of the
Vm, so it does not narrow the parameter by the back door — a non-foundry Vm
that conforms still works, which testBytesToHexAcceptsConformingVm pins.

Tests

11 new tests in test/lib/LibHexString.bytesToHex.t.sol, and
test/concrete/NonConformingVm.sol — a toString(bytes)-shaped contract
returning a string fixed at construction. It is its own file because the repro
attached to the issue declared two contracts in one, which rainix static
rejects. The library call is forced through the existing LibHexStringExternal
wrapper, because bytesToHex is internal and expectRevert only sees a revert
that happens in a subcall.

Each of the issue's three rows gets a test, plus the cases the check implies:
truncated but correctly prefixed, overlong, a wrong first prefix character with a
correct second, a wrong second with a correct first, the empty-data boundary
where the required length is 2 rather than 0, both accepting cases, and a fuzz
test over (bytes data, string toStringReturn) asserting the whole property —
for any data and any string a Vm might return, the library either reverts or
returns exactly that string with its first two characters removed.

The wrong-first-character test exists because the mutation pass found it
missing: dropping the check on hexBytes[0] initially SURVIVED, since every
non-conforming string in the suite had a wrong second character too and the
second check alone rejected all of them. Second commit on this branch, with the
mutant re-run to confirm it now dies.

No existing test was weakened, edited or deleted. The suite goes 84 -> 95.

Pre-existing red, unrelated, checked not assumed

forge test --via-ir fails testBuildFileForContractCommittedArtifactIsCurrent,
because the committed src/generated/CodeGennable.sol embeds a BYTECODE_HASH
produced by the pipeline foundry.toml configures. Verified this fails
identically on the base commit af4e5a9 with these changes reverted (83 passed /
1 failed there, 93 / 1 here — the same single test). Not introduced here and not
fixed here; the default pipeline, which is what CI runs, is 94/94 green.

QA

  • Discriminating tests: 10 new tests, every one asserting an exact revert payload
    (abi.encodeWithSelector with a literal expected length, not the source's own
    formula) or an exact returned string, never "it reverted". Full suite 95 tests
    / 15 suites, 0 failures, from an 84/0 baseline on af4e5a9; forge fmt --check
    clean; CI green on this branch, rainix / static and rainix / test both
    success on the head commit c3aaa42 at
    https://github.com/rainlanguage/rain.sol.codegen/actions/runs/31958846546
    (static is what enforces one contract per file, hence the separate
    NonConformingVm.sol). The 31 LibHexString tests were also run under the
    pipelines LibHexString.bytesToHex gets a test file #37 established as the bar — --via-ir, --evm-version shanghai
    where there is no mcopy, and --via-ir --evm-version paris — 31/31 under
    each, because the guard introduces a second live reference to the same buffer
    the memory-safe block mutates.

  • Mutations applied: 36, run with mutation-probe against the whole suite from a
    green baseline — 18 on the new check, and all 18 of LibHexString.bytesToHex gets a test file #37's mutants on the
    adjacent prefix strip re-run, because a guard inserted immediately in front of
    that assembly could have masked a mutant that used to die. 34/36 killed on the
    first pass; both survivors are named below and one of them was a real gap that
    this branch then closed. Killer lists are the first the probe reports, not the
    complete set.

    #mutationverdict / killed by
    N012 * len + 2 -> 2 * len + 1KILLED — testLiteralParserFunctionPointersConstantString, testOperandHandlerFunctionPointersConstantString, testOpcodeFunctionPointersConstantString
    N022 * len + 2 -> 2 * len + 3KILLED — same three
    N032 * len + 2 -> 2 * len (prefix unaccounted)KILLED — testSubParserWordParsersConstantString, testOpcodeFunctionPointersConstantString, testBytesConstantString
    N042 * len + 2 -> len + 2 (one char per byte)KILLED — testLiteralParserFunctionPointersConstantString, testOperandHandlerFunctionPointersConstantString, testOpcodeFunctionPointersConstantString
    N052 * len + 2 -> 3 * len + 2KILLED — testBytesConstantString, testBytesConstantStringAtMaxLength, testBytesConstantStringCarriesData
    N06whole guard deleted (this is the pre-PR source)KILLED — testBytesToHexRejectsEveryNonConformingVmOutput, testBytesToHexRevertsOnEmptyVmOutput, testBytesToHexRevertsOnOneCharacterVmOutput
    N07guard condition -> falseKILLED — same three
    N08guard condition -> trueKILLED — testLiteralParserFunctionPointersConstantString, testIntegrityFunctionPointersConstantString, testOperandHandlerFunctionPointersConstantString
    N09length clause droppedKILLED — testBytesToHexRevertsOnEmptyVmOutput, testBytesToHexRevertsOnOverlongVmOutput, testBytesToHexRevertsOnTruncatedVmOutput
    N10length != expected -> length < expectedKILLED — testBytesToHexRejectsEveryNonConformingVmOutput, testBytesToHexRevertsOnOverlongVmOutput
    N11length != expected -> length == expectedKILLED — testOperandHandlerFunctionPointersConstantString, testSubParserWordParsersConstantString, testLiteralParserFunctionPointersConstantString
    N12first prefix character clause droppedSURVIVED on the first pass — a real gap, see below. KILLED after testBytesToHexRevertsOnWrongFirstPrefixCharacter
    N13second prefix character clause droppedKILLED — testBytesToHexRevertsOnWrongSecondPrefixCharacter
    N14bytes1("0") -> bytes1("1")KILLED — testSubParserWordParsersConstantString, testLiteralParserFunctionPointersConstantString, testOpcodeFunctionPointersConstantString
    N15bytes1("x") -> bytes1("X")KILLED — testIntegrityFunctionPointersConstantString, testOpcodeFunctionPointersConstantString, testSubParserWordParsersConstantString
    N16both prefix characters read from index 0KILLED — testOpcodeFunctionPointersConstantString, testOperandHandlerFunctionPointersConstantString, testBytesConstantString
    N17offending string dropped from the error payloadKILLED — testBytesToHexRevertsOnOneCharacterVmOutput, testBytesToHexRevertsOnTruncatedVmOutput, testBytesToHexRevertsOnUnprefixedVmOutput
    N18expected length dropped from the error payloadKILLED — testBytesToHexRevertsOnEmptyVmOutput, testBytesToHexRevertsOnEmptyVmOutputForEmptyData, testBytesToHexRevertsOnOneCharacterVmOutput
    M01-M04strip offset 2 -> 0, 1, 3, 32KILLED — testBytesToHexAcceptsConformingVm, testBytesToHexAllocationBoundary, testBytesToHexCharset
    M05-M08length term dropped, -1, -3, +2KILLED — testBytesToHexAcceptsConformingVm, testBytesToHexAcceptsConformingVmForEmptyData, testBytesToHexCharset
    M09-M11length read from / written to the wrong pointer, write deletedKILLED — same three
    M12pointer not reassignedKILLED — testBytesToHexAcceptsConformingVm, testBytesToHexIsVmToStringWithoutPrefix, testBytesToHexKnown
    M13input ignored (toString(bytes("")))KILLED — testIntegrityFunctionPointersConstantString, testSubParserWordParsersConstantString, testLiteralParserFunctionPointersConstantString
    M14return hexString -> return ""KILLED — testBytesToHexAcceptsConformingVm, testBytesToHexConcatenatesIntoHexLiteral, testBytesToHexIsVmToStringWithoutPrefix
    M15stray mstore(0x80, 0)KILLED — testBytesToHexDoesNotMutateInput, testBytesToHexLeavesNeighbouringMemoryAlone, testBytesConstantStringAtMaxLength
    M16assembly ("memory-safe") -> assemblySURVIVED — non-behaviour control, correct verdict
    M17free memory pointer moved backwardsKILLED — testBytesToHexAcceptsConformingVm, testBytesToHexAllocationBoundary, testBytesToHexCharset
    M18caller's input bytes clobberedKILLED — testBytesToHexDoesNotMutateInput, testBytesToHexLongData, testBytesToHexRoundTrips

    N12 was a genuine hole: every non-conforming string in the first version of the
    suite had a wrong SECOND prefix character as well as a wrong first, so the
    second check alone rejected all of them and the first was never exercised
    alone. Fixed by testBytesToHexRevertsOnWrongFirstPrefixCharacter
    ("Zxaabb"), and N12 and N13 re-run afterwards: 2/2 killed, N12 by exactly
    that test.

    M16 is the same non-behaviour control LibHexString.bytesToHex gets a test file #37 used and its survival is still the
    correct verdict, not a gap: the annotation is a promise to the optimiser and
    removing it only makes the optimiser more conservative. What the annotation
    claims is probed directly by M15, M17 and M18, all killed.

    No mutant of the new guard survived on the prefix strip, so the guard does not
    mask any of LibHexString.bytesToHex gets a test file #37's coverage — all 18 M-mutants still die, several of them now
    additionally through the new conforming-Vm tests.

  • Oracle: intent came from the definition of toString(bytes) and from what solc
    accepts, not from what the library returns. The fuzz test derives conformance
    independently (length == 2n + 2 && [0] == '0' && [1] == 'x') and asserts the
    stripped string it computes itself, so it cannot agree with a wrong
    implementation. The accepting cases are asserted through a non-foundry Vm as
    well as the real one, which is what proves the guard gates on the string's
    shape and not on the caller's Vm being foundry's. Note the N01-N05, N08,
    N11, N14-N16 killers are all LibCodeGen tests: mutating the required shape
    makes the real vm fail the check, which is the independent confirmation that
    what the guard demands is exactly what foundry returns.

  • Category check: the issue lists three offending returns; the category is every
    way a return can fail the two facts the strip depends on. Enumerated as: too
    short to index (0 and 1 characters), right length with no prefix, right length
    with a wrong first prefix character only, right length with a wrong second only,
    correctly prefixed but too short, too long, and the data.length == 0 boundary
    where the required length is 2. All seven have a test, and the fuzz test covers
    the category as a property rather than as the list.

Summary by CodeRabbit

  • Bug Fixes

    • Improved hexadecimal conversion validation to reject malformed or unexpected output.
    • Added clear error reporting for invalid prefixes and incorrect hexadecimal lengths.
    • Preserved support for valid conversions, including empty data and compatible custom implementations.
  • Tests

    • Expanded coverage for malformed output, incorrect lengths, invalid prefixes, and property-based scenarios.

thedavidmeisterand others added 2 commits August 16, 2026 16:22
…efix
bytesToHex takes its Vm as a parameter, so the string it strips two
characters from is whatever the caller's Vm returns. A shorter return
underflows the length word and an unprefixed one silently discards two
characters of real data. Require "0x" followed by two hexadecimal
characters per input byte, which is what foundry's own Vm returns.
Closes#43
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mutation pass survived dropping the check on the first prefix
character: every case with a wrong first character also had a wrong
second one, so the second check alone rejected them all. Add the mirror
of the wrong-second-character case.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@thedavidmeisterthedavidmeister self-assigned this Aug 16, 2026
@coderabbitai

coderabbitaiBot commented Aug 16, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 9d8d0d29-2ad2-48e5-a8ed-9f662a4c0208

📥 Commits

Reviewing files that changed from the base of the PR and between 127d9a0 and e01ae9e.

📒 Files selected for processing (3)
  • src/lib/LibHexString.sol
  • test/concrete/NonConformingVm.sol
  • test/lib/LibHexString.bytesToHex.t.sol

Walkthrough

bytesToHex now validates VM-generated hexadecimal strings before removing the 0x prefix. Tests cover malformed output, empty data, conforming output, and property-based cases.

Changes

Hex string validation

Layer / File(s)Summary
Validate VM output
src/lib/LibHexString.sol
Adds UnexpectedHexString and validates the VM result length and 0x prefix before stripping the prefix.
Test VM output validation
test/concrete/NonConformingVm.sol, test/lib/LibHexString.bytesToHex.t.sol
Adds a configurable VM test double and tests malformed, empty, conforming, and property-based output cases.

Estimated code review effort: 3 (Moderate) | ~20 minutes

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 2026-08-16-libhexstring-vm-output-check

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@thedavidmeister

Copy link
Copy Markdown
ContributorAuthor

@coderabbitai review

@coderabbitai

coderabbitaiBot commented Aug 16, 2026

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

The check read the length twice — once as `hexBytes.length` and again as
`mload(hexString)` in the block below it — and reached the prefix through two
bounds-checked index reads, while the block directly underneath re-derived the
pointer it had just walked away from. The check and the code whose correctness
depends on it were two constructs a later edit can drift apart.
One block now: one length load feeding both the comparison and the strip, the
prefix as the top two bytes of the data word against `0x3078`, and the strip
running only on the valid path rather than after a separate guard.
The length is checked in an outer `if` and the prefix read inside it, because
`and` in Yul evaluates both arms — a single `and` would read the data word even
for a zero-length string, past that string's own allocation, which is the claim
`memory-safe` makes. `expectedLength` is at least 2, so a length that matches it
puts the word inside the allocation.
The revert stays in solidity. `UnexpectedHexString` carries a dynamic string,
and hand-encoding one in assembly is a dozen lines of pointer arithmetic in a
build-time function where gas is not a consideration.
95 tests pass, 31 of them LibHexString's.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@thedavidmeister

Copy link
Copy Markdown
ContributorAuthor

Reviewed e01ae9e: ready — Closes #43.

bytesToHex takes Vm as a parameter, so the string it strips a prefix from is whatever the caller's Vm returned. Stripping two characters from a shorter string underflows the length word; stripping them from an unprefixed one silently discards two characters of real data into generated Solidity that still compiles. That second row is the one that matters, and it is why the check is an equality on data.length * 2 + 2 rather than a >= 2 floor.

The decision to fix rather than close is argued on the record and holds: the trusted value is a function parameter, not the cheatcode address, so it is untrusted by construction; #37 already asserts exactly this property over 2048 runs, but only where it protects this repo's CI rather than a consumer's build; and the function is internal pure at build time, so there is no counterweight.

The final commit reshapes the guard into the existing assembly block on the user's call. One length load feeds both the comparison and the strip, the prefix is the top two bytes of the data word against 0x3078 rather than two bounds-checked index reads, and the strip runs only on the valid path instead of after a separate guard — so the check and the code whose correctness depends on it can no longer drift apart.

The length is checked in an outer if with the prefix read nested inside it, because and in Yul evaluates both arms: a single and would read the data word even for a zero-length string, past that string's own allocation, breaking the memory-safe claim the block asserts. expectedLength is at least 2, so a matching length puts the word inside the allocation. The revert stays in solidity because the error carries a dynamic string.

95 tests pass, 31 of them LibHexString's, and forge fmt --check is clean. All four checks green.

Two limits on the evidence, stated rather than implied. The 36-mutant matrix in the body was run against c3aaa42, before the reshape — the guard's BEHAVIOUR is unchanged and the 31 tests that pin it still pass, but no mutant has been applied to the shape that is merging. I am re-running it now and will report here either way. And CodeRabbit never reviewed this PR: it hit an account-wide limit contended by four sibling PRs, so the green check has no review behind it. The @coderabbitai review I posted was against the previous head.

@thedavidmeister
thedavidmeister merged commit 0382198 into mainAug 16, 2026
5 checks passed
@coderabbitai

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bashset -euo pipefail
printf'%s\n''== PR metadata =='
gh pr view 52 --repo rainlanguage/rain.sol.codegen --json baseRefOid,headRefOid,changedFiles,title,url
printf'%s\n''== Changed files =='
git diff --name-status "$(git merge-base HEAD origin/HEAD)" HEAD 2>/dev/null || git diff --name-status HEAD~1 HEAD
printf'%s\n''== LibHexString outline =='
ast-grep outline src/lib/LibHexString.sol --items all --view expanded
printf'%s\n''== Relevant source locations =='
rg -n -C 12 'bytesToHex|UnexpectedHexString|memory-safe' src/lib/LibHexString.sol
printf'%s\n''== Relevant test locations =='
fd -a 'LibHexString.*bytesToHex.*\.sol|NonConformingVm\.sol'test| sort |whileread -r f;doecho"--- $f"
rg -n -C 3 'bytesToHex|UnexpectedHexString|NonConformingVm'"$f"||truedoneprintf'%s\n''== Diff against PR base =='
base="$(gh pr view 52 --repo rainlanguage/rain.sol.codegen --json baseRefOid --jq .baseRefOid)"
git diff --find-renames --find-copies --unified=80 "$base"...HEAD -- src/lib/LibHexString.sol test

Length of output: 41037


@thedavidmeister I reviewed e01ae9e5d5f734f022400e498701950c0064da2d.

I found no blocking issues.

The nested conditions prevent a prefix-word read until the length check proves that the allocation contains at least two bytes. The valid path changes the pointer and length only after both checks pass. The invalid path preserves the original string for UnexpectedHexString.

The tests cover the relevant malformed return categories and the empty-data boundary. The fuzz test also matches the guard predicate.


✏️ Learnings added
Learnt from: thedavidmeister
URL: https://github.com/rainlanguage/rain.sol.codegen/pull/52
Timestamp: 2026-08-16T16:59:55.042Z
Learning: In `src/lib/LibHexString.sol`, `LibHexString.bytesToHex(Vm vm, bytes memory data)` treats `Vm` as caller-supplied. It must validate that `vm.toString(data)` has length `2 * data.length + 2` and begins with `0x` before it removes the prefix, because a `Vm` cast can target a non-Foundry contract.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

You are interacting with an AI system.

@thedavidmeister

Copy link
Copy Markdown
ContributorAuthor

@coderabbitai review

@coderabbitai

coderabbitaiBot commented Aug 16, 2026

Copy link
Copy Markdown
⚠️ Action not completed

Pull request is closed.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LibHexString.bytesToHex underflows the string length if a caller-supplied Vm returns fewer than two characters

1 participant

@thedavidmeister