Reconstruct lexer diagnostics lazily for seekable input - #5234
Merged
Merged
Conversation
nlohmann
marked this pull request as draft
July 4, 2026 07:02
nlohmann
force-pushed
the
claude/funny-jemison-73a586
branch
2 times, most recently
from
July 4, 2026 07:52
abf0e2b to
521995c
Compare
`lexer::get()` copied every scanned character into `token_string` on the whole successful-parse hot path, yet that buffer is consumed only by `get_token_string()` when rendering the "last read" fragment of a parse error. On well-formed input the per-byte copy (plus the `unget()` pop) is pure overhead that is always discarded. For seekable input adapters - random-access, single-byte iterators such as those backing `std::string`, `const char*`, and `std::vector<char>` - the offending token is now reconstructed on demand from the input when an error is reported, using a saved start offset, and the eager copy is skipped. Streaming adapters (file, istream, wide-string, and user-defined adapters) keep the eager copy; the strategy is chosen at compile time via `input_adapter_supports_seek`, so adapters without the capability are unaffected. Error messages are byte-for-byte identical across all adapters, verified by a new parity regression test. Microbenchmark (4 MB mixed JSON, parsed from a std::string): ~149 -> ~160 MB/s, about +8%. Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nlohmann
force-pushed
the
claude/funny-jemison-73a586
branch
from
July 4, 2026 08:36
521995c to
eaefd09
Compare
nlohmann
marked this pull request as ready for review
July 4, 2026 21:08
gregmarr
approved these changes
Jul 4, 2026
Owner
Author
|
@gregmarr Thanks for the reviews! |
nlohmann
added a commit
that referenced
this pull request
Jul 5, 2026
…-6a6c1c Resolves a conflict in include/nlohmann/detail/input/input_adapters.hpp between this branch's memcpy fast path for get_elements() and develop's lazy-diagnostics feature (#5234), which added a `begin` iterator member, `get_consumed_count()`, and `copy_consumed_range()` to the same class. The two features are orthogonal (lexer.hpp uses the new diagnostics helpers; binary_reader.hpp uses get_elements()) and both are preserved. single_include/nlohmann/json.hpp is regenerated from the merged sources. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The lexer maintained a per-character
token_stringdiagnostic buffer on 100% of successful parses, but that buffer is only read on error.lexer::get()pushed every scanned character intotoken_string(andunget()popped it), yet the buffer is consumed exclusively byget_token_string()when rendering the "last read" fragment of aparse_error. On well-formed input this per-byte copy is pure overhead that is always discarded.This PR implements the "bigger win" option from the todo: reconstruct the offending token lazily on error from a saved start offset for random-access/contiguous input, keeping the eager per-byte copy only for streaming adapters that cannot seek back.
Approach
std::string,const char*,std::vector<char>, arrays): the eager copy is skipped.reset()records only the token's start offset; on error the token is rebuilt from the input viaiterator_input_adapter::copy_consumed_range().FILE*,std::istream, wide-string, and user-defined adapters: unchanged eager-copy behavior.input_adapter_supports_seek(an overload-based detection that is tolerant of adapters lacking the flag, so user-defined adapters keep working).Correctness
Error messages are byte-for-byte identical across all adapter types. Verified against the existing exhaustive parser/diagnostic tests plus a new regression test (
last-read diagnostics are identical across input adapters) that cross-checksstd::string,const char*,std::vector<char>,std::list(bidirectional),std::istringstream(streaming), andstd::u16string/std::u32string(wide) — covering control-character escaping, number overflow, BOM, and whitespace/structural accumulation.Benchmark
Microbenchmark parsing a 4 MB mixed JSON document (objects, arrays, strings, numbers) from a
std::string, best-of-many medians:≈ +8% parse throughput (theoretical ceiling from removing the buffer entirely is ~+11%).
🤖 Generated with Claude Code