Per language counters spec by boyter · Pull Request #776 · boyter/scc

boyter · 2026-09-14T22:49:27Z

operation warp speed

…est safe Phase 0 of the per-language counters work. No new language ships. This is the phase that makes the other thirteen safe to write. counters_shared.go is the half of a counter that does not change from one language to the next: the outer loop, the end of line accounting, the splice, the binary bail, the string and block comment bodies, the eligibility guard that was duplicated across two files, and the counterTally and counterState types a range scoped counter needs. The hot half, the stop table switch and the complexity matcher, stays hand written per language. That split is the design. Every backwards read now takes a floor and is clamped to it. javaWordStarts becomes wordStartsAt(content, index, floor), hasPrefixAt gains the same bound, and the raw content[index-1] reads in cComplexityAnchored go through byteBefore. Those were safe only because the reduced check set at the start of a line kept the index above zero, which is an invariant of one counter and not of the machinery around it. floor is also what stops a backwards read walking into another language's bytes once a counter is handed a range rather than a file. Two tunings neither counter had: - the blank run skip, which the generic loop has had for some time. Leading indentation is 19.7% of the bytes of a Java file and 16.8% of a Python one, and both counters walked every one of them through isWhitespace. - skipToTerminator, which jumps a block comment with bytes.Index plus bytes.Count instead of walking it. A fifth of a C file sits inside one. It deliberately does not look for a nul: isBinary is called only from the code state in both loops, so scanning the skipped region would break conformance rather than improve it. Java is anchored the way C already was. It stopped on the first byte of every check, which is f, i, s, w, e, t and c; it now stops on f, w, l, y and h and reads the keyword backwards from there. Measured over 29MB of real Java the stop rate goes from 29.6% of all bytes to 10.6%. Benchmarks, Apple M1, valkey for C and guava for Java, 8 interleaved rounds. The absolute numbers are not comparable with the 599 and 344 MB/s recorded on a Ryzen 9 9950X3D; the ratios are. C generic 350.1 MB/s specialised 496.2 MB/s 1.42x (ceiling 716.3) Java generic 342.5 MB/s specialised 620.8 MB/s 1.81x (ceiling 930.0) What this phase moved, against itself: C specialised 482.7 -> 496.2 MB/s within noise C specialised, complexity off 647.5 -> 716.3 MB/s +10.6% p=0.003 Java specialised 448.2 -> 620.8 MB/s +38.5% p=0.000 Java specialised, complexity off 799.0 -> 930.0 MB/s +16.4% p=0.000 The generic loop is untouched and reads flat across both binaries, which is what says the measurement is sound. The closure the shared outer loop dispatches through was measured against an inlined variant and costs nothing: the states are entered per token, not per byte. Tests. The harness had three defects and all three are fixed: it compared five fields and not Binary, which decides whether a file is reported at all; it labelled every failure java even when called from the C test; and it left a package global moved on the way out. Added, none of which existed: - structural conformance, which holds each counter against languages.json with no corpus and no env var, and asserts the anchoring collision table of the architecture doc. The expected answers are recorded for all sixteen planned languages, empty for thirteen, fr for Python, =ei for Ruby, r for Rust. - two fuzz targets with countLoopGeneric as the oracle, seeded from examples/language. Eight of the thirteen languages to come have no corpus anywhere, so this is the only differential they will get. - a test that the seven condition eligibility guard is honoured, one condition at a time, including that --trace still writes the same log. Verified non vacuous: removing the guard fails all seven. - direct tests of the shared spine, covering the nested block comment that no language counted today has and the bulk skip's line accounting. - the anchor selection method, recorded and reproducible against a corpus. Fixtures for the five languages with no sample at all, plus a plain .cs and a plain .rb, since the fuzz seeds and the corpus walk fail on the same set. Verified: 846 C and C Header files and 3275 Java files counted both ways with complexity on and off, zero disagreements, and whole CLI output byte identical with the flag on and off under --no-complexity, --complexity, --cognitive-complexity, --duplicates and --no-large. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GmrexsiiZT5HbrieEh3mHe

JavaScript and TypeScript scored identical source differently. foo?.bar was 0 as JavaScript and 1 as TypeScript, and a switch case was one lower as JavaScript, for the same bytes. JavaScript spelled ?. ?? and ??= as prefix checks. A prefix check requires !isIdentifierContinue on the byte in front of it, and ?. is essentially always written tight against an identifier, so the entry could never fire. ?? and ??= carried a trailing space, so a??b missed as well. TypeScript has the three as postfix checks, which have no such test and do fire. JavaScript also had no case check at all, which TypeScript, Java and C Header all have. So JavaScript takes the postfix form and gains case, which is the direction that makes a dead entry live rather than deleting a live one with extra steps. foo?.bar 0 -> 1 fn?.() 0 -> 1 obj?.["k"] 0 -> 1 a??b 0 -> 1 a??=b 0 -> 1 case 1: 0 -> 1 This changes scc's default output for every JavaScript file, not just those counted with --exp-per-language-counters, which is why it is its own commit and why it lands before the JavaScript counter is written. Encoding the divergence into two counters would freeze it there. TypeScript is untouched. On vuejs/core complexity moves from 344 to 357 across 36 JavaScript files while TypeScript's 10623 does not move, and no line, code, comment or blank count changes anywhere. Note for later, not fixed here: while( and switch( are missing from both languages, and neither has a try or catch check. That is under counting rather than divergence and wants its own argument. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GmrexsiiZT5HbrieEh3mHe

…ivision Phase 1. The counter itself is Java with a third quote and a different check list, which the shared spine of Phase 0 makes small. The part worth reading is M16. Anchors are f w l c | & = ?, each the rarest byte of its check measured over 19.8MB of React: f for if and for at 1.07% against i at 3.22%, w for while and switch at 0.52% against s at 3.31%, l for else at 2.45% against e at 7.68%, c for case at 2.44%. No check holds another's anchor in a position where reading back from it matches, so the scan carries on through a matched token rather than stepping over it. Stop rate falls from 24.9% of all bytes to 13.1% on React and from 25.4% to 13.0% on three.js. M16 is the reason the counter is worth writing. A slash in JavaScript opens a regular expression or divides, and which it does depends on the token in front of it, which no trie can carry. The generic loop reads every slash as a division, so a quote inside a pattern opens a string that never closes and swallows the rest of the file, and a slash pair inside one opens a comment. const packageVersionMatch = manifest.match( /\[workspace\.package\]\nversion = "([^"]+)"/ ); The generic loop opens a string on the first quote of that pattern and never recovers: 45 code and 4 blank against the counter's 40 and 9, and 2 complexity against 7, because everything below was inside a phantom string. Counted over 5610 files of React and three.js the fix moves 47 of them, 0.83%. Every one was read by hand and in every one the generic loop is the wrong answer. The eligibility flag is jsRegexLiterals, on, and only a test turns it off. The divergence rule of the spec assumes a divergence is a named input the differential can skip, and M16 is not that: it fires wherever a pattern holds a quote, a comment opener or a check token. So the corpus differential runs twice, once with the fix off where exact agreement with the generic loop is still required and holds on all 5610 files with complexity on and off, and once with it on where the disagreements are counted and reported. One thing arrived at by accident and left in on purpose. < and > are operators a slash can follow, so the closing tag of a JSX element reads as a pattern opening and the element text between two tags is skipped rather than scanned. Over those 5610 files it never made a count worse, and it makes some much better: the generic loop reads the apostrophe in element text, as in doesn't, as opening a string that eats the rest of the file. Written down at jsRegexOperators rather than left to be rediscovered. Benchmark NOT recorded. The machine carried a load average of 110 while this was measured, so the three corpora gave 1.26x, 1.37x and 1.49x with 25% to 62% variance and the figure is not worth writing down. The counter beats the generic loop on every corpus and every run, which is the direction the ship criterion asks for, but the margin has to be re-measured on a quiet machine before it is recorded. The fixtures under examples/linejudge/ are reconstructions written from the case descriptions in the spec, not the suite's own files, which are not checked out here. They reproduce the counts the spec records and pin both answers, so the divergence is asserted rather than tolerated, but a claim about the recorded suite score wants the real suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GmrexsiiZT5HbrieEh3mHe

Adding a language meant editing five lists that had to agree: the predicate chain in CountStats, counterSpecs, fuzzCounterLanguages, countersBoundsLanguages and the sample table in the anchor test, plus a name in the flag help. Six more counters are coming and nothing checked that any of those stayed in step. counterSpec now carries the counter itself and the extension a corpus of it is sampled by, so counterSpecs is the only place a counter is registered. Everything else reads it: - counterDispatch, a map built once, replaces the chain of per-language predicates. One lookup per file behind the guard rather than a walk, which is what §8 of the architecture doc asks for at sixteen languages. - useCCounter, useJavaCounter and useJavaScriptCounter are gone. They were the same guard written three times with the language swapped. - the fuzz target, the bounds walk and the anchor measurement all take their language list from the registry, so a new counter is covered by them without anyone remembering. The flag help enumerated the languages, which does not survive sixteen names. It now points at --list-counters, which prints the list and the count from the same registry. No behaviour change: output is byte identical to the previous commit over valkey, guava, React, examples and this repo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GmrexsiiZT5HbrieEh3mHe

scc panics with index out of range [-1] on a Brainfuck file holding a single +, a Swift file holding a single ?, and 23 other one byte inputs. Present in 4.1.0 and unrelated to any counter work: the generic loop does it. Trie.Match reports the depth its walk reached. Where the matched token runs to the very end of the slice it was handed, the walk stops for want of bytes rather than on a byte the token does not carry, so the depth comes back one short. For a token of a single byte that is zero, and every caller that steps the cursor on by offsetJump - 1 then drove the index to -1 and crashed the count. Eight languages carry a one byte complexity check and all eight crash: APL, Alchemist, Brainfuck, Factor, K, Stata, Swift and jq. Rust and Zen C spell theirs as a postfix check, which is a different path and was never affected. Clamped with max(offsetJump, 1) at the nine sites that step over a matched token, which is the idiom prepareString already uses a few lines away for exactly this reason. Measured: 25 of the 35 single byte inputs that the language database says are a complexity check crashed before, none crash now, and output is byte identical over valkey, guava, React, three.js, examples and this repo. The fix changes nothing that was not already falling over. Found by the fuzzer added in the counters work, which is the first fuzz target scc has had. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GmrexsiiZT5HbrieEh3mHe

…comments Phase 2. Three languages, one quote each, which makes them the smallest counters so far. What they add is M8, block comments that count their own openers, and M14, complexity checks that are not words. Anchors, each the rarest byte of its check measured on a real corpus: Kotlin f w l y h | & = w 0.42% for when and while against i at 3.68%, y 0.71% for try against t at 5.72% Scala f w l | & = > < < 0.04% and > 0.20%, already their own first byte, w 0.62% for switch and while against s at 3.85% Swift f w l h g ? | & = ? 0.08% is the whole check, g 0.72% for guard against u and d at 2.03% No check holds another's anchor where reading back from it would match, so the scan runs on through a matched token rather than stepping over it. Kotlin's finally carries an l and a y, but else wants an e in front and try wants an r; when and while carry an h, but catch wants catc. Scala's >= and <= carry an =, but == wants an = or a ! in front, so each pair counts once. Stop rates: Kotlin 23.9% to 9.8%, Scala 22.8% to 9.6%, Swift 24.3% to 12.0%. M8 needed the nested comment state of Phase 0 rewritten, and it deserves saying plainly that it was wrong. It reset the depth to one on every call rather than carrying it across the line, and where nothing closed the comment it handed back the last newline, which sits behind closers the same call had already counted, so they were read twice. No language passed nested until now, so nothing shipped with it; the fuzzer found it on /*00000/*0\n*/0 the moment one did. The seed is checked in. Split in two rather than patched. counterCommentState is the flat one and keeps the bulk skip of tuning 8; counterNestedCommentState carries depth in and out and stops at the newline. The architecture doc says nested comments need the nearer of opener and closer and are otherwise fine, and that is not right: a depth is only meaningful at a position, so a scan that hops several openers and runs out of file cannot report both the newline it should stop on and the depth it reached there. Nested comments give up tuning 8 and keep the within-line vector scan. Written down at the function. Verified: 17,775 files counted both ways with complexity on and off, zero disagreements. Kotlin 1,376 across okhttp and Exposed, Scala 15,744 across apache/spark and scala/scala, Swift 655 across swift-nio and Alamofire. C, Java and JavaScript output is byte identical to the previous commit, so the flat path did not move. Not one of those 17,775 files holds a nested block comment. M8 has no differential coverage anywhere and rests entirely on the hand written tables and the fuzzer, which is worth knowing before Rust in Phase 7 nests as well. Benchmarks written and deliberately not run: the machine is loaded and every figure it gives is noise. No ratio is claimed for these three yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GmrexsiiZT5HbrieEh3mHe

Phase 3. Two languages, one extra quote each, which is M3 and M4 in their simplest form: C# verbatim @" with ignoreEscape and a two byte start, Go's backtick raw string with ignoreEscape and a one byte start. Anchors, each the rarest byte of its check measured over 256MB of roslyn and 92MB of the Go standard library: C# f w l | & = @ w 0.32% for switch and while against s at 3.25%, f 0.73% for if, for and foreach against i at 4.99%, @ 0.04%, the rarest byte in the language, opens verbatim Go g f w l | & = w 0.39% for switch against s at 2.83%, g 1.08% for go against o at 2.61% C# carries no case check and spells switch, while and else with a space only, with no bracket or brace form, so those three take spaceOpens rather than cOpens. Using cOpens would have counted a switch( the language database does not have. while holds the l of else, but else wants an e in front of its l and while has an i there. Go's select is the one anchor not chosen as the rarest byte of its own check. Its rarest is c at 1.78% against l at 1.83%, but l is already bought for else and c is not otherwise in the table, and the table is what the scan pays for rather than the individual check. else and select both carry an e in front of the l so the byte behind cannot separate them; the byte after does, s against e, and the two matches are mutually exclusive. Stop rates: C# 19.1% to 7.5%, the best ratio of the nine counters, Go 20.3% to 11.8%. Two things about the oracle, both reproduced rather than fixed, since a counter that is right where the generic loop is wrong is a counter that disagrees with it. The generic loop does not understand C# doubled quotes: with ignoreEscape set it closes at the first " of a "" pair and the second opens a fresh string, which balances on @"a""b" by accident and would not on an odd number of pairs. And Go's backtick is escape tested on the way in, so a backtick behind a backslash opens nothing even though a backslash means nothing inside a raw string. Both are in the shape tables. counterStringState needed nothing. It handles a multi byte end; a multi byte start is the code and blank state's job, and the subtle part is only where the cursor lands, which is pinned against prepareString directly rather than inferred. spaceOpens was owned by the JavaScript counter and C# wanted the identical predicate, so it moved to counters_shared.go next to cOpens and braceOpens. JavaScript output is byte identical, along with every other language, so nothing moved with it. Verified: 76,972 files counted both ways with complexity on and off, zero disagreements. C# 51,056 across roslyn and dotnet/runtime, Go 25,916 across kubernetes and the Go standard library. A further 1.1M fuzz executions across all nine counters found nothing. 05-phasing.md says Phase 3 adds M4 for both languages. Go's backtick is a one byte start, so only C# adds M4; the machinery table in 02-machinery.md lists C# alone and is the one that is right. Benchmarks written and deliberately not run while the machine is loaded. No ratio is claimed for these two yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GmrexsiiZT5HbrieEh3mHe

…regex openers Phase 4. TypeScript is JavaScript with a different check list and inherits M16; PHP adds M12, two line comment tokens. Anchors, measured over 28MB of the TypeScript compiler and 17.7MB of laravel: TypeScript f w l c | & = ! ? w 0.31% for switch and while against s at 3.51%, c 2.16% for case against a at 4.01% PHP f w l | & = w 0.40% against s at 4.20%, f 0.92% for if and for against i at 3.61% TypeScript's equality family is the one place anchoring does not apply. === holds == and !== holds !=, so two checks can match at the same position, which anchoring assumes cannot happen. The generic loop takes the longest match at the first byte and steps the cursor past it, so for those four the counter does the same thing at the same place with the same skip. ! at 0.06% is the second rarest byte in the language, so buying it costs nothing. PHP spells switch, while and else with a space only, like C#, so they take spaceOpens rather than cOpens. Stop rates: TypeScript 24.5% to 11.8%, PHP 22.2% to 10.4% on laravel and 23.5% to 11.4% on WordPress. The shared half of the two ECMAScript counters moves to counters_ecmascript.go: the three quotes, the postfix checks and the whole of M16. Each counter keeps only its stop table and its complexity matcher. And Phase 1 got one thing wrong, which this reverts. It kept < and > as bytes a regular expression may follow, so a JSX closing tag read as a pattern opening, and argued that over 5610 files it never made a count worse. That does not survive a template literal holding markup, which is ordinary in both languages: const a = `<div>${repeat(`<span></span>`, 3)}</div>` The slash of </span> sits behind a <, so the scan ran on to the slash of </div> and swallowed the backtick between them. The line then ends inside a string that is not open. The counter reported 3 code and no comment where the generic loop correctly reported 2 code and 1 comment. That is the counter being wrong, which no argument about JSX covers, so the two bytes are gone. Measured: every wrong divergence disappears and every right one is kept. Exactly 2 files of 5610 JavaScript change, both the accidental JSX ones, and both now agree with the generic loop instead of diverging unenumerated. No other language moves. The cost is that a JSX apostrophe in element text now follows the generic loop's phantom string rather than accidentally avoiding it, which is the generic loop's bug and belongs to the embedded language phase. Verified: PHP 4,991 files across laravel and WordPress, zero disagreements. TypeScript 18,811 files across the TypeScript compiler and angular, zero disagreements with the regex fix off. With it on, 77 diverge, all of them the generic loop being wrong. The worst is angular's ng_optimized_image_spec.ts, where .replace(/"/g, '') opens a phantom string: the generic loop calls 1,522 of its 2,696 lines comment and 193 blank, where the file holds 76 line comments and exactly 420 blank lines, which is what the counter reports. A further 1.3M fuzz executions across all eleven counters found nothing. Benchmarks written and deliberately not run while the machine is loaded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GmrexsiiZT5HbrieEh3mHe

…Escape Phase 5. C++ and C++ Header have byte identical checks, quotes, line_comment, multi_line and linesplice, and differ only in extensions and C++ Header's eight heuristics, so this is one counter registered twice. Anchors f w l y h | & =, each the rarest byte of its check over 20MB of protobuf and googletest: w 0.25% for while and switch against s at 3.78%, y 0.62% for try against t at 5.47%, h 0.73% for catch against c at 1.76%. The result worth writing down is that anchoring works on a string opening and not only on a keyword. Every raw string starts with R, u, U or L, and those four are 3.32% of a C++ file, half as much again as all eight complexity anchors put together. So a raw string is anchored on the quote that ends its opening token, which the scan already stops on at 0.63%, and the prefix is read backwards. R, u, U and L never enter the table at all. That needed the layer 2 conformance test widened, and it rightly failed the first attempt: assertion 2 only exempted the first byte of a complexity check, not of a quote. counterSpec now carries QuoteAnchors, assertion 2 exempts a declared anchored quote, and assertion 3 holds it to being in the stop table, to being a byte of the token, and to being a real quote of the language. A new raw form added to languages.json still fails the build. Stop rates: C++ 24.1% to 9.8%, C++ Header 23.2% to 9.3%. This is the phase that changes code which already ships, per the architecture doc §7.1. countLoopC hardcoded ignoreEscape to false at line end accounting, which is right for C and wrong for C++, since C++ has both linesplice and five raw quotes. countLoopShared's linesplice bool becomes a spliceRule carrying an optional pointer to the flag, the C++ counter owns the bool in its closure and writes it whenever it opens a string, and resetCounterLineState reads it at each line end. The flag has to be per string rather than per language, which §7.1 does not say: C++ has plain quotes too and a single per-language bool would be wrong for those. It is load-bearing, checked by removing it: a raw string wrapping blank lines then reports them blank, where they are code. Two of the four span-lines fixtures still passed without it by coincidence, so the blank-lines-inside-a-raw -string case is the one that actually pins it. Verified. The eleven existing counters are byte identical between the previous commit and this one across eighteen corpora, so threading the flag moved nothing. C++ itself: 3,698 files by extension with zero disagreements across protobuf, googletest, nlohmann/json and opencv, and 10,524 rows with zero differences using real detection, which matters because protobuf alone splits its .h files into 380 C++ Header, 196 C Header and 116 Objective C. 150 corpus files hold a real raw string. 47 hand written shapes cover every encoding prefix, a delimiter at 16 bytes and at 17, and a raw string holding its own closer. 40 malformed raw strings, including truncations at end of file and nul bytes inside a delimiter, produce no panic and no disagreement, which matters because the delimiter is read out of the file and is attacker controlled. 1.44M fuzz executions with raw string seeds found nothing. 02-machinery.md §3 does not record that the fall back matters as much as the success: where no delimiter is readable the generic loop still opens a string and closes it at )" with ignoreEscape set, so R" bad runs to end of file rather than behaving like a plain quote. Three fixtures depend on that. Benchmarks written and deliberately not run while the machine is loaded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GmrexsiiZT5HbrieEh3mHe

Phase 6. Ruby adds M11, block comment delimiters of more than one byte, and is the first of the three languages where anchoring is not sound for free. Anchors f w l | & =, each the rarest byte of its check over 19.3MB of rails: f 1.18% for if and for against i at 3.82%, w 0.56% for switch and while against s at 4.50%, l 2.83% for else against e at nearly 9%. ! at 0.06% is rarer than = at 0.58% and != would be cheaper anchored on it, but == has to be anchored on an = and = has to be in the table regardless because it opens =begin, so reading != back from its = costs nothing and keeps ! out. Stop rate 24.5% to 10.4% on rails, 22.2% to 10.3% on brew. The architecture doc names Ruby's colliding bytes as =, e and i, and offers two ways out: anchor the colliding checks on their first byte and pay the stops, or prove the backwards read cannot cross a token boundary. This proves it, in four parts, written out at buildRubyStop. 1. e and i are never anchors, so the scan never stops on one and never begins a read at one. 2. A read can never begin inside =begin or =end. Every byte of them but the leading = is b, e, g, i, n or d, none an anchor; the leading = is, but the code state tests =begin before any check and the two are mutually exclusive on the byte after, b against = or !. 3. A read can never reach in from outside. Every read is one byte deep plus the word boundary: f wants an i, w an s, l an e, = an = or a !. =begin ends in n and =end in d, neither of which any check carries behind its anchor. The case that gets furthest is =belse, where the l really does have an e behind it and only the word boundary on the b stops it. 4. Nothing inside a comment is read at all: the comment state finds =end with bytes.Index and runs no matcher. Asserted rather than only argued. A test walks every anchor placed behind each delimiter plus the named near misses, 54 cases. The one thing that makes Ruby different from every other counter: = stays in the --no-complexity stop table. Everywhere else the anchors are dropped wholesale under that flag, but Ruby's = also opens =begin, and a scan that stopped stopping on it would read a whole block comment as code. It has its own test. Two things about the oracle, reproduced rather than fixed. The generic loop imposes no first column rule, so an indented =begin and one sitting after code both open a comment, where real Ruby wants column one. And there is no heredoc token at all, so <<~SQL and its body are ordinary code and a quote inside one opens a string. Both wrong about Ruby, both right about the oracle. Verified: 5,980 files across rails, brew and jekyll with zero disagreements, complexity on and off, and 6,025 rows with real detection. 73 hand written shapes, 15 adversarial delimiter cases of my own, and 2.25M fuzz executions, none disagreeing. The fourteen other counters are byte identical across thirteen corpora. Worth recording that M11 has almost no corpus coverage: one file in 5,980 holds =begin at a line start. Like the nested comments of Phase 2 it rests on the hand written tables and the fuzzer, which matters for Python, whose docstrings are its entire block comment mechanism. The collision table of 03-architecture §2.1 is a good alarm but far more conservative than the real hazard: it intersects every byte a check is spelled with, where only the anchor bytes and the one or two a read touches can matter. For Ruby that reduces =ei to = alone, and = is handled by ordering. Benchmarks written and deliberately not run while the machine is loaded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GmrexsiiZT5HbrieEh3mHe

…a lifetime Phase 7, and the hardest one: M3, M4, M5, M8, M13 and M15 together. Twenty quote forms, nineteen multi byte starts up to br########" at eleven bytes, sixteen multi byte ends, nested block comments, a postfix ? with its ?Sized exclusion, and the character literal fix. Anchors f w p l h | & = ?, each the rarest byte of its check over 145MB of rust-lang/rust: w 0.42% for while, h 0.98% for match against m at 1.46%, p 1.43% for loop against l at 2.25%, f 1.46% for if and for. loop and match are anchored on their last byte, which besides the frequency keeps them out of the line start check set, since that only has to carry checks anchored on their own first byte. Rust is the second of the three collision languages, on r, and the argument is written out in four parts at buildRustStop. The short form is that the fast choice and the safe choice turned out to be the same choice. Raw strings are anchored on the quote that ends their opening token, exactly as C++ does, so r is in neither stop table: it is never an anchor, so the scan never stops on one and never begins a read at one, which dissolves the collision outright. The one read that touches an r walks back over at most eight hashes to exactly one r and no further, clamped to floor. And finding an r inside a word is not a mistake, because neither loop asks for a word boundary in front of a quote, so for" opens a raw string in both and they agree by construction. Quote anchoring is worth more here than anywhere else: r is 3.16% of a Rust file and b 0.82%, so 3.98% saved against 7.37% for all nine complexity anchors put together. Complexity stop rate 20.9% to 12.3%, with the quote saving on top and not modelled by that measurement. M15 is the conformance prize. languages.json gives Rust b' but no plain ', so '"' lets the inner quote open a string that never closes. Adding a plain ' would fix LineJudge 4010 and 4020 and break 4050, which scc passes today, because a lifetime 'a would then open a string. The counter tells them apart on what follows: one byte and a closing quote, or an escape, is a literal; identifier bytes with no closing quote is a lifetime. 4050 is the regression test and has one of its own. Gated behind rustCharLiterals the way the regex fix is gated, so the corpus differential runs once with it off requiring exact agreement and once with it on reporting what moved. A bug found on the way, and it is not a divergence. ' newline ' was read as a character literal holding a newline, which on rust-analyzer's lexer fixtures swallowed line endings: char_literals.rs came out as 3 lines against the file's 47. A wrong line count is not something the divergence rule covers at all, and the spec never says so; it governs code, comment and blank and is silent on Lines. Bounded the read and rejected a newline in both the plain and escape paths. Verified. 39,898 files across rust-lang/rust, tokio, ripgrep and serde: zero disagreements with M15 off, complexity on and off, and with it on 118 diverge, 0.30%, every one attributable to M15 by construction. Not one file anywhere changes its Lines count, which I checked field by field across all 39,898. The worst divergence is tidy/src/triagebot.rs, which holds exactly 27 line comments and 23 blank lines: the counter reports 27 and 23, the generic loop 120 and 9, because trim_matches('"') opens a phantom string. 2.75M fuzz executions, 47 hand written shapes, 36 raw string bounds cases. The fifteen other counters are byte identical across fifteen corpora. Rust enumerates its raw quotes per hash count rather than setting Delimited, so rawStringEnd stays C++ only and is not reused here, which a reader might assume from M4 and M5 sitting next to M6. Benchmarks written and deliberately not run while the machine is loaded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GmrexsiiZT5HbrieEh3mHe

Phase 8. Python adds M4, M5, M7 and M10: ten quote forms, eight multi byte starts, six multi byte ends, six docstrings, and no block comment at all, so docstrings are the entire comment mechanism beyond the hash. Anchors f w l h y x d o, each the rarest byte of its check over 42MB of cpython: w 0.45% for while and with, x 0.58% for except against c at 1.89%, y 0.59% for try and finally against t at 4.58%, h 0.96% for match. Three checks share f and two share y; if and elif both carry an i behind the f, so elif is tested first and the two are mutually exclusive, if wanting a word boundary in front of its i which the l of elif is not. Stop rate 29.1% to 17.0% on cpython, 31.9% to 17.8% on django. Python is the third and last collision language, on f and r, and it dissolves the way Rust's did: every prefixed string is caught on the quote that follows its prefix rather than on the prefix itself, so r is in neither stop table and f is there only as a complexity anchor. The four part argument is at buildPythonStop. Twenty adversarial cases of the forr""" deff""" finallyr"x" shape all agree. SDocString is the one part of the generic loop no counter modelled, and it needed three things from the shared spine. One was a live trap: resetState maps SDocString to SBlank through its default, because the generic loop never calls it on that state, so routing a docstring through it would have turned every line after the first into a blank. Every counter now resets through resetCounterState. The docstring state is the generic one with the byte loop taken out: bytes.Index to the closer, bytes.Count for the newlines, bulkLines to tally them. The jump loops over candidate closers rather than taking the first, since a closer behind a backslash does not close, but each hop is a vector scan. A triple quote opened after code goes through the shared string state instead and is not skipped, which is correct but slower; most triple quoted bytes in real Python are docstrings, so the hot case is covered. The debug gap of 03-architecture §6 is closed the way it recommends. The counter emits both of docStringState's lines and Debug stays out of the eligibility guard, so --debug no longer silently selects a different loop. Verified by capturing stdout across six shapes. Three things the spec does not record. The blank state is where quote anchoring does not reach, and only Python notices: every other counter can defer a prefixed string to the code state a byte later because both produce SString, while Python's blank produces SDocString and its code produces SString, and that difference is M7 itself. So the blank state looks one byte forwards where the code scan looks one byte backwards. Second, prepareString and the trie disagree about which quote matched: the trie takes the longest token and prepareString the first flagged quote in database order, so r"""""" counts 1 code where f"""""" and """""" count 1 comment. Reproduced and pinned. Third, docStringState never consults ignoreEscape, so a backslash before the closer stops even a raw docstring closing. Leading indentation measures 20.9% of cpython against the 16.8% the spec predicted, so the blank run skip matters more than Phase 0 expected. Verified: 5,865 files across cpython, django, numpy and flask with zero disagreements, complexity on and off, and 6,144 rows with real detection. Not one of 5,867 rows changes its Lines count. 2.23M fuzz executions, 73 hand written shapes, 20 collision cases. The fifteen other counters are byte identical across 23 corpora. That is sixteen counters. Benchmarks are written for every one of them and none has been run: the machine has carried an unrelated load throughout, and a ratio measured on it would be noise. No counter has a recorded margin yet, so none of them has passed the ship criterion of 00-goal.md §3. They stay behind --exp-per-language-counters until that measurement is made on a quiet machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GmrexsiiZT5HbrieEh3mHe

scc made two read calls per file where mezura made one, 127,454 against 63,918 over the Linux kernel. Reported by mezura's author with a syscall table in #769, along with the approach taken here. The read loop ran until it was told zero, because a short read does not prove the end of a file: a filesystem may chop a read anywhere, and stopping at the first short one would make the count depend on the filesystem underneath. The buffer is always readSlack bytes longer than the file, so the second read was always the one that returned zero. What does prove the end is the FIRST read coming up short while still delivering everything the stat promised, since a file read whole in one call has plainly ended. Every read after the first is read to a zero-length read as before: a file arriving in pieces is one whose pieces say nothing about where it ends, and a chop can land exactly on the size of a file that has since grown. Restricting it to the first read is not a detail. The obvious form of this, breaking on any short read once the stat's promise is met, truncates a 32KB file to 16KB when the stat under-reports and the reads are chopped, which is the third of those three conditions. TestReadFileShortReadsAreNotTheEnd caught it. A stat size of zero promises nothing and buys no shortcut, so /proc and sysfs still read to the end. TestReadFileSyntheticFile covers that with /proc/kallsyms, which exists only on Linux and skips everywhere else, so the new test uses a fifo instead: it stats as zero, hands back only what has been written, and every read of it is short. Four claims about the size are run against it, covering a file that grew, one that shrank, and one that never had a size at all. No measurable wall clock change on macOS: interleaved with warm caches, rust, three.js, opencv and cpython all land within noise of where they were. A read that returns zero from the page cache costs almost nothing, and scc already spent less system time per file than mezura, 7.6us against 11.0us. The syscall count is what halves. Whether that shows up as time is a question for Linux, cold caches and network filesystems. Output is byte identical across seventeen corpora. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GmrexsiiZT5HbrieEh3mHe

countLoopGeneric switched on the state of every byte and had no case for SComment or SCommentCode. Nothing in a line comment can move the state, so every byte of one walked the whole loop body - the bounds check, the isBlankRun lookup, the isWhitespace test, the Cognitive global, the switch dispatch - to arrive at the same state it started in and do nothing. Instrumenting the loop over llvm-project showed how much of the work that was: 543,345,194 iterations over 1,147,253,434 bytes, 2.11 bytes an iteration, and 485,750,380 of those iterations - 89.4% - were in SComment, with another 4.1% in SCommentCode. Nine tenths of the loop ran to do nothing at all. LLVM IR is the reason, since it comments with ; and llvm/test is written in it. The fix is the one the loop already uses for a run of blanks: find the end and carry on from there. IndexByte to the newline, or to endPoint where the file ends without one. Where the case sits matters more than what it does. Written as an else-if beside the isBlankRun skip it cost 2% on corpora with few line comments, because every byte then paid for a test that almost never fired, and spelling it as an unsigned range check did not help. Inside the switch it is free: states 1 to 8 are dense, so the compiler lays the switch out as a jump table and the new case costs a table entry rather than a comparison. On the Linux kernel it is very slightly positive for that reason. byteType is the one caller that cannot have the skip. ContentByteType wants a classification recorded for every byte of the file, so the skip is guarded on it being nil, the same guard the isBlankRun skip carries. TestClassifyContentMixedLine catches it if the guard goes. Measured on llvm-project, GOMAXPROCS=1, CPU seconds, interleaved with the arms rotated: 5.07 to 3.45, 1.47x. The win is where the comments are. On cpython and ruby it is 1.05x, on the Linux kernel 1.01x, on kubernetes and lucene nothing, because /* */ is bulk scanned by commentState already and those trees do not carry the line comment volume llvm does. Output is byte identical on llvm-project, linux, kubernetes, cpython, ruby and lucene, summary and per-file, with and without the counters. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

…d eighteenth The sixteen counters were chosen by language popularity. These two were chosen by where the generic loop was actually spending its time on a tree that had been measured, which turns out to be a different question. On llvm-project scc reads 1851MB, and 1094MB of it had no counter. LLVM IR is 720MB of that across 47,714 files, Assembly another 208MB. Sizing them by bytes said both were worth about the same. Sizing them by what the generic loop does with those bytes said otherwise, and the second number is the one that predicted the result. What costs the generic loop is not bytes but stops. LLVM IR declares sixteen complexity checks and nine of them begin with a letter common in the language - l for llvm.loop and lshr, b for br, s for switch and shl, i for indirectbr, c for callbr and catchswitch and cleanupret, r for resume, a for ashr and and, o for or, x for xor - so a naive stop set covers 23.75% of a .ll file and the loop stops on a quarter of every one before it has done any work. Anchoring the checks onto the seven rarer bytes behind them - w k b h m r a - takes that to 6.08%. and and or have no rare byte anywhere in them, a at 2.24% and r at 2.00%, and the two of them are 4.24 of the 6.08 points that are left. They are the floor. Measured over llvm-project the stop rate falls 25.4% to 9.0%. Assembly has the same small token set and gains far less: 14.2% naive to 8.2% anchored, and its files are 36% whitespace that both loops skip a vector at a time. It is 1.14x where LLVM IR is 3.94x. It is here because it was written while the argument for LLVM IR was being checked, not because it earns its place the way the other one does. Assembly's checks are not C's, though they are spelled from the same words. languages.json gives Assembly switch, while and else once each with a trailing space where C gives each of them twice, once with a space and once with the bracket or brace, so while(a) and else{a} are not checks here and the counter takes spaceOpens rather than cOpens and braceOpens. Assembly comments with ; alone - // opens nothing - and carries both " and ' as quotes. LLVM IR has no block comment at all, which leaves three of the shared states unreachable. Neither language has an anchoring collision: no byte that opens a check is also a delimiter byte, so the backwards reads need no argument beyond the one the shared helpers already make. Measured on llvm-project, GOMAXPROCS=1, CPU seconds, interleaved: 5.07 to 3.29 for LLVM IR alone, 1.54x. Assembly adds 1.03x on top. In the counting loop alone LLVM IR is 3.94x generic, 330MB/s to 1300MB/s, against the 1.6x to 2.05x the sixteen average. Every .ll file of llvm-project, 47,714 of them, agrees with the generic loop, and again with complexity off. Assembly agrees over 13,230 .s and 327 .S of llvm-project and 1,359 .S of the Linux kernel. Both fuzz against the generic loop as oracle, 4.66M and 4.16M executions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

…for them Three of the 366 languages carry heuristics - C++ Header, Objective C and Qt Translation Source - and they exist to tell .h apart between C, C++ and Objective-C, and .m between MATLAB and Objective-C. On llvm-project that is 19,731 files, and the profile put 0.88s of 8.74s in regexp on their account. For a question with three possible answers that is a great deal of work. It is not the size of what is searched. detector.go already caps toCheck at 20,000 bytes and the mean over those files is 3,604. It is not the literal pre-check either, which costs 68ms for the whole corpus and is accurate: where a pre-check passes the regex goes on to match 97% of the time, because LLVM's headers really are C++ and the regex really does have to run. It is that the patterns cannot be searched for efficiently. Compare std::\w+ at 1.3us a run against ^[ \t]*(try|constexpr) at 37.9us. The first has a literal prefix, so the engine memchrs for std:: and tries the match where it lands. A pattern that opens ^\s* or ^[ \t]* has no literal prefix at all, so the engine has nowhere to skip to and retries at every one of the 3,604 byte offsets instead of the 120 line starts where a line-anchored pattern could possibly begin. The six anchored heuristics were 414ms of the 451ms. So give them somewhere to start. At plan build time (?m)^\s*REST becomes (?m)\A(?:REST), which makes the engine try exactly one offset, and it is run at the line starts the plan's own walk already visits, at the positions where one of the heuristic's declared literals sits. The pre-check and the regex become one operation over one pass rather than two. Per pattern, over all 21,915 ambiguous files, with the hit count identical in every case: template 624ms to 23.8ms, try|constexpr 814ms to 34.2ms, catch 841ms to 32.7ms, class|namespace 402ms to 20.1ms, public: 514ms to 25.7ms, the Objective-C alternation 896ms to 71.9ms. Twelve to twenty-six times each. The rewrite leans on literals meaning "one of these is where the pattern begins", which is a little stronger than the pre-check's "one of these is present at some line start", so it fires only on the two lead shapes it recognises and falls back to the ordinary search for anything else: a rest that opens \b, \B, \A, \z or \Z, an empty literal, or one that starts with whitespace the lead would have skipped. #include <cstdint|string|...> is declared anchored:false and keeps the ordinary search, since its literals sit anywhere on the line. Measured on llvm-project, GOMAXPROCS=1, CPU seconds, both arms in one binary so code layout is held still: 1.10x on its own. It is worth more as the counting loop gets faster, 1.15x on top of the LLVM IR counter and the comment skip, because the 0.42s it saves is fixed while what it is a fraction of shrinks. On the Linux kernel it is nothing, as it should be: those headers are C, the pre-check rejects them and no regex ran in the first place. Byte identical across six corpora, 296,599 files. detector_anchored_test.go cross-checks the anchored walk against the original regex over a built corpus and 400 randomly assembled files, 8,208 answers, and holds anchoredForm to refusing the shapes it does not understand. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

TestReadFileShortReadsAreNotTheEnd failed four runs in ten on the commit that added it, always the same case: stats short, the file grew and the reads are chopped. It read 16384, or 20480, or 28672 bytes of a 32768 byte fifo and called it the end. Two things were wrong, one in the comment and one in the claim underneath it. The comment said the buffer is always larger than the size by readSlack, so a short first read must have hit the end. It is not. ReadFile hands readFileInto a pooled buffer at reader.buf[:cap(reader.buf)], rounded up to a power of two, so for a size of 16384 the buffer is 32768 rather than 16896 and n < wanted holds for any first read from the size up to nearly twice it. That is the spread of byte counts in the failures, and it is why the failure moved around: which chop arrives first is a race with the writer. The claim was that restricting the rule to the first read removes the case where a chop lands on the size of a file that has since grown. It does not. The first read is as able to land there as any later one, and when it does, every condition on the shortcut holds and the read stops early. That case cannot be fixed while the change exists, which is the point worth writing down. A short read that has met the stat's promise is indistinguishable from the end of the file; separating them takes another read, and not taking another read is the whole of the optimisation. So the comment now says what the rule actually rests on - a regular file returns less than was asked for only at the end of it - and says plainly what it does not cover and who could meet it, which is a network filesystem serving a file that is growing. scc cannot: size comes from a stat of a regular file, and every source that chops reads stats as zero and is turned away by size > 0. The test keeps the case and asserts what is actually promised for it, that whatever comes back is a correct prefix and never wrong bytes. The other three still require the whole file. Twelve runs, no failures. Worth knowing when weighing this: the change this guards bought no measurable time. Interleaved, 25 reps, GOMAXPROCS=1, warm cache on Linux, the halved read count moved neither user nor system time out of the noise on the Linux kernel or on llvm-project. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

Copilot

🟡 Changes recommended

Two critical and one moderate unresolved findings remain, along with a documentation nit.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds specialized per-language counters, parser and detector optimizations, Unix read handling changes, and extensive regression coverage.

Changes:

Adds counter dispatch, listing, conformance, and fuzz tests.
Optimizes parsing, anchored detection, and file reads.
Adds language examples and regression fixtures.

File summaries

File	Summary
`processor/workers.go`	Integrates specialized counters and parser fast paths.
`processor/testdata/fuzz/FuzzSpecialisedCounter/e34bc4a414f3ddab`	Adds a fuzz regression input.
`processor/testdata/fuzz/FuzzSpecialisedCounter/c18522695dec4f60`	Adds a fuzz regression input.
`processor/processor.go`	Handles counter-listing commands.
`processor/parser_onebyte_test.go`	Tests one-byte EOF handling.
`processor/parser_edge_test.go`	Tests comment and EOF boundaries.
`processor/filereader_unix.go`	Optimizes Unix short-read termination. Critical (3 votes): size-based early exit can truncate short reads.
`processor/filereader_chunked_test.go`	Tests chunked FIFO reads.
`processor/detector.go`	Uses anchored heuristic results.
`processor/detector_plan.go`	Adds anchored heuristic planning. Moderate (2 votes): form-feed/vertical-tab indentation can produce false matches.
`processor/detector_plan_test.go`	Tests heuristic literal screening.
`processor/detector_anchored_test.go`	Tests anchored heuristic equivalence.
`processor/counters_typescript_test.go`	Tests the TypeScript counter.
`processor/counters_swift.go`	Implements Swift counter support.
`processor/counters_swift_test.go`	Tests the Swift counter.
`processor/counters_scala.go`	Implements Scala counter support.
`processor/counters_scala_test.go`	Tests the Scala counter.
`processor/counters_ruby_test.go`	Tests the Ruby counter.
`processor/counters_python_test.go`	Tests the Python counter.
`processor/counters_php.go`	Implements PHP counter support.
`processor/counters_php_test.go`	Tests the PHP counter.
`processor/counters_llvmir_test.go`	Tests the LLVM IR counter.
`processor/counters_kotlin.go`	Implements Kotlin counter support.
`processor/counters_kotlin_test.go`	Tests the Kotlin counter.
`processor/counters_java_test.go`	Tests the Java counter.
`processor/counters_guard_test.go`	Tests counter eligibility guards.
`processor/counters_go_test.go`	Tests the Go counter.
`processor/counters_fuzz_test.go`	Differential-fuzzes counters.
`processor/counters_ecmascript.go`	Implements ECMAScript counter support. Critical (1 vote): escaped newlines can allow regex literals to cross lines.
`processor/counters_csharp_test.go`	Tests the C# counter.
`processor/counters_cpp_test.go`	Tests the C++ counter.
`processor/counters_cpp_bounds_test.go`	Tests C++ counter bounds.
`processor/counters_conformance_test.go`	Tests counter conformance.
`processor/counters_c_test.go`	Tests the C counter.
`processor/counters_bounds_test.go`	Tests counter bounds.
`processor/counters_assembly_test.go`	Tests the Assembly counter.
`processor/counters_anchor_test.go`	Tests counter anchors.
`processor/constants.go`	Updates counter-related constants.
`main_test.go`	Covers command behavior.
`languages.json`	Updates language metadata.
`examples/linejudge/7020-regex_holding_a_comment_opener.ts`	Adds a TypeScript regression fixture.
`examples/linejudge/7020-regex_holding_a_comment_opener.js`	Adds a JavaScript regression fixture.
`examples/linejudge/7010-regex_literal_holding_a_quote.ts`	Adds a TypeScript regression fixture.
`examples/linejudge/7010-regex_literal_holding_a_quote.js`	Adds a JavaScript regression fixture.
`examples/linejudge/4020-two_char_literals_holding_escapes.rs`	Adds a Rust regression fixture.
`examples/linejudge/4010-char_literal_holding_a_quote.rs`	Adds a Rust regression fixture.
`examples/language/swift.swift`	Adds a Swift language example.
`examples/language/scala.scala`	Adds a Scala language example.
`examples/language/rust.rs`	Adds a Rust language example.
`examples/language/ruby.rb`	Adds a Ruby language example.
`examples/language/php.php`	Adds a PHP language example.
`examples/language/kotlin.kt`	Adds a Kotlin language example.
`examples/language/csharp.cs`	Adds a C# language example.
`config.go`	Registers counter-related CLI flags. Nit (1 vote): checked-in README help is stale.

Review details

Files not reviewed (1)

processor/constants.go: Generated file

Suppressed comments (2)

config.go:356

This adds a public CLI flag and expands the specialized-counter registry, but the checked-in README help remains stale: it omits --list-counters and still says --exp-per-language-counters only covers C, C Header, and Java. Please regenerate/update the README command reference so users can discover the new flag and do not receive an incorrect counter list.

	flags.BoolVar(boolVar(&processor.SpecialisedCounters), "exp-per-language-counters", false, "experimental, may change or be removed: count with the scanner written for the file's language where there is one, see --list-counters")
	flags.BoolVar(boolVar(&processor.ListCounters), "list-counters", false, "print the languages that have a scanner of their own and exit")

processor/detector_plan.go:480

scanLineStarts skips a superset of the whitespace accepted by the rewritten ^[ \t]* forms (isAnchorSkip also includes \r, \f, and \v). Because anchoredForm removes the prefix and the regex is then matched at j, a line such as \fclass Foo is reported as an anchored C++/Objective-C heuristic even though the original regex cannot match it. Preserve and validate the original indentation prefix (or use the exact whitespace class for each anchored form) before setting hit, otherwise language detection can change on form-feed/vertical-tab-indented files.

Files reviewed: 69/70 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

ecmaRegexEnd walked a candidate pattern and skipped the byte behind every backslash without looking at it. A backslash sitting immediately before a newline therefore stepped over the newline, the case that rejects a pattern for crossing a line never saw it, and the search carried on down the file until it found a slash on some later line and called everything in between one pattern. A regular expression literal may not cross a line, and a line terminator is not a thing a backslash can escape, so there was never a pattern here to find. What it cost was a line of the count: var re = /foo\ bar/; if (x) { y(); } The generic loop answers three lines and three of code. The counter answered two and two, having swallowed the first newline as an escaped byte, which makes it a disagreement with the generic loop and so a bug in the counter by the rule the other seventeen are held to. The newline is now tested before it is skipped. Reported by Copilot on the pull request, which had it as hiding the code and complexity on the following line; that is the smaller half of it, since the line itself goes missing too. No corpus moves: output is identical on kubernetes, lucene, llvm-project, linux, cpython and ruby, the JavaScript and TypeScript corpus differentials still agree, and FuzzSpecialisedCounter found nothing in 4.07M executions. The shape is invalid JavaScript, so real code does not carry it, and the table entry in TestJavaScriptRegexLiterals fails without the fix and passes with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

This reverts the read loop of 4c83231 and the test weakening of 8d31470, and keeps TestReadFileShortReadsAreNotTheEnd with all four cases demanding the whole file, which is what they were written to demand. The rule was that a first read coming up short of the buffer while still delivering everything the stat promised has reached the end of the file. It does not hold. The first read is as able as any later one to land at or past a size the file has since grown beyond, and when it does, every condition on the shortcut is met and the read stops early. The test caught it on the commit that added it, failing eight runs in thirty, returning 16384, 20480 or 24576 bytes of a 32768 byte fifo. 8d31470 read that as unfixable and changed the case to assert a prefix instead. Both halves of that were wrong. It is fixable: fstat on the descriptor already open says whether the thing being read is a regular file and how large it is now, which is exactly the information a short read does not carry, and sixty runs of the strict test pass with it. And a prefix is not what ReadFile promises - processFile counts whatever comes back as the whole file - so the assertion could no longer fail at all, on a truncation of any length including none. A test that cannot fail replaced one that had already found a real defect twice. The reason for reverting rather than taking the fstat fix is that neither is worth having. Measured on this machine, an fstat on an open descriptor costs about 124ns against about 102ns for the end-of-file read it would replace, so the correct fix is slower than not optimising at all: 972ns a file against 925ns for this revert, where the unsound version was 830ns. The whole budget is around 95ns a file, some six milliseconds across the sixty-odd thousand files of the Linux kernel, spread over thirty-two threads. Interleaved, twenty-five reps, GOMAXPROCS=1 and warm caches, it moved neither user nor system time out of the noise on the kernel or on llvm-project, which is what the original commit already said about macOS and what the syscall table in #769 does not translate into. The exposure was small but real and silent. newFileJobGuessed rejects anything that is not a regular file and size > 0 turns away the fifos and most of /proc, so scc's own path is nearly always safe. What is left is network and userspace filesystems, where a short read on a regular file is ordinary and a file growing between the stat and the read is ordinary too, and a 512 byte window either side of the kernel's MAX_RW_COUNT cap on a local one. The result in each case is a file counted short with no error raised, which for a counting tool is the worst way to be wrong. Output is byte identical on llvm-project, linux, kubernetes, cpython, ruby and lucene. read is back to 179,670 calls on the kernel from 90,248. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

… what it skipped ebe6fc8 turned the line walk from something that proposed a candidate into the thing that decides, and left the skip table alone. The table was written for the old job, and its comment says so: skipping more than a pattern accepts "would only ever make the prefilter say possible where the regex will say no, which costs a regex run and never a wrong answer". That stopped being true the moment the walk started answering, and the comment is what should have been read again. anchorSkip steps over a space, a tab, a carriage return, a form feed and a vertical tab. Neither lead accepts all five. [ \t]* accepts the first two. \s* accepts four of them, because Go's \s is space, tab, newline, carriage return and form feed and leaves the vertical tab out. So the walk would step over a form feed, land on class, try the anchored pattern there and settle a heuristic at a position its pattern could never have begun at. Three header files show it, and they are in the property test now: int x;\n \f class Foo {\n};\n C Header became C++ Header int x;\n \r class Foo {\n};\n C Header became C++ Header int x;\n \v template <typename T>… C Header became C++ Header The divergence only ever invents a match, never loses one, so a .h or .m file could be pushed toward C++ Header or Objective C and never away. Nothing in llvm-project, linux, kubernetes or cpython carries the shape - form feeds appear in real headers, GNU page-break style, but alone on their own line, which does not diverge - which is why 296,599 files came back clean and the bug shipped. The walk now records what it stepped over and each slot is settled only where its own lead accepts it. Deleting the three bytes from the table instead would have traded these invented matches for lost ones on the \s* lead, which is the worse direction. TestAnchoredFormAgreesWithTheSearch missed it because its piece list has \r\n but no lone \r, no \f and no \v, so no file it built could tell an over-skip from a legal one. Adding those three bytes makes it fail without this fix and pass with it. Found by an audit of ebe6fc8 rather than by the corpus, which had nothing to say here. Output is byte identical on all six corpora, both against the commit before the heuristics change and against the commit after it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

Copilot

🔵 Needs a closer look

Three moderate review findings remain unresolved.

Review details

Files not reviewed (1)

processor/constants.go: Generated file

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

processor/detector_plan.go:489

Because j is advanced by the shared isAnchorSkip loop before this check, rewritten ^[ \t]*... heuristics can be evaluated after bytes they do not allow. A line beginning with \vprivate: (or \rprivate:/\fprivate:) fails the original C++ heuristic but reaches private here, and the rewritten \A regex then reports a false language-detection hit. The line walk needs an indentation set tied to each anchored lead, or these [ \t]* forms must stay on the ordinary search path.

processor/counters_ecmascript.go:104

When a regex literal starts immediately after a newline (or after indentation on a new line), ecmaRegexAllowed walks back to that newline and consults this table. Because \n is absent, it classifies the slash as division and never skips the literal, so contents such as /["']/ can open a string and contents such as /[/*]/ can open a comment. Treat the line boundary as a context where a regex may start, as the tsBlankState/jsBlankState callers already expect.

	for _, b := range []byte("(,=:[!&|?{};+-*%~^") {

processor/counters_rust.go:404

This bound excludes the final byte from character-literal recognition. CountStats passes endPoint == len(content)-1, so a valid Rust literal such as '? has its closing quote at endPoint and is rejected; the code then reaches ? and counts it as postfix complexity. Extend the inspection limit through endPoint (while still clamping to len(content)) and add an EOF character-literal regression case.

Files reviewed: 68/69 changed files
Comments generated: 0 new
Review effort level: Lite

…eir names An audit of 4433956 found no counting bug in either new counter, over a reference matcher run against every prefix and suffix of every check, thirty million fuzz executions on an adversarial seed corpus and three hundred and sixty thousand mutations of real files. What it found instead was that several of the things the commit asserts are not actually asserted anywhere. The hand written tables compare the counter against the generic loop, which is the rule the eighteen are held to and catches a counter that has gone wrong. It cannot catch both loops being wrong together, and cases named "catchswitch does not also count switch" or "while with a bracket is not a check" read as a claim about a number while testing only an agreement: each passes just as well if the pair of them answer two, or nothing at all. counters_values_test.go pins the twenty-two of those where counting a token twice, or not at all, would be a silent wrong answer rather than a disagreement. Both loops answer them today. One case was testing something other than its name. "llvm.loop has no trailing space" put llvm.loop inside a double quoted string, so it never reached the matcher and counted nothing: it tested the string state. It is now a line that actually spells llvm.loop with no space behind it, and the string version is kept under a name that says what it does. The bounds walks were not extended when the anchors were. Their alphabets are chosen deliberately - the comment says h and y are the anchors Java and Kotlin read furthest back from - and neither carried k or m, which are invoke and llvm.loop, the two LLVM IR arms that read furthest back, nor the dot that llvm.loop is spelled with. So the walk never put those arms at a position where the read runs off the front of the file, which is the one thing those tests exist to do. Adding k, m, the dot and a percent finds nothing, which is the answer that was wanted rather than the answer that was assumed. asmBlankState asked asmString whether the quote in front of it was escaped, and fell back to code when it was. blankState in the generic loop opens the string unconditionally, and so does every sibling counter including llvmBlankState in this same commit. The question cannot be answered differently here - the blank state is only entered with a newline, a space, a tab, a carriage return, the slash that closed a block comment or the BOM in front of the byte, never a backslash - so it was a difference from the oracle that the entry conditions happened to hide, and it is now spelled the way the other seventeen spell it. Output is byte identical on all six corpora. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

An audit of 3323c82 ran about a hundred and fifty million differential counts against the generic loop, including every byte string up to seven long over a per-language alphabet and the same again concentrated on lines that open a comment, and found no disagreement. It found two things about the code around it instead. TestClassifyContentCommentOnly asked only whether the line held any comment byte at all. The two bytes of the // are classified before the loop reaches the comment state, so the assertion held with the byteType guard taken out and the whole body of the comment left unclassified. Removing the guard now fails it alongside TestClassifyContentMixedLine, where before it failed only the latter. Its siblings for multiline comments and strings already assert every byte, so this is the shape the file already had. The else branch of the skip does not change an answer - dropping it re-enters the case on each following byte and arrives at the same place - and exists to save the iterations. It is also the one line in the loop that puts index somewhere the walk did not, and it is only safe because endPoint is Bytes-1, so landing on it lands on the last byte and the line-end test takes it. Were endPoint ever to become len(content) the branch would jump the loop past its own bound and the count would stop early with no error raised. The audit's mutation of exactly that produced a silent undercount, so the dependency is written down. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

scc -d answered a different number every time it ran. Ten runs over cpython gave ten different line counts, 3,044,453 to 3,044,625, from the same binary on the same tree. Not a race in the duplicate map, which is properly locked: a race in what the map was being asked about. The digest was built inside codeState, a byte at a time, from the bytes that survived shouldForeach's mask, so it covered the code and not the comments or the whitespace. The comment above it said as much and reasoned that it did not matter, since two duplicates skip the same way. That holds. What does not hold is the converse, and the converse is what the check needs: two files that are not duplicates can skip the same way too. Any pair with the same code and different comments hashed alike, so one of them was dropped, and which one depended on the worker that reached the lock first. They do not count alike, so the totals moved. Hashing the whole file makes a duplicate set a set of files that are actually identical. Identical files count identically, so it stops mattering which one survives and the answer stops moving: ten runs, one answer, on cpython, ruby and kubernetes, with every worker still running. It is also faster, which was not the point but is worth having. The old digest allocated a one byte slice and made a hash.Write call for every code byte of every file; one Sum256 over the content replaces all of that, and the write is gone from the hot loop along with the Duplicates test that guarded it. With -d on llvm-project: 0.81s to 0.44s wall, 22.26s to 10.36s of CPU. The other way of fixing this, one worker of each kind so the arrival order is fixed, was measured at thirteen times slower - llvm-project going from 0.96s to 12.50s - which buys determinism by giving up the thing the tool is for. Fewer files are removed now, because fewer files were ever duplicates: cpython drops 133 where it used to drop 801, kubernetes 6,327 where it used to drop 6,838. The ones that came back are the pairs that shared their code and differed somewhere the old digest could not see. Output with -d is unchanged in every other respect and output without -d is byte identical on llvm-project, linux, kubernetes, cpython, ruby and lucene. One thing is still arrival-ordered, and is not fixed here: --by-file with -d lists whichever of an identical set was counted first, so the row names a different path from run to run while every number on it stays the same. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

…cted Five audits went over the branch. Three of the report's figures were wrong, two were quoted with more confidence than they had earned, and one claim about mezura was simply false. This is the corrected version, re-measured on an idle machine. What changed from the first draft: TableGen is not part of the explanation. The draft said mezura recognises neither LLVM IR, MLIR nor TableGen, so those three account for the byte gap. scc does not recognise TableGen either - languages.json has no entry and the 1,815 .td files of the tree count as zero bytes in both tools - so it explains none of the difference. LLVM IR and MLIR are 64% of the gap, not all of it; the rest is Plain Text, JSON, Markdown, YAML and mezura excluding generated files and honouring .gitignore. The per-byte comparison now says which clock it is on. 1.58x slower before and 1.40x faster after are user CPU. On wall clock scc was already faster per byte than mezura before any of this, because it parallelises better. Both pairs are internally consistent; the draft quoted one without naming it, which is how it came to look like it contradicted its own headline. The headline ratios are re-measured: 1.51x for the comment skip where the draft said 1.47x and an auditor said 1.57x, 1.57x for the counters, 1.10x for the heuristics, 1.84x for all three. Non-additivity holds at 43%. cpython and ruby are not 1.05x. They are 1.01 and 1.03 with error bars that cover them, and kubernetes is 1.02 rather than nothing. The draft measured them with /usr/bin/time, which resolves 10ms - one tick on a 200ms run is 5%. "No win on the kernel" is right for wall clock and wrong for CPU, where there is a small win of one to two percent that is lower in eleven of eleven paired reps. The anchor stop-rate percentages in the LLVM IR counter's commit message came from a 32MB alphabetical prefix, mostly llvm/test, and do not hold over the corpus. The ranking they justify does hold. The numbers are flagged rather than restated, since re-measuring them is its own job. The read-syscall section is rewritten because 4c83231 has been reverted. It halved the read count and bought no measurable time, and its rule was unsound: the first read can land at or past a stale size. An fstat on the open descriptor fixes it in twenty lines, and is slower than reverting, because an fstat costs more than the end-of-file read it would replace. Two measurement lessons are written down at the end, both learned the expensive way. Separately built binaries differ by about 10% on this codebase from code layout alone, so anything smaller has to be measured inside one binary. And a corpus cannot find a bug that no file in it contains: 296,599 files agreeing proved less than three hand-built files disagreeing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

Thirty-two of the 366 languages declare no comment, no string and no complexity check: Plain Text, Markdown, JSON, JSONL, CSV, ReStructuredText, AsciiDoc, MDX, Patch, Jupyter, SPDX and the rest of the formats that are read rather than executed. Nothing in one of their files can change the state of the scan, so counting one is deciding, for each line, whether it holds a byte that is not whitespace. The generic loop arrives at the same answer the long way, walking every byte through a state machine, a blank-run skip and a trie with nothing in it to match. IndexByte finds the end of each line a vector at a time and the line is read only as far as its first byte that is not whitespace, which on a line of prose or of json is the first byte. A blank line is read to its end, and a blank line is short. It is not a counter in the sense the eighteen are. There is no language in it, nothing to get wrong per language and no anchor argument to make, so one function answers for all thirty-two and for any language that later declares nothing either. It is reached on ProcessMask being zero, which is the same thing said in the form the loop already holds, so a language that gains a comment leaves the path on its own. What it is worth is CPU rather than wall clock, and the profile says why plainly: with the path in, no counting function appears in the top sixteen of a run over nothing but these files. What is left is the directory walk at 50%, the syscalls at 43%, gitignore matching at 21% and reading the files at 21%. They are small files, so the per-file cost of finding and opening one dominates what it costs to count it. Over kubernetes restricted to these extensions, 61MB: 183ms of user CPU to 132ms, and the wall clock moves 31.9ms to 28.3ms, which is inside its own error bars. Whole trees, where these files are a quarter of the bytes and far less of the work: kubernetes 763ms of CPU to 701ms, cpython 305ms to 272ms, both with no wall clock change at all. It is an energy saving, not a speed one, and it is behind --exp-per-language-counters with the others. The nul is the only thing here that needed care, and the generic loop sees fewer of them than it looks. blankState carries no binary check, so the first byte of a line that is not whitespace is never tested however it is spelled, and codeState, which does test, is entered on the bytes after that one. Its walk also stops at endPoint, so the last byte of a file is never tested: cpython carries a csv fuzz corpus ending \n\n\0 that is counted rather than dropped, and a path that checks the whole file loses it. Both limits are in the search rather than reasoned about at the call site, and the hand written table has four nul cases holding them there. Output is byte identical on llvm-project, linux, kubernetes, cpython, ruby, lucene and an Obsidian vault that is 95% markdown, per-file and summary, with the counters on and off. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

Picks up boyter/gocodewalker#28, written against the theories this branch's profiling pointed at: the walk was half the time of a run over a tree of markdown and json, and inside it the gitignore matching was about a third. Measured here, walk alone, with every file excluded so nothing is counted: kubernetes 26.1ms to 19.7ms 1.33x user 98.1ms to 67.7ms llvm-project 77.7ms to 64.1ms 1.21x user 369ms to 298ms linux 43.3ms to 36.7ms 1.18x user 227ms to 183ms Whole runs with -c --exp-per-language-counters, where the walk is a smaller share of the work: linux 1.08x, kubernetes 1.06x, llvm-project 1.06x, cpython 1.03x. Those four are inside the ten percent band that separately built binaries drift by on this codebase, so the direction being the same in all four is the evidence, not any one of them. The walk numbers are outside it and stand on their own. It also drops danwakefield/fnmatch, which the common gitignore patterns no longer need, so scc carries one fewer dependency. Output is byte identical on llvm-project, linux, kubernetes, cpython, ruby, lucene and an Obsidian vault, per-file and summary, which is the thing worth checking: every one of these changes is in the code that decides whether a file is walked at all, so a wrong answer here does not show as a slower run but as a file that is counted or skipped when it should not be. No file changed its decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

getExtension held a sync.Map keyed on the file name, so it kept one entry per distinct name in the tree rather than one per extension. On the trees where that matters it therefore missed most of the time: llvm-project has 146,245 distinct names across 184,219 files and the Linux kernel 78,049 across 99,112, so four lookups in five found nothing. Each of those paid for a load, the work the cache was avoiding, and a store, and both map operations boxed a string into an interface on the way, which allocates. What was left behind was 146,245 entries held for the life of the process. What it stood in front of is two backwards scans and, for a name carrying two extensions, one small concatenation. A name with a single extension returns a slice of itself and allocates nothing. A profile of the walk over llvm-project put getExtension at 23% of the run, and under it sync.Map.Store at 43%, the boxing at 29% and sync.Map.Load at 29% - almost none of it the extension itself. Taking the cache out moves the walk's user CPU by 21% on llvm-project, 15% on the kernel and 8% on kubernetes, which is close enough to what the profile said to believe the attribution. Wall clock barely moves: 1.02x to 1.03x on a walk and 1.02x on a whole run, all inside the band that separately built binaries drift by here. The walk runs on every core and waits on syscalls, so CPU given back does not come out as time. It is the allocations and the retained map that are worth having. Keyed on the extension instead there would be about a thousand entries and almost every lookup would hit, but there is nothing left to cache: the work is cheaper than the lookup that would front it. Output is byte identical on llvm-project, linux, kubernetes, cpython, ruby, lucene and an Obsidian vault. The existing table of cases for getExtension - single extension, none, leading dot, two extensions, second pass - is unchanged and still passes, which is what pins the behaviour here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJMG14V2aPAAt4jRALJN3P

boyter and others added 17 commits September 14, 2026 10:58

pr-insights Bot added VH/complexity Very high complexity XL/size Extra large change labels Sep 14, 2026

boyter requested a lite review from Copilot September 14, 2026 22:49

Copilot started reviewing on behalf of boyter September 14, 2026 22:50 View session

Copilot AI reviewed Sep 14, 2026

View reviewed changes

Comment thread processor/counters_ecmascript.go

Comment thread processor/filereader_unix.go Outdated

boyter requested a lite review from Copilot September 14, 2026 23:08

Copilot started reviewing on behalf of boyter September 14, 2026 23:09 View session

boyter and others added 2 commits September 15, 2026 09:15

Copilot AI reviewed Sep 14, 2026

View reviewed changes

boyter and others added 2 commits September 15, 2026 09:31

boyter marked this pull request as ready for review September 14, 2026 23:42

boyter and others added 5 commits September 15, 2026 10:19

boyter merged commit 7d52303 into master Sep 15, 2026
3 of 4 checks passed

boyter deleted the per-language-counters-spec branch September 15, 2026 05:35

Navigation Menu

Uh oh!

Per language counters spec - #776

Per language counters spec#776
boyter merged 27 commits into
masterfrom
per-language-counters-spec

boyter commented Sep 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

boyter commented Sep 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Files not reviewed (1)

Suppressed comments (2)

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Files not reviewed (1)

Suppressed comments (3)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants