Skip to content

docs: design the micro-compiler architecture for the compilers - #158

Merged
OmarAlJarrah merged 5 commits into
mainfrom
docs/micro-compiler-architecture
Jul 30, 2026
Merged

docs: design the micro-compiler architecture for the compilers#158
OmarAlJarrah merged 5 commits into
mainfrom
docs/micro-compiler-architecture

Conversation

@OmarAlJarrah

@OmarAlJarrahOmarAlJarrah commented Jul 30, 2026

Copy link
Copy Markdown
Member

Summary

The OpenAPI compiler is one Go package whose files are a filing convention rather than a boundary,
and a single struct has accumulated every lowering concern. The per-file split did not reduce that;
the method count grew. The individual functions are not the problem — no body in the package exceeds
50 lines against a 70-line cap. The failure is type surface, which no rule in force measures.

This adds two documents. docs/micro-compiler-design.md is the architecture: every lowering step
becomes a function over an explicit position, returning its diagnostics instead of accumulating
them, with type interning as the single shared effect, and concerns become packages so the compiler
enforces the boundary. docs/micro-compiler-plan.md is the order the work lands in and what each
step waits on; the detail for each unit lives in its issue rather than being duplicated, and the
blocking relationships are recorded as issue dependencies so the next available piece of work can be
queried rather than inferred.

Three findings surfaced while writing them, each verified against the tree rather than inferred:

  • The canonical naming grammar is implemented three times across the OpenAPI, GraphQL and Protobuf
    compilers. Two copies are identical; the third treats . as a word separator and the other two do
    not. Naming.Canonical is part of the IR contract, so the same field means different things
    depending on which compiler produced it, and irverify checks only casing — never segmentation —
    so both spellings pass.
  • The architecture test's prefix matching treats an allowlist entry for compilers as licence for
    compilers/graphql, so one compiler may import another today with nothing to catch it.
  • The general two-order diff the testing rules prescribe does not exist. It is present as three
    hand-written cases at the site where the pointer-collision bug was found, and the determinism check
    that resembles its general form recompiles the same bytes, which is a different property.
    Separately, nothing anywhere asserts that two distinct source constructs cannot mint the same type
    ID.

Each is dispositioned rather than noted: the first two are prerequisites, and the naming divergence
is filed on its own since it is a live contract violation regardless of whether this work proceeds.

The documents are tracked under docs/ rather than kept as working notes because the previous design
for this area was written to an ignored path, could not be committed, and had to be recovered from
the history of a since-deleted file.

Test plan

Documentation only — no Go files are touched, so no code path changes and the gate is unaffected:

git diff --name-only origin/main...HEAD | grep -c '\.go$' # 0

Every derivation command in the design document was executed against a095636 and its output
matches the surrounding claim; the documents give the command rather than the number wherever a
count would otherwise rot. Every issue referenced in the plan was checked to resolve to the issue it
is described as.

The OpenAPI compiler is one Go package whose ~20 files are a filing
convention rather than a boundary, and one struct has accumulated 151
methods. The per-file split did not reduce that count; it grew. The
individual functions are not the problem, since no body in the package
exceeds 50 lines against a 70-line cap. The failure is type surface,
which no rule in force measures.
This records the architecture that replaces it: every lowering step is a
pure function over an explicit position, returning its diagnostics rather
than accumulating them, with interning as the single shared effect.
Concerns become packages so the compiler enforces the boundary, since
file boundaries demonstrably do not.
It also records three findings that shape the work. The canonical naming
grammar is written three times across the OpenAPI, GraphQL and Protobuf
compilers, two copies identical and one behaviourally divergent, while
the verifier that guards neutral naming checks only casing and is silent
on segmentation. The architecture test's prefix matching lets one
compiler import another undetected. And the general two-order diff the
testing rules prescribe does not exist, which matters because moving
interning is exactly what order dependence is sensitive to.
The document is tracked under docs/ rather than left as a working note,
because the previous design for this work was lost to an ignored path and
had to be recovered from the history of a deleted file.
The first draft asserted a shape without specifying it. This settles the
parts a reader would otherwise have to invent: what a micro-compiler is,
what Ctx looks like, which packages exist and which way they import, and
what each step has to prove before it lands.
Testing carried the least weight and now carries the most, because most
of this work is proving behaviour did not change rather than changing it.
Two oracles the repository lacks are specified here. The general
two-order diff exists only as three hand-written cases at the site where
the pointer-collision bug was found, and the determinism check that looks
like its general form recompiles the same bytes, which is a different
property. Nothing at all asserts that two distinct source constructs
cannot mint the same type ID, and moving lowering between packages is
exactly when an ID derivation gets rewritten.
Three claims from the first draft were wrong and are corrected. The
annotation readers cannot move without the site type they read, which is
declared alongside reference resolution. Promoting the naming grammar is
not behaviour-neutral, since the three copies disagree and at most one
survives unchanged. And constraints carries lowerer methods, so it joins
a package during the conversion phase rather than founding one.
Also records the blind spots that must not be mistaken for coverage: the
harness returns before the invariant checks when a fixture trips an error
diagnostic, the annotation matrix cannot address a carrier position, and
the verifier checks that provenance is in range but never that a pointer
is correct.
The design says what to build; this says what order it lands in and what
each step waits on. The detail for each unit of work lives in its issue
rather than being repeated here, and the blocking relationships are
recorded as real issue dependencies rather than only as prose, so the
next available piece of work can be queried instead of inferred.
Three things gate everything else and are worth stating plainly. The
architecture test cannot currently enforce compiler isolation, so the
package boundaries this rests on are unenforceable until it can. The
two-order diff and the type-ID collision check both have to exist and be
proven against the current code before any lowering moves, because a
guard written during a refactor and first observed passing has
demonstrated nothing.
The lint caps land last rather than first. They are calibrated against
the finished shape, and enforcing them earlier would only encode the
shape being replaced.
@OmarAlJarrahOmarAlJarrah added the documentation Improvements or additions to documentation label Jul 30, 2026
An audit of the filed work against the design found four issues already
in the backlog that land inside this restructuring rather than beside it,
and one piece of work with no issue at all.
The provenance report is the one worth naming. It proposes a method on
the lowerer, which the restructuring removes, so its direction no longer
holds — but its problem gets sharper, because hand-built provenance is
the single change here with nothing behind it: the verifier checks that
a source index is in range and never that a pointer is right. The
reference-resolution cleanup and the test-scaffolding report are likewise
reframed rather than dropped, and the golden line-ending gap is noted
because byte equality is what proves roughly twenty pull requests neutral.
The missing work was documentation. Both the architecture document and
CLAUDE.md carry a package-layout tree that this effort invalidates, and
CLAUDE.md states the documents are the spec and are read first, so a
diagram that no longer matches the tree misleads rather than merely ages.
Also unblocks the naming-divergence defect, which had been made to wait
on the promotion that would fix it. That contradicted the reason it was
filed separately: it ships today and must not depend on this programme
running to completion.
Provenance pointer correctness was the one property in the restructuring
with nothing behind it. The verifier checks that a source index is in
range and never that a pointer is right, so a signature change passing
the wrong coordinate produced a node that was wrong but valid, and only a
human reading a golden diff would notice.
It does not have to be that way. Every named entity derives its ID from
its source pointer, so an ID and its own provenance must agree, and that
is exactly checkable. The type-ID oracle now carries both assertions: an
ID minted from the wrong pointer disagrees with its provenance, and a
grammar that collapses two pointers breaks injectivity. Neither implies
the other and one traversal gets both.
Where it lives is left open deliberately. The verifier imports only the
IR while the grammar is headed for the compiler framework, so the harness
is the fallback. The better answer is to decide whether the identifier
shape belongs in the IR itself, which would enforce it for every compiler
rather than only under test — that belongs with the promotion, not here.
The golden line-ending gap is promoted from background cleanup to a
prerequisite, blocking both entry points. Byte equality is the sole proof
of neutrality across all fifty-nine snapshots, and a comparison that can
fail environmentally teaches a reader to dismiss golden diffs at the one
point where every diff has to be read as a finding.
@OmarAlJarrah
OmarAlJarrah merged commit 40f6e06 into mainJul 30, 2026
1 check passed
@OmarAlJarrah
OmarAlJarrah deleted the docs/micro-compiler-architecture branch July 30, 2026 11:55
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@OmarAlJarrah