fix(compiler): reach the nesting limit instead of overflowing on the way to it

[Guikingone]

Fixes #686.

A deeply nested array literal aborted the compiler process:

$ python3 -c "d=200; print('<?php'); print('\\$a = ' + '['*d + '1' + ']'*d + ';')" > deep.php
$ elephc deep.php
thread 'main' has overflowed its stack
fatal runtime error: stack overflow, aborting

No diagnostic, no file, no position -- on source PHP itself compiles and runs.

The limit already existed

reject_excessive_nesting caps bracket depth at MAX_COMPILER_NESTING = 1024 and
reports "maximum compiler nesting depth exceeded". The cap was unreachable: every
recursive AST pass costs one frame per level -- the parser, fold_expr, the
magic-constant walker, Checker::infer_type, the optimizer's rewriters, EIR lowering
-- and the default 8 MiB main stack ran out around 140.

So the guard written to make hostile nesting a diagnostic could only ever fire for
input the compiler had already survived. Measured before:

depth  130   compiles
depth  200   STACK OVERFLOW      <- abort
depth 1000   STACK OVERFLOW      <- abort
depth 1100   "maximum compiler nesting depth exceeded"

The fix is the stack, not the limit

Lowering the cap to what the stack survives would reject valid PHP: php-src compiles
1000 levels without complaint, and 130 is not a defensible line. A compile now runs
inside compiler_stack::with_compiler_stack, which gives it COMPILER_STACK_BYTES
(256 MiB) -- sized against the compiler's OWN limit, reserved rather than committed,
so an ordinary compile faults in only the pages its depth actually touches.

Measured after:

depth  200   compiles, prints 1
depth 1000   compiles, prints 1
depth 1024   compiles, prints 1
depth 1100   "maximum compiler nesting depth exceeded"

Both depths were run and produce host PHP 8.5.10's answer.

A mapping, not a thread

with_compiler_stack is stacker::maybe_grow: a memory mapping obtained on demand,
with the same reserve-don't-commit behaviour a thread stack has. The first cut of this
PR spawned a 256 MiB thread instead, and review found two things wrong with that.

• The spawn could fail. The thread version fell back to running inline on the
  caller's small stack -- silently re-arming the abort this PR exists to remove. A
  mapping has no spawn, so the failure path is gone rather than patched.
• A thread could only wrap main. It needs Send + 'static and gives back
  nothing, so the in-process path could not use it. maybe_grow is generic over the
  return type and needs no Send, so the same wrapper works around anything.

A panic also unwinds through the caller normally instead of having to be caught and
resumed.

The budget sits on the PHASE

The second review round found the in-process regression could not tell whether the
per-pass guards were still there: it ran through a harness that wrapped the whole
pipeline. The isolated test it asked for -- one phase at a time, on a 256 KiB
thread, a size the fixtures set themselves so RUST_MIN_STACK cannot hide anything --
found that they were not enough anyway:

• fold_constants overflowed in prelude_prune::usage::scan_expr, reached through
  superglobals::seed_cli_populated_superglobals -- a recursive walker inside the
  "guarded" phase, with no guard of its own;
• types::check and propagate_constants overflowed in
  conditional::exprs::rewrite_expr, another phase entirely.

A phase is not one walker, and guarding every recursive Expr walker is not the fix:
a scan finds around 180 of them, the next pass added reintroduces the bug, and no test
can prove the absence.

So each recursive PHASE now wraps itself, at the single function its spellings funnel
through: the parser, conditional::apply, magic-constant substitution, name
resolution, constant folding, check_types_with_options (which every check*
reaches), constant propagation, the three control-flow passes, reachability pruning
and EIR program lowering. An embedder calling one phase on a small worker thread is
safe without knowing this exists. The per-walker guards and the shared
parser::grow_stack_for_recursion are gone.

COMPILER_STACK_HEADROOM -- half the reservation -- keeps that free on the normal
path: main wraps the whole compile, so the phases inside it see plenty of headroom
and use the run's own stack instead of reserving one each.

A driver wraps too, for what it does BETWEEN phases

main and the codegen harness keep their own wrapper, and the reason is now written
down rather than assumed: removing the harness's aborts the 1024-level fixture in
ExprKind::clone. That is not a phase. It is what a driver does between phases with
an AST that deep -- moving it, cloning it, dropping it -- recursing through derived
Clone and Drop code no guard can be placed inside.

Verified

nesting_below_the_limit_compiles_and_runs compiles a 1000-level literal through the
BINARY and runs it, asserting both that stderr mentions no stack overflow and that the
program prints 1. The existing deeply_nested_source_reports_compiler_depth_limit
still gets its diagnostic at 20_000 levels, so the two ends of the limit are pinned
together.

test_deeply_nested_literal_compiles_and_runs_in_process compiles AND RUNS 1024 levels
through the whole in-process pipeline -- not a type check, which is what the first cut
pinned and which stops before six recursive passes.

tests/embedder_stack_tests.rs pins the phases one at a time on the small thread. Four
of its five fixtures abort with has overflowed its stack when with_compiler_stack is
reduced to calling its body directly. The fifth, the parser, passes either way at that
size -- measured, with brackets and with parentheses -- which matches the issue, whose
aborts were always in the passes below; it is documented as the guard it is rather than
as a reproduction.

cargo test --test codegen_tests -- codegen::regressions codegen::types codegen::optimizer passes (1022), as do 1915 unit tests, 1534 error tests and all 6
compiler security limit tests.

Docs

docs/internals/the-parser.md already documented the cap; it now also says why the cap
needed the stack to be reachable, that every recursive phase carries its own budget and
which ones they are, and that a driver wraps itself for the AST handling between phases
-- with the ExprKind::clone abort as the reason. No docs/php/ or examples/ change:
nothing about what PHP code MEANS has changed, and an example would have to be a
1024-level literal.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KSAAWPyNBq6dP2b5puN3wr

[greptile-apps]

Greptile Summary

This PR prevents valid, deeply nested PHP source from overflowing the compiler stack before reaching the existing nesting limit.

• Runs the binary and in-process compiler drivers with a reserved 256 MiB stack budget.
• Applies the same budget independently at recursive phase entry points for embedders.
• Adds binary, full-pipeline, and isolated small-thread regression coverage.
• Documents the phase-level and driver-level stack guarantees.

Confidence Score: 5/5

The PR appears safe to merge; the previously reported stack-safety and regression-coverage gaps are resolved without a new actionable issue.

The compiler now supplies sufficient stack at both driver and recursive-phase boundaries, and the focused regressions cover binary compilation, the complete in-process pipeline, standalone phase invocation, and deep-value lifetime behavior. All previous findings were either fully fixed and conceded or manually resolved, and no outstanding merge-safety failure remains.

Important Files Changed

Filename	Overview
src/compiler_stack.rs	Defines the shared reserved-stack budget and generic wrapper used by compiler drivers and recursive phases.
src/parser/mod.rs	Preserves the existing nesting rejection before parsing accepted input on the compiler stack.
src/optimize.rs	Wraps the public recursive optimizer phase entry points so standalone embedder calls receive sufficient stack.
src/types/checker/mod.rs	Establishes the stack budget at the common type-checking entry point.
src/ir_lower/program.rs	Runs EIR program lowering inside the phase-level compiler stack wrapper.
tests/embedder_stack_tests.rs	Exercises standalone recursive phases and deep-value destruction on explicitly constrained worker stacks.
tests/compiler_security_limits_tests.rs	Verifies below-limit nesting compiles through the binary while the existing excessive-depth diagnostic remains reachable.
tests/codegen/regressions/syntax_edges.rs	Verifies a deeply nested program completes the full in-process compilation and execution pipeline.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[PHP source] --> B[Parser nesting-limit check]
    B -->|Above 1024| C[Compiler depth diagnostic]
    B -->|Within limit| D[Phase entry]
    D --> E{Enough stack headroom?}
    E -->|Yes| F[Reuse current compiler stack]
    E -->|No| G[Reserve compiler stack mapping]
    F --> H[Recursive compiler phase]
    G --> H
    H --> I[Next phase or compiled output]

Loading

_{Reviews (6): Last reviewed commit: "docs(compiler): correct the stack sizes ..." | Re-trigger Greptile}