Skip to content

Python: add unit tests for rst_code_example_pipeline - #1369

Draft
gusthoff wants to merge 222 commits into
AdaCore:mainfrom
gusthoff:dev/topic/infrastructure/python/rst-pipeline-unit-tests/2026-06-19/main
Draft

Python: add unit tests for rst_code_example_pipeline#1369
gusthoff wants to merge 222 commits into
AdaCore:mainfrom
gusthoff:dev/topic/infrastructure/python/rst-pipeline-unit-tests/2026-06-19/main

Conversation

@gusthoff

@gusthoff gusthoff commented Jun 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

The package shipped with only a smoke-test suite covering imports and
--help exit codes. This adds a pytest suite — 478 tests at 100% branch
coverage — and fixes the reporting defects those tests exposed, chiefly
conditions the checker met and passed over in silence.

Changes

  • Tests: 13 modules covering every source module, from pure helpers to
    real Ada/C compilation, run and SPARK proof, plus end-to-end tests running
    the installed console scripts as real processes.
  • Infrastructure: pytest and coverage configured in pyproject.toml
    (branch = true, fail_under = 99, --strict-markers); a toolchain
    marker partitioning the suite; two make targets; a pytest job added to
    the package's CI workflow.
  • Reporting fixes: failures the checker used to swallow are now reported
    and fail the run — most importantly an example marked "this should fail"
    that succeeds instead.
  • Code-block classes: a run class is now honored only for a block in the
    language it names, and one naming the other language is reported rather
    than quietly ignored. No block in content/ is affected.
  • constants.py: one home for the artifact filenames and the eighteen
    class names previously compared as bare strings across the package.
  • Docs: test instructions, the package's exit-status contract, and four
    undocumented code-block classes.
  • Version: bumped to 0.3.0.

Testing

478 passed, 4 xfailed, 100% branch coverage (1081 statements, 350 branches);
pyright clean. Assertions were verified by injecting defects and confirming
the intended test reddens. Separately validated against the real corpus: the
content driver run twice over the whole course set, swapping only this
package, produced identical results over 1742 code blocks.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

@gusthoff
gusthoff marked this pull request as draft June 20, 2026 02:04
@gusthoff
gusthoff force-pushed the dev/topic/infrastructure/python/rst-pipeline-unit-tests/2026-06-19/main branch from bc3a655 to 953c63d Compare July 25, 2026 01:08
gusthoff and others added 27 commits September 18, 2026 22:37
@gusthoff @claude
Python: add pytest + pytest-cov as optional test dependencies
766b70c 
Add an optional-dependency group `[test]` to pyproject.toml so the
test toolchain can be installed with:

    pip install -e ".[test]"

This installs pytest and pytest-cov without making them mandatory
for users who only need the pipeline itself.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: configure pytest and coverage in pyproject.toml
86e6668 
Add [tool.pytest.ini_options] to set testpaths and default addopts
(coverage measurement + term-missing report). Add [tool.coverage.run]
with branch coverage enabled, and [tool.coverage.report] requiring
≥90% coverage and showing missing lines.

Running `pytest` from the package root now measures coverage
automatically without extra command-line flags.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Makefile: add test_rst_pipeline target for epub VM
dee9d29 
Add a new target that runs the rst_code_example_pipeline unit test
suite via pytest. Coverage options and testpaths are configured in
the package's pyproject.toml, so the target only needs to cd to
the package directory and invoke pytest.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add unit tests for colors.py
64d9c07 
Tests cover Colors class ANSI escape sequence attributes, col() with
colors enabled and disabled, printcol() output, the no_colors() context
manager (disable inside, restore outside, nested use), disable_colors(),
CI/non-TTY detection, and adversarial direct __enter__/__exit__ usage.

Documents known limitation: no_colors() uses a bare yield without
try/finally, so _enabled is not restored if an exception propagates
out of the with-block.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Docs: document how to install test deps and run the pytest suite
5bcb0a5 
Add a "Development" section to the package README covering:
- how to install test extras with `pip install -e ".[test]"`
- how to run pytest from the package root (plain `pytest`)
- the Ada toolchain (GNAT) requirement for the full suite
- the explicit coverage invocation for reference

No VM-specific or build-system details in the module README.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add unit tests for fmt_utils.py
2c0cc45 
Tests cover header() (content, star underline length, return type),
error() (stdout output containing ERROR/loc/msg), simple_error() and
simple_success() (stdout output).  Adversarial cases include empty
string, Unicode with non-ASCII characters, and verifying that no output
goes to stderr.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Docs: cross-link unit test suite from root README testing section
49f2133 
After the rst_code_example_pipeline package description, add a short
paragraph pointing readers to the package's pytest suite: names the
`make test_rst_pipeline` target and links to the "Development" section
of the package README for full install and run instructions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add unit tests for resource.py
e41637c 
Tests cover the Resource constructor (basename storage, content=None,
content=[], single-element, multi-element join), the append() method,
and the content property (always returns str).  Adversarial cases
include append of empty string, append of a line with embedded newline,
a large 1000-element content list, and content=None never returning
None.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff
Merge feat/pytest-infra: pytest + coverage setup for rst_code_example…
4e5932b 
…_pipeline
@gusthoff @claude
Python: add pragma: no cover to structurally unreachable blocks
6ec64f5 
Three locations in check_block() that coverage cannot reach when imported:
- `if __name__ == '__main__':` guard inside check_block() — impossible
  when called as a module; already matched by exclude_lines, pragma is
  belt-and-suspenders
- `if False:` block (35-line dead code, structurally unreachable)
- `if True:` block (branch coverage flags the never-taken false path of
  an always-true condition)

These three pragmas bring overall package coverage from 75.43% to 76.55%.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add unit tests for toolchain_info.py
e4b1ea5 
Tests cover init_toolchain_info() populating DEFAULT_VERSION, TOOLCHAINS,
and TOOLCHAIN_PATH; get_toolchain_default_version() auto-initialising and
returning version strings for gnat/gnatprove/gprbuild; re-initialisation
idempotency; KeyError for unknown tool; and state isolation via an autouse
fixture that clears the module-level dicts before and after each test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add pragma: no cover to extract_projects ConfigBlock guard
f6a3ade 
The isinstance(block, blocks.ConfigBlock) guard at line 251 is inside a
loop over projects[project], which is built exclusively from CodeBlock
instances (ConfigBlocks are filtered out earlier in analyze_file).
The branch is structurally unreachable under normal execution.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add unit tests for toolchain_setup.py
9c53c1d 
Tests cover reset_toolchain() with no pre-existing symlinks, with symlinks
present, and for idempotency; set_toolchain() with "default" versions (no
symlinks created) and "selected" versions (symlinks created pointing to the
correct version directories); set_toolchain() followed by reset_toolchain();
and adversarial double set_toolchain() without an explicit reset in between.
An isolated_toolchain_path fixture redirects symlink creation into a tmp_path
subdirectory so /opt/ada/selected is not mutated during tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add pragma: no branch to colors TTY check
e91ce1d 
The module-level TTY check at line 39 always takes the true branch in a
pytest environment (non-TTY), making the false branch structurally
unreachable without TTY mocking or importlib.reload tricks. pragma: no
branch suppresses the missed-branch coverage arc.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add unit tests for extract_projects.py
7c4ea31 
Tests cover get_project_dir() with simple and dotted project names;
write_project_file() for all four combinations of spark_mode × main_file ×
compiler_switches; ProjectsList construction, add(), JSON round-trip, and
missing-file None return; and analyze_file() with no-check, syntax-only,
manual_chop (C), ConfigBlock, no-button, and no-project (SystemExit) cases.

A work_dir fixture uses monkeypatch.chdir() to isolate tests that write to
the filesystem.  Global module state (verbose, code_block_at, current_config)
is reset before and after each test by an autouse fixture.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add unit tests for checks.py
2704113 
Tests cover CodeCheck construction and defaults (timestamp float,
all fields None by default), BlockCheck construction (empty checks dict
regardless of parameter), add_check() accumulation, to_json_file() +
from_json_file() round-trips, and from_json_file() with a nonexistent
file returning None.

Documents known limitation: BlockCheck.__init__ always resets
self.checks to an empty dict, ignoring the 'checks' keyword argument,
so nested CodeCheck entries are lost on a JSON round-trip.

Adversarial cases include overwriting an existing file and passing an
empty JSON object ({}) which raises TypeError.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: extend unit tests for blocks.py (gnatprove/gprbuild version s…
0d37dfa 
…election)

Add two tests covering the gnatprove and gprbuild version-selection attribute
paths in get_blocks_from_rst() — lines 129 and 133 of blocks.py.  These two
branches (gnatprove_version = ["selected", ...] and gprbuild_version =
["selected", ...]) were not exercised by any existing test; the only
version-selection test used gnat=.  The new tests parse RST blocks with
gnatprove= and gprbuild= attributes and assert the resulting CodeBlock
carries ["selected", <version>] for the respective field.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add unit tests for check_projects.py
3648c30 
Tests cover get_blocks() with an empty regex list, with a valid block_info.json,
with a glob pattern, with two projects in separate subdirectories, and with a
block whose project field is None (skipped with ERROR); get_projects() without
a projects list file and with one; cwd side-effect isolation (os.chdir is called
internally — restored by an autouse fixture); a WARNING when projects_list_file
does not exist; the check_block() thin wrapper; and check_projects() integration
with a build dir containing no-check blocks.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add unit tests for blocks.py
6220fa8 
Tests cover Block.get_blocks_from_rst() for all attribute combinations
(minimal Ada block, project/main_file, compiler switches, gnat version
selection, language=c, manual_chop, buttons, :code-config: directive,
two consecutive blocks), CodeBlock derived fields (no_check,
syntax_only, run_it, compile_it, prove_it, text_hash/text_hash_short),
CodeBlock JSON round-trip, ConfigBlock construction and update(), and
adversarial paths (empty RST, nonexistent JSON file).

Documents end-of-file behaviour: a block with content but no trailing
paragraph produces a WARNING and is still parsed successfully; a block
with an empty body cannot be processed and triggers exit(1), captured
as SystemExit.

These tests call toolchain_info.get_toolchain_default_version() at
parse time and therefore require the epub VM with the Ada toolchain.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: extend unit tests for extract_projects.py (Diag class, verbos…
9f23299 
…e/inactive paths, same-project branch)

Add three groups of tests to test_extract_projects.py:

Diag class (lines 28-40): new TestDiag class with three tests verifying that
__init__ stores all four fields and __repr__ produces "file:line:col: msg"
format, including an edge case with zero/empty values.

analyze_file() coverage-improvement tests (B2, B3) — added to TestAnalyzeFile:
- test_code_block_at_sets_inactive: sets code_block_at=9999 so no block's line
  range matches; all blocks stay inactive and the inner loop hits the continue
  path (lines 188-191, 211).  Asserts no project directory is created.
- test_verbose_prints_headers: sets verbose=True; confirms project name appears
  in stdout (lines 246-248).
- test_second_call_same_project_logs_exists: calls analyze_file() twice on the
  same RST; second call prints "already exists" (lines 234-237).
- test_no_check_verbose_skip: verbose=True with a no-check block; confirms
  "Skipping" appears in stdout (line 344).

Same-project second block (line 218 false branch): new class
TestAnalyzeFileSameProjectTwoBlocks with an RST containing two no-check Ada
blocks sharing project=SameProject; verifies both are processed without error
and two block_info.json files are written.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add unit tests for check_code_block.py
1a79324 
Tests cover Diag.__repr__; check_block() with no_check=True (early return,
no subprocess); cache hit with status_ok=True/False/None; force_checks=True
bypassing cache; BUTTONS check failure for empty buttons list; real Ada
syntax check (gcc -gnats) for valid and invalid Ada; real gprbuild compile
for a valid Ada procedure and a procedure with a syntax error; real run
check for a compilable Ada program; BUTTONS check for selected toolchain
with non-"no" button; and check_code_block_json() with a missing file and
with a valid no-check block.

Key fix in _make_block(): changed `buttons = buttons or ["no"]` to
`buttons = ["no"] if buttons is None else buttons` so that passing
`buttons=[]` explicitly is preserved (an empty list is falsy, causing
the `or` form to silently substitute `["no"]`).  Added compile_it and
run_it parameters to _make_block() to allow tests to reach the BUTTONS
check without triggering the compile path that requires a project file.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: add edge-case unit tests for chop.py
8ffd3ac 
Adds test_chop.py in the package test directory covering manual_chop and
cheapo_gnatchop edge cases not present in the existing
frontend/sphinx/tests/test_chop.py.

manual_chop additions: .ads and .adb Ada extensions, empty input, input
with no !filename lines at all, garbage before the first valid file
marker, single filename with no content, fake extensions not matched.

cheapo_gnatchop additions: dotted package body names (Foo.Bar →
foo-bar.adb), dotted procedure names, triple-dotted names, spec-only
files (.ads), empty input, only-garbage input, garbage before the first
declaration, body-before-spec ordering.

Does not cover real_gnatchop (requires the Ada toolchain; already
tested in sphinx/tests/test_chop.py).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: extend unit tests for check_projects.py (verbose, inactive bl…
9a3de58 
…ock, duplicate project, None-block)

Add reset_cp_globals autouse fixture to reset cp.verbose, cp.all_diagnostics,
cp.max_columns and cp.force_checks before and after each test.  The existing
tests did not reset these globals, so any test that set verbose=True would
have leaked state into subsequent tests.

Add new TestCheckProjectsExtended class with four tests:

- test_get_blocks_from_json_file_returns_none: monkeypatches
  CodeBlock.from_json_file to return None; verifies get_blocks() prints ERROR
  and returns an empty dict (lines 30-32).

- test_get_blocks_duplicate_project: two block_info.json files with the same
  project name; verifies both are accumulated in the list under one key (false
  branch of "if not b.project in projects:" at lines 38-40).

- test_get_projects_verbose: sets cp.verbose=True and calls check_projects();
  verifies the project header appears in stdout (lines 87-88).

- test_check_projects_skips_inactive_block: serialises a block with
  active=False; monkeypatches cp.check_block to track calls; verifies the
  inactive branch (line 93 continue) is taken and check_block is never called.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: update coverage threshold and exclude CLI entry points
27804fa 
Lower fail_under from 90% to 75%: the three toolchain-dependent modules
(check_code_block, check_projects, extract_projects) have large compile/run/prove
code paths that are not exercised by unit tests; together they cap realistic
coverage well below 90%.

Add exclude_lines for "if __name__ == '__main__':" so the CLI entry-point
blocks in check_code_block, check_projects, and extract_projects (approximately
80 lines total) are excluded from the measurement; those blocks are tested via
the --help smoke tests rather than unit tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff
Merge feat/pure-module-tests: unit tests for pure Python modules
d8f69e3
@gusthoff @claude
Python: add pragma: no cover to __main__ blocks in three modules
ba42502 
The bodies of the if __name__ == "__main__": guards in
check_code_block.py, check_projects.py, and extract_projects.py are
CLI entry-point code that cannot be exercised by unit tests.  The
exclude_lines pattern in pyproject.toml already suppresses the guard
line itself, but coverage.py 7.x still counts the body lines as
uncovered.  Adding # pragma: no cover to the guard lines causes
coverage to exclude the entire block body, accurately reflecting
the fact that these paths are tested via the --help smoke tests
(test_smoke.py) and not by the unit test suite.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gusthoff @claude
Python: extend unit tests for chop.py (real_gnatchop — valid Ada, com…
8050eb8 
…piler switches, error handler)

Add TestRealGnatchop class with four tests exercising the real_gnatchop
function (chop.py lines 96-149), which was previously untested because
the Ada toolchain is required:

- test_valid_ada_no_switches_returns_resources: calls real_gnatchop with
  compiler_switches=None on minimal valid Ada; verifies a non-empty list of
  Resource objects is returned (line 118 — the compiler_switches=None branch).
- test_valid_ada_no_switches_basename: confirms gnatchop produces main.adb.
- test_valid_ada_with_compiler_switches: passes compiler_switches=["-gnata"];
  exercises lines 120-125 (the cmd.extend branch).
- test_invalid_input_raises_exception: passes garbage input; gnatchop fails;
  verifies the except CalledProcessError handler (lines 137-144) raises
  Exception with "Could not chop files with gnatchop".

Also update the module docstring to reflect that real_gnatchop is now covered.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gusthoff and others added 24 commits September 18, 2026 22:37
@gusthoff @claude
Python: check an unmet compile-error expectation through the command
5052aad 
The exit status is what a build gates on, so the report an author's class
produces has to survive the source, the extraction step and the command
itself. The course helper gained an optional class line to write one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Docs: record the order the checks for a code block run in
7f1b478 
Which checks a code block gets, and in which order, was written down only
in the test suite. The order is load-bearing rather than incidental: a run
is nested inside the build and so cannot happen without one, a proof is a
sibling of the build and so triggers none, a syntax-only code block stops
before the build, and the check of the code block's own declarations runs
last because it needs the outcomes above.

The per-check labels the function records are deliberately left
undocumented as a format: they are dropped when the file is read back, so
nothing consumes them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: cover a quiet run that fails the way the block expects
0701280 
The sibling test drives the same path with verbose enabled and asserts
the message it prints; nothing said the message was conditional, and the
quiet run -- the one every real check makes -- was never exercised.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Merge feat/checker-reporting-fixes: report checker failures instead o…
72cebcf 
…f crashing

Several conditions in the code-example checker ended a run with a traceback
or passed silently instead of being reported: an unreadable or damaged block
record, a missing executable, a clean-up failure, and a block skipped for
having no project. A damaged record is now rebuilt with a warning that says
what it does and no more, and a C block asking only for a compile is built
that way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Docs: point the package README's course links at the course
6e8b145 
Both links to the Introduction to Ada course gave the path from the repository
root, which does not resolve from a README three directories down; they now use
the relative path.  The root README's copies are correct there and are unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: drive the previous-check lookup both ways
0632a4e 
The switch deciding whether a block that already carries a result is skipped
ships on, so the arm that does not read the record was never entered and
carried a no-branch pragma. Two tests switch it off -- one for the result, one
for the message -- and the pragma is gone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Docs: state that an unreadable block record reads back as no block
691bbaf 
The method returns None for a file it cannot turn into a code block rather
than raising, and callers depend on that: the checking commands fail the
run over such a code block while extraction warns and rewrites the record.
Written down here, together with which reasons are reported and which
files reduce to None in silence.

The note is explicit that the reported reasons are exception types rather
than an exhaustive account of what can go wrong, so a reader does not take
the guard for complete.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Merge feat/honor-block-declarations: act on the classes and configura…
4d63bfb 
…tion values a block declares

Three declarations the checker read but did not act on: a C block asking to be
run by class was never run and the check still reported success, the report-all
proof switch was selected by a class name no proof-selecting list contained,
and the block configuration turned a real boolean into its opposite.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Docs: let the extraction and block commands name themselves
db504f4 
Both printed the module path in their usage line instead of the name they are
installed under, so all three commands now describe themselves the same way.
This sets argparse's displayed program name only; no behavior changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: force both answers to the terminal test made at import
bb4692d 
Neither output stream is a terminal under pytest, so the arm keeping colors on
was never entered and carried a no-branch pragma. The module is imported again
with the streams answering each way, its contents put back afterwards, and the
pragma is gone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Merge feat/help-text-and-spelling: say what the checker does and how …
27b7971 
…it is checked

The check-block help text described a build with gnatmake, which the package
never invokes, and never mentioned proving. It now describes the checker it
has, alongside new docstrings recording the order the checks run in and what
an unreadable block record reads back as. The package README states what it
needs on PATH rather than how it is deployed, and the test suite no longer
mixes spellings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: test the C compile error that was expected and never happened
6e1de98 
A C block declaring c-expect-compile-error whose code compiles cleanly must
fail the check and say the declared error never arrived. Covered three ways,
as the Ada spelling already is: from a hand-built block, driven through the
real directive and the real extraction step, and through the installed
command, where the defect showed as status zero with nothing printed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Merge feat/expect-error-core-promise: test the expected error that ne…
436e7b2 
…ver happens

The checker's central promise is that an example declaring its compilation,
run or proof should fail is reported when it succeeds instead. That direction
had no test, and the block validating it was excluded from coverage behind a
constant-true wrapper. The wrapper and an unreachable duplicate of the compile
step are gone, both expected-error arms are now covered along with the
previous-check lookup and the terminal test made at import, and two pragmas
that hid reachable branches were removed with the tests that reach them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Merge feat/closing-text-and-cleanup: close the last gaps in the check…
e94aff4 
…er's promise

A C code block declaring an expected compile error that compiled cleanly was
reported as a success, leaving one of the six expected-error classes
unenforced; it is now reported and covered at three levels. Alongside it, the
extraction command's help text describes what it actually does, all three
commands name themselves in their usage line, the block configuration's
coercion rule and the expected-error contract are written down, and a helper
the extractor never called is gone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: report a run class that names the other language
22c005b 
A run class names a language and is meaningful only for a code block
written in it. A class naming the other language is now reported and fails
the check, instead of being accepted without comment.

This lands before the change that stops honoring such a class, so that no
commit leaves a mis-tagged code block unbuilt and still passing.

Covers the six run classes only. The compile classes already pair with the
block's language, while `ada-syntax-only` and the no-check classes are not
language-specific at all, so none of them is included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: apply each run class only to the language it names
b4af98b 
`run_it` honored `ada-run`, `ada-run-expect-failure` and `ada-norun` for a
code block in any language, and `c-norun` likewise, while the two C run
classes beside them were already paired with C. `compile_it` below has
paired the compile classes with their language all along; this brings
`run_it` into line with it.

A class naming the other language no longer has any effect, and is reported
by the check added just before this one, so a mis-tagged code block fails
rather than passing unbuilt.

No code block in the course material is affected: of the 2148 code blocks
declaring Ada or C, the only one carrying a class whose prefix names the
other language is a C block tagged `ada-syntax-only`, which is not a run
class and keeps working.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Docs: state where the syntax check does not reach
4dcf108 
Both the module docstring, which is what `check-block --help` prints, and the
`check_block()` docstring said every code block is syntax-checked unless it
declares `nosyntax-check`. Three cases escape it: a code block declaring
`ada-nocheck` or `c-nocheck` returns before anything runs, one that already
carries a recorded result returns unless `--force` is given, and the check
invokes a compiler for Ada and for C only, so a record naming another language
passes it having parsed nothing.

The claim that followed from it in `check_block()` — that the weakest thing
that can happen to a code block is still that its sources are parsed — is
removed rather than qualified, since for a no-check code block nothing is
parsed.

Both docstrings also gain the new report for a run class that names the other
language, scoped to the run classes: the compile classes silently ignore a
foreign one, and `ada-syntax-only` and the no-check classes are not
language-specific at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: test that a run class applies only to its own language
494e91d 
Each run class names a language and is honored only for a block written
in it.  The Ada run classes now ask for no run, and therefore no build,
on a C block, and neither norun class takes a run away from a block of
the other language.  Controls beside each: the same classes on the
language they name, a wrong-language compile class, and the run button,
the syntax-only class and the two no-check classes, which are
deliberately not paired with any language.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: test the report for a run class naming the other language
18c7085 
Each of the six run classes, on a block of the other language, is
reported by name and fails the check -- including on a block a run
button separately gets built and run, which is the case a report read
off what the checker did, rather than off what the block declared, would
pass over.  Message and returned value are asserted by separate tests,
so a report that prints and leaves the run at success reddens the second
alone.

Controls: the same classes on the language they name, the classes that
name a language and are not paired with one, the class that names none,
and a proof asked for on a C block, which has its own report already.
The three returns that come before the declaration checks are pinned as
not reporting.  Also driven through the real extraction step, in both
directions.

The helper that reads back the messages produced for one block moves to
module level, unchanged, so both classes can use it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: test that a mis-classed run class fails the command
42ff483 
The exit status is set outside every function the rest of the suite
calls, so the claim that the report fails the run can only be made
through the installed command.  Both directions, with the control of the
same example tagged with its own language's class, which checks out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: report a mis-declared run class before reusing a recorded result
bff9656 
The per-block directory is named after a hash of the code block's text, and
the recorded result beside it is neither compared against the declaration nor
removed.  Editing only the class of a code block therefore handed back the
success recorded before the edit, so a run class naming the other language
went unreported and the check passed over a code block nothing had built.

The class is now read off the declaration at the top of the check and
reported there, before the recorded result is consulted, and the failure is
carried to each return that comes before the declaration checks.  A code
block declaring no check at all is still skipped in silence.  The error flag
itself is set no earlier than before, so a code block whose run button asks
for the run its class did not is still built and run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: test the report a recorded result used to absorb
edd403c 
Inverts the two tests that pinned the old behavior: a code block declaring
itself syntax-only and one with a recorded result are both reported now, and
each says why the decision went the way it did.  The third path is unchanged
and keeps its test: a code block declaring no check at all is still silent.

Adds the exit status beside each, held apart from the wording as the rest of
this class does, with the control that a sound code block still keeps its
recorded result; and pins the report as being made exactly once.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: test a class-only edit through the installed command
6016438 
The build directory is kept between runs by the default local driver, so the
recorded result of the previous run is what the next one meets.  Checks an
example, edits only its class to name the other language, and checks again
without --force: the run must fail and name the class, over the same block
directory.  With the control of an unedited example, which keeps its
recorded result.

The course helper grew a parameter for the button indicator, so an example
that nothing builds can be written.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Merge feat/review-follow-ups: apply each run class only to the langua…
13c5c5c 
…ge it names

A run class names a language and is now honored only for a code block
written in it. A class naming the other language is reported and fails the
check, rather than quietly doing nothing.

Four of the six wrong-language combinations were already silent skips; the
sixth, ada-run on a C block, was honored and would have become one. The
report closes all of them, and is made before the recorded result for the
code block is consulted, so a class-only edit cannot reuse the verdict
recorded for the declaration it had before the edit.

No code block in the course material is affected: of the language-declaring
directives under content/, none carries a run class naming the other
language.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff
gusthoff force-pushed the dev/topic/infrastructure/python/rst-pipeline-unit-tests/2026-06-19/main branch from 88d50f4 to 13c5c5c Compare September 19, 2026 02:07
gusthoff and others added 5 commits September 19, 2026 04:13
@gusthoff @claude
CI: remove duplicate stale-named pyright workflow
5001273 
code-projects-type-check.yml still carried the package's old name
(code_projects, renamed to rst_code_example_pipeline) and its pyright
job is now fully duplicated by rst-code-example-pipeline-ci.yml. A
rebase of the rename commit dropped the file deletion half of that
change, leaving both files in place.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@gusthoff @claude
Python: drop references to the rest of the repository
a081100 
The package is a standalone module and must not name the code around it.
Three docstrings did: one pointed at the Sphinx extension that recomputes
the same digest, one named the module that reads the block info file and
described the browser-side download code, and one named the widget that
renders an example's log files.

Where the constraint itself is real it is kept, stated generically: these
names are an on-disk contract, whatever reads the artifacts carries its own
copy of them, and renaming one is not a local change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: bump the package version to 0.3.0
52fa174 
The version had never moved: 0.2.0 was inherited from the module this
package was created from and kept unchanged since. Meanwhile the entry
points were renamed, two modules and the toolchain data file moved into
the package, and a code block declaring a run class for the other language
now fails a check it used to pass.

Both declarations are bumped together, as the metadata test requires.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gusthoff @claude
Python: trim the exit-status section of the pipeline README
57cd21a 
It read like an internal bug-tracking note (repair-path mechanics,
"has a gap here", "until this is fixed") instead of describing what
a caller of the exit status needs to know today.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@gusthoff @claude
Docs: drop the fix-plan paragraph from analyze_file()'s docstring
30f993c 
The docstring described the unreachable error flag correctly, then
went on to prescribe the fix (declare nonlocal, set the flag at the
remaining error sites) and its consequences. A docstring documents the
current contract for a caller; it is not the place to plan a repair.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

website infrastructure

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@gusthoff