Python: add unit tests for rst_code_example_pipeline

.github/workflows/rst-code-example-pipeline-ci.yml

@@ -0,0 +1,62 @@
+name: rst_code_example_pipeline CI
+
+on:
+  push:
+    paths:
+      - 'frontend/python/rst_code_example_pipeline/**'
+  pull_request:
+    branches:
+      - main
+    paths:
+      - 'frontend/python/rst_code_example_pipeline/**'
+
+jobs:
+  pyright:
+
+    runs-on: ubuntu-26.04
+
+    strategy:
+      matrix:
+        python-version: ['3.14']
+
+    steps:
+    - uses: actions/checkout@v7
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v7
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install pyright
+      run: pip install pyright
+    - name: Install rst_code_example_pipeline
+      run: pip install -e 'frontend/python/rst_code_example_pipeline[test]'
+    - name: Run pyright on rst_code_example_pipeline
+      working-directory: frontend/python/rst_code_example_pipeline
+      run: pyright .
+
+  pytest:
+
+    runs-on: ubuntu-26.04
+
+    strategy:
+      matrix:
+        python-version: ['3.14']
+
+    steps:
+    - uses: actions/checkout@v7
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v7
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install OS deps
+      run: |
+        sudo apt-get update && \
+        sudo apt-get install -y \
+        crudini
+    - name: Install GNAT FSF
+      run: |
+        ${GITHUB_WORKSPACE}/.github/workflows/install_toolchain.sh --gnat --gnatprove --gprbuild
+    - name: Install rst_code_example_pipeline
+      run: pip install -e 'frontend/python/rst_code_example_pipeline[test]'
+    - name: Run rst_code_example_pipeline test suite
+      working-directory: frontend
+      run: make test_rst_pipeline

.gitignore

@@ -1,5 +1,6 @@
 .DS_Store
 *.pyc
+*.egg-info/
 env
 .idea
 .vagrant*

CONTRIBUTING.md

@@ -310,6 +310,11 @@ The following button-related parameters are available for this directive:
 | `prove_flow_report_all_button` | examine SPARK data and control flow and report all findings |
 | `submit_button`                | submit code for a lab                                       |
 
+When `compile_button` is used and nothing else asks for the code to be run —
+no `run_button`, and no class that asks for a run — the code is compiled but
+not linked. No executable is produced, so such a code block does not have to
+contain a main subprogram (in Ada) or a `main` function (in C).
+
 ### Project parameter and code accumulation
 
 A `project` parameter must be provided. For this parameter, we use the
@@ -586,7 +591,8 @@ block in the generated HTML or e-book output.
 
 The following classes are available for the testing phase:
 
-  - `ada-nocheck`: testing of this specific code block is completely skipped.
+  - `ada-nocheck` and `c-nocheck`: testing of this specific code block is
+    completely skipped, for Ada or C code respectively.
 
   - `nosyntax-check`: code must not be checked for syntax errors. (Note that
     the code block is still compiled in the testing phase.)
@@ -597,9 +603,21 @@ The following classes are available for the testing phase:
 If an error is expected during the testing phase, one of the following classes
 must be used:
 
-  - `ada-expect-compile-error`: a compilation error is expected.
+  - `ada-expect-compile-error` and `c-expect-compile-error`: a compilation
+    error is expected, in Ada or C code respectively.
+
+  - `ada-run-expect-failure` and `c-run-expect-failure`: a run-time error is
+    expected, in Ada or C code respectively.
 
-  - `ada-run-expect-failure`: a run-time error is expected.
+  - `ada-expect-prove-error`: a proof error is expected. The code block must
+    also be proved, either through one of the prove buttons or through one of
+    the `ada-prove` classes listed below.
+
+These classes state a requirement on the testing phase, not a hint about the
+generated output: the expected error has to actually occur. If it does not —
+the code compiles, runs or proves cleanly — that absence is reported as an
+error and fails the check, so one of these classes left behind after the code
+example was fixed makes the testing phase fail rather than passing quietly.
 
 When the `no_button` parameter is used, the following classes are available to
 compile or run the code examples:
@@ -618,7 +636,13 @@ output.
 When the `run_button` parameter is used, the following classes are available:
 
   - `ada-norun` and `c-norun`: to explicitly deactivate the run of Ada or C
-    code, respectively, during the testing phase.
+    code, respectively, during the testing phase. These classes also take
+    precedence over a class that asks for a run, such as `ada-run` or `c-run`.
+
+    The code is built in order to be run, so deactivating the run also
+    deactivates the build unless something else asks for the code to be
+    compiled — a `compile_button`, or the `ada-compile` or `c-compile` class.
+    Without one of those, the code block is only checked for syntax errors.
 
 ## Lab exercises
 

README.md

@@ -224,3 +224,71 @@ check-code                                         \
 
 For more examples and alternative configurations, please refer to the
 [README of the rst_code_example_pipeline package](frontend/python/rst_code_example_pipeline/README.md)
+
+#### Running the package's unit tests
+
+The package also has its own pytest-based unit test suite.  Run the whole
+suite with `make test_rst_pipeline`, from the `frontend/` directory:
+
+```sh
+# Full suite -- the validation run, and the only one
+make test_rst_pipeline
+```
+
+This needs an Ada toolchain *installation*, not merely a compiler on `PATH`.
+The package exists to extract, build and run source-code examples, so the
+tests covering that work invoke the toolchain for real -- and some of them
+create and remove symlinks under the installation tree configured in the
+package's `src/rst_code_example_pipeline/data/toolchain.ini` (`/opt/ada` and
+below).  A distribution-packaged GNAT on `PATH`, without that tree, is not
+enough: the suite goes red.  This is the only run that validates the
+module (the `rst_code_example_pipeline` package).  The epub VM has such an
+installation; so does the `pytest` job in
+`.github/workflows/rst-code-example-pipeline-ci.yml`, which provisions one
+and then runs this same target on a GitHub runner.
+
+For developers working on a machine without an Ada toolchain, a second
+target runs just the subset of tests that need no toolchain:
+
+```sh
+# Toolchain-free subset -- does not validate the module
+make test_rst_pipeline_smoke
+```
+
+**This smoke run does not validate the module.**  It compiles nothing, so it
+proves nothing about the module's actual job.  Its reach is narrower than
+"every test that does not need a toolchain" may suggest: it executes roughly
+half of the module's lines and leaves `check_code_block.py`, the file that
+drives the toolchain, almost entirely unexecuted.  Ordinary Python inside
+the toolchain-facing files -- diagnostic parsing, message formatting,
+writing the JSON check report -- is deselected wholesale along with the
+tests that cover it, so breaking any of that still leaves the smoke run
+green.  A green smoke run must never be mistaken for a passing suite.
+Coverage is switched off for it, because a coverage figure measured from a
+subset would invite the same misreading.
+
+So before considering a change tested, run the full suite where a toolchain
+installation exists: on the epub VM, or on a machine provisioned the way the
+CI runner is.  `.github/workflows/install_toolchain.sh` is the specification
+for that provisioning -- it reads every path and version from the package's
+`toolchain.ini` and unpacks the GNAT FSF builds into that tree.  It is
+written for the CI runner (it expects `GITHUB_WORKSPACE`, needs `crudini`,
+and exports the `bin` directories through `GITHUB_PATH`), so read it rather
+than run it unchanged on a workstation.
+
+Both targets are thin wrappers around `pytest`.  `make` itself runs from
+`frontend/`, where the `Makefile` lives -- there is no `Makefile` in the
+package directory.  It is the recipes that `cd` into
+`frontend/python/rst_code_example_pipeline`; `pytest` runs there.  Without
+the Makefile, run the equivalent commands from that directory:
+
+```sh
+# Full suite -- the validation run, and the only one
+pytest
+
+# Smoke subset -- does not validate the module
+pytest -m "not toolchain" --no-cov
+```
+
+See the "Development" section of the package README for installation and
+usage details.

frontend/Makefile

@@ -215,6 +215,12 @@ test_parser:                                                                   #
 	@coverage run --source=widget,rst_code_example_pipeline.chop,rst_code_example_pipeline.resource -m unittest discover --start-directory sphinx
 	@coverage report --fail-under=90 -m
 
+test_rst_pipeline:                                                             ## Test the rst_code_example_pipeline package (epub VM).
+	@cd python/rst_code_example_pipeline && pytest
+
+test_rst_pipeline_smoke:                                                       ## Toolchain-free smoke test; does NOT validate the package.
+	@cd python/rst_code_example_pipeline && pytest -m "not toolchain" --no-cov
+
 ##@ Build website
 publish:                                                                       ## [DEPRECATED] Publish contents to the learn website.
 	@echo "Publishing current branch to learn..."

frontend/python/rst_code_example_pipeline/README.md

@@ -2,9 +2,8 @@
 
 ## Introduction
 
-The [rst_code_example_pipeline](frontend/python/rst_code_example_pipeline) package contains
-scripts to extract, build and run the code blocks from the ReST files. These are the main
-entry points:
+The `rst_code_example_pipeline` package contains scripts to extract, build and
+run the code blocks from the ReST files. These are the main entry points:
 
 - `extract-code` extracts all code blocks and stores into the specified
   build directory;
@@ -13,18 +12,23 @@ entry points:
 
 - `check-block` checks a single (previously extracted) code block.
 
-The package is installed in editable mode as part of the VM provisioning:
+Install the package from the repository, in editable mode:
 ```sh
 pip install -e frontend/python/rst_code_example_pipeline
 ```
 
+The entry points drive an Ada toolchain directly and expect it on `PATH`:
+`extract-code` splits an Ada code block with `gnatchop`, and the two checking
+commands syntax-check and compile with `gcc`, build with `gprbuild`, clean up
+with `gprclean`, and prove with `gnatprove`.
+
 
 ## Simple usage
 
 To build and run the source-code examples from a course, just run
 `extract-code` followed by `check-code`. For example, to test
 the source-code examples from the
-[Introduction to Ada course](content/courses/intro-to-ada), run:
+[Introduction to Ada course](../../../content/courses/intro-to-ada), run:
 
 ```sh
 extract-code                                                  \
@@ -41,6 +45,26 @@ for each code block (source-code example) that is extracted from the ReST files.
 and checks the source-code example described in each of those JSON files.
 
 
+## Exit status
+
+All three entry points report the outcome of a run through their exit status:
+
+- `check-code` and `check-block` exit `1` if any checked code block failed a
+  check, and `0` otherwise. A code block that could not be read or checked
+  also counts as a failure.
+
+- `extract-code` exits `1` when it cannot run at all — for example, a code
+  block has no project name, or neither `--build-dir` nor
+  `--extracted_projects` was given — and `0` otherwise.
+
+An invalid command line is rejected before any work is done, with exit
+status `2`.
+
+Some malformed code blocks are reported only through an `ERROR` line in the
+output, while `extract-code` itself still exits `0`. Check the output, not
+only the exit code, to catch these.
+
+
 ## Verbose mode
 
 All the scripts have a `--verbose` / `-v` switch. For example:
@@ -62,7 +86,7 @@ check-code                                                    \
 It's possible to store the list of extracted projects into a JSON file and
 use that file for checking the projects. For example, to build the source-code
 examples from the
-[Introduction to Ada course](content/courses/intro-to-ada), run:
+[Introduction to Ada course](../../../content/courses/intro-to-ada), run:
 
 ```sh
 extract-code                                                  \
@@ -168,3 +192,35 @@ check-block                                                   \
   --max-columns 80                                            \
   test_output/projects/Courses/Intro_To_Ada/Imperative_Language/Greet/cba89a34b87c9dfa71533d982d05e6ab/block_info.json
 ```
+
+
+## Development
+
+### Installing with test dependencies
+
+The package declares an optional `test` extras group that installs
+[pytest](https://docs.pytest.org/) and
+[pytest-cov](https://pytest-cov.readthedocs.io/).
+Install the package in editable mode together with those extras:
+
+```sh
+pip install -e ".[test]"
+```
+
+### Running the unit tests
+
+Coverage options and test paths are configured in `pyproject.toml`, so a plain
+`pytest` invocation from the package root is enough:
+
+```sh
+pytest
+```
+
+Some modules require an Ada toolchain (GNAT) to be on `PATH`; run the full
+suite in an environment where GNAT is available.
+
+To pass coverage options explicitly:
+
+```sh
+pytest --cov=rst_code_example_pipeline --cov-report=term-missing tests/
+```