build-jupyter-cache: container-mode failures are silent, undebuggable, and use unpinned sub-actions

[mmcky]

Summary

When build-jupyter-cache runs in container mode and a lecture fails to execute, the failure is effectively invisible and undebuggable: no execution report is uploaded, the auto-issue alert does not get created, and the action runs sub-actions from a floating @main rather than the consumer's pinned tag. In lecture-dp this combination meant the weekly cache build failed for ~2 months (see #28) with no alert issue and no downloadable traceback, so nobody noticed until PRs were observed running full cold builds.

These are three separable robustness gaps in build-jupyter-cache.

1. Sub-actions are referenced at @main, not pinned

build-jupyter-cache/action.yml calls its sub-actions at @main even though consumers pin the parent action to a release tag (e.g. lecture-dp uses build-jupyter-cache@v0.6.0):

Line	Reference
132	uses: quantecon/actions/setup-environment@main
147	uses: quantecon/actions/build-lectures@main
156	uses: quantecon/actions/build-lectures@main
165	uses: quantecon/actions/build-lectures@main

A consumer that pins @v0.6.0 for reproducibility still silently gets whatever is on main for the actual build/setup steps. This makes the cache builder non-reproducible and means a change on main can alter a "pinned" consumer's behavior. The internal references should resolve to the same release ref as the parent action.

2. Cache-build failures upload no execution report

The HTML build step inside build-jupyter-cache invokes build-lectures without upload-failure-reports, so when a notebook raises CellExecutionError there is no *.err.log artifact to inspect. This is doubly confusing because the auto-generated failure issue body (scripts/create-failure-issue.sh) tells the maintainer to "Download the build artifact for detailed execution reports" — but no such report is uploaded on failure.

By contrast build-lectures used directly in PR CI is typically called with upload-failure-reports: true, so PR failures are debuggable while cache failures are not. Recommend defaulting upload-failure-reports: true (or exposing it as an input) for the build steps inside build-jupyter-cache.

3. The failure-alerting step itself fails

build-jupyter-cache is meant to open a GitHub issue on failure (create-issue-on-failure: true). In a real run it did not. In lecture-dp run 26381283100 the alerting step exited 127:

/home/runner/work/_actions/quantecon/actions/v0.6.0/build-jupyter-cache/scripts/create-failure-issue.sh: No such file or directory
##[error]Process completed with exit code 127.

Net effect: every cache-build failure went un-alerted (zero build-failure issues were ever created in lecture-dp, despite months of failing runs). Worth hardening so the alerting path cannot silently no-op: verify the script ships and is executable in tagged releases, confirm gh is on PATH inside the quantecon-build container, and surface a clear error if the issue cannot be created.

Impact / context

This is the tooling layer that should have surfaced #28 (unpinned container scipy/numpy breaking un_insure.md). Because of gaps 2 and 3, that failure produced neither an alert nor a usable traceback in the cache path, and gap 1 makes the cache builder's behavior drift with main. Fixing these would make container-mode cache failures loud and debuggable.

Tracked downstream in QuantEcon/lecture-dp (CI cache issue, linked).

[mmcky]

Closed by #122 (merged as a5da9f3). All three parts are done, but the diagnosis in this issue was wrong in ways worth recording — anyone acting on it as written would have fixed roughly one third of the problem.

The three parts

1. Sub-actions referenced at @main — fixed before this session; build-jupyter-cache/action.yml pins setup-environment@v0 and build-lectures@v0 at all four call sites.

2. Cache-build failures upload no execution report — fixed in #122 via a new upload-failure-reports input, defaulting true and passed to all three inner build-lectures calls. One correction: the reports were not entirely absent before. They were reachable inside the full _build artifact — hundreds of MB including .jupyter_cache — while the issue body said only "download the build artifact" and named nothing. Genuinely absent only when upload-artifact was off. It was mostly a discoverability failure, and the issue body now names the artifacts actually produced with per-builder reproduce commands.

3. The alerting step itself fails — fixed, but not for the stated reason.

What was actually broken

This issue attributes the exit 127 to the missing gh CLI and recommends installing gh in both images or falling back to curl. There were three independent bugs on that 14-line path, and gh absence was the one that never even executed.

The observed 127 was path resolution, not a missing binary. The step invoked its script through ${{ github.action_path }}, which expands to the runner's path (/home/runner/work/_actions/...). In a job container: the action is mounted at /__w/_actions/..., so bash got a path that does not exist — No such file or directory, exit 127, before the script ran. Upstream: actions/runner#2185.

Worth recording for anyone who hits this elsewhere: $GITHUB_ACTION_PATH — the environment variable — does not have this problem. Env values are rewritten through the container path map before docker exec, so the env-var form arrives correctly as /__w/.... Verified live: preview-netlify runs bash "$GITHUB_ACTION_PATH/../scripts/detect-changed-lectures.sh" successfully inside container: quantecon-build. Only the ${{ }} expression form is broken. The preview actions were already correct and were left alone.

Missing gh was real but latent. Fixing the path alone would have died one line later on gh issue list. Neither image installs it.

A third bug broke alerting on hosted runners too, and nobody had noticed it. gh issue create --label resolves label names client-side and fails with could not add label: 'x' not found if they do not exist. The action defaults to build-failure,automated, and no consumer has either label. So even on a hosted runner with gh present and the path resolving, this would have failed. Verified both directions: POST /repos/{owner}/{repo}/issues with those labels creates the issue and the labels implicitly (colour #ededed, empty description — GitHub's auto-created fingerprint), while gh issue create --label refuses.

Replacing the script with actions/github-script removes all three at once: no CLI, no action_path, and the REST API creates missing labels. Each remedy this issue proposed would have fixed exactly one — and installing gh in both images, the option the wording steers toward, would have meant rebuilding and republishing two multi-GB images to observe no change at all.

A fourth problem, not in this issue

A failed build could be reported as the wrong failure. Composite steps carry an implicit success() in a plain if:, so once any earlier step failed — including the alerting step — Fail if builds failed silently skipped, and the job's only red was whatever incidentally broke. In lecture-dp run 26381283100 the visible error was the exit-127 from "Create failure issue" while the actual build failure never surfaced at all. Both steps are now always() && ….

Scale, for the record

Worse than "~2 months" as originally reported. The canary repo's cache.yml failed eight consecutive weeks (2026-02-22 → 2026-04-12) filing zero issues, while linkcheck.yml in the same repo — using a REST-API action rather than the CLI — filed a report every one of those weeks. Same repo, same token, same period. That contrast is what identified the mechanism.

Guarantees now in place

• Alerting cannot silently no-op: a step asserts an issue was really filed and fails the job with an explicit error otherwise. The URL is exposed as a new failure-issue-url output.
• Consumers need issues: write while create-issue-on-failure is true — templates/cache.yml already has it, and a missing permission is now a loud failure naming the fix rather than silence.
• bjc-fail-guard in the PR harness asserts a failing cache build fails the step, reports html-status=failure, and saves no cache over the last good one. That failure path previously had no coverage at all, which is how three bugs accumulated on it unseen.

Successor

One gap in this area remains, tracked separately in #123: a setup-environment failure still alerts nobody. If setup fails the composite aborts before verify-builds runs, so all-passed is unset ('', not 'false') and every guard — including the new one — skips. Different trigger, different fix.

PLAN's P0 section is now empty, and its entry carries this corrected diagnosis so the wrong model is not inherited from a struck-through row.