CI/CD: adopt the NxWitness "decouple merge from build/publish + PR smoke build" pattern (path-gated smoke, base-branch targeting, required-status aggregator)

[ptr727]

Summary / problem statement

A CI/CD pattern was just implemented in ptr727/NxWitness (PRs #398, #402, #406) that we should adopt into ProjectTemplate so it propagates to downstream projects aligned to this template (PlexCleaner specifically, also NxWitness itself which already has it).

The pattern solves three related problems:

1. Slow PR time-to-feedback. PRs build the full Docker matrix — including QEMU‑emulated linux/arm64 — before the PR knows it is green. In NxWitness that was a ~36‑row matrix (5 products × {ubuntu, lsio} × {main, develop} × version tags, each amd64+arm64); hours of emulated builds per PR. ProjectTemplate is far smaller (one image, two arches) but still pays a full multi‑arch QEMU build on every PR regardless of what changed.
2. No‑op republishes to consumers. When every push/merge to main/develop republishes images, a Dependabot github‑actions/nuget bump, a codegen version bump, or a docs change pushes an image "update" with no real content change. Docker Hub consumers see a churned tag, CI minutes are wasted, and the build is needlessly exposed to flaky registry pulls. (NxWitness also had a weekly codegen PR and a separate weekly publish double‑building the same content within ~24h.)
3. No path discrimination. Doc‑only / workflow‑only / library‑only PRs trigger the whole build matrix.

The core idea is to decouple MERGE from BUILD/PUBLISH and replace the full‑matrix PR build with a fast, path‑gated, representative smoke build, while publishing happens only on a schedule or a manual "push‑like" workflow_dispatch.

Important caveat for this template (do not blindly copy NxWitness): ProjectTemplate uses a continuous‑release model. publish-release.yml deliberately republishes on every push to main and develop (NBGV‑versioned: main = Release/latest, develop = prerelease/develop), and merge-bot-pull-request.yml intentionally uses an App token so merge commits do fire publish-release.yml. The NxWitness notion that "merge should NOT republish" fights this repo's design and should NOT be adopted wholesale here. The high‑value, low‑risk parts to adopt are the path‑gated PR smoke build and the per‑artifact path filtering. The "sole scheduled/manual publisher" half is documented below for completeness and for downstream projects, but is an optional/secondary decision for this template (see §3).

---

The pattern in detail (reference: NxWitness)

A. PR smoke build (fast feedback), path‑gated, base‑branch‑targeted, representative subset

• A changes job (uses dorny/paths-filter@v3) detects whether image‑affecting files changed and emits booleans. In NxWitness:

  - uses: dorny/paths-filter@v3
   id: filter
   with:
   filters: |
   image:
   - 'Docker/**'
   - 'Make/Matrix.json'
   - 'Make/Version.json'
   base:
   - 'Docker/NxBase.Dockerfile'
   - 'Docker/NxBase-LSIO.Dockerfile'

• The smoke-build job runs only when image files changed (if: needs.changes.outputs.image == 'true'). Otherwise unit tests run alone. A Dependabot / nuget / docs / workflow‑only PR is unit‑tests‑only and deliberately does NOT build Docker — path filtering cannot distinguish a logic change in a build workflow from an action‑version bump in the same file, so build‑workflow validation is left to actionlint + the scheduled full build.

• The smoke build builds a representative subset: one Ubuntu + one LSIO variant (NxMeta + NxMeta-LSIO), amd64 only, push: false. Rationale: all product Dockerfiles are generated from one template differing only by base + build args, so one‑per‑base exercises essentially all build logic. Residual gaps (a product‑specific download‑URL 404, arm64‑specific breakage) are covered by the scheduled full build and are not a publish gate.

• The smoke build targets the PR base branch via smoke_branch: ${{ github.base_ref }}: a PR onto develop validates the develop rows, a PR onto main validates the main rows. Empty base_ref (a workflow_dispatch) falls back to building both branches.

• The base image is built in smoke only when a base Dockerfile changed (build_base: ${{ needs.changes.outputs.base == 'true' }}): product builds pull the published base from the registry, so building it otherwise is wasted work — but a base bump (e.g. ubuntu:noble) must be compile‑validated.

smoke-build:
 needs: [changes]
 if: ${{ needs.changes.outputs.image == 'true' }}
 uses: ./.github/workflows/build-docker-task.yml
 secrets: inherit
 with:
 push: false
 smoke: true
 smoke_branch: ${{ github.base_ref }}
 build_base: ${{ needs.changes.outputs.base == 'true' }}

B. Required‑status‑check aggregator (GitHub limitation workaround)

GitHub can't require a conditionally‑skipped job directly. NxWitness has a single check-workflow-status aggregator that needs: [changes, test-release, smoke-build], runs if: always(), treats success/skipped as pass, fails on failure/cancelled — but explicitly fails if the changes (path‑detection) job itself failed, so an image‑changing PR can never merge with the smoke build silently skipped because of a paths‑filter error. This aggregator job is the required status check in branch protection. (ProjectTemplate already has this aggregator; it just needs changes and smoke-build added to its needs.)

C. Reusable‑workflow parameterization (the mechanics)

build-docker-task.yml gains inputs: push (bool), smoke (bool), smoke_branch (string), ref (string), build_base (bool). NxWitness filters its matrix JSON with jq:

# $ref and $sb below are jq variables (passed via --arg), not shell vars.
# shellcheck disable=SC2016
if [[ "$SMOKE" == "true" ]]; then
 FILTER='.Images |= (map(select((.Name == "NxMeta" or .Name == "NxMeta-LSIO") and ($sb == "" or .Branch == $sb))) | unique_by([.Name, .Branch]))'
elif [[ -n "$REF" ]]; then
 FILTER='.Images |= map(select(.Branch == $ref))'
else
 FILTER='.'
fi
echo "matrix=$(jq --arg ref "$REF" --arg sb "$SMOKE_BRANCH" --compact-output "$FILTER" ./Make/Matrix.json)" >> "$GITHUB_OUTPUT"

Key mechanics (relevant even though ProjectTemplate has no matrix):

• Platforms are switched by smoke: ${{ inputs.smoke && 'linux/amd64' || 'linux/amd64,linux/arm64' }} — applied to QEMU, Buildx, and build-push-action (all three).
• Checkout uses ref: ${{ inputs.ref || github.ref }} so an empty input falls back to the triggering ref.
• The Docker Hub login step is gated on inputs.push so non‑publishing/smoke builds (and forked PRs without secrets) don't need credentials; public base images still pull anonymously. (Note: ProjectTemplate currently always logs in — see §3 item 4.)
• The base build job is always present but its steps/inclusion are gated; build-docker depends on it with if: always() && (needs.build-base.result == 'success' || == 'skipped') so it still runs when the base build is intentionally skipped.
• get-version-task.yml and build-base-images-task.yml gain a ref input (checkout) and a platforms input (smoke = linux/amd64; full = linux/amd64,linux/arm64). (ProjectTemplate's get-version-task.yml already takes a ref input.)
• Use jq --arg for branch values (never interpolate into the jq program) and a documented # shellcheck disable=SC2016.

D. Sole scheduled/manual publisher (both branches) — NxWitness model

In NxWitness the publish workflow has no push: trigger — only schedule (weekly cron) + workflow_dispatch. One global concurrency group with cancel-in-progress: false (queue, don't cancel) prevents a manual dispatch from cancelling an in‑flight publish and leaving a partially pushed tag set. It builds the base image(s) once from main (shared base tag, built from the release branch even when dispatched from another ref), then builds + pushes the full matrix for both main and develop in the same run (two reusable‑workflow calls, ref: main / ref: develop, build_base: false), then updates the GitHub release, registry readme, and a date badge. Release metadata is pinned to main: get-version ref: main, checkout ref: main, target_commitish: main, prerelease: false.

For ProjectTemplate this half is optional and conflicts with the continuous‑release design (see top caveat). For downstream single‑image projects considering it, scale it down (see §4).

E. Codegen / Dependabot interaction

Auto‑merge of Dependabot + codegen PRs is unchanged. In NxWitness their merges no longer trigger builds (the scheduled publish batches them), so codegen frequency can be increased independently (e.g. daily) since codegen merges are cheap. In ProjectTemplate, by contrast, merges intentionally do trigger publish-release.yml via the App‑token merge bot (merge-bot-pull-request.yml), so this part does not transfer — but the PR smoke build still makes the PR validation of those bot PRs cheap (a github‑actions/nuget bump becomes unit‑tests‑only).

F. Containment / safety argument

In NxWitness, publishing happens only on the scheduled/manual run and a build must succeed to publish, so a full‑build failure means no publish + a failure notification — consumers never receive a broken image. Smoke is fast PR feedback, not a publish gate. For ProjectTemplate (which still publishes on merge), the equivalent safety is that the merge‑time build-release-task.yml still runs the full multi‑arch build before pushing; the smoke build just front‑loads cheap validation onto the PR.

G. Linting note

NxWitness has no CI lint job: structured‑file linting (C#, Markdown, Docker, GitHub Actions, spelling) is editor‑only via workspace‑recommended extensions in the *.code-workspace file, and actionlint is run for deeper workflow checks (incl. shellcheck on run: steps). The smoke pattern relies on actionlint to validate build‑workflow changes that the smoke build deliberately skips. Confirm/keep this convention when adopting.

---

3. Concrete adoption steps for ProjectTemplate

Grounded in ProjectTemplate's actual current workflows (verified against the repo):

• test-pull-request.yml — triggers pull_request → [main, develop] + workflow_dispatch; concurrency cancel‑in‑progress: true. Jobs: test-release (calls test-release-task.yml) and an already‑present check-workflow-status aggregator (needs: [test-release]). No changes/path‑filter job, no smoke job.
• test-release-task.yml — unit-test job (CSharpier check, dotnet format verify, dotnet test) → build-release-task.yml with github: false, nuget: false, dockerhub: false. So every PR builds NuGet + PyPI + executable + a full 2‑arch Docker image (amd64+arm64 via QEMU), just without publishing.
• build-release-task.yml — already gates publishing via boolean inputs github / nuget / dockerhub; fans out to build-nugetlibrary-task.yml, build-pypilibrary-task.yml, build-executable-task.yml, build-docker-task.yml, and a conditional github-release job. The "publish is conditional" half of the pattern is already done.
• build-docker-task.yml — get-version → build-docker. linux/amd64,linux/arm64 is hardcoded in three places (QEMU, Buildx, docker/build-push-action). It always logs in to Docker Hub (comment: "to benefit from higher rate limits ... for pulls and cache") even when push: false. Single image docker.io/ptr727/projecttemplate; tags latest|develop + SemVer2; registry buildcache tag for cache-from/cache-to.
• get-version-task.yml — already accepts a ref input (defaults to github.ref).
• publish-release.yml — triggers push → [main, develop] + workflow_dispatch; calls build-release-task.yml with all three publish flags true, plus a sibling publish-pypi job and a date-badge job. Republishes on every merge — intentional, keep.
• publish-periodic-docker-release.yml — workflow_dispatch + weekly cron (0 2 * * MON); build-docker-task.yml with push: true to refresh ubuntu:rolling, then date badge. This is already the scheduled/manual Docker publish path — no new scheduled path needed.
• merge-bot-pull-request.yml — App‑token auto‑merge of Dependabot/codegen so merges fire publish-release.yml (the recursion guard would suppress GITHUB_TOKEN‑authored merges). Intentional; do not break.

Adoption checklist

• Add a platforms (or smoke) input to build-docker-task.yml. Replace the three hardcoded linux/amd64,linux/arm64 strings (QEMU, Buildx, build‑push) with ${{ inputs.smoke && 'linux/amd64' || 'linux/amd64,linux/arm64' }} (or a platforms string input). No matrix jq filtering is needed here — single image.
• Gate the Docker Hub login on push (NxWitness‑style: if: ${{ inputs.push }}), OR consciously keep the current always‑login behavior for pull rate limits and document that forked PRs without secrets will not get a smoke Docker build. Decide explicitly — this is a deviation from NxWitness.
• Add a path‑filter changes job to test-pull-request.yml using dorny/paths-filter@v3 with per‑artifact filters, since this repo is multi‑artifact (Docker, NuGet, PyPI, executable). Suggested filters (verify exact paths against the repo layout):
  • docker: Docker/**, plus shared sources the image compiles from (e.g. Console/**, Directory.Build.props, *.sln, version files)
  • nuget: NuGetLibrary/** (+ shared props/version)
  • pypi: PyPiLibrary/**
  • executable: shared C# sources / props
• Add a smoke-build job (or smoke inputs threaded through test-release-task.yml) that runs the Docker build amd64‑only, push: false, gated on the docker path filter (if: needs.changes.outputs.docker == 'true'). Wire smoke_branch: ${{ github.base_ref }} only if branch‑specific build args matter (the Dockerfile uses BUILD_CONFIGURATION = main ? Release : Debug, so base‑branch targeting does have a small effect here — worth threading ref/base_ref so the PR validates the correct configuration).
• Gate the rest of the PR build matrix on its path filters too (NuGet/PyPI/executable), so a docs‑only or single‑artifact PR doesn't rebuild everything. This is higher leverage here than in NxWitness because the repo is multi‑artifact.
• Extend the existing check-workflow-status aggregator in test-pull-request.yml: add changes and smoke-build (and any new conditional artifact jobs) to needs; treat skipped as pass; fail if changes failed (so a paths‑filter error can't let a docker‑changing PR merge with the smoke build skipped). Keep this job as the required status check in branch protection.
• PR smoke cache hygiene: the smoke build should cache-from the shared registry buildcache but not cache-to it (or use a PR‑scoped cache), so PRs don't pollute the release buildcache. NxWitness uses PR‑scoped type=gha,scope=pr-<number> for this and skips PR cache‑to on forks.
• Do NOT remove the push: trigger from publish-release.yml. The continuous‑release‑on‑merge design and the App‑token merge bot depend on it. (This is the deliberate deviation from NxWitness §D for this template.) Optionally document this decision inline.
• Keep actionlint / editor‑only linting as the validator for build‑workflow changes that smoke deliberately skips (§G).
• Validate on real CI the way NxWitness did (see §G of validation below) before propagating to downstream repos.

Optional (only if the maintainer wants to move ProjectTemplate toward the NxWitness "merge ≠ publish" model)

• Remove push: from publish-release.yml; make publish-periodic-docker-release.yml (or a new scheduled publisher) the sole publisher; switch its concurrency to a single global group with cancel-in-progress: false. This is a design change with downstream implications (NBGV continuous versioning, the merge‑bot rationale) and should be a separate, deliberate decision — not bundled with the smoke‑build adoption.

---

4. Applicability to PlexCleaner + scale‑up/scale‑down guidance

PlexCleaner (current state, verified by investigation)

PlexCleaner already has most of the decoupling half:

• One multi‑arch image (docker.io/ptr727/plexcleaner, amd64+arm64). Publishing is already separated: PRs never push; merge‑time publish-release.yml builds Docker but does not push (only the GitHub release + 7‑runtime executable 7z), while publish-periodic-docker-release.yml (push + workflow_dispatch + weekly cron) is the only path that pushes Docker.
• Where the pattern still applies:
  1. No PR smoke/partial build — every PR runs the full amd64+arm64 QEMU Docker build and the full 7‑runtime executable matrix (win-x64, linux-x64, linux-musl-x64, linux-arm, linux-arm64, osx-x64, osx-arm64). Add an amd64‑only smoke Docker build and/or a reduced PR executable matrix. build-docker-task.yml is already parameterized, so adding a platforms/smoke input threaded from test-release-task.yml is small.
  2. No path filtering — doc/workflow‑only PRs still run the full matrix. Add dorny/paths-filter; the check-workflow-status aggregator already exists to keep one required check green when jobs skip.
  3. Redundant merge‑time Docker build — on merge the image is built twice (non‑pushed in publish-release.yml's release path, then pushed in publish-periodic-docker-release.yml). Consider dropping the throwaway build from the release path (rely on PR smoke + the publish workflow).
  4. Mutable develop tag republish on every merge — if PlexCleaner wants the NxWitness "don't republish on routine merges" behavior, narrow the push: trigger on publish-periodic-docker-release.yml (e.g. main‑only or tag‑gated) and rely on cron/dispatch for develop. (Unlike ProjectTemplate, PlexCleaner's Docker publish is already off the merge path for the actual push, so this is a smaller change there.)
  • No adaptation needed for the manual/scheduled publish path or registry caching — both already exist.

General scale‑up / scale‑down

• Highest value for projects with MANY Docker images/variants/arches (long full builds, many no‑op republishes) — i.e. NxWitness.
• For single‑image projects (ProjectTemplate, PlexCleaner) the core ideas still help but scale down: one representative build instead of a per‑product subset; likely no branch matrix (build the PR base branch's config only if build args differ by branch — they do in ProjectTemplate's Dockerfile, so keep ref/base_ref threading); amd64‑only PR build deferring multi‑arch to publish; the required‑status aggregator and "don't rebuild on non‑code merges" still apply per the project's release model.

---

5. Reference

NxWitness PRs implementing the pattern:

• Track verbatim sections in the fleet burn-down report #398 — core (decouple merge from build/publish, PR smoke build, scheduled/manual sole publisher, required‑status aggregator, reusable‑workflow parameterization)
• Promote develop to main: per-section fidelity, issue generator, editorconfig fix, PhotoCleaner audit #402 — base‑branch‑targeted smoke (smoke_branch wired to github.base_ref)
• Apply the PhotoCleaner round-1 feedback batch #406 — docs

Key NxWitness files to study:

• .github/workflows/test-pull-request.yml — changes (paths‑filter) + smoke-build + check-workflow-status aggregator
• .github/workflows/build-docker-task.yml — push/smoke/smoke_branch/ref/build_base inputs, jq matrix filter, smoke ? amd64 : amd64,arm64 platform switch, push‑gated login
• .github/workflows/build-base-images-task.yml — ref + platforms inputs, push‑gated login, shared base tag
• .github/workflows/get-version-task.yml — ref input
• .github/workflows/publish-release.yml — schedule+dispatch only (no push:), single global concurrency cancel-in-progress: false, base built once from main, both branches built, release metadata pinned to main

Reminder for the implementer: ProjectTemplate already has conditional publishing, a ref‑aware get-version-task.yml, a check-workflow-status aggregator, and a weekly scheduled Docker publish — so adoption here is the PR smoke build + per‑artifact path filtering, not the "merge ≠ publish" redesign, which conflicts with this template's intentional continuous‑release model.