Skip to content

Fix fresh-install failures in the mirror playbooks (task ordering, SELinux, certbot failure handling) - #5

Open
georgetasioulis wants to merge 3 commits into
cloudlinux:mainfrom
georgetasioulis:fix/playbook-selinux-and-ordering
Open

Fix fresh-install failures in the mirror playbooks (task ordering, SELinux, certbot failure handling)#5
georgetasioulis wants to merge 3 commits into
cloudlinux:mainfrom
georgetasioulis:fix/playbook-selinux-and-ordering

Conversation

@georgetasioulis

@georgetasioulisgeorgetasioulis commented Aug 5, 2026

Copy link
Copy Markdown

Summary

While standing up a new SWNG mirror on a fresh AlmaLinux 10 server with ansible/complete-swng-rsync, I hit seven issues that prevent the playbooks from completing (or silently break syncing afterwards). All of them exist in the other playbook variants too, so this PR fixes them across complete-swng-rsync, combined-mirror, specific-version-rsync(Recomended), and yum-reposync.

1. Disk-space check runs before the mirror directory exists

The first task runs df -h {{ mirror_base_path }}, but the directory is only created a few tasks later — on a fresh server the play aborts immediately with df: /var/www/mirrors: No such file or directory. Fixed by creating the directory before the check.

2. SELinux relabel is not recursive

The relabel task adds the public_content_rw_t fcontext rule but runs restorecon non-recursively on /var/www only. Subdirectories — including the ACME webroot — keep their var_t label, so nginx returns 403 for Let's Encrypt HTTP-01 challenges and certbot can never obtain a certificate. Re-running the playbook doesn't help since the relabel repeats non-recursively. Fixed with restorecon -RFv.

3. rsync fails under systemd on enforcing SELinux (exit code 10)

systemd-spawned rsync transitions into the confined rsync_t domain, which by default cannot make outbound connections:

type=AVC msg=audit(...): avc: denied { name_connect } for pid=31169 comm="rsync"
dest=873 scontext=system_u:system_r:rsync_t:s0
tcontext=system_u:object_r:rsync_port_t:s0 tclass=tcp_socket permissive=0

Every timer-triggered sync dies instantly (rsync error: error in socket IO (code 10)), while the Ansible-invoked initial sync works because it runs unconfined — masking the bug until the first timer firing. Fixed by setting the rsync_client and rsync_anon_write booleans in the rsync-based playbooks (not needed for yum-reposync, which doesn't run in rsync_t).

4. HTTPS vhost deployed even when certbot failed

The certbot tasks use ignore_errors: yes, but the HTTPS vhost template is deployed unconditionally afterwards. When certificate issuance fails (e.g. because of issue 2), nginx is pointed at nonexistent certificate files and fails to start entirely — which in turn blocks any webroot-based retry, leaving the server in a state the playbook cannot recover from. Fixed by gating the HTTPS vhost (and the HTTP→HTTPS redirect) on the certificate actually existing, with a visible warning when it is skipped.

5. Confined rsync cannot write the upstream-owned tree (silent no-op syncs)

With issue 3 fixed, syncs connect but still fail: rsync -a preserves upstream ownership, so the tree is owned by non-root uids (e.g. 11122/11373) with mode 775 directories. The confined rsync_t receiver runs as root but is not granted the dac_override capability, so every mkstemp fails with Permission denied (13) — and the denial is dontaudited, so the audit log stays empty while the sync grinds through the entire file list writing nothing:

type=AVC msg=audit(...): avc: denied { dac_override } for comm="rsync" capability=1
scontext=system_u:system_r:rsync_t:s0 tclass=capability permissive=0

(captured with semodule -DB). The rsync_full_access boolean does not help — on AlmaLinux 10 it grants file-access rules but not the capability.

Rather than granting rsync_t the dac_override capability (broader than the problem requires), the fix stops preserving upstream ownership, which has no value on a mirror that only serves files over HTTP: --no-owner --no-group is added to the initial sync commands and every systemd service template, so the tree is root-owned and confined rsync needs no extra capability. Existing deployments synced with the old owner-preserving flags are normalized once via a find-guarded recursive chown.

6. Requires= in the timer units lets a service stop silently disable syncing

Every timer declares Requires=<sync>.service. With Requires, an explicit systemctl stop of the sync service propagates and stops the timer too — all future scheduled syncs are silently disabled and the mirror goes stale with nothing reporting a failure (observed in practice after stopping the service once during maintenance). It also makes starting the timer trigger an immediate sync run. Timer units need no dependency on the service they trigger; the line is removed.

7. verify.yml aborts on mirrors that are not a dedicated mountpoint

verify.yml hard-fails when mirror_base_path is not a mountpoint, so on a valid single-disk setup (mirror on the root filesystem) verification aborts before any of the useful checks run. Downgraded to a warning.

Testing

  • Hit all five issues on a real deployment (fresh AlmaLinux 10.2, complete-swng-rsync); verified each fix resolves the corresponding failure and the full playbook now converges: cert issued, HTTPS serving, timer-driven syncs running confined under rsync_t and writing the tree. For issue 5, verified both directions on the same server: the confined service fails on the upstream-owned tree and writes normally after chown -R root:root.
  • ansible-playbook --syntax-check passes for all four modified playbooks.

🤖 Generated with Claude Code

Four issues that break the playbooks on a fresh AlmaLinux 10 server:
1. Disk-space check ran before the mirror directory was created,
aborting the play on first run (df on a nonexistent path).
Reordered directory creation before the check.
2. The SELinux relabel used a non-recursive restorecon, leaving
subdirectories (including the ACME webroot) labeled var_t. Nginx
then returned 403 for Let's Encrypt HTTP-01 challenges and no
certificate could ever be issued. Now relabels recursively.
3. rsync run from the systemd service is confined to rsync_t, which
cannot make outbound connections by default (AVC name_connect
denial on port 873, rsync exit code 10). Every timer-triggered
sync failed instantly, while manual/Ansible-invoked rsync worked
because it runs unconfined - masking the bug until the timer
fired. Set the rsync_client and rsync_anon_write booleans in the
rsync-based playbooks.
4. The HTTPS vhost was deployed even when certbot failed (its
failure is ignored), pointing nginx at nonexistent certificate
files. nginx then failed to start at all, which also blocks any
webroot retry - a chicken-and-egg state requiring manual
recovery. The HTTPS config is now gated on the certificate
actually existing, with a warning when it is skipped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@georgetasioulis
georgetasioulisforce-pushed the fix/playbook-selinux-and-ordering branch from aa8451d to ff0ce56CompareAugust 6, 2026 00:00
With the rsync_client boolean set, timer-driven syncs connect but
still fail to write: rsync -a preserves upstream ownership, so the
tree ends up owned by non-root uids (e.g. 11122/11373) with mode 775
directories. The systemd-run rsync receiver is confined to rsync_t,
which lacks the dac_override capability, so root cannot create files
in directories it does not own:
avc: denied { dac_override } for comm="rsync" capability=1
scontext=system_u:system_r:rsync_t:s0 tclass=capability permissive=0
The denial is dontaudited by default, so the audit log stays empty
(visible only with semodule -DB) while every mkstemp fails EACCES and
the sync grinds through the whole file list writing nothing. The
rsync_full_access boolean does not help - on AlmaLinux 10 it grants
file access rules but not the capability.
Upstream UID/GID values have no meaning on a mirror that only serves
files over HTTP, so stop preserving them instead of widening policy:
- Add --no-owner --no-group to the initial sync commands and every
systemd service template; transferred files are then owned by the
receiving user (root) and confined rsync_t writes root-owned
directories without any extra capability.
- Normalize existing trees once (guarded by a find probe so the
recursive chown only runs when non-root-owned files exist, e.g. on
deployments synced with the previous owner-preserving flags).
Verified on a real AlmaLinux 10.2 deployment: the confined service
fails on the upstream-owned tree and writes normally after
chown -R root:root.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@georgetasioulis
georgetasioulisforce-pushed the fix/playbook-selinux-and-ordering branch from ff0ce56 to f6fbf32CompareAugust 6, 2026 00:02
Two smaller issues found while operating a deployed mirror:
1. Every timer unit declares Requires=<sync>.service. With Requires,
an explicit systemctl stop of the sync service propagates and
stops the timer as well, silently disabling all future scheduled
syncs - the mirror then goes stale with nothing reporting a
failure. (It also makes starting the timer trigger an immediate
sync run.) Timer units need no dependency on the service they
trigger; remove it.
2. verify.yml hard-fails when the mirror path is not a dedicated
mountpoint, aborting before any of the useful checks run. A
mirror on the root filesystem (e.g. a single-disk VM) is a valid
setup - downgrade the check to a warning so verification can
proceed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@georgetasioulis