Skip to content

Remove the VPS Backup Pull, Now Maintained Outside This Repository#111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ptr727 merged 2 commits into from
Aug 24, 2026
Merged

Remove the VPS Backup Pull, Now Maintained Outside This Repository #111

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
16985e5
Remove the VPS backup pull, now maintained outside this repository
ptr727 Aug 24, 2026
9fab953
Fix overclaim in TODO.md flagged by CodeRabbit
ptr727 Aug 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .secrets/README.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -9,10 +9,10 @@ exposing one.

The documented local convention is `~/.secrets/`, not this directory. CI supplies the same
values directly from the GitHub Environment instead, reading no file here at all.
`ENV_FILE=<name> deploy/make-release.sh` and `ops/install.sh` both resolve a relative `ENV_FILE`
against `$HOME/.secrets`, refuse a traversing one, and default to
`~/.secrets/Blog.local.production.env`. An absolute `ENV_FILE` is honored as given rather than
resolved against `~/.secrets/`, an escape hatch rather than the documented shape.
`ENV_FILE=<name> deploy/make-release.sh` resolves a relative `ENV_FILE` against `$HOME/.secrets`,
refuses a traversing one, and defaults to `~/.secrets/Blog.local.production.env`. An absolute
`ENV_FILE` is honored as given rather than resolved against `~/.secrets/`, an escape hatch rather
than the documented shape.
`~/.secrets/` is shared across every repo on the host, so each of this repo's files carries the
`Blog.` prefix:

Expand Down
50 changes: 6 additions & 44 deletions .secrets/example.env
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -4,15 +4,9 @@
# This file states the shape and a placeholder. Add a value here and describe it there,
# or checks/check-env-docs.py fails.
#
# It fills two destinations, marked below, because a value belongs to whichever machine
# holds it. Copy the section you need rather than the whole file.
#
# ~/.secrets/Blog.<server>.<environment>.env on a workstation, one file per environment
# /etc/vps-backup-pull.env on the backup host, or let ops/install.sh write it
#
# A value appearing in both sections is spelled the same way in both, deliberately. One
# name per thing means the side that writes and the side that reads cannot disagree, which
# is also why ops/install.sh copies values across rather than translating them.
# One destination: ~/.secrets/Blog.<server>.<environment>.env on a workstation, one file
# per environment. The off-host VPS backup pull, and the environment file it reads, are
# maintained outside this repository -- see OPERATIONS.md, Working With the VPS.
#
# Naming convention: the prefix names whatever owns the value, not whatever reads it.
# SITE_ is the generic deploy contract, DEPLOY_ is the release tooling, CADDY_ is the
Expand DownExpand Up@@ -87,39 +81,7 @@ VPS_TRAEFIK_LOG=/var/log/traefik/access.log
# The two agent channel files on the VPS.
VPS_COMMS_DIR=/srv/agent-comms

# =============================================================================
# Both destinations
# =============================================================================
# These name the off-host copy, so the pull writes them and the log review reads them.

# Off-host archives and the plaintext hostconfig tree beside them.
BACKUP_ARCHIVE_ROOT=/path/to/backup/vps

# Off-host copy of the rotated logs. Mode 700, since query strings are logged in full.
# Off-host copy of the rotated logs, written by the pull maintained outside this
# repository. Read by the log review. Mode 700 on the backup host, since query strings
# are logged in full.
LOG_ARCHIVE_ROOT=/path/to/backup/vps-logs

# The rotated access logs on the VPS, and the source of that copy.
VPS_TRAEFIK_LOG_ARCHIVE=/var/log/traefik/archive

# =============================================================================
# /etc/vps-backup-pull.env
# =============================================================================
# On the backup host. Also needs VPS_SSH_HOST and the three values above.
#
# systemd parses this file itself rather than passing it to a shell, so there is no
# expansion and no command substitution: a $ or a backtick is a literal character. It does
# strip matching quotes, which is why a value containing spaces is quoted and arrives
# without them.
#
# VPS_SSH_HOST, BACKUP_ARCHIVE_ROOT and LOG_ARCHIVE_ROOT have no defaults in the pull. An
# address and a destination belong to one host, and a wrong-but-valid destination is a
# backup nobody can find, so it names what is missing and refuses to run.

# The layout on the VPS, the same for any host running this stack.
VPS_ARCHIVE_DIR=/var/backups/pangolin
VPS_BLOG_LOG_DIR=/var/log/blog/legacy

# Key auth only, since the VPS has password auth disabled.
# Quoted because it contains spaces: this file is sourced by a shell for the secrets/
# half, where a bare value would run everything after the first space as a command.
SSH_OPTS="-o ConnectTimeout=15 -o BatchMode=yes"
22 changes: 1 addition & 21 deletions ENVIRONMENT.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -33,31 +33,11 @@ Held in `~/.secrets/Blog.<server>.<environment>.env`, one file per environment,
| `CAPTURE_AUTHOR_SLUG` | the old platform's author slug, used to backfill the author archive and its pagination | Optional, and an account name rather than a site value, which is why it is a variable at all. Unset, [`capture/classify.py`](./capture/classify.py) skips the backfill and says so, rather than emitting a list that is silently short by the author URLs. Environment-independent. |
| `VPS_SSH_HOST` | the VPS administrative login | Not the deploy account. See "Two credentials" below. Environment-independent. |
| `VPS_TRAEFIK_LOG` | today's live access log on the VPS, still being appended to | Never pulled, because rotation is what makes a file eligible. An analysis covering today reads it over SSH. Nothing sources it. |
| `VPS_TRAEFIK_LOG_ARCHIVE` | the rotated access logs on the VPS, and the source of the off-host copy | Also read by the pull, below. |
| `VPS_COMMS_DIR` | the two agent channel files on the VPS | Nothing sources it, and the transfer commands in `OPERATIONS.md` are spelled out rather than using it. See "The one place indirection is wrong" below. |
| `BACKUP_ARCHIVE_ROOT` | the off-host encrypted archives and the plaintext `hostconfig` tree beside them | Written by the pull, read by a rebuild. |
| `LOG_ARCHIVE_ROOT` | the off-host copy of the rotated logs | Written by the pull, read by the log review. |
| `LOG_ARCHIVE_ROOT` | the off-host copy of the rotated logs | Written by a pull maintained outside this repository, read by the log review. |

Three more are named in the template but commented out, because CI resolves them from the GitHub Environment and a local run deploys to a path and needs none of them: `DEPLOY_SSH_HOST`, `DEPLOY_SSH_USER`, `DEPLOY_SSH_KNOWN_HOSTS`. They are listed there so the local file and the environment describe the same shape.

## The backup host

Held in `/etc/vps-backup-pull.env`, read by `vps-backup-pull` through the unit's `EnvironmentFile`. Template: [`.secrets/example.env`](./.secrets/example.env). [`ops/install.sh`](./ops/install.sh) generates it by copying from the repository environment file, which is why the four shared names are spelled identically in both.

| Value | Names | Notes |
| --- | --- | --- |
| `VPS_SSH_HOST` | where to pull from | Required. No default. |
| `BACKUP_ARCHIVE_ROOT` | where the archives and host config land | Required. No default. |
| `LOG_ARCHIVE_ROOT` | where both log sets land | Required unless `--no-logs`. No default. Mode 700, because query strings are logged in full. |
| `VPS_ARCHIVE_DIR` | the encrypted archives on the VPS | Defaults to the documented layout. |
| `VPS_TRAEFIK_LOG_ARCHIVE` | the rotated edge access logs on the VPS | Defaults to the documented layout. |
| `VPS_BLOG_LOG_DIR` | one-off Caddy container dumps on the VPS, kept from before rotation existed | Defaults to the documented layout. |
| `SSH_OPTS` | the SSH options the transfer uses | `BatchMode` makes an unusable key fail immediately rather than hanging a timed run on a password prompt nobody sees. |

**The three marked required carry no default on purpose.** An address and a destination belong to one host, and a wrong-but-valid destination is a backup nobody can find, so the pull names what is missing and refuses to run rather than falling back to something plausible.

**`systemd` parses this file itself rather than passing it to a shell**, so there is no expansion and no command substitution, and a `$` or a backtick is a literal character. It does strip matching quotes, verified rather than assumed, so a value containing spaces is quoted and arrives without them. That matters because [`.secrets/example.env`](./.secrets/example.env) is also sourced by a shell for the other destination, where an unquoted value would run everything after the first space as a command.

## The GitHub Environments

Held on the `production` and `staging` environments. The deploy workflow reads no file.
Expand Down
10 changes: 3 additions & 7 deletions OPERATIONS.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -210,28 +210,24 @@ ssh "$VPS_SSH_HOST" true && echo reachable
| --- | --- | --- |
| `VPS_SSH_HOST` | the administrative login | the VPS |
| `VPS_TRAEFIK_LOG` | today's live access log, still being appended to | the VPS |
| `VPS_TRAEFIK_LOG_ARCHIVE` | the rotated access logs, and the source of the off-host copy | the VPS |
| `VPS_COMMS_DIR` | the two agent channel files | the VPS |
| `LOG_ARCHIVE_ROOT` | the off-host copy of the rotated logs | the backup host |
| `BACKUP_ARCHIVE_ROOT` | the off-host encrypted archives and the plaintext hostconfig tree beside them | the backup host |

**There are two credentials to this host and picking the wrong one is the first mistake to avoid.** `DEPLOY_SSH_USER`, held per environment and used only by the deploy, reaches a confined account behind an `rrsync` forced command that can write one release tree and read nothing else. `VPS_SSH_HOST` is the ordinary administrative login used for everything on this page. They are deliberately separate credentials with different blast radii, so reaching for the deploy account to read a log fails in a way that reads like an outage, and reaching for the admin account to deploy grants far more than the deploy needs.

**The off-host copy is made by a script in this repository, [`ops/vps-backup-pull`](./ops/vps-backup-pull), on a `systemd` timer on the backup host.** It copies three things off the VPS into `BACKUP_ARCHIVE_ROOT` and `LOG_ARCHIVE_ROOT`: the encrypted archives, a plaintext copy of the same non-secret host files, and the rotated access logs. What it does, the three behaviors that look like bugs and are not, how to install it, and how to check it ran are in [`ops/README.md`](./ops/README.md). Read the unit and its last run on the backup host rather than trusting a schedule written down anywhere, including here.
**The off-host copy is made by a pull, maintained outside this repository, on a `systemd` timer on the backup host.** It copies the VPS's encrypted archives, a plaintext copy of its non-secret host files, and the rotated access logs, the last of those landing in `LOG_ARCHIVE_ROOT`. What it does beyond that, how it is installed, and how the VPS itself is provisioned and hardened are the backup host's own configuration to document, not this repository's -- this page covers only what Log Review needs: that `LOG_ARCHIVE_ROOT` exists and is where the rotated logs land. Read the unit and its last run on the backup host rather than trusting a schedule written down anywhere, including here.

**It is a pull rather than a push, and nothing on the VPS knows it happens.** That direction is the security property rather than an implementation detail: the backup host holds a key the VPS trusts, and the VPS holds no credential reaching any other system, so a compromise of the web server cannot walk into the backups that exist to survive it.

**Both sides use one set of names, so there is nothing to reconcile.** The pull writes `BACKUP_ARCHIVE_ROOT` and `LOG_ARCHIVE_ROOT` and the log review reads the same two, spelled the same way, and [`ops/install.sh`](./ops/install.sh) generates the pull's `EnvironmentFile` from this repository's `~/.secrets/` file by copying rather than translating. Every value is described once, in [`ENVIRONMENT.md`](./ENVIRONMENT.md), and [`checks/check-env-docs.py`](./checks/check-env-docs.py) fails if one is declared without a description or described without existing.
**`LOG_ARCHIVE_ROOT` is spelled the same way on both sides, so there is nothing to reconcile.** The pull writes it and the log review reads it, under the one name. Every value this repository reads or writes is described once, in [`ENVIRONMENT.md`](./ENVIRONMENT.md), and [`checks/check-env-docs.py`](./checks/check-env-docs.py) fails if one is declared without a description or described without existing.

```sh
set -a; . ~/.secrets/Blog.local.production.env; set +a
ls -d "$LOG_ARCHIVE_ROOT" "$BACKUP_ARCHIVE_ROOT"
ls -d "$LOG_ARCHIVE_ROOT"
```

**Today's traffic is never in the off-host copy, and that is deliberate.** Rotation is what makes a file eligible to be pulled, so a live log would be copied as a torn prefix and fetched again on the next run. An analysis covering today therefore reads `VPS_TRAEFIK_LOG` over SSH and everything older from `LOG_ARCHIVE_ROOT`, and treats the two as one series joined on `StartUTC` rather than on which file a line came from.

**The plaintext `hostconfig` tree under `BACKUP_ARCHIVE_ROOT` is the readable copy of the VPS's own configuration**, carrying the same non-secret files the encrypted archives hold. It exists so a rebuild does not depend on the encryption key, which is not on the backup host and must never be put there, because beside the ciphertext it would make the encryption decorative. What that tree covers is whatever the VPS advertises, read from the host rather than duplicated here, so it tracks the host instead of drifting from a list.

**The channel transfers are the one exception, and they must stay literal.** The permission allowlist in `.claude/settings.local.json` matches the text of a command rather than what it expands to, so substituting `"$VPS_SSH_HOST:$VPS_COMMS_DIR/..."` into those two `rsync` lines turns an allowed command into one that prompts, while looking like a tidy-up that changed nothing. Use the values above everywhere else, and leave the two commands under "The Channel Between the Two Sides" spelled out exactly as they are written there.

**What this section does not cover, and where it lives instead.** Reading the logs for content is "Log Review"; exchanging rounds with the agent that owns the host is "The Channel Between the Two Sides"; the boundary of which side fixes what is "Who Owns What"; and what a rebuild restores, including the host-key step that blocks both deploy and rollback, is "Backup and Restore".
Expand Down
2 changes: 0 additions & 2 deletions README.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -153,7 +153,6 @@ flowchart LR
| [`checks/`][checks] | the URL contract and the gates that enforce it |
| [`deploy/`][deploy] | the release script, the web-server config, and the redirect maps |
| [`capture/`][capture] | the migration's provenance tooling, and how the site was derived from the old platform's exports |
| [`ops/`][ops] | the pull that copies the server's backups and access logs off it, and its schedule |
| [`ENVIRONMENT.md`][environment] | every configuration value, described once |

Every configuration value is described in [ENVIRONMENT.md][environment]. The deploy procedure and the server layout are in [OPERATIONS.md][operations].
Expand DownExpand Up@@ -231,7 +230,6 @@ Licensed under the [MIT License][license]\
[license]: ./LICENSE
[migration-post]: ./content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md
[operations]: ./OPERATIONS.md
[ops]: ./ops/
[workflow]: ./WORKFLOW.md

<!-- External -->
Expand Down
Loading