From 73432ec4e375ee3e9bae116e823a9017832ac98b Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 07:31:34 -0700 Subject: [PATCH 1/5] Stop crediting #65 with a fix that never ran The item described the mechanism #75 replaced: installing the Ubuntu package and calling `git restore-mtime static`. CI uses a pinned action now, and make-release.sh requires v2025.08 or newer by either invocation form. The worse half was a claim of mine, added earlier the same day, that release 20260809-030521 was the first production deploy to exercise the restore. It exercised nothing. That release ran v2022.12, which refuses to run under current git and exits 0 regardless, and the host measured the result as 0 of 3,275 shared inodes. The entry asserted a working feature on the strength of a merge rather than a measurement, which is the exact distinction this repo keeps having to relearn. The item now credits #75 and #76, says plainly that #65 shipped broken, and carries what was actually measured: 1052 of 3269 linked on the local production mirror with the tool installed by hand, the release stamped 2026-08-01 rather than at build time, and 584 MB then 18 MB for two releases. It also records the part that will otherwise read as a regression: the first restored release cannot link, because its predecessor was built with unrestored mtimes, so the first deploy after the promotion is expected to link zero and the one after it is the real test. The retained #65 diagnosis keeps its wording and gains a lead-in saying so. It contains a "today" that means 2026-08-08 and names a fix that did not run, both of which read as current state without it. Found by Copilot review on #77. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/TODO.md b/TODO.md index e556aaf..3199726 100644 --- a/TODO.md +++ b/TODO.md @@ -42,8 +42,13 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. - **Media is checked live now, which unblocks the item below.** [`checks/golden-media-live.txt`](./checks/golden-media-live.txt) is fetched by `check-live-urls.sh` against a running server, covering both media trees and the `@uploads` rule, and asserting status, a non-zero body and an image content type so that a 403 from a bad mode, a 404 from a lost transfer, a truncated file and a soft-404 error page are each caught. Verified against production, and each of the four failure shapes was reproduced rather than assumed. The record of why it was needed follows. - **~~Nothing checks that media survived the trip to the server.~~ Closed 2026-08-08, by the item above.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. -- **~~Restore file mtimes in CI so `--link-dest` links.~~ Done, [#65][pr-65], and on `main` since the 2026-08-09 promotion.** `deploy-site-task.yml` installs `git-restore-mtime` and runs `git restore-mtime static` before the upload. Measured with two independent clones each way: 0 of 1791 files linked without the restore, 1052 with, and that 1052 corroborates from three directions, being what Hugo reports as static files, what links on a locally built release, and the file count in `static/`. **Release `20260809-030521` is the first production deploy to exercise it**, done 2026-08-09, so it is also the first where a badly moded file can ride a link into every later release, which is the trap below. The live media check landed first, deliberately, and is what watches for exactly that. **The shared-inode count is the open half**: it is server-side, the host had been measuring 0 across every pipeline release, and roughly 1052 of 1791 is what the change predicts. Asked in the channel rather than inferred from this side. The reasoning is kept below because it is what the next change to the deploy's linking behavior will need. - - The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. +- **~~Restore file mtimes so `--link-dest` links.~~ Done, but by [#75][pr-75] and [#76][pr-76] rather than by [#65][pr-65], which shipped it broken.** CI restores with `chetan/git-restore-mtime-action`, pinned by SHA, and [`deploy/make-release.sh`](./deploy/make-release.sh) requires the tool on `PATH` or as a git subcommand at **v2025.08 or newer**. Both then assert the outcome: nothing under `static/` may be newer than HEAD's own commit, since a checkout always happens after the commit it checks out. + - **#65 never restored anything and exited 0 for four releases.** It installed the Debian and Ubuntu package, git-tools v2022.12, which shells out to `git whatchanged`; current git refuses that without `--i-still-use-this`, a flag there is no way to pass through. It printed `1,052 files to be processed`, processed none, and succeeded, twelve refusals inside one green step. **Release `20260809-030521` did not exercise it**, an earlier version of this entry said it did, and the host measured the truth as 0 of 3,275 shared inodes. Local git still permits `whatchanged`, which is why #65's two-clone measurement was honest and measured everything except the machine that runs it. + - **Verified end to end on the local production mirror, 2026-08-09**, with the tool installed by hand: `1052 of 3269 files hard-linked`, and the release carrying `site/media` and `site/external` stamped `2026-08-01`, the commit date rather than the build date, which is what proves the restore reached the release and not merely the working tree. Two releases cost 584 MB then 18 MB. + - **The first restored release cannot link, by construction**, because it is compared against a predecessor built with unrestored mtimes. `make-release.sh`'s zero-shared-files guard refuses that one, so the changeover takes a single `NO_LINK_DEST=1` release to seed a restored generation. The deploy has no such guard and will simply link nothing for one deploy. **So the first deploy after the promotion is expected to link 0**, and the one after it is the real test. + - **The shared-inode count on the VPS is still the open half** and is the host's to measure, since nothing here can read an inode there. Told to them in the channel as §S, which corrects §R for predicting the jump one deploy too early. Roughly 1052 of 3275 is what the change predicts, at the *second* restored deploy. + - Making `--link-dest` effective is also what makes a badly moded file able to ride a link into every later release, which is the trap below. The live media check landed first, deliberately, and is what watches for exactly that. + - **The original diagnosis, kept as written on 2026-08-08 because it is what the next change to the deploy's linking behaviour will need.** Read it as the record of why the change was made, not as current state: the "today" in it is that date, and the fix it names is #65's, which turned out not to run. The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. - **~~Publish a release from `main`, once the pipeline has soaked.~~ Done 2026-08-09, with the maintainer lifting the soak condition explicitly.** `1.0.54` is cut from `main` and marked Latest, and `1.0.54-gcd9285a9fe` from `develop`, each tag pinned to its own branch tip. Before this, `1.0.11` had been the newest from `main` since 2026-08-01, predating every deploy change. **The soak reasoning is kept because it was not satisfied, it was overridden**, and the same argument returns at the next release: what a release from `main` cannot yet claim is a switchover that has been *trusted* rather than merely green, which the log review under **Recurring operations** establishes and no gate can. `1.0.54` therefore names a working pipeline and a rehearsal host, not a completed cutover. - **Gate the deploy ref by environment in both directions, so a deploy cannot do what a release structurally cannot.** [`deploy-site.yml`](./.github/workflows/deploy-site.yml)'s `assert-ref` job refuses production from any ref but `main`, and **staging accepts any ref at all**. Add the missing arm so staging accepts only `develop`: `develop` deploys staging, `main` deploys production, and neither can reach the other's environment. @@ -201,6 +206,8 @@ The deploy root is deliberately absent from this table. The rsync destination is [pr-68]: https://github.com/ptr727/Blog/pull/68 [pr-69]: https://github.com/ptr727/Blog/pull/69 [pr-70]: https://github.com/ptr727/Blog/pull/70 +[pr-75]: https://github.com/ptr727/Blog/pull/75 +[pr-76]: https://github.com/ptr727/Blog/pull/76 From 67eaa9e92ae51a6c42ec2573610657b9ff6f2de2 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 07:34:32 -0700 Subject: [PATCH 2/5] Use the file's US spelling The lead-in added one commit ago wrote "behaviour" two words away from the "behavior" it was introducing, in a file that uses the US form throughout. Nothing gates spelling here: cspell runs over README.md and HISTORY.md only, so this is convention rather than a rule, which is why it needed a reader to catch it. Found by Copilot review on #78. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index 3199726..acf8466 100644 --- a/TODO.md +++ b/TODO.md @@ -48,7 +48,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **The first restored release cannot link, by construction**, because it is compared against a predecessor built with unrestored mtimes. `make-release.sh`'s zero-shared-files guard refuses that one, so the changeover takes a single `NO_LINK_DEST=1` release to seed a restored generation. The deploy has no such guard and will simply link nothing for one deploy. **So the first deploy after the promotion is expected to link 0**, and the one after it is the real test. - **The shared-inode count on the VPS is still the open half** and is the host's to measure, since nothing here can read an inode there. Told to them in the channel as §S, which corrects §R for predicting the jump one deploy too early. Roughly 1052 of 3275 is what the change predicts, at the *second* restored deploy. - Making `--link-dest` effective is also what makes a badly moded file able to ride a link into every later release, which is the trap below. The live media check landed first, deliberately, and is what watches for exactly that. - - **The original diagnosis, kept as written on 2026-08-08 because it is what the next change to the deploy's linking behaviour will need.** Read it as the record of why the change was made, not as current state: the "today" in it is that date, and the fix it names is #65's, which turned out not to run. The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. + - **The original diagnosis, kept as written on 2026-08-08 because it is what the next change to the deploy's linking behavior will need.** Read it as the record of why the change was made, not as current state: the "today" in it is that date, and the fix it names is #65's, which turned out not to run. The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. - **~~Publish a release from `main`, once the pipeline has soaked.~~ Done 2026-08-09, with the maintainer lifting the soak condition explicitly.** `1.0.54` is cut from `main` and marked Latest, and `1.0.54-gcd9285a9fe` from `develop`, each tag pinned to its own branch tip. Before this, `1.0.11` had been the newest from `main` since 2026-08-01, predating every deploy change. **The soak reasoning is kept because it was not satisfied, it was overridden**, and the same argument returns at the next release: what a release from `main` cannot yet claim is a switchover that has been *trusted* rather than merely green, which the log review under **Recurring operations** establishes and no gate can. `1.0.54` therefore names a working pipeline and a rehearsal host, not a completed cutover. - **Gate the deploy ref by environment in both directions, so a deploy cannot do what a release structurally cannot.** [`deploy-site.yml`](./.github/workflows/deploy-site.yml)'s `assert-ref` job refuses production from any ref but `main`, and **staging accepts any ref at all**. Add the missing arm so staging accepts only `develop`: `develop` deploys staging, `main` deploys production, and neither can reach the other's environment. From 110faeef9136a4be3a0d66a3191eae937a32c6c2 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 07:37:50 -0700 Subject: [PATCH 3/5] Describe the assertion's actual scope, not a stronger one The entry said both paths assert that nothing under static/ is newer than HEAD's commit. CI does. make-release.sh excludes modified and untracked paths, deliberately, because a working tree can legitimately hold a static file newer than any commit and skipping the check whenever the tree is dirty would make it useless during an edit loop. So the claim was stronger than the code, which is the same defect this file keeps producing: prose that describes the rule someone meant rather than the one that runs. It now states the shared invariant as tracked and unmodified files, and names why the two paths differ: a fresh checkout has nothing uncommitted in it, so CI needs no exclusion to reach the same guarantee. Found by Copilot review on #78, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index acf8466..7fa4fb7 100644 --- a/TODO.md +++ b/TODO.md @@ -42,7 +42,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. - **Media is checked live now, which unblocks the item below.** [`checks/golden-media-live.txt`](./checks/golden-media-live.txt) is fetched by `check-live-urls.sh` against a running server, covering both media trees and the `@uploads` rule, and asserting status, a non-zero body and an image content type so that a 403 from a bad mode, a 404 from a lost transfer, a truncated file and a soft-404 error page are each caught. Verified against production, and each of the four failure shapes was reproduced rather than assumed. The record of why it was needed follows. - **~~Nothing checks that media survived the trip to the server.~~ Closed 2026-08-08, by the item above.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. -- **~~Restore file mtimes so `--link-dest` links.~~ Done, but by [#75][pr-75] and [#76][pr-76] rather than by [#65][pr-65], which shipped it broken.** CI restores with `chetan/git-restore-mtime-action`, pinned by SHA, and [`deploy/make-release.sh`](./deploy/make-release.sh) requires the tool on `PATH` or as a git subcommand at **v2025.08 or newer**. Both then assert the outcome: nothing under `static/` may be newer than HEAD's own commit, since a checkout always happens after the commit it checks out. +- **~~Restore file mtimes so `--link-dest` links.~~ Done, but by [#75][pr-75] and [#76][pr-76] rather than by [#65][pr-65], which shipped it broken.** CI restores with `chetan/git-restore-mtime-action`, pinned by SHA, and [`deploy/make-release.sh`](./deploy/make-release.sh) requires the tool on `PATH` or as a git subcommand at **v2025.08 or newer**. Both then assert the outcome, since a checkout always happens after the commit it checks out: **no tracked, unmodified file under `static/` may be newer than HEAD's own commit.** CI compares the whole tree, because a fresh checkout has nothing uncommitted in it; the local path excludes modified and untracked paths, which a working tree can legitimately hold newer than any commit, so the check stays usable during an edit loop rather than being skipped whenever the tree is dirty. - **#65 never restored anything and exited 0 for four releases.** It installed the Debian and Ubuntu package, git-tools v2022.12, which shells out to `git whatchanged`; current git refuses that without `--i-still-use-this`, a flag there is no way to pass through. It printed `1,052 files to be processed`, processed none, and succeeded, twelve refusals inside one green step. **Release `20260809-030521` did not exercise it**, an earlier version of this entry said it did, and the host measured the truth as 0 of 3,275 shared inodes. Local git still permits `whatchanged`, which is why #65's two-clone measurement was honest and measured everything except the machine that runs it. - **Verified end to end on the local production mirror, 2026-08-09**, with the tool installed by hand: `1052 of 3269 files hard-linked`, and the release carrying `site/media` and `site/external` stamped `2026-08-01`, the commit date rather than the build date, which is what proves the restore reached the release and not merely the working tree. Two releases cost 584 MB then 18 MB. - **The first restored release cannot link, by construction**, because it is compared against a predecessor built with unrestored mtimes. `make-release.sh`'s zero-shared-files guard refuses that one, so the changeover takes a single `NO_LINK_DEST=1` release to seed a restored generation. The deploy has no such guard and will simply link nothing for one deploy. **So the first deploy after the promotion is expected to link 0**, and the one after it is the real test. From f88dba834a05bd6cc187911daad8417034ee4994 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 07:40:41 -0700 Subject: [PATCH 4/5] Say static/ where static/ is what is walked "CI compares the whole tree" reads as the repository. Both assertions run `find static`, and nothing outside that directory is examined by either, which matters because the sentence is the thing a future edit to the guard would be read against. Verified against both implementations rather than from memory: .github/workflows/deploy-site-task.yml:133 find static -type f deploy/make-release.sh:186 find static -type f The difference between the two paths is the exclusion, not the scope, and the sentence now says so. Found by Copilot review on #78, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index 7fa4fb7..3b7a1ac 100644 --- a/TODO.md +++ b/TODO.md @@ -42,7 +42,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. - **Media is checked live now, which unblocks the item below.** [`checks/golden-media-live.txt`](./checks/golden-media-live.txt) is fetched by `check-live-urls.sh` against a running server, covering both media trees and the `@uploads` rule, and asserting status, a non-zero body and an image content type so that a 403 from a bad mode, a 404 from a lost transfer, a truncated file and a soft-404 error page are each caught. Verified against production, and each of the four failure shapes was reproduced rather than assumed. The record of why it was needed follows. - **~~Nothing checks that media survived the trip to the server.~~ Closed 2026-08-08, by the item above.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. -- **~~Restore file mtimes so `--link-dest` links.~~ Done, but by [#75][pr-75] and [#76][pr-76] rather than by [#65][pr-65], which shipped it broken.** CI restores with `chetan/git-restore-mtime-action`, pinned by SHA, and [`deploy/make-release.sh`](./deploy/make-release.sh) requires the tool on `PATH` or as a git subcommand at **v2025.08 or newer**. Both then assert the outcome, since a checkout always happens after the commit it checks out: **no tracked, unmodified file under `static/` may be newer than HEAD's own commit.** CI compares the whole tree, because a fresh checkout has nothing uncommitted in it; the local path excludes modified and untracked paths, which a working tree can legitimately hold newer than any commit, so the check stays usable during an edit loop rather than being skipped whenever the tree is dirty. +- **~~Restore file mtimes so `--link-dest` links.~~ Done, but by [#75][pr-75] and [#76][pr-76] rather than by [#65][pr-65], which shipped it broken.** CI restores with `chetan/git-restore-mtime-action`, pinned by SHA, and [`deploy/make-release.sh`](./deploy/make-release.sh) requires the tool on `PATH` or as a git subcommand at **v2025.08 or newer**. Both then assert the outcome, since a checkout always happens after the commit it checks out: **no tracked, unmodified file under `static/` may be newer than HEAD's own commit.** CI compares every file under `static/`, because a fresh checkout has nothing uncommitted in it; the local path walks the same directory but excludes modified and untracked paths, which a working tree can legitimately hold newer than any commit, so the check stays usable during an edit loop rather than being skipped whenever the tree is dirty. - **#65 never restored anything and exited 0 for four releases.** It installed the Debian and Ubuntu package, git-tools v2022.12, which shells out to `git whatchanged`; current git refuses that without `--i-still-use-this`, a flag there is no way to pass through. It printed `1,052 files to be processed`, processed none, and succeeded, twelve refusals inside one green step. **Release `20260809-030521` did not exercise it**, an earlier version of this entry said it did, and the host measured the truth as 0 of 3,275 shared inodes. Local git still permits `whatchanged`, which is why #65's two-clone measurement was honest and measured everything except the machine that runs it. - **Verified end to end on the local production mirror, 2026-08-09**, with the tool installed by hand: `1052 of 3269 files hard-linked`, and the release carrying `site/media` and `site/external` stamped `2026-08-01`, the commit date rather than the build date, which is what proves the restore reached the release and not merely the working tree. Two releases cost 584 MB then 18 MB. - **The first restored release cannot link, by construction**, because it is compared against a predecessor built with unrestored mtimes. `make-release.sh`'s zero-shared-files guard refuses that one, so the changeover takes a single `NO_LINK_DEST=1` release to seed a restored generation. The deploy has no such guard and will simply link nothing for one deploy. **So the first deploy after the promotion is expected to link 0**, and the one after it is the real test. From beccfb05d05568963652224844aa06e1e8482492 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 07:46:20 -0700 Subject: [PATCH 5/5] Match the file's number format, and say permissions rather than mode Counts in the new text alternated between 1052 and 1,052 next to 0 of 3,275 in the same block, which makes two measurements of the same quantity look like different quantities at a glance. The comma form is what this file mostly uses, ten instances to six, and 1,052 already appeared with one, so the new text follows it. The preserved 2026-08-08 diagnosis is deliberately left alone. Its lead-in says it is kept as written, and reformatting inside it would make that false for the sake of consistency it explicitly opts out of. And "a badly moded file" reads as fashion rather than permissions. It is now "a file that acquires the wrong permissions", which is what the sentence is about: a hard link carries its inode's mode, so a wrong one rides the chain into every later release. Found by Copilot review on #78, as suppressed comments. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/TODO.md b/TODO.md index 3b7a1ac..055fe2f 100644 --- a/TODO.md +++ b/TODO.md @@ -44,10 +44,10 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **~~Nothing checks that media survived the trip to the server.~~ Closed 2026-08-08, by the item above.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. - **~~Restore file mtimes so `--link-dest` links.~~ Done, but by [#75][pr-75] and [#76][pr-76] rather than by [#65][pr-65], which shipped it broken.** CI restores with `chetan/git-restore-mtime-action`, pinned by SHA, and [`deploy/make-release.sh`](./deploy/make-release.sh) requires the tool on `PATH` or as a git subcommand at **v2025.08 or newer**. Both then assert the outcome, since a checkout always happens after the commit it checks out: **no tracked, unmodified file under `static/` may be newer than HEAD's own commit.** CI compares every file under `static/`, because a fresh checkout has nothing uncommitted in it; the local path walks the same directory but excludes modified and untracked paths, which a working tree can legitimately hold newer than any commit, so the check stays usable during an edit loop rather than being skipped whenever the tree is dirty. - **#65 never restored anything and exited 0 for four releases.** It installed the Debian and Ubuntu package, git-tools v2022.12, which shells out to `git whatchanged`; current git refuses that without `--i-still-use-this`, a flag there is no way to pass through. It printed `1,052 files to be processed`, processed none, and succeeded, twelve refusals inside one green step. **Release `20260809-030521` did not exercise it**, an earlier version of this entry said it did, and the host measured the truth as 0 of 3,275 shared inodes. Local git still permits `whatchanged`, which is why #65's two-clone measurement was honest and measured everything except the machine that runs it. - - **Verified end to end on the local production mirror, 2026-08-09**, with the tool installed by hand: `1052 of 3269 files hard-linked`, and the release carrying `site/media` and `site/external` stamped `2026-08-01`, the commit date rather than the build date, which is what proves the restore reached the release and not merely the working tree. Two releases cost 584 MB then 18 MB. + - **Verified end to end on the local production mirror, 2026-08-09**, with the tool installed by hand: `1,052 of 3,269 files hard-linked`, and the release carrying `site/media` and `site/external` stamped `2026-08-01`, the commit date rather than the build date, which is what proves the restore reached the release and not merely the working tree. Two releases cost 584 MB then 18 MB. - **The first restored release cannot link, by construction**, because it is compared against a predecessor built with unrestored mtimes. `make-release.sh`'s zero-shared-files guard refuses that one, so the changeover takes a single `NO_LINK_DEST=1` release to seed a restored generation. The deploy has no such guard and will simply link nothing for one deploy. **So the first deploy after the promotion is expected to link 0**, and the one after it is the real test. - - **The shared-inode count on the VPS is still the open half** and is the host's to measure, since nothing here can read an inode there. Told to them in the channel as §S, which corrects §R for predicting the jump one deploy too early. Roughly 1052 of 3275 is what the change predicts, at the *second* restored deploy. - - Making `--link-dest` effective is also what makes a badly moded file able to ride a link into every later release, which is the trap below. The live media check landed first, deliberately, and is what watches for exactly that. + - **The shared-inode count on the VPS is still the open half** and is the host's to measure, since nothing here can read an inode there. Told to them in the channel as §S, which corrects §R for predicting the jump one deploy too early. Roughly 1,052 of 3,275 is what the change predicts, at the *second* restored deploy. + - Making `--link-dest` effective is also what makes a file that acquires the wrong permissions able to ride a link into every later release, which is the trap below. The live media check landed first, deliberately, and is what watches for exactly that. - **The original diagnosis, kept as written on 2026-08-08 because it is what the next change to the deploy's linking behavior will need.** Read it as the record of why the change was made, not as current state: the "today" in it is that date, and the fix it names is #65's, which turned out not to run. The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. - **~~Publish a release from `main`, once the pipeline has soaked.~~ Done 2026-08-09, with the maintainer lifting the soak condition explicitly.** `1.0.54` is cut from `main` and marked Latest, and `1.0.54-gcd9285a9fe` from `develop`, each tag pinned to its own branch tip. Before this, `1.0.11` had been the newest from `main` since 2026-08-01, predating every deploy change. **The soak reasoning is kept because it was not satisfied, it was overridden**, and the same argument returns at the next release: what a release from `main` cannot yet claim is a switchover that has been *trusted* rather than merely green, which the log review under **Recurring operations** establishes and no gate can. `1.0.54` therefore names a working pipeline and a rehearsal host, not a completed cutover.