From 34a1b0f8b12776db698ee879c6434d1da01bf368 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 09:35:41 -0700 Subject: [PATCH 1/5] Filter the log by the header, and say a hand probe has to carry it The outward pass told a reader to separate our traffic from real visitors by user agent, which is the coincidence X-Blog-Check was added to replace: the CI runner's curl and the host's own curl are byte-identical, so only the client address separates them and the CI half rotates every run. The header is the mechanism now, and its value carries provenance, so a run is identifiable rather than merely excludable. The gap worth recording is that check-live-urls.sh sends it and a bare curl does not, so an interactive probe lands in the visitor set unless whoever runs it passes -H. Two untagged probes turned up against 3,100 tagged ones in the 2026-08-09 deploy window, measured on the host side. The user-agent method stays, scoped to what it can still read: the days logged before the field existed. Kept as sub-bullets under the first filter rather than as four more bold paragraphs, because the section opens by saying it is four filters and a reader has to be able to count them. Co-Authored-By: Claude Opus 5 (1M context) --- OPERATIONS.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/OPERATIONS.md b/OPERATIONS.md index 22b83b8..a98b8a2 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -265,7 +265,13 @@ Two properties of the Caddy side are worth knowing before parsing it. Its access The outward pass is four filters over the edge log, and each one exists because skipping it produced a wrong answer once. -**Exclude this repository's own deploy gate first.** `check-live-urls.sh` requests the whole URL contract on every deploy, so an unfiltered day is mostly a recording of our own `curl`. Filter on user agent: on 2026-08-08, 9,285 of 9,996 requests were `curl/8.5.0` and the 711 that remained are the entire real dataset. A count that omits this step is measuring the pipeline rather than the readers, and it will be an order of magnitude too large. +**Exclude this repository's own deploy gate first.** `check-live-urls.sh` requests the whole URL contract on every deploy, so an unfiltered day is mostly a recording of our own `curl`. A count that omits this step is measuring the pipeline rather than the readers, and it will be an order of magnitude too large. + +The mechanism is the `X-Blog-Check` request header, which every synthetic request announces itself with, so `jq 'select(.["request_X-Blog-Check"] == null)'` is the whole filter. Its value is `/` rather than a boolean, so a run is identifiable rather than merely excludable: `github/-` from CI, `vps/smoke` from the host side, `proxmox/` from here. + +- **A hand probe carries it only because whoever runs it adds it.** `check-live-urls.sh` sends it on every request and a bare `curl` sends nothing, so an interactive probe passes `-H "X-Blog-Check: proxmox/"`, naming the purpose rather than the host. Two untagged probes turned up against 3,100 tagged ones in the 2026-08-09 deploy window. +- **Absence is not proof of a human**, since a scanner sends no header either, so this pairs with the scanner-shape filters below rather than replacing them. The field is forgeable and must never reach auth, rate limiting, robots handling, or caching. +- **Before 2026-08-09 the log carries no such field**, and user agent is the only key for those days: on 2026-08-08, 9,285 of 9,996 requests were `curl/8.5.0`, leaving 711 real ones. That key is a coincidence rather than a rule, since the CI runner's curl and the host's are byte-identical and only the rotating client address separates them, which is why the header exists. **A referer does not implicate this site unless it points somewhere else.** The rule worth applying is that a 404 carrying a referer is a broken link and a 404 without one is a typed or probed address, and it fails on scanners, which set `Referer` to the request URL itself. Every one of the 36 referer-bearing site-host 404s on 2026-08-08 was self-referential, so the unrefined rule reported three dozen broken links on a site that had none. Discard the matches before counting, and **normalize the scheme rather than comparing it**, because a scanner reaching an HTTPS site routinely sends an `http://` referer for the same address. Comparing against the request's own scheme therefore matches nothing and leaves every false positive in place: on the 2026-08-08 data the naive form kept all 36 where the normalized form kept none. From 568a62f3031e6cbcede1565ca5be16114437c860 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 09:42:08 -0700 Subject: [PATCH 2/5] Show real tag values, and say the field depends on the edge logging it The examples were angle-bracket placeholders, and the tag is validated: exactly one slash, letters, digits, dot, underscore, hyphen. Pasting one of those literally is refused, so the section now shows values that work and says the shape is enforced rather than conventional. The larger omission is that the filter depends on something this repository does not own. `request_X-Blog-Check` appears only because the edge is configured to log that header, so an absent field means either an untagged request or a capture that stopped, and the log alone does not distinguish them. Reading a day of absence as a day of real traffic is the wrong answer that shape produces. Both found by review, which is also the general form the host side named: before believing a negative, establish that the check could have produced a positive. Co-Authored-By: Claude Opus 5 (1M context) --- OPERATIONS.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/OPERATIONS.md b/OPERATIONS.md index a98b8a2..eef7ae0 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -267,7 +267,9 @@ The outward pass is four filters over the edge log, and each one exists because **Exclude this repository's own deploy gate first.** `check-live-urls.sh` requests the whole URL contract on every deploy, so an unfiltered day is mostly a recording of our own `curl`. A count that omits this step is measuring the pipeline rather than the readers, and it will be an order of magnitude too large. -The mechanism is the `X-Blog-Check` request header, which every synthetic request announces itself with, so `jq 'select(.["request_X-Blog-Check"] == null)'` is the whole filter. Its value is `/` rather than a boolean, so a run is identifiable rather than merely excludable: `github/-` from CI, `vps/smoke` from the host side, `proxmox/` from here. +The mechanism is the `X-Blog-Check` request header, which every synthetic request announces itself with, so `jq 'select(.["request_X-Blog-Check"] == null)'` is the whole filter. Its value is a source and an id rather than a boolean, so a run is identifiable rather than merely excludable, and real values look like `github/31322640628-1` from CI, `vps/smoke` from the host side, and `proxmox/media-dev` from here. The shape is enforced by `check-live-urls.sh`, which takes exactly one `/` and only letters, digits, `.`, `_`, `-`, so a placeholder written with angle brackets is a description rather than something to paste. + +- **The field exists only because the edge is configured to log that header**, which is the host side's to hold and not this repository's. An absent field therefore has two meanings, an untagged request or a capture that stopped, and they are not distinguishable from the log alone. Confirm the capture is live before reading a day's absence as a day of real traffic. - **A hand probe carries it only because whoever runs it adds it.** `check-live-urls.sh` sends it on every request and a bare `curl` sends nothing, so an interactive probe passes `-H "X-Blog-Check: proxmox/"`, naming the purpose rather than the host. Two untagged probes turned up against 3,100 tagged ones in the 2026-08-09 deploy window. - **Absence is not proof of a human**, since a scanner sends no header either, so this pairs with the scanner-shape filters below rather than replacing them. The field is forgeable and must never reach auth, rate limiting, robots handling, or caching. From 8558458e6323ee83c90174bf9509c6f1283b478c Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 09:45:15 -0700 Subject: [PATCH 3/5] Put a pasteable value in the pasteable command The prose examples were corrected and the curl one-liner was not, which is the one a reader actually copies. It now carries proxmox/media-dev. ENVIRONMENT.md keeps its angle brackets deliberately. There they describe the enforced grammar and the value the script derives under Actions, neither of which anyone types, and the settable example beside them is already a real one. Found by Copilot review on #81, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) --- OPERATIONS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPERATIONS.md b/OPERATIONS.md index eef7ae0..ff66e44 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -271,7 +271,7 @@ The mechanism is the `X-Blog-Check` request header, which every synthetic reques - **The field exists only because the edge is configured to log that header**, which is the host side's to hold and not this repository's. An absent field therefore has two meanings, an untagged request or a capture that stopped, and they are not distinguishable from the log alone. Confirm the capture is live before reading a day's absence as a day of real traffic. -- **A hand probe carries it only because whoever runs it adds it.** `check-live-urls.sh` sends it on every request and a bare `curl` sends nothing, so an interactive probe passes `-H "X-Blog-Check: proxmox/"`, naming the purpose rather than the host. Two untagged probes turned up against 3,100 tagged ones in the 2026-08-09 deploy window. +- **A hand probe carries it only because whoever runs it adds it.** `check-live-urls.sh` sends it on every request and a bare `curl` sends nothing, so an interactive probe passes `-H "X-Blog-Check: proxmox/media-dev"`, naming the purpose of the probe rather than the host, in the same character set as any other tag. Two untagged probes turned up against 3,100 tagged ones in the 2026-08-09 deploy window. - **Absence is not proof of a human**, since a scanner sends no header either, so this pairs with the scanner-shape filters below rather than replacing them. The field is forgeable and must never reach auth, rate limiting, robots handling, or caching. - **Before 2026-08-09 the log carries no such field**, and user agent is the only key for those days: on 2026-08-08, 9,285 of 9,996 requests were `curl/8.5.0`, leaving 711 real ones. That key is a coincidence rather than a rule, since the CI runner's curl and the host's are byte-identical and only the rotating client address separates them, which is why the header exists. From 6169dac1b0836a11c892d5c2845f52c5c8e4db4c Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 09:49:17 -0700 Subject: [PATCH 4/5] Claim only what the tagging actually covers The lead said every synthetic request announces itself, and the bullet below it said a hand curl sends nothing unless someone adds the header. The section contradicted itself in the direction that flatters the filter, which is the direction that produces a wrong answer nobody checks. It now says the scripted checks send it on every request they make, and that the filter is only as complete as the tagging is. Found by Copilot review on #81, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) --- OPERATIONS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPERATIONS.md b/OPERATIONS.md index ff66e44..a34cc84 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -267,7 +267,7 @@ The outward pass is four filters over the edge log, and each one exists because **Exclude this repository's own deploy gate first.** `check-live-urls.sh` requests the whole URL contract on every deploy, so an unfiltered day is mostly a recording of our own `curl`. A count that omits this step is measuring the pipeline rather than the readers, and it will be an order of magnitude too large. -The mechanism is the `X-Blog-Check` request header, which every synthetic request announces itself with, so `jq 'select(.["request_X-Blog-Check"] == null)'` is the whole filter. Its value is a source and an id rather than a boolean, so a run is identifiable rather than merely excludable, and real values look like `github/31322640628-1` from CI, `vps/smoke` from the host side, and `proxmox/media-dev` from here. The shape is enforced by `check-live-urls.sh`, which takes exactly one `/` and only letters, digits, `.`, `_`, `-`, so a placeholder written with angle brackets is a description rather than something to paste. +The mechanism is the `X-Blog-Check` request header, which the scripted checks send on every request they make, so `jq 'select(.["request_X-Blog-Check"] == null)'` is the filter. It is only as complete as the tagging is, which is the first bullet below. Its value is a source and an id rather than a boolean, so a run is identifiable rather than merely excludable, and real values look like `github/31322640628-1` from CI, `vps/smoke` from the host side, and `proxmox/media-dev` from here. The shape is enforced by `check-live-urls.sh`, which takes exactly one `/` and only letters, digits, `.`, `_`, `-`, so a placeholder written with angle brackets is a description rather than something to paste. - **The field exists only because the edge is configured to log that header**, which is the host side's to hold and not this repository's. An absent field therefore has two meanings, an untagged request or a capture that stopped, and they are not distinguishable from the log alone. Confirm the capture is live before reading a day's absence as a day of real traffic. From 3bafebd1ec0ba6236aa559b9d74cae8e6934131c Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sun, 9 Aug 2026 09:53:17 -0700 Subject: [PATCH 5/5] Say which half of the tag carries the purpose "Naming the purpose rather than the host" implied the whole value is a purpose label, when proxmox is the source half and only the id half is free. A reader following that would have written something like media-dev/probe, inverting the grammar the section just described. Found by Copilot review on #81, as a suppressed comment. Co-Authored-By: Claude Opus 5 (1M context) --- OPERATIONS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/OPERATIONS.md b/OPERATIONS.md index a34cc84..efcb837 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -271,7 +271,7 @@ The mechanism is the `X-Blog-Check` request header, which the scripted checks se - **The field exists only because the edge is configured to log that header**, which is the host side's to hold and not this repository's. An absent field therefore has two meanings, an untagged request or a capture that stopped, and they are not distinguishable from the log alone. Confirm the capture is live before reading a day's absence as a day of real traffic. -- **A hand probe carries it only because whoever runs it adds it.** `check-live-urls.sh` sends it on every request and a bare `curl` sends nothing, so an interactive probe passes `-H "X-Blog-Check: proxmox/media-dev"`, naming the purpose of the probe rather than the host, in the same character set as any other tag. Two untagged probes turned up against 3,100 tagged ones in the 2026-08-09 deploy window. +- **A hand probe carries it only because whoever runs it adds it.** `check-live-urls.sh` sends it on every request and a bare `curl` sends nothing, so an interactive probe passes `-H "X-Blog-Check: proxmox/media-dev"`. The source half stays `proxmox`, which is where the probe came from, and the id half is where the purpose goes. Two untagged probes turned up against 3,100 tagged ones in the 2026-08-09 deploy window. - **Absence is not proof of a human**, since a scanner sends no header either, so this pairs with the scanner-shape filters below rather than replacing them. The field is forgeable and must never reach auth, rate limiting, robots handling, or caching. - **Before 2026-08-09 the log carries no such field**, and user agent is the only key for those days: on 2026-08-08, 9,285 of 9,996 requests were `curl/8.5.0`, leaving 711 real ones. That key is a coincidence rather than a rule, since the CI runner's curl and the host's are byte-identical and only the rotating client address separates them, which is why the header exists.