feat(seo): make the sites legible to search engines and agents

[LKSNDRTMLKV]

Targets Google, Bing, Baidu and Apple. Prepared for the next web deploy — not urgent to merge.

🚨 The content signals were decorative, and said the opposite of the goal

The landing robots.txt carried:

# search:   yes
# ai-input: no
# ai-train: no

Two problems.

They were not machine-readable. Content-signal is a real directive, not a
comment, and its syntax is search=yes — not search: yes. Written as three
comments in the wrong form, no parser read them. So they neither signalled
anything to a crawler nor reserved anything under Directive 2019/790 Art. 4,
which asks for machine-readable means. The block cited that article while failing
its one requirement.

And had they worked, ai-input: no would have blocked the goal. Per
Cloudflares definitions, search covers "hyperlinks and short excerpts" and
explicitly excludes AI-generated summaries; ai-input is what governs AI
Overviews and assistant answers. Google AI Overviews now sit above the blue
links. Opting out of ai-input while asking for search visibility means
appearing only below the panel that answers the question.

Now, on both sites:

Content-signal: search=yes, ai-input=yes, ai-train=no

Indexed and quotable; not training material. That also makes the Art. 4
reservation actually machine-readable for the first time.

The enforceable half: per-crawler directives

Content signals are advisory. These are honoured by the vendors:

Directive	Governs	Still in search?
Applebot-Extended	Apple foundation-model training only; never fetches a page	✅ Apple state it does not affect Siri, Spotlight or Safari
Google-Extended	Gemini training/grounding	✅ Does not affect Google Search

Both now Disallow. This is the working version of what the old comments were
reaching for: indexed by Apple and Google, not training data for either.

The docs site had no robots.txt at all on main

It is added here, mirroring the landing one with its own sitemap. That is the
content-rich property — the regulatory pages — and it was the one with nothing.

llms.txt on both sites

New. Markdown index pointing at the pages worth quoting, with a short
## How to cite this project accurately section that states the things a model
otherwise guesses wrong: pre-1.0, no users, no customers; core is Apache-2.0
but the engine is source-available and not OSI open-source; "Odal Node" is a
project name, not a registered company.

Every URL listed was checked against the pages that actually exist.

Structured data

• Docs site had none. Adds SoftwareSourceCode + TechArticle, with the three regulations the regulatory pages are grounded in named as citation.
• Landing adds SoftwareSourceCode alongside the existing Organization and WebSite.

🚨 The index page already carried a SoftwareApplication block. Left unlinked,
a crawler would see two unrelated entities both called "Odal Node" and pick
one. Both now share @id: https://odal-node.io/#software, so they are one node
described from two angles — the deployable application, and the source with its
licence.

SoftwareSourceCode rather than a second SoftwareApplication: this is a
library and a self-hosted engine, not an installable app with an operating
system and a price.

⚠️ Two things I did not do

"logbook" is not in the site copy. It is in the structured-data keywords
and llms.txt, which are machine-facing. It is not in any human-facing
prose, because WEB_CONTENT_STRATEGY.md §7 is a controlled vocabulary with an
explicit Use list, and "logbook" appears in neither that list nor the Never list.
Adding a new consumer-facing synonym is a brand decision — if you want it, §7
should gain it first, and then the copy.

The existing SoftwareApplication declares offers.price: "0" EUR. Its
description qualifies this correctly (Apache-2.0 core, BSL engine with a
self-host grant), but a bare zero price for software whose commercial use needs
a licence is the kind of simplification worth a second look. Left untouched —
changing a pricing claim is not an SEO decision.

Verification

pnpm -r build and pnpm -r check both pass (0 errors, 0 warnings). Verified in
the built output, not the source:

• both robots.txt ship the Content-signal line verbatim
• both llms.txt ship
• docs index.html carries one JSON-LD block: SoftwareSourceCode + TechArticle
• landing index.html carries four, with SoftwareApplication and SoftwareSourceCode sharing one @id
• none of the new files contain a term from §7s Never list

Not covered here

Content-level keyword work — headings, on-page terminology, content gaps — is
the marketing:seo-audit job, and belongs against the docs site where the
substance is. Baidu additionally needs Webmaster Tools verification and Simplified
Chinese content; an ICP licence is unobtainable without a China legal entity, so
that route stays closed for now.

[cloudflare-workers-and-pages]

Deploying odal-node-docs with    Cloudflare Pages

Latest commit:	17b7d30
Status:	✅  Deploy successful!
Preview URL:	https://4271a91a.odal-node-docs.pages.dev
Branch Preview URL:	https://feat-discoverable-to-search.odal-node-docs.pages.dev

View logs

[cloudflare-workers-and-pages]

Deploying odal-node-landing with    Cloudflare Pages

Latest commit:	17b7d30
Status:	✅  Deploy successful!
Preview URL:	https://34a86fe6.odal-node-landing.pages.dev
Branch Preview URL:	https://feat-discoverable-to-search.odal-node-landing.pages.dev

View logs

[LKSNDRTMLKV]

Found a defect in my own work here, and added the gate that catches it.

The landing llms.txt pointed at a page that does not exist

I listed https://odal-node.io/trust/. On main that file is _trust.astro —
an underscore-prefixed draft Astro deliberately does not build. I wrote the link
from a page listing taken on a different branch, where trust is published.

That is precisely the failure this file is worst at: llms.txt exists to tell a
machine which pages are worth reading, so a dead entry is not a cosmetic typo —
it sends a model to a 404 and invites it to report the page as missing.

Removed. Every remaining URL was then checked against dist/:

landing:  3 on-site URLs, all built
docs:    19 on-site URLs, all built

scripts/check-llms-txt.mjs

Nothing else checks this file. It is a static asset in public/, so no build
step parses it and no link checker reaches it — which is why the error survived
review, mine included.

The script compares every on-site URL against dist/, so it answers the only
question that matters: does the page exist in the artifact being deployed.
Wired into pnpm run check:llms and into CI after the build step.

Mutation-tested rather than just run. Re-adding the exact line I had written:

FAIL: 1 URL(s) in site/dpp-landing/public/llms.txt were not built:
  https://odal-node.io/trust/

Either the page is a draft (an `_`-prefixed file is not built)
or the path is wrong. Do not ship an index that points at nothing.
exit=1

It also fails when it finds zero on-site URLs — the script is a loop, and a
loop over nothing succeeds. That is the vacuous-pass shape found four times
elsewhere in this project today; no reason to ship a fifth.

Also checked, no action needed

• All 11 published crates build on docs.rs (HTTP 200 each). A failed docs.rs
  build is a large discoverability loss for a Rust library and is invisible from
  the repo — worth confirming rather than assuming.
• The landing sitemap is correct: 3 URLs, and the four _-prefixed drafts
  are properly excluded.
• Crate readme fields are absent everywhere, which is fine — Cargo
  auto-detects README.md, and crates.io renders it.

pnpm -r build, pnpm -r check (0 errors) and check:llms all pass.