add custom Docusaurus plugin generating llms(-full).txt

[NicNomadic]

NB: replaces #448 (closed)

Solves issue #438

That is, adds a custom plugin for Docusaurus that scans the doc sidebar and generates standard files for ingestion by agents, called llms.txt and llms-full.txt.

The custom plugin is justified by the fact that it does some specific cleanup actions, such as removing the markers that we use for rebranding the doc back and forth.

[saroupille]

Read the plugin end to end — the approach is sound and the risk to the site is nil (postBuild-only: it writes two extra files at the end of the build; a bug breaks the build, not the content). Approving so it can be merged on Thursday as-is. Three non-blocking notes:

1. slug: frontmatter isn't honored.fileToUrl assumes path = URL. Concretely, docs/overview/index.md has slug: /, so llms.txt will link the homepage as https://docs.etherlink.com/overview — which 404s on the live site (verified). It's the only affected page today (tutorials/predictionMarket and evm/bridging/bridging overrides happen to coincide with their paths). Easy fix: parse slug in extractFrontmatter and use it for the printed URL, while keeping the path-derived key for sidebar matching.

2. Pages missing from sidebars.js are silently dropped from both outputs (if (!doc) continue covers the reverse case, but a doc file absent from the sidebar just vanishes). A console.warn listing orphaned docs would catch drift as pages get added later.

3. Raw JSX remains in llms-full.txt for .mdx pages (<InlineCopy>, <Link> tables…) — imports are stripped but components aren't. Acceptable noise for LLM consumption; just noting it's known.

🤖 Review researched with Claude Code on behalf of @saroupille

[NicNomadic]

NB: replaces #448 (closed)

Solves issue #438

That is, adds a custom plugin for Docusaurus that scans the doc sidebar and generates standard files for ingestion by agents, called llms.txt and llms-full.txt.

The custom plugin is justified by the fact that it does some specific cleanup actions, such as removing the markers that we use for rebranding the doc back and forth.

[saroupille]

Read the plugin end to end — the approach is sound and the risk to the site is nil (postBuild-only: it writes two extra files at the end of the build; a bug breaks the build, not the content). Approving so it can be merged on Thursday as-is. Three non-blocking notes:

1. slug: frontmatter isn't honored.fileToUrl assumes path = URL. Concretely, docs/overview/index.md has slug: /, so llms.txt will link the homepage as https://docs.etherlink.com/overview — which 404s on the live site (verified). It's the only affected page today (tutorials/predictionMarket and evm/bridging/bridging overrides happen to coincide with their paths). Easy fix: parse slug in extractFrontmatter and use it for the printed URL, while keeping the path-derived key for sidebar matching.

2. Pages missing from sidebars.js are silently dropped from both outputs (if (!doc) continue covers the reverse case, but a doc file absent from the sidebar just vanishes). A console.warn listing orphaned docs would catch drift as pages get added later.

3. Raw JSX remains in llms-full.txt for .mdx pages (<InlineCopy>, <Link> tables…) — imports are stripped but components aren't. Acceptable noise for LLM consumption; just noting it's known.

🤖 Review researched with Claude Code on behalf of @saroupille

[NicNomadic]

NB: replaces #448 (closed)

Solves issue #438

That is, adds a custom plugin for Docusaurus that scans the doc sidebar and generates standard files for ingestion by agents, called llms.txt and llms-full.txt.

The custom plugin is justified by the fact that it does some specific cleanup actions, such as removing the markers that we use for rebranding the doc back and forth.

[saroupille]

Read the plugin end to end — the approach is sound and the risk to the site is nil (postBuild-only: it writes two extra files at the end of the build; a bug breaks the build, not the content). Approving so it can be merged on Thursday as-is. Three non-blocking notes:

1. slug: frontmatter isn't honored.fileToUrl assumes path = URL. Concretely, docs/overview/index.md has slug: /, so llms.txt will link the homepage as https://docs.etherlink.com/overview — which 404s on the live site (verified). It's the only affected page today (tutorials/predictionMarket and evm/bridging/bridging overrides happen to coincide with their paths). Easy fix: parse slug in extractFrontmatter and use it for the printed URL, while keeping the path-derived key for sidebar matching.

2. Pages missing from sidebars.js are silently dropped from both outputs (if (!doc) continue covers the reverse case, but a doc file absent from the sidebar just vanishes). A console.warn listing orphaned docs would catch drift as pages get added later.

3. Raw JSX remains in llms-full.txt for .mdx pages (<InlineCopy>, <Link> tables…) — imports are stripped but components aren't. Acceptable noise for LLM consumption; just noting it's known.

🤖 Review researched with Claude Code on behalf of @saroupille

[NicNomadic]

NB: replaces #448 (closed)

Solves issue #438

That is, adds a custom plugin for Docusaurus that scans the doc sidebar and generates standard files for ingestion by agents, called llms.txt and llms-full.txt.

The custom plugin is justified by the fact that it does some specific cleanup actions, such as removing the markers that we use for rebranding the doc back and forth.

[saroupille]

Read the plugin end to end — the approach is sound and the risk to the site is nil (postBuild-only: it writes two extra files at the end of the build; a bug breaks the build, not the content). Approving so it can be merged on Thursday as-is. Three non-blocking notes:

1. slug: frontmatter isn't honored.fileToUrl assumes path = URL. Concretely, docs/overview/index.md has slug: /, so llms.txt will link the homepage as https://docs.etherlink.com/overview — which 404s on the live site (verified). It's the only affected page today (tutorials/predictionMarket and evm/bridging/bridging overrides happen to coincide with their paths). Easy fix: parse slug in extractFrontmatter and use it for the printed URL, while keeping the path-derived key for sidebar matching.

2. Pages missing from sidebars.js are silently dropped from both outputs (if (!doc) continue covers the reverse case, but a doc file absent from the sidebar just vanishes). A console.warn listing orphaned docs would catch drift as pages get added later.

3. Raw JSX remains in llms-full.txt for .mdx pages (<InlineCopy>, <Link> tables…) — imports are stripped but components aren't. Acceptable noise for LLM consumption; just noting it's known.

🤖 Review researched with Claude Code on behalf of @saroupille

[NicNomadic]

NB: replaces #448 (closed)

Solves issue #438

That is, adds a custom plugin for Docusaurus that scans the doc sidebar and generates standard files for ingestion by agents, called llms.txt and llms-full.txt.

The custom plugin is justified by the fact that it does some specific cleanup actions, such as removing the markers that we use for rebranding the doc back and forth.

[saroupille]

Read the plugin end to end — the approach is sound and the risk to the site is nil (postBuild-only: it writes two extra files at the end of the build; a bug breaks the build, not the content). Approving so it can be merged on Thursday as-is. Three non-blocking notes:

1. slug: frontmatter isn't honored.fileToUrl assumes path = URL. Concretely, docs/overview/index.md has slug: /, so llms.txt will link the homepage as https://docs.etherlink.com/overview — which 404s on the live site (verified). It's the only affected page today (tutorials/predictionMarket and evm/bridging/bridging overrides happen to coincide with their paths). Easy fix: parse slug in extractFrontmatter and use it for the printed URL, while keeping the path-derived key for sidebar matching.

2. Pages missing from sidebars.js are silently dropped from both outputs (if (!doc) continue covers the reverse case, but a doc file absent from the sidebar just vanishes). A console.warn listing orphaned docs would catch drift as pages get added later.

3. Raw JSX remains in llms-full.txt for .mdx pages (<InlineCopy>, <Link> tables…) — imports are stripped but components aren't. Acceptable noise for LLM consumption; just noting it's known.

🤖 Review researched with Claude Code on behalf of @saroupille

[NicNomadic]

NB: replaces #448 (closed)

Solves issue #438

That is, adds a custom plugin for Docusaurus that scans the doc sidebar and generates standard files for ingestion by agents, called llms.txt and llms-full.txt.

The custom plugin is justified by the fact that it does some specific cleanup actions, such as removing the markers that we use for rebranding the doc back and forth.

[saroupille]

Read the plugin end to end — the approach is sound and the risk to the site is nil (postBuild-only: it writes two extra files at the end of the build; a bug breaks the build, not the content). Approving so it can be merged on Thursday as-is. Three non-blocking notes:

1. slug: frontmatter isn't honored.fileToUrl assumes path = URL. Concretely, docs/overview/index.md has slug: /, so llms.txt will link the homepage as https://docs.etherlink.com/overview — which 404s on the live site (verified). It's the only affected page today (tutorials/predictionMarket and evm/bridging/bridging overrides happen to coincide with their paths). Easy fix: parse slug in extractFrontmatter and use it for the printed URL, while keeping the path-derived key for sidebar matching.

2. Pages missing from sidebars.js are silently dropped from both outputs (if (!doc) continue covers the reverse case, but a doc file absent from the sidebar just vanishes). A console.warn listing orphaned docs would catch drift as pages get added later.

3. Raw JSX remains in llms-full.txt for .mdx pages (<InlineCopy>, <Link> tables…) — imports are stripped but components aren't. Acceptable noise for LLM consumption; just noting it's known.

🤖 Review researched with Claude Code on behalf of @saroupille

[NicNomadic]

NB: replaces #448 (closed)

Solves issue #438

That is, adds a custom plugin for Docusaurus that scans the doc sidebar and generates standard files for ingestion by agents, called llms.txt and llms-full.txt.

The custom plugin is justified by the fact that it does some specific cleanup actions, such as removing the markers that we use for rebranding the doc back and forth.

[saroupille]

Read the plugin end to end — the approach is sound and the risk to the site is nil (postBuild-only: it writes two extra files at the end of the build; a bug breaks the build, not the content). Approving so it can be merged on Thursday as-is. Three non-blocking notes:

1. slug: frontmatter isn't honored.fileToUrl assumes path = URL. Concretely, docs/overview/index.md has slug: /, so llms.txt will link the homepage as https://docs.etherlink.com/overview — which 404s on the live site (verified). It's the only affected page today (tutorials/predictionMarket and evm/bridging/bridging overrides happen to coincide with their paths). Easy fix: parse slug in extractFrontmatter and use it for the printed URL, while keeping the path-derived key for sidebar matching.

2. Pages missing from sidebars.js are silently dropped from both outputs (if (!doc) continue covers the reverse case, but a doc file absent from the sidebar just vanishes). A console.warn listing orphaned docs would catch drift as pages get added later.

3. Raw JSX remains in llms-full.txt for .mdx pages (<InlineCopy>, <Link> tables…) — imports are stripped but components aren't. Acceptable noise for LLM consumption; just noting it's known.

🤖 Review researched with Claude Code on behalf of @saroupille

[NicNomadic]

NB: replaces #448 (closed)

Solves issue #438

That is, adds a custom plugin for Docusaurus that scans the doc sidebar and generates standard files for ingestion by agents, called llms.txt and llms-full.txt.

The custom plugin is justified by the fact that it does some specific cleanup actions, such as removing the markers that we use for rebranding the doc back and forth.

[saroupille]

Read the plugin end to end — the approach is sound and the risk to the site is nil (postBuild-only: it writes two extra files at the end of the build; a bug breaks the build, not the content). Approving so it can be merged on Thursday as-is. Three non-blocking notes:

1. slug: frontmatter isn't honored.fileToUrl assumes path = URL. Concretely, docs/overview/index.md has slug: /, so llms.txt will link the homepage as https://docs.etherlink.com/overview — which 404s on the live site (verified). It's the only affected page today (tutorials/predictionMarket and evm/bridging/bridging overrides happen to coincide with their paths). Easy fix: parse slug in extractFrontmatter and use it for the printed URL, while keeping the path-derived key for sidebar matching.

2. Pages missing from sidebars.js are silently dropped from both outputs (if (!doc) continue covers the reverse case, but a doc file absent from the sidebar just vanishes). A console.warn listing orphaned docs would catch drift as pages get added later.

3. Raw JSX remains in llms-full.txt for .mdx pages (<InlineCopy>, <Link> tables…) — imports are stripped but components aren't. Acceptable noise for LLM consumption; just noting it's known.

🤖 Review researched with Claude Code on behalf of @saroupille