diff --git a/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx b/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx index c3064ce6..75926be7 100644 --- a/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx +++ b/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx @@ -84,16 +84,16 @@ Add `html2rss` to your Claude Desktop configuration (`~/Library/Application Supp When an agent needs articles immediately without saving a feed configuration: -1. Call `scrape_url` with `strategy: "auto"`. Auto runs Faraday then Botasaurus when `BOTASAURUS_SCRAPER_URL` is set on the MCP process. +1. Call `scrape_url` with `strategy: "auto"`. Auto runs Faraday then Botasaurus when `BOTASAURUS_SCRAPER_URL` is set on the MCP process, prefers native RSS/Atom when present, and may rewrite a weak homepage URL via entry resolution. 2. An empty item list is still success for articles-now. Read `payload.items`. Follow `next_step` (including `read_runtime` / `html2rss://runtime` when Botasaurus is unset). -3. Call `inspect_url` when the result is weak or you need recon (`final_url`, `status`, `scheme_downgrade`, native RSS/Atom). +3. Call `inspect_url` when the result is weak or you need recon (`final_url`, `status`, `scheme_downgrade`, `alternate_feeds`, `surface_category`). Follow `next_step` — often `scrape_url` when `alternate_feeds` is present. ### Pattern B: Iterative Feed Config Authoring When an agent is tasked with creating a durable YAML feed configuration: 1. **Capture:** Call `capture_config`. YAML is `payload.yaml` (`items` + `enhance: true`). Check `payload.articles_count` and `has_selectors`. -2. **Recon (optional):** Call `inspect_url` for `final_url`, `status`, `scheme_downgrade`, and `alternate_feeds` if the draft is weak. +2. **Recon (optional):** Call `inspect_url` for `final_url`, `status`, `scheme_downgrade`, `alternate_feeds`, and `surface_category` if the draft is weak. Follow `next_step` (often `scrape_url` when alternates are present). 3. **Rewrite:** If the destination is html2rss-configs, add `directory.topics` and explicit channel `title`/`url`. Strive to keep `enhance: true` (set `false` only when chrome leaks into items). 4. **Validate:** Pass `yaml` (or `config`) to `validate_config`. Exactly one of those arguments. 5. **Apply:** Call `apply_config`. `isError` plus `payload.item_count: 0` means the config is not shippable — channel title in the XML is not success. diff --git a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx index e1a635f2..950aa938 100644 --- a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx +++ b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx @@ -76,7 +76,12 @@ Prefer `Html2rss.feed_result` when one scrape must render as both RSS and JSON F - `to_rss` / `to_json_feed(feed_url:)` — render formats - `status` — scrape telemetry (`Html2rss::Status`) -`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`), and `admission_drops` (Cleanup reason → count). +`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include: + +- `scraper_tallies`, `selected_strategy`, `attempt_count`, `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`) +- `admission_drops` (Cleanup reason → count) +- `entry_url` / `scrape_url` (pasted vs effective URL after entry resolution) +- `entry_resolution` (`applied`, `probe_count`, `reason`, optional `winner_score`) For URL-only auto discovery, `Html2rss.auto_feed_result(url)` returns the same `FeedResult` (CLI `html2rss auto --explain` prints `status.to_h` on stderr). diff --git a/src/content/docs/ruby-gem/reference/auto-source.mdx b/src/content/docs/ruby-gem/reference/auto-source.mdx index 2007795c..c09d39ae 100644 --- a/src/content/docs/ruby-gem/reference/auto-source.mdx +++ b/src/content/docs/ruby-gem/reference/auto-source.mdx @@ -22,13 +22,15 @@ To enable it, add `auto_source: {}` to your configuration: One fetch, then scrapers run in **tiers**. Within a tier, results merge. The pipeline stops when Cleanup has admitted enough articles (`limit`, default `25`). -1. **In-page structured:** `schema`, `microdata`, `microformats2`, `json_state`, `xhr_articles` -2. **Follow-up IO:** `wordpress_api`, `sitemap`, `meta_oembed` -3. **Heuristic:** `semantic_html` -4. **Heuristic:** `html` — skipped when an earlier tier already admitted at least one clean article +1. **Native syndication:** `native_feed` — discovers same-origin RSS/Atom (`rel=alternate` plus common feed paths) and parses them before HTML heuristics. Direct feed entry URLs (Content-Type or body sniff) skip HTML AutoSource and parse immediately. +2. **In-page structured:** `schema`, `microdata`, `microformats2`, `json_state`, `xhr_articles` +3. **Follow-up IO:** `wordpress_api`, `sitemap`, `meta_oembed` +4. **Heuristic:** `semantic_html` +5. **Heuristic:** `html` — skipped when an earlier tier already admitted at least one clean article What each scraper does: +- **`native_feed`:** Prefer native RSS/Atom when present. Tallies appear as `AutoSource::NativeFeed` under `--explain`. - **`schema`:** Parses `