Comments scraping + Substack-styled rendering + macOS driver fixes + README doc updates

[sanket-k]

Summary

This PR adds three related improvements to the scraper:

1. Comment scraping (--comments)

• New --comments flag fetches and caches each post's comment thread.
  Public threads are free; paid-only threads require --premium.
• New --comments-sort flag (best | most_recent_first).
• Comments render into the per-post HTML page, and comment_count is
  exposed to the index UI (with a new "Sort by Comments" option).
• Ships backfill_comment_counts.py to repair older data files.

2. macOS driver support & reliability

• Correctly locates Chrome/Edge under /Applications and resolves the
  right driver platform (mac-arm64 vs mac-x64, mac64_m1).
• Driver recovery: recreates crashed sessions and periodically restarts
  to shed leaked state; suppresses noise via the quiet flag.

3. Substack-styled HTML rendering

• Rewrites per-post HTML to match Substack's default look: Spectral serif
  body (19px/1.6), orange (#ff6719) links, left-aligned text, 728px column,
  and a centered cover/title/subtitle/byline header built from structured
  metadata rather than inlined markdown.
• New render_posts.py standalone regenerate script (no network required).
• New --render-only / --render-all CLI flags.

4. README.md — major overhaul :

• adds Features overview, System Architecture (mermaid), Installation,
  Usage examples for comments & sorting,
• Workflow sequence (mermaid), full CLI Reference table,
  Browser & Driver Support, Backfilling Comment Counts,
• Substack-style Rendering docs, Output Layout, Online Version, and Viewing Markdown
  Files in Browser.
• Corrects the stale "Microsoft Edge" browser note.

Changes

• substack_scraper.py — comment helpers, structured render path, driver fixes
• render_posts.py — new network-free regenerate script
• backfill_comment_counts.py — new data repair utility
• assets/css/essay-styles.css, assets/css/style.css, author_template.html — restyled
• assets/js/populate-essays.js — "Sort by Comments" + comment_count handling
• tests/test_substack_scraper.py — 13+ new tests (comments, rendering, regenerate)
• README.md — CLI reference table, architecture/workflow mermaid diagrams

Testing

• All new and existing tests pass: pytest tests/
• Manually verified scraping + rendering on macOS (Apple Silicon).

Backward compatibility

The structured render path falls back to the flat (meta=None) path for legacy
compatibility, so existing data files keep working.

[timf34]

Thanks for the substantial work here @sanket-k — there's a lot of genuinely useful material in this PR. That said, at +2,477/−265 across 13 files it bundles several independent changes, and I can't review or maintain it as one unit. I also want to keep this project's output design stable: the Substack-lookalike restyling changes the default look of every generated page, which is a bigger design decision than I want to take on as a side effect of other features.

Could you split it? In order of how readily I could merge them:

1. macOS driver fixes (Chrome/Edge under /Applications, mac-arm64/mac-x64 resolution, session recovery) — this would be a great standalone PR and I'd expect it to merge quickly.
2. Comments scraping (--comments, --comments-sort) — the most-requested feature; I'd happily review it on its own, ideally without the index-UI changes in the first pass.
3. The Substack-styled rendering + render scripts — I'm unlikely to take this as a change to the defaults; it could potentially work as an opt-in flag, but let's discuss before you invest more time there.

Happy to review the first two as soon as they're up. Thanks again!