Content Engine

Script that:

1. Reads a list of sources from a file (each line: type url — type is rss or html)
2. Parses the latest N items: RSS/Atom with feedparser, HTML blog listing pages with BeautifulSoup + readability
3. Calls an LLM to generate 10 post ideas from the parsed content (structured: source links, source insight, post idea, description, format, how to use)

Uses OpenAI for LLM calls and LangFuse for tracing (via langfuse.openai).

Quick start

See docs/RUNNING.md for full setup, environment variables, pipeline steps, and troubleshooting.

uv sync
cp .env.example .env # set OPENAI_API_KEY
python run.py # writes ideas.md

Configure sources in urls.txt (rss or html per line). For new HTML sources, use docs/ADDING_SOURCES.md.

Documentation

Doc	Purpose
docs/RUNNING.md	Install, env, run pipeline, output, troubleshooting
docs/ADDING_SOURCES.md	Adding and validating new RSS/HTML sources

Project structure

content-engine/
├── run.py # Entry point (loads .env, calls main)
├── main.py # Pipeline: load sources → fetch → LLM → output
├── config.py # Constants (DEFAULT_*, USER_AGENT, SOURCE_TYPES)
├── models.py # FeedEntry, Source dataclasses
├── sources.py # load_sources() from urls file
├── fetcher.py # fetch_entries() — RSS + HTML, merge by date
├── parsers/
│ ├── __init__.py
│ ├── rss.py # fetch_entries_rss()
│ └── html.py # fetch_entries_html()
├── prompt_loader.py # load_prompt(name, **variables)
├── llm.py # generate_post_ideas()
├── prompts/ # Prompt templates ({{placeholder}})
│ ├── post_ideas_system.txt
│ └── post_ideas_user.txt # {{count}}, {{content}}, {{sources}}
├── urls.txt
└── ...

Run from the project root so that imports resolve (python run.py).