feat: genericize into self-hostable OSS project - #6
Open
datj9 wants to merge 1 commit into
Open
Conversation
Turn tech-bytes into a real OSS project anyone can fork, configure, and run
with zero AWS. The default deploy path is now GitHub Pages via a scheduled
GitHub Actions workflow; AWS/CDK is demoted to an optional advanced path.
Architecture
- pipeline/ (renamed from lambdas/): framework-agnostic Python package with a
single CLI entrypoint `python -m pipeline.run [all|source]`.
- pipeline/providers/: pluggable LLM (openai, anthropic, local/OAI-compatible)
behind one Protocol. Selected by config; keys from env (SSM fallback on AWS).
- pipeline/storage/: pluggable storage (local_fs default, s3 optional) behind
one Protocol. Selected by $STORAGE. boto3 fully lazy — free path needs no AWS.
- config/techbytes.config.yml: single user-facing config (sources + branding +
LLM choice). Icons live here as emoji literals, not in site code.
- pipeline/shared/{config,rebuild,utils,secrets}.py: cross-cutting helpers.
CloudWatch metrics no-op outside AWS; rebuild trigger no-op on free path.
Site (Astro)
- Branding (title/tagline/url) read from config at build, PUBLIC_* env overrides.
- Hardcoded `categoryIcons` map deleted; icons come from data (which comes from
config) as emoji literals.
- Data dir resolved via $DATA_DIR; Pages base path auto-detected from
GITHUB_REPOSITORY.
- Contribute page repo URL resolved from $GITHUB_REPOSITORY / PUBLIC_GITHUB_REPO.
Workflows
- digest.yml (NEW): cron -> run pipeline -> commit data/ -> build -> Pages.
- deploy-aws.yml (renamed): optional advanced path; pulls domain/zone/repo from
vars/context, no hardcoded values.
- preview.yml: ruff + pytest + site build + best-effort cdk synth.
Infra (CDK, optional)
- Fully parameterized via context: domain.name, hosted.zone.name, github.repo,
ssm.prefix. No hardcoded domain/account/repo.
- Lambda handler paths updated for the pipeline/ rename.
OSS scaffolding
- LICENSE (MIT), README (quickstart + config + provider setup), CONTRIBUTING,
regenerated .env.example.
- All personal identifiers removed (domain, AWS profile, repo slug, SSM paths,
model id, rebuild URL). Grep gate clean across source + build output.
Tests
- 34 pass: existing release_radar/gh_trending schema tests updated for the new
config-driven icon; new test_pipeline.py covers provider factory, storage
factory, and local_fs behavior.
Decisions
- License: MIT. Brand: keeps "Tech Bytes" as the demo default (forkers override
via config). Dir renamed lambdas/ -> pipeline/. Anthropic default model:
claude-sonnet-4-6.datj9force-pushed
the
feat/genericize-oss
branch
from
August 16, 2026 23:17
d7a1562 to
fdd6384Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turns tech-bytes into a real OSS project: anyone can fork, edit one config file, plug in their LLM key (OpenAI / Anthropic / local), and launch with zero AWS. GitHub Actions builds the digest on a cron and publishes to GitHub Pages. AWS/CDK stays as an optional advanced path.
Plan:
~/.claude/plans/purring-kindling-cray.mdWhat changed
Pipeline (
lambdas/→pipeline/)pipeline/providers/):openai,anthropic,local(OpenAI-compatible: Ollama, LM Studio, Groq, OpenRouter) behind one Protocol. Selected byllm.providerin config; keys from env, SSM fallback on AWS.pipeline/storage/):local_fs(default, writesdata/) ands3(AWS path) behind one Protocol.boto3fully lazy — free path imports no AWS libs.python -m pipeline.run [all|release_radar|hn_digest|gh_trending|email_digest].config/techbytes.config.ymlowns sources + branding + LLM choice. Icons are emoji literals in config (the site reads them directly; the old hardcodedcategoryIconsmap is gone).trigger_rebuildno-op on the free path.Site (Astro)
PUBLIC_*env overrides for CI.GITHUB_REPOSITORYfor project Pages sites.$DATA_DIR(default../data).contribute.astrorepo URL resolved from env — no hardcodeddatj9/tech-bytes.Workflows
digest.yml(NEW) — free path: cron → run pipeline → commitdata/→ build → deploy Pages. No AWS.deploy-aws.yml(renamed fromdeploy.yml) — optional AWS path; reads domain/zone/repo from vars/context.preview.yml— ruff + pytest + site build + best-effort cdk synth on every PR.Infra (CDK, optional)
domain.name,hosted.zone.name,github.repo,ssm.prefixfrom context. No hardcoded values; clear error if missing. Seeinfra/README.md.OSS scaffolding
LICENSE(MIT),README.md(60-second quickstart + config reference + provider setup),CONTRIBUTING.md, regenerated.env.example.Decisions (locked at execution)
"Tech Bytes"as the demo default (forkers override via config)lambdas/→pipeline/(git mv preserves history)claude-sonnet-4-6Verification
All green locally:
ruff check pipeline/pytest pipeline/tests/pnpm run build(site)tsc --noEmit(infra)python -m pipeline.run release_radar(smoke, local storage, fake key)data/release-radar.json, no boto3 importfinaldivision|datj9|super-personalinsite/dist//tech-bytes/SSM namespace)preview.ymlruns ruff + pytest + site build + cdk synth on this PR.Test plan for reviewer
cp config/techbytes.config.example.yml config/techbytes.config.yml→ setOPENAI_API_KEY(orLLM_PROVIDER=anthropic+ANTHROPIC_API_KEY) →python -m pipeline.run all→ confirmdata/*.jsonwritten, no AWS deps required.cd site && npm i && npm run build→ confirm generic branding, nofinaldivision/ hardcoded icons indist/.llm.providerbetweenopenai/anthropic/localin config; confirm the factory picks the right provider and default model.GITHUB_REPOSITORY=myuser/myrepobeforepnpm run build; confirm Pages base path and contribute page use the fork's identity.Breaking changes
config/technologies.ymlremoved → migrate toconfig/techbytes.config.yml(see example).data/release-radar.jsonicon field now an emoji literal (was acategoryIconskey string). Seed data migrated; first pipeline run regenerates everything.release_radar.handler.handlertopipeline.release_radar.handler.handler— CDK bundling updated to match. Existing AWS deployments needcdk deployto pick up the new handler paths.