RSS Feed Mapping, Scraper, and Publishing Tools

This repo owns the source/data layer for the novel announcement system.

It stores novel and host metadata, generates RSS feeds, checks host updates, sends a few direct Discord reports/tools, and provides the installable mapping package used by the Discord announcement repos.

The repo now uses split TOML mapping files instead of keeping all novel data directly inside novel_mappings.py.

---

Repository Structure

Important files and folders:

rss-feed/
├─ .github/workflows/
│ ├─ update_free_feed.yml
│ ├─ update_paid_feed.yml
│ ├─ update_comments.yml
│ ├─ create_novel_toml.yml
│ ├─ update_novel_card.yml
│ ├─ publish_novel_card.yml
│ ├─ publish_membership_update.yml
│ ├─ monthly_revenue.yml
│ ├─ nu_weekly_readers.yml
│ └─ send_token_alert.yml
├─ novel_mappings.py
├─ mappings/
│ ├─ __init__.py
│ ├─ output_feeds.toml
│ ├─ hosts/
│ │ └─ mistmint_haven.toml
│ └─ novels/
│ ├─ amlwc.toml
│ ├─ atvhe.toml
│ ├─ ec.toml
│ ├─ hiaflg.toml
│ ├─ tdlbkgc.toml
│ ├─ tvitpa.toml
│ └─ wsmsc.toml
├─ host_utils/
│ ├─ __init__.py
│ ├─ host_dragonholic.py
│ ├─ host_nu_comments.py
│ ├─ host_titv.py
│ └─ mistmint_haven/
│ ├─ __init__.py
│ ├─ common.py
│ ├─ client.py
│ ├─ free_chapters.py
│ ├─ paid_chapters.py
│ └─ comments.py
├─ feed_common.py
├─ free_feed_generator.py
├─ paid_feed_generator.py
├─ comments.py
├─ message_renderer.py
├─ message_templates/
│ ├─ membership_update.toml
│ ├─ nu_weekly_readers.toml
│ ├─ publish_novel_card.toml
│ ├─ revenue_report.toml
│ └─ token_alert.toml
├─ novelupdates/
│ ├─ nu_weekly_readers.py
│ └─ nu_readers.json
├─ revenue/
│ ├─ report.py
│ ├─ state.json
│ └─ hosts/
├─ token/
│ ├─ send_token_alert.py
│ └─ token_alert_state.json
├─ tools/
│ ├─ audit_dead_host_utils.py
│ ├─ create_novel_toml.py
│ ├─ publish_membership_update.py
│ ├─ publish_novel_card.py
│ └─ update_novel_card.py
├─ free_chapters_feed.xml
├─ paid_chapters_feed.xml
├─ aggregated_comments_feed.xml
├─ novel_card_targets.json
├─ requirements.txt
├─ pyproject.toml
└─ README.md

---

What This Repo Produces

Generated RSS files:

free_chapters_feed.xml
paid_chapters_feed.xml
aggregated_comments_feed.xml

These are read by:

discord-webhook
mistmint-discord

The RSS files are the bridge between scraping/API logic and Discord announcements.

---

Discord Integrations, Routes, and Mentions

External Discord repos are configured in:

config/integrations.json

The important split is:

primary_discord
→ your private/master Discord target; normally discord-webhook; receives announcements for all novels
host_discord_targets
→ optional extra host/server-specific targets, such as mistmint-discord or a future host Discord

A host-specific Discord target is an extra destination, not a replacement for primary_discord. If a novel's host has no entry in host_discord_targets, the direct announcement tools still post to the primary Discord only.

Example thread-based host target:

{
"mistmint_discord": {
"raw_base": "https://raw.githubusercontent.com/Cannibal-Turtle/mistmint-discord/main",
"paths": {
"server_json": "config/server.json",
"thread_id_map": "config/thread_id_map.json"
}
},
"host_discord_targets": {
"mistmint_haven": {
"integration": "mistmint_discord",
"routes": {
"membership_update": {
"type": "thread_map",
"map_key": "thread_id_map",
"default_path": "config/thread_id_map.json"
},
"special_announcement": {
"type": "thread_map",
"map_key": "thread_id_map",
"default_path": "config/thread_id_map.json"
}
}
}
}
}

Example channel-based host target:

{
"wuxiaworld_discord": {
"raw_base": "https://raw.githubusercontent.com/Cannibal-Turtle/wuxiaworld-discord/main",
"paths": {
"server_json": "config/server.json",
"roles_json": "config/roles.json",
"novel_discord_map": "config/novel_discord_map.toml",
"state": "state.json"
}
},
"host_discord_targets": {
"wuxiaworld": {
"integration": "wuxiaworld_discord",
"routes": {
"membership_update": {
"type": "channel",
"channel_key": "membership_updates"
},
"special_announcement": {
"type": "channel",
"channel_key": "announcements"
}
}
}
}
}

For channel routes, channel_key points to a key in that Discord repo's config/server.json. New Discord repos may use either flat channel IDs:

{
"guild_id": "123456789012345678",
"global_mention": "<@&444444444444444444>",
"announcements": "222222222222222222",
"membership_updates": "333333333333333333"
}

or nested channel IDs:

{
"guild_id": "123456789012345678",
"global_mention": "<@&444444444444444444>",
"channels": {
"announcements": "222222222222222222",
"membership_updates": "333333333333333333"
}
}

For thread routes, map_key points to a path entry in that integration's paths, usually thread_id_map, and the tool looks up the novel short_code in that JSON file. Missing host-specific thread/channel targets should mean “no extra host post,” not “skip the primary Discord post.”

Mention Resolution

For each target Discord integration, mention data is loaded from that target repo, not from the novel host mapping:

load config/server.json
→ global_mention
load config/roles.json
→ complete / ongoing / nsfw / membership / special_announcement / etc.
load config/novel_discord_map.toml
→ per-novel role / emoji / role URL
then the event decides which pieces to include

Recommended ownership:

File in target Discord repo	Owns
config/server.json	guild_id, channel IDs, raw global_mention such as @everyone, @here, <@USER_ID>, or <@&ROLE_ID>
config/roles.json	named role IDs such as complete, ongoing, nsfw, membership, membership_update, special, special_announcement
config/novel_discord_map.toml	per-novel Discord role/emoji/role URL keyed by novel short code
state.json	completion state used to choose complete vs ongoing status role

Event examples:

membership_update
→ global_mention + membership/membership_update role + maybe novel role + status/nsfw if configured by the tool
special_announcement
→ global_mention + special/special_announcement role + maybe novel role + status/nsfw if configured by the tool
completed_novel
→ novel role + complete role + maybe global_mention
regular chapter
→ novel role + nsfw role if needed + maybe global_mention

In this repo, direct manual announcement mention strings are assembled by:

Script	Responsible functions
tools/publish_membership_update.py	resolve_global_mention(), resolve_event_role_mention(), resolve_novel_role_mention(), resolve_status_role_id(), build_global_mention(), allowed_mentions_for_mention()
tools/publish_special_announcement.py	same pattern as membership: it resolves global/event/novel/status roles, builds the mention string, and sets allowed_mentions

config_loader.py only loads generic integration data. It should not grow new host-specific helpers for every future server.

Regular chapter announcements and completed-novel announcements are mostly assembled inside the downstream Discord repos that consume the RSS feeds, such as discord-webhook and mistmint-discord.

---

Mapping Files

novel_mappings.py

novel_mappings.py is the loader/front door.

Other scripts and repos can still import:

fromnovel_mappingsimportHOSTING_SITE_DATA

This keeps dependent scripts working while the actual editable data lives under:

mappings/

novel_mappings.py loads:

mappings/output_feeds.toml
mappings/hosts/*.toml
mappings/novels/*.toml

and builds HOSTING_SITE_DATA automatically.

---

Output Feed Config

mappings/output_feeds.toml

This file stores generated RSS feed URLs.

These are global repo-level feeds, not host-specific feeds.

free_feed = "https://raw.githubusercontent.com/Cannibal-Turtle/rss-feed/main/free_chapters_feed.xml"paid_feed = "https://raw.githubusercontent.com/Cannibal-Turtle/rss-feed/main/paid_chapters_feed.xml"comments_feed = "https://raw.githubusercontent.com/Cannibal-Turtle/rss-feed/main/aggregated_comments_feed.xml"

Novel TOML files use flags like:

has_free = truehas_paid = truehas_comments = true

Then novel_mappings.py injects the correct feed URLs automatically.

---

Host Mapping Files

Host-level data lives in:

mappings/hosts/

Example:

mappings/hosts/mistmint_haven.toml

Example structure:

host = "Mistmint Haven"translator = "Cannibal Turtle"translator_url = "https://example.com/account/@CannibalTurtle"host_logo = "https://example.com/logo.png"coin_emoji = "🪙"ticket_emoji = "🎟️"free_feed_url = "https://example.com/feed/"novel_free_feed_url = "https://example.com/novels/{slug}/feed"paid_feed_url = "https://example.com/feed/paid/"novel_paid_feed_url = "https://example.com/novels/{slug}/paid-feed"chapters_api_url = "https://api.example.com/api/novels/slug/{slug}/chapters"free_chapters_source = "feed"paid_chapters_source = "api"chapter_mode = "auto"comments_api_url = "https://api.example.com/..."novels_api_url = "https://api.example.com/api/my-novels"# Comment source modes:# "trans" = use tokened author dashboard endpoint; best metadata/reply tracking, token required# "public" = use no-token /comments/novel/{identifier}; novel-page comments only, no chapter thread lookup# "auto" = try "trans" first; if token is missing/expired, fall back to "public"comments_source = "auto"# Name of the GitHub secret that stores this host's login token/cookie.token_secret = "MISTMINT_COOKIE"

Host files should contain data shared by all novels on that host.

Examples:

• host
• translator
• translator_url optional translator/profile/library URL; stored in mappings and not emitted into RSS unless the feed generators are changed later
• host_logo
• coin_emoji
• ticket_emoji
• host/global feed URLs, such as free_feed_url and paid_feed_url
• host-defined per-novel feed templates, such as novel_free_feed_url and novel_paid_feed_url
• API URL templates, such as chapters_api_url = ".../{slug}/..."
• feed/API mode settings
• comments_api_url
• comments_source
• token_secret

Do not put per-novel data here.

---

Novel Mapping Files

Novel-level data lives in:

mappings/novels/

Each novel gets its own TOML file.

Example:

mappings/novels/amlwc.toml

Example:

host = "Mistmint Haven"title = "After the Male Leads Went Crazy, They All Turned into Male Ghosts"short_code = "AMLWC"novelupdates_url = "https://www.novelupdates.com/series/after-the-male-leads-went-crazy-they-all-turned-into-male-ghosts"novel_url = "https://www.mistminthaven.com/novels/after-the-male-leads-went-crazy-they-all-turned-into-male-ghosts"featured_image = "https://web-novel-mistmint.s3.ap-southeast-1.amazonaws.com/novels/example-cover.jpg"novel_id = "4221504f-49cd-4c8b-9c98-89e8b67705df"chapter_count = "93 Chapters"last_chapter = "Chapter 93"start_date = "20/6/2026"has_free = truehas_paid = trueis_nsfw = falseis_membership = falsediscord_color = "#c90016"tags = ["chinese", "quick transmigration", "supernatural"]
site_genres = ["Horror", "Supernatural", "Transmigration"]
history_file = "arc_history/amlwc_history.json"custom_description = """Optional multiline description here.TOML supports triple-quoted multiline strings, so summaries are easier to paste and edit than JSON."""

---

Important Novel Fields

Required Fields

Field	Purpose
host	Must match a host file, e.g. "Mistmint Haven"
title	Novel title used in feeds and status matching
short_code	Stable short code used across bots and workflows
novel_url	Main novel page
featured_image	Cover image URL
novel_id	Host/API novel ID when available; important for Mistmint API/comment tools
has_free	Whether this novel appears in the free feed
has_paid	Whether this novel appears in the paid feed
is_nsfw	Whether this novel should be categorized as NSFW
is_membership	Whether this novel is currently membership-only/available for membership

Optional Fields

Field	Purpose
novelupdates_url	Novel Updates page URL
chapter_count	Display text for completion/status cards
last_chapter	Completion checker target
start_date	Used to calculate “After X of updates...” in completion messages
has_comments	Comments feed flag; defaults to true unless explicitly set to false
tags	Discord-supported genre tags, such as chinese, modern, romance, bl; downstream Discord repos use this for role mentions
site_genres	Full original Mistmint Haven genre names from the API, kept for reference even if some are not Discord-supported tags
history_file	Arc checker history file
discord_color	Novel-specific embed color for Discord repos
theme_color	Optional alternate novel color field
custom_description	Multiline description for manual publishing/status cards

---

Empty Optional Fields

Use empty strings instead of deleting optional fields when you want scripts to safely skip related behavior.

start_date = ""discord_color = ""site_genres = []
history_file = ""

Meaning:

Field	Empty Behavior
start_date = ""	Completion announcement omits the duration phrase
discord_color = ""	Discord repos use their normal/default color logic
site_genres = []	No Mistmint host genres are stored
history_file = ""	Arc checker skips arc tracking for the novel

---

NSFW and Membership Tracking

NSFW

NSFW is controlled in the novel TOML:

is_nsfw = true

This can flow into RSS categories and Discord announcements.

Membership

Membership status is controlled in the novel TOML:

is_membership = true

The manual membership tool can update this automatically when a novel is announced as membership-available. Banners for membership are automatically blurred if novel is nsfw.

---

Helper Functions from novel_mappings.py

Downstream repos can use:

fromnovel_mappingsimportHOSTING_SITE_DATA

and helpers such as:

get_novel_details_by_short_code(short_code)
find_novel_by_short_code(short_code)
short_code_has_free_chapters(short_code)
short_code_has_paid_chapters(short_code)
short_code_has_comments_feed(short_code)
resolve_short_code(title, host)

Short codes are the stable bridge between this repo and the Discord repos.

---

Installable Package

pyproject.toml makes this repo installable as:

cannibal-turtle-rss-feed

Install from GitHub:

pip install --upgrade git+https://github.com/Cannibal-Turtle/rss-feed.git@main

The package currently installs:

novel_mappings.py
mappings/

including:

mappings/hosts/*.toml
mappings/novels/*.toml
mappings/output_feeds.toml

It is mainly a shared mapping package for the Discord repos.

The scraper/feed engine files such as host_utils/, tools/, and feed generator scripts are part of the repo, but are not currently packaged by pyproject.toml unless packaging is expanded.

---

Requirements

Install local script dependencies with:

pip install -r requirements.txt

Current main dependencies:

feedparser
PyRSS2Gen
aiohttp
beautifulsoup4
requests

Some workflows/scripts may also install:

discord.py
python-dateutil
tomli

tomli is only needed below Python 3.11.

---

Host Utilities

Host-specific scraper/API logic lives in:

host_utils/

Current shape:

host_utils/
├─ __init__.py
├─ host_dragonholic.py
├─ host_nu_comments.py
├─ host_titv.py
└─ mistmint_haven/
├─ __init__.py
├─ common.py
├─ client.py
├─ free_chapters.py
├─ paid_chapters.py
└─ comments.py

host_utils/__init__.py is the host registry/dispatcher.

Use:

fromhost_utilsimportget_host_utilsutils=get_host_utils("Mistmint Haven")

The Mistmint implementation is split into:

File	Purpose
common.py	Shared helpers, parsing, diagnostics, settings, state helpers
client.py	Mistmint API/client helpers
free_chapters.py	Free chapter feed/API logic
paid_chapters.py	Paid chapter scraping/API/update logic
comments.py	Comments, replies, sticker/comment link logic
__init__.py	Assembles MISTMINT_UTILS

The host registry should lazy-load hosts so one host does not require another host’s dependencies at import time.

---

RSS Feed Generators

The chapter generators are separated by chapter type, not by source method:

free_feed_generator.py
→ builds free_chapters_feed.xml
→ formats free/public chapter items
paid_feed_generator.py
→ builds paid_chapters_feed.xml
→ formats paid/premium chapter items, including paid history and coin/price fields

Shared generator rules live in:

feed_common.py

feed_common.py owns shared helpers such as:

• reading the canonical completion state URL from mappings/output_feeds.toml
• loading discord-webhook/state.json
• resolving completion keys:
  • paid → paid_completion
  • free + novel has paid feed → free_completion
  • free + novel has no paid feed → only_free_completion
• detecting whether a source is host/global or novel-scoped from mapping shape
• shared sorting and NSFW marker helpers

Source Scope Rule

The generators treat scope separately from chapter type.

host/global feed
→ fetch once
→ scan entries
→ match entry title to mapped novel
→ no completion gate
novel-level feed
→ loop novels
→ check completion state before fetching that novel feed
→ parse entries for that novel
novel-level API
→ loop novels
→ check completion state before API request
→ fetch chapter data
→ filter by chapter type

This keeps host/global feeds simple: if the feed has an entry, the generator includes it. Completion state is only a fetch-saving gate for novel-scoped sources.

Source Config

Host TOML controls which source method each chapter type uses:

free_chapters_source = "feed"paid_chapters_source = "api"

The source method is not the same as chapter type.

For example, Mistmint's chapter API returns all chapters. The free and paid logic filter the same chapter data differently:

free API logic
→ keep chapters where isFree is true
paid API logic
→ keep chapters where isFree is false and the chapter is not hidden

A host/global feed is recognized when a feed URL is defined in mappings/hosts/*.toml, such as:

free_feed_url = "https://www.mistminthaven.com/feed/"

A host can also define reusable per-novel feed templates alongside its global feeds:

novel_free_feed_url = "https://www.mistminthaven.com/novels/{slug}/feed"novel_paid_feed_url = "https://www.mistminthaven.com/novels/{slug}/paid-feed"

These do not replace the global feeds. In feed_api mode the matching per-novel feed is the first fallback when its global feed is capped or unavailable. A host only needs to define the URLs it actually provides.

A novel-level feed is recognized when a feed URL is defined in a specific mappings/novels/*.toml file.

A URL template like this is stored at host level but fetched per novel because it needs the novel slug:

chapters_api_url = "https://api.mistminthaven.com/api/novels/slug/{slug}/chapters"

Feed/API Fetch Behavior

Simple rule:

global feed URL = one async fetch
slugged / per-novel feed URL = async concurrent fetches
paid / free API source = async concurrent scraping

Examples:

free_feed_url = "https://www.mysite.com/feed/"paid_feed_url = "https://www.mysite.com/feed/premium/"

These are global feeds, so each one is fetched once.

free_feed_url = "https://www.mysite.com/feed/{novel_slug}/"paid_feed_url = "https://www.mysite.com/feed/premium/{novel_slug}/"

These are per-novel feeds, so the generator creates async tasks per novel and fetches them concurrently.

A host may expose both scopes at once:

free_feed_url = "https://www.mysite.com/feed/"novel_free_feed_url = "https://www.mysite.com/novels/{slug}/feed"paid_feed_url = "https://www.mysite.com/feed/paid/"novel_paid_feed_url = "https://www.mysite.com/novels/{slug}/paid-feed"chapters_api_url = "https://api.mysite.com/novels/{slug}/chapters"

With either chapter source set to feed_api, its generator uses the same scope-aware chain:

host/global free or paid feed
→ if capped or unavailable, fetch mapped novels through the matching per-novel feed template
→ use chapters_api_url only for novels whose per-novel feed is missing, failed, or also looks capped
→ de-duplicate by GUID/link

The free and paid generators each write a temporary fallback report for the Discord alert step. The report identifies whether the current batch used per-novel feeds, API fallback, or both; it is not committed to the repository.

For shared free + paid feeds:

free_feed_url = "https://www.mysite.com/feed/{novel_slug}/"paid_feed_url = "https://www.mysite.com/feed/{novel_slug}/"

The generator fetches the same feed for free and paid, but host_utils must know how to separate free entries from paid entries. Host-specific free/paid detection belongs in host_utils/<host_name>/.

Shared generator helpers belong in feed_common.py.

free_feed_generator.py

Builds:

free_chapters_feed.xml

Uses novel TOML entries with:

has_free = true

Supports:

host/global free feed
novel-level free feed
host utility/API loader, such as Mistmint free API mode

The XML item structure is kept inside this generator so the free RSS output format stays stable.

paid_feed_generator.py

Builds:

paid_chapters_feed.xml

Uses novel TOML entries with:

has_paid = true

Supports:

host/global paid feed
per-novel paid feed as a primary source or global-feed fallback
novel-level paid API/scraper fallback for uncovered, failed, or capped novel feeds
manual/state fallback when a host uses it

When paid_chapters_source = "feed_api" and both feed scopes are configured, the paid generator follows global paid feed → per-novel paid feed → API, the same way the free generator handles its sources.

Paid-only behavior stays inside this generator, including:

paid_history.json + paid GUID handling (for manual entry only)
coin/price fields
paid-specific item formatting

comments.py

Builds:

aggregated_comments_feed.xml

Uses novel TOML entries with:

has_comments = true

The comments generator already asks novel_mappings.py for comment source URLs with novel fallback, so it can use host-level or novel-level comment API/feed config depending on the mapping.

---

Sample RSS Item Fields

Generated RSS items may include:

title
volume
chapter
chaptername
link
description
category
translator
short_code
featured_image_url
pub_date
host
host_logo_url
guid
guid_is_permalink

These are consumed by the Discord repos.

---

Feed Sorting

Paid/free feed sorting should not depend on TOML insertion order.

Where possible, sorting should use parsed dates, chapter numbers, and stable tie-breakers such as alphabetical title/short code.

---

Mistmint Haven Quick Setup

Modes

Mistmint host config can control source modes:

# Chapter source modes:# "feed" = use a host-provided feed/RSS-style source for this chapter type# "api" = use the host chapter API/data source, then filter by chapter typefree_chapters_source = "feed"paid_chapters_source = "api"# Paid chapter fallback mode:# "auto" = try API/chapter data first; cookie is optional if the endpoint allows it# "manual" = force mistmint_state.json/manual paid chapter fallback instead of APIchapter_mode = "auto"# Comment source modes:# "trans" = use tokened author dashboard endpoint; best metadata/reply tracking, token required# "public" = use no-token /comments/novel/{identifier}; novel-page comments only, no chapter thread lookup# "auto" = try "trans" first; if token is missing/expired, fall back to "public"comments_source = "auto"

Typical meaning:

Setting	Purpose
free_chapters_source	Whether free chapters come from a feed-style source or the chapter API/data source
paid_chapters_source	Whether paid chapters come from a feed-style source or the chapter API/data source
chapter_mode	Mistmint paid fallback mode; auto tries API/chapter data, manual forces manual/state fallback
comments_source = "trans"	Uses comments_api_url / comments/trans/all-comments; best metadata and reply tracking, but token/cookie is required
comments_source = "public"	Uses public no-token /comments/novel/{identifier} endpoints. This is a novel-page comments fallback only: it does not resolve chapterId, does not call chapter comment endpoints, and should not be treated as a full chapter comments feed.
comments_source = "auto"	Tries trans first, then falls back to public mode if the token is missing/expired

The public Mistmint comments endpoint is internal host logic, not user-facing repo config. The code builds it from BASE_API as /comments/novel/{identifier} and tries the mapped novel_id first, then the novel slug. Based on the current public API shape, this endpoint returns comments attached to the novel itself, not full chapter comment threads. Keep this in Python unless Mistmint changes endpoint structure often enough that it becomes worth exposing a separate URL template.

Public comments mode is intentionally limited. It does not resolve chapterId, it does not call /comments/chapter/{chapterId}, and it does not build reply chains. Use trans mode for the complete author-wide comments feed and richer reply tracking.

Mapping

For each Mistmint novel, the novel TOML should include:

host = "Mistmint Haven"short_code = "CODE"novel_url = "https://www.mistminthaven.com/..."has_free = truehas_paid = truehas_comments = true

Required Repo Secrets

Mistmint API/private data may need:

Secret	Purpose
MISTMINT_COOKIE	Mistmint login/session cookie or token, depending on script
DISCORD_BOT_TOKEN	Required for direct Discord tools/reports
GH_PAT	Used when dispatching workflows or editing external repo files where needed

tools/create_novel_toml.py uses the host config's token_secret first. For Mistmint, that normally means MISTMINT_COOKIE. It can also use MISTMINT_TOKEN if you provide bearer-token auth instead.

---

Token-Expiry Alerts

Token alert logic lives in:

token/send_token_alert.py
token/token_alert_state.json
message_templates/token_alert.toml

Template-specific user settings live in:

[settings]
global_mention = "||<@&1329392448798982214>||"

Token alert checks are host-agnostic. Any host file can opt in by defining a token secret:

token_secret = "HOST_COOKIE_OR_TOKEN_SECRET"

Optional host setting:

token_alerts = "auto"

Supported values:

Value	Meaning
"auto" or omitted	Alert for hosts with token_secret, except when comments_source = "public"
true	Always alert when token_secret is present
false	Never send token alerts for this host

Examples:

# Token is needed for private/API comment fetching. Alerts run.token_secret = "MISTMINT_COOKIE"comments_source = "auto"token_alerts = "auto"

# Comments are intentionally public-only. Alerts are skipped in auto mode.token_secret = "SOME_OPTIONAL_COOKIE"comments_source = "public"token_alerts = "auto"

# Token is used by another script even if comments are public. Alerts still run.token_secret = "HOST_API_TOKEN"comments_source = "public"token_alerts = true

token-expiring alerts only work for JWT-style tokens/cookies with an exp field. token-invalid alerts are sent when a host comments loader raises an AUTH_ERROR, and follow the same token_alerts host policy.

The workflow:

.github/workflows/send_token_alert.yml

can be triggered by dispatch or manually.

---

Message Templates in rss-feed

This repo has direct Discord tools/reports, so it also has templates:

message_templates/membership_update.toml
message_templates/nu_weekly_readers.toml
message_templates/publish_novel_card.toml
message_templates/revenue_report.toml
message_templates/token_alert.toml

Unlike discord-webhook, this repo does not use config/embeds.json.

Template-specific user/repo settings should live in each TOML file under:

[settings]

Examples:

[settings]
global_mention = "||<@&1329392448798982214>||"embed_color = "2D3F51"novel_discord_map_url = "https://raw.githubusercontent.com/Cannibal-Turtle/discord-webhook/main/config/novel_discord_map.toml"

Python should read these settings with:

load_template_settings("template_name")

This keeps fork-specific IDs, colors, and URLs out of Python.

---

Novel Updates Weekly Readers

Script:

novelupdates/nu_weekly_readers.py

Template:

message_templates/nu_weekly_readers.toml

State:

novelupdates/nu_readers.json

Workflow:

.github/workflows/nu_weekly_readers.yml

Purpose:

• fetch Novel Updates reading-list counts
• compare with saved state
• generate weekly deltas
• post a Discord report
• save updated counts

Template settings include:

[settings]
global_mention = "||<@&1329392448798982214>||"embed_color = "2D3F51"novel_discord_map_url = "https://raw.githubusercontent.com/Cannibal-Turtle/discord-webhook/main/config/novel_discord_map.toml"allow_role_pings = trueno_data_text = "_No data this week (no NU counts retrieved)._"

The novel_discord_map_url lets this repo resolve novel short codes to Discord role mentions without storing Discord role data inside rss-feed mappings.

---

Monthly Revenue Report

Script:

revenue/report.py

Template:

message_templates/revenue_report.toml

State:

revenue/state.json

Workflow:

.github/workflows/monthly_revenue.yml

Purpose:

• collect host revenue data
• calculate monthly deltas
• post a Discord report
• save baseline/state data

Template settings include:

[settings]
global_mention = "||<@&1329392448798982214>||"embed_color = "C9D3FF"novel_discord_map_url = "https://raw.githubusercontent.com/Cannibal-Turtle/discord-webhook/main/config/novel_discord_map.toml"

Revenue rows are controlled by template blocks such as:

[row_basic]
[row_membership]
[first_run]
[month_header]
[monthly_total]
[empty_report]

---

Create Novel TOML Tool

Script:

tools/create_novel_toml.py

Workflow:

.github/workflows/create_novel_toml.yml

Purpose:

• fetch an existing dashboard novel from a configured host API
• create mappings/novels/<short_code>.toml
• fill host-provided fields automatically, including title, slug URL, novel ID, description, start date, NSFW flag, and the cover image
• guess the Novel Updates URL from the title
• keep only Discord-supported mention tags from discord-webhook/config/tag_roles.json in tags
• preserve the full Mistmint Haven API genre list in site_genres
• optionally create arc_history/<short_code>_history.json

For Mistmint Haven, the host file must include:

novels_api_url = "https://api.mistminthaven.com/api/my-novels"token_secret = "MISTMINT_COOKIE"

Workflow inputs:

Input	Required?	Purpose
host	Yes	Hosting site, e.g. Mistmint Haven
title	Yes	Novel title exactly/as shown in the host dashboard
short_code	Yes	New short code, e.g. AMLWC
chapter_count	No	Optional display text, e.g. 93 Chapters; blank writes ""
last_chapter	No	Optional target text, e.g. Chapter 93; blank writes ""
discord_color	No	Optional hex color, e.g. #c90016; blank writes ""
quick_transmigration	No	Checkbox. Tick this only when the novel is quick transmigration; adds quick transmigration to tags.
infinite_flow	No	Checkbox. Tick this only when the novel is infinite flow; adds infinite flow to tags.
has_arcs	Yes	If true, creates arc_history/<short_code>_history.json and sets history_file
dry_run	Yes	If true, previews the TOML in the Actions log without committing
overwrite	Yes	If true, allows replacing an existing mappings/novels/<short_code>.toml

Recommended first run:

dry_run = true
overwrite = false

Then rerun with:

dry_run = false

once the generated TOML looks right.

Tags and Mistmint genres

tags should only contain tags that exist in the Discord repo's config/tag_roles.json, because downstream Discord repos use tags for role mentions.

site_genres stores the full original genre names from the Mistmint API. It does not need to match Discord role tags and is kept as a reference copy of what Mistmint lists on the novel.

World-hopping uses two Actions checkboxes. Leave both unchecked when the novel is not quick transmigration or infinite flow. If it is world-hopping, tick exactly one checkbox.

When a world-hopping checkbox is ticked, the selected tag is written directly into tags:

tags = ["chinese", "quick transmigration", "modern", "romance", "bl"]
site_genres = ["Modern", "Romance", "Yaoi", "Transmigration"]

If quick transmigration or infinite flow is selected, the tool removes plain transmigration from tags automatically. This keeps a world-hopping novel from ending up with both the broad transmigration role and the specific world-hopping role.

Leaving both world-hopping checkboxes unchecked writes normal tags only:

tags = ["chinese", "transmigration", "modern", "romance", "bl"]
site_genres = ["Modern", "Romance", "Yaoi", "Transmigration"]

---

Automatic Novel Status Updater

Script:

tools/update_novel_card.py

State/targets:

novel_card_targets.json

Workflow:

.github/workflows/update_novel_card.yml

Purpose:

• resolve title + host to short_code
• calculate current novel status
• edit existing Discord status cards in place
• avoid reposting duplicate novel cards

Workflow inputs:

title: "Novel title"host: "Mistmint Haven"

novel_card_targets.json stores Discord message targets by short code.

---

Manual Novel Card Tool

Script:

tools/publish_novel_card.py

Template:

message_templates/publish_novel_card.toml

Workflow:

.github/workflows/publish_novel_card.yml

Workflow inputs:

short_code: "AMLWC"channel_id: "optional extra Discord channel/thread ID"

What it does:

• resolves a novel by short code
• pulls novel metadata from novel_mappings.py
• pulls Discord role/emoji/role URL from novel_discord_map_url
• renders a manual novel card
• posts to the configured archive channel and/or optional extra channel
• records/updates the status target where relevant

Template settings include:

[settings]
archive_channel_id = "1463476725253144751"novel_discord_map_url = "https://raw.githubusercontent.com/Cannibal-Turtle/discord-webhook/main/config/novel_discord_map.toml"

---

Membership Update Tool

Script:

tools/publish_membership_update.py

Template:

message_templates/membership_update.toml

Workflow:

.github/workflows/publish_membership_update.yml

Workflow inputs:

short_code: "AMLWC"banner_url: "https://..."

What it does:

• resolves a novel by short code
• posts a membership announcement to the primary Discord target
• optionally posts an extra host-specific copy to a configured host Discord channel/thread
• uses Components V2 payloads
• can write membership state back to the novel TOML
• resolves mentions from the target Discord integration's server.json, roles.json, novel_discord_map.toml, and state.json when those files are configured

Template settings can still include a fallback public mention:

[settings]
public_global_mention = "||@everyone||"

That setting is the last fallback. Preferred global/public pings should live in the target Discord repo's config/server.json as global_mention.

Banner cropping for membership updates is configured in message_templates/membership_update.toml:

[settings.banner]
ratio = "4:1"# use "original" to preserve the full featured image

The workflow's optional banner_ratio input can override this per run. Ratio values such as 8:3 crop/resize to that shape; original performs no crop or resize. The same controls are available for special announcements in message_templates/special_announcement.toml and publish_special_announcement.yml. With crop_position = "auto", a lightweight Pillow-only heuristic compares text-like horizontal activity near the top and bottom: stronger top activity selects lower center, stronger bottom activity selects upper center, and unclear results default to upper center. Manual crop positions remain available.

---

Workflow Inputs

create_novel_toml.yml

Input	Purpose
host	Hosting site, e.g. Mistmint Haven
title	Novel title exactly/as shown in host dashboard
short_code	Novel short code, e.g. AMLWC
chapter_count	Optional chapter count text
last_chapter	Optional last-chapter text
discord_color	Optional novel embed color
quick_transmigration	Whether to add quick transmigration to tags
infinite_flow	Whether to add infinite flow to tags
has_arcs	Whether to create an arc history file
dry_run	Preview without committing
overwrite	Allow replacing an existing mapping file

publish_novel_card.yml

Input	Purpose
short_code	Novel short code, e.g. AMLWC
channel_id	Optional extra Discord channel/thread ID

publish_membership_update.yml

Input	Purpose
short_code	Novel short code, e.g. AMLWC
banner_url	Optional finished banner URL
banner_ratio	Optional ratio override such as 8:3, or original for no crop/resize
crop_position	Auto or manual crop position when using a ratio

publish_special_announcement.yml

Input	Purpose
short_code	Novel short code
banner_url	Optional finished banner URL
banner_ratio	Optional ratio override such as 8:3, or original for no crop/resize
crop_position	Auto or manual crop position when using a ratio

update_novel_card.yml

Input	Purpose
title	Novel title
host	Hosting site, e.g. Mistmint Haven

---

Workflows

Workflow	Purpose
update_free_feed.yml	Regenerates free RSS feed, scheduled daily
update_paid_feed.yml	Regenerates paid RSS feed, scheduled hourly
update_comments.yml	Regenerates comments RSS feed, scheduled hourly
create_novel_toml.yml	Creates a new novel TOML from a configured host API
update_novel_card.yml	Edits existing Discord novel status cards
publish_novel_card.yml	Manually posts a novel/status card
publish_membership_update.yml	Manually posts membership announcement
publish_special_announcement.yml	Manually posts a special announcement
monthly_revenue.yml	Posts monthly revenue report
nu_weekly_readers.yml	Posts weekly NU reader-count report
send_token_alert.yml	Sends token warning/error alerts

---

Adding a New Novel on an Existing Host

Preferred method for a configured API host:

1. Run:

   create_novel_toml.yml
   

2. Start with:

   dry_run = true
   overwrite = false
   

3. Check the generated TOML in the Actions log.

4. Rerun with:

   dry_run = false
   

5. Confirm the new file exists in:

   mappings/novels/<short_code>.toml
   

Manual fallback:

1. Create a novel TOML file in:

   mappings/novels/
   

2. Add the core fields:

   host = "Mistmint Haven"title = "Novel Title"short_code = "CODE"novelupdates_url = "https://www.novelupdates.com/series/novel-title"novel_url = "https://www.mistminthaven.com/novels/novel-title"featured_image = "https://..."novel_id = ""chapter_count = ""last_chapter = ""start_date = ""has_free = truehas_paid = trueis_nsfw = falseis_membership = falsediscord_color = ""tags = ["chinese"]
   site_genres = []
   history_file = ""custom_description = """Description here."""

3. If the novel has arcs, set:

   history_file = "arc_history/code_history.json"

   and create the matching JSON file with:

   {}

4. Add Discord role/emoji/role URL data in the Discord repo that owns the announcement role map, usually:

   discord-webhook/config/novel_discord_map.toml
   

5. If the novel's host has an extra host-specific Discord target, add the required channel/thread mapping in that host Discord repo. For a thread target, that usually means:

   <host-discord-repo>/config/thread_id_map.json
   

   Channel IDs belong in that Discord repo's config/server.json; config/integrations.json only stores which channel_key to read.

6. Run the relevant feed workflow.

7. Run publish_novel_card.yml if you need a manual card.

8. Confirm novel_card_targets.json updates if the novel has a status card.

---

Adding a New Hosting Site

1. Add host config:

   mappings/hosts/new_host.toml
   

2. Add novel configs:

   mappings/novels/code.toml
   

3. Decide the source scope/method for free and paid chapters.

   Host/global feed example:

   # mappings/hosts/new_host.tomlfree_chapters_source = "feed"free_feed_url = "https://example.com/feed/"

   Global feed with per-novel and API fallback:

   # mappings/hosts/new_host.tomlfree_feed_url = "https://example.com/feed/"novel_free_feed_url = "https://example.com/novels/{slug}/feed"paid_feed_url = "https://example.com/feed/paid/"novel_paid_feed_url = "https://example.com/novels/{slug}/paid-feed"chapters_api_url = "https://api.example.com/novels/{slug}/chapters"

   Then set the relevant free_chapters_source and/or paid_chapters_source to feed_api in config/source_modes.json.

   Novel-level feed example:

   # mappings/novels/code.tomlfree_feed_url = "https://example.com/novel/code/feed/"

   Novel-level API example:

   # mappings/hosts/new_host.tomlpaid_chapters_source = "api"chapters_api_url = "https://api.example.com/novels/{slug}/chapters"

4. Add or update host utilities in:

   host_utils/
   

5. Register the host in:

   host_utils/__init__.py
   

6. Make sure feed generators can load the host utils.

7. Add any required token/cookie secret name in the host TOML.

8. Add Discord-side config only where needed:

   • discord-webhook is the primary/private Discord target and can hold all-novel role data.
   • A host-specific Discord repo is optional. Add one under host_discord_targets only if that host needs an extra channel/thread post.
   • If the host has no extra Discord target, direct announcement tools still post to the primary Discord.

---

Manual Publishing Checklist

When adding a new novel:

1. Run create_novel_toml.yml, or manually create a novel TOML file in mappings/novels/.

2. Add a unique short_code.

3. Check tags, site_genres, chapter_count, last_chapter, and discord_color before publishing. For world-hopping, tick exactly one world-hopping checkbox so the matching role tag appears inside tags.

4. Add Discord role/emoji/role URL data in the Discord repo.

5. Run publish_novel_card.yml.

6. Confirm novel_card_targets.json was updated.

7. If the novel enters membership, run publish_membership_update.yml with:

   • short_code
   • banner_url

8. Confirm the novel TOML now has:

   is_membership = true

---

Troubleshooting

ModuleNotFoundError: feedparser

Install requirements:

pip install -r requirements.txt

For workflows that import multiple scripts, also make sure the workflow installs needed dependencies before running Python.

Host import breaks an unrelated script

host_utils should lazy-load host modules.

A script asking for Mistmint should not import Dragonholic/TITV dependencies unless it requests those hosts.

TOML template placeholders render empty

A placeholder only works if Python passes the value in the render context.

Example:

color = "{embed_color}"

needs Python to pass:

{"embed_color": 0x2D3F51}

Template-specific defaults should live under:

[settings]

and Python should read them with load_template_settings(...).

Discord role/config URL should not be hardcoded in Python

Repo/user-specific IDs and URLs should live in template [settings], not in Python.

Good:

[settings]
global_mention = "||<@&1329392448798982214>||"novel_discord_map_url = "https://raw.githubusercontent.com/Cannibal-Turtle/discord-webhook/main/config/novel_discord_map.toml"

Completion state URL is missing

feed_common.py expects the canonical Discord completion state URL in:

[completion_state_url]
discord_webhook = "https://raw.githubusercontent.com/Cannibal-Turtle/discord-webhook/main/state.json"

If this URL is missing or unreachable, generators continue without completion skipping instead of failing the whole feed run.

JSON state file crashed

Empty files are invalid JSON.

Use:

{}

---

Design Guarantees

• novel_mappings.py remains import-compatible for dependent scripts.
• Actual mapping data lives in TOML files.
• Host-level config lives in mappings/hosts/.
• Novel-level config lives in mappings/novels/.
• Novel descriptions can use TOML multiline strings.
• NSFW status comes from is_nsfw.
• Membership status comes from is_membership.
• Output feed URLs and the canonical Discord completion state URL are centralized in mappings/output_feeds.toml.
• feed_common.py owns shared generator helpers; free/paid XML item formatting stays in the individual generator files.
• Completion state is a fetch-saving gate for novel-scoped sources, not a filter for host/global feeds.
• Paid/free feed sorting should not depend on mapping insertion order.
• history_file = "" safely means no arc tracking.
• start_date = "" safely means no duration phrase in completion messages.
• site_genres is the full Mistmint API genre list; tags is the Discord-supported mention list.
• Leaving both world-hopping checkboxes unchecked adds no world-hopping tag to tags.
• update_novel_card.py edits existing Discord messages instead of reposting.
• novel_card_targets.json stores message targets by short code.
• Discord role IDs, custom emojis, and role URLs belong in Discord bot repos, not in rss-feed mappings.
• Direct-report template settings belong in message_templates/*.toml, not hardcoded Python.
• Mistmint comments can run in trans, public, or auto mode via comments_source.
• Public Mistmint comment fallback is best-effort and currently limited to public /comments/novel/{identifier} results, which should be treated as novel-page comments rather than a full chapter comments feed. Tokened trans mode remains the most complete source for author-wide comments and reply tracking.

---

System Overview

Host sites / APIs
↓
rss-feed host utils + feed_common.py
↓
free / paid / comments RSS XML
↓
discord-webhook + mistmint-discord
↓
Discord announcements

Manual tools and reports also live in rss-feed:

NU weekly readers
monthly revenue
membership update
manual novel card
token alerts
status updater

---

Example Result

The final Discord status card is created once, then updated automatically when new free chapters are posted.

The status updater preserves the existing card layout and only edits the status field.