feat(email): durable email outbox with push-based retry consumer

[hhvrc]

What this does

Transactional emails (account activation, password reset, email-change verification, email-change notice) were sent inline during the HTTP request. If the mail provider was down, the process restarted, or the network blipped, the email was simply lost - with no record and no way to resend.

This PR makes email delivery durable and retryable via a transactional outbox delivered by Hangfire: the request handler writes an EmailOutboxMessage row in the same transaction as the business change and commits; the Cron host picks it up and hands it to Hangfire, which sends it and retries until it succeeds or is exhausted. Rows are never auto-deleted, so failures stay visible and resendable.

The split is deliberate: the outbox row owns transactional integrity and the audit fact ("this user is owed this email, here is its delivery state"); Hangfire owns the delivery machinery (retry back-off, persistent scheduling, crash requeue, and an operator dashboard) instead of a hand-rolled worker.

How it works

1. Enqueue, don't send.AccountService creates the request row (e.g. UserPasswordReset) and an EmailOutboxMessage (status Pending) in one transaction, then publishes a Redis notification. The API host does nothing else - it never sends.
2. Hand-off in Cron.EmailOutboxConsumer (a coalescing background service in the Cron host) is woken by the Redis notification (email-outbox-pending), with a 1-minute sweep as a safety net for any missed notification. The push and the sweep share one capacity-1 wake signal, so they can never fire on top of each other. It claims Pending rows with FOR UPDATE SKIP LOCKED and enqueues a Hangfire job per email, flipping each row to Queued.
3. Hangfire delivers.EmailOutboxJob.SendAsync(id) carries [AutomaticRetry(Attempts = 10)]: a transient failure throws so Hangfire reschedules it on its built-in back-off curve; a permanent/skip outcome marks the row Failed; success marks it Sent. On the final attempt (detected via PerformContext.RetryCount) the row is marked Failed too, so an exhausted message never stays stuck looking Queued. Crash mid-send is recovered by Hangfire's requeue. The precise attempt count + per-retry history live in the Hangfire dashboard.
4. Single send path.IEmailOutboxDispatcher is the only place email is sent. It maps the message type + dynamic jsonb payload to a template, mints the secret token lazily at send time (re-minted on every resend), and classifies the result. IEmailService returns EmailSendResult (Sent / Transient / Permanent) instead of throwing.

Security note

Because a reset/verification link can't be reconstructed from a stored hash, tokens are minted at send time and only the hash is persisted. The queue therefore never contains a usable link, and a resend produces a fresh token - the desired behaviour for security links anyway. Existing token tables are unchanged: the hash is seeded on insert and rotated on send.

---

Why the diff is large (and what's actually in it)

52 files, +2996 / −339. The line count is dominated by generated code and a project move, not new logic:

Category	Lines	Notes
EF migration + model snapshot (autogenerated)	~1,743 (~58%)	dotnet ef migrations add output; the bulk is the Designer/snapshot EF rewrites wholesale. Not hand-written, not reviewed line-by-line.
Email stack moved API → Cron	large	Providers, templates, dispatcher, options, and .liquid files relocated so sending lives in the Cron host. Mostly moves (namespace-only changes); a few git-detected renames, the rest paired add/delete.
New delivery logic	~280	EmailOutboxConsumer (159) + EmailOutboxJob (120) - the real surface area to review.
Tests / docs	-	EmailOutboxMessageTests; XML docs explaining the design so it's self-documenting.

So the substantive new code is the ~280-line consumer + Hangfire job; the rest is generated schema, a project relocation, the outbox entity/enums, and documentation.

---

Outbox + Hangfire (the design)

This is a transactional outbox with Hangfire as the relay/executor - the two layers own different things and are combined on purpose:

• The outbox row owns transactional integrity + the audit fact. The email intent commits in the same transaction as the change that caused it, so there's no window where the business row commits but the email is lost, or where an email is scheduled for a change that then rolls back. The row is also the domain answer to "is this user owed this email, and what's its state" - one query gives type / status / last error / sent-or-failed.
• Hangfire owns the delivery machinery. Once the consumer hands a row off, Hangfire provides the retry back-off, the persistent retry schedule, crash requeue, and the dashboard (with per-job history and one-click requeue) - the parts a bespoke worker would otherwise reinvent. This also keeps email delivery consistent with how the rest of the codebase already runs background work (Hangfire in the Cron host).

The API host stays free of email-send work entirely; it only writes the row and rings the bell.

---

Scope / roadmap

One self-contained PR. Follow-ups (now trivial - they just insert a row): admin "send any email type", admin "list email types", and a Cron prune job for old Sent rows.

Deployment note: the Cron host now sends email, so it needs the OpenShock:Mail config section (and, for a real provider, Mailjet/SMTP credentials). The API host no longer needs that config or the templates.

Testing

• Full solution builds clean.
• Migration regenerated via tooling (then renamed back to its original timestamp to keep the diff a modification, not a delete + re-add); model has no pending changes; existing token tables untouched.
• Common.Tests green (265 passing).
• Existing Docker-based MailTests are consistent with async delivery (they already wait for mail, parse the minted token, expect the same link formats) but were not run in this environment.

[ghost]

Ready to review this PR? Stage has broken it down into 12 individual chapters for you:

	Title
1	Define email outbox schema and models
2	Add Redis notification for pending mail
3	Relocate email stack from API to Cron
4	Update email providers with result classification
5	Implement outbox delivery state machine
6	Implement outbox dispatcher and consumer
7	Wire outbox delivery into Cron host
8	Switch API to enqueue outbox messages
9	Update integration tests for outbox flow
10	Add Cron integration tests
11	Update solution and project structure
12	Other changes

_{Chapters generated by Stage for commit 93de6d1 on Jun 30, 2026 10:14pm UTC.}