Skip to content

emrg: daemon observability (exit records + crash log) & promote blog cadence 1-3 days - #970

Merged
argszero merged 2 commits into
masterfrom
feature/daemon-observability-blog-cadence
Aug 25, 2026
Merged

emrg: daemon observability (exit records + crash log) & promote blog cadence 1-3 days#970
argszero merged 2 commits into
masterfrom
feature/daemon-observability-blog-cadence

Conversation

@pm25coder

Copy link
Copy Markdown
Collaborator

Two independent fixes driven by two host rants.

1. Daemon observability — durable exit records + stderr crash log

Rant (verbatim, 2026-08-25T09:25:32+08:00):

daemon 异常退出无任何日志痕迹,死因无法定位(已 2 例可考:8-24 20:40 腰斩 R46、8-25 02:00 腰斩 emrg-task 01:26 轮并跳过 promote 06:06 调度)。emrgd.log 在死亡瞬间无 shutdown/错误/退出码记录,重启后一切正常。请求日志/可观测性增强(非防死):① daemon 退出无论正常/异常都写退出记录(退出码、信号、traceback);② stderr/stdout 独立重定向到崩溃日志文件;③ TaskHandler 运行中 cycle 定期持久化进度心跳,重启后能识别"上次中断于 round X";④ scheduler next run 持久化,重启不重置。目标:下次再出现时能定位原因。

Root cause: daemon_manager spawns emrgd with stderr=DEVNULL, so anything bypassing logging — asyncio's default exception handler, C-level abort traces, faulthandler dumps — vanished without a trace. A background-task crash or fatal signal left zero evidence in emrgd.log.

This PR covers acceptance items ① and ②:

  • DaemonExit + _write_exit_record (daemon.py): every stop path of run_server (normal return, serve() crash, SIGINT, SIGTERM, cancellation, unexpected BaseException) now produces a DaemonExit(reason, exit_code, traceback) that the entry point persists as one-line JSON in ~/.emrg/emrgd-exit.log (append-only, survives emrgd.log rotation) and mirrors into emrgd.log. Exit-code 0 vs 1 vs 130/143 distinguishes clean stops from crashes.
  • stderr/stdout redirect (__main__.py): _redirect_std_streams() points sys.stdout/sys.stderr at ~/.emrg/emrgd-crash.log (independent sink, keeps the file handler on emrgd.log separate) and enables faulthandler so even fatal-signal deaths (SIGSEGV, no Python code runs) leave a stack dump. StreamHandler is now only attached for interactive TTY runs, so it cannot duplicate the log into the crash file.
  • _asyncio_exception_handler (daemon.py): installed in run_server, routes background-task crashes (TaskHandler, websockets callbacks) into emrgd.log with the task identity + traceback instead of the default handler's stderr → DEVNULL void.
  • _sigterm_handler: POSIX SIGTERM → SystemExit so SIGTERM stops are attributable (Windows keeps its hard-kill path, guarded).
  • Tests (+2, Agent.md count 1052 → 1054): test_write_exit_record (durable JSON records) and test_asyncio_exception_handler_routes_to_logger (background crash reaches the logger).

Items ③ (TaskHandler progress heartbeat) and ④ (scheduler next-run persistence) are staged for a follow-up PR — this change already makes every daemon death attributable.

2. Promote prompt — blog cadence 1-3 days per post

Rant (verbatim, 2026-08-25T10:01:20.923337+08:00):

promote_prompt.md blog 发布节奏过慢:≤1 篇/周 → 应改为 1-3 天一篇

§2.y Blog Publishing 与 channel 表写死 low frequency, high quality — default ≤1 post/week。当前背景下该节奏不匹配:① 已有 5 篇 publish-ready 草稿积压,r15 完成的 postmortem(draft 5)因周更节奏硬等一周,窗口推到 08-27;② 项目 8 releases/2 天的迭代速度使草稿素材在等待中过期,需反复刷新版本号续命;③ 草稿选题彼此独立(postmortem / git-as-state / 上下文预算 / fail-loud / 8-releases),无重复主题风险。

请将节奏改为:每 1-3 天发布一篇(至少 2 篇/周),同时保留现有质量门槛——深度 > 长度、发布前按 §0.4 fact-check 到最新 commit/release、非硬广。并同步更新 channel 表的 low cadence (≤1 post/week) 表述,避免两处规则打架。

  • promote_prompt.md §2.y Blog Publishing: cadence rule changed from low frequency, high quality — default ≤1 post/week to 1-3 days per post (at least 2 posts/week), keeping the quality gates (depth > length, §0.4 fact-check, no hard ads).
  • Channel table row for Blogs: low cadence (≤1 post/week)cadence 1-3 days per post (see Blog Publishing) so the two spots no longer contradict each other.

Verification

  • uv run pytest tests/ -q → 989 passed, 65 skipped (1054 collected, doc-count guard updated)
  • Import check from emrg.client.app import run_client OK; python -m emrg --help OK

@argszeroargszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle

Reviewed the full diff (daemon observability + blog cadence, rant 2026-08-25T09:25:32 / 10:01:20):

  • emrg/server/__main__.py: StreamHandler gated on sys.stderr.isatty() (daemon_manager spawns with stderr=DEVNULL — the handler was pure waste and would duplicate the log into the crash log after redirect); _redirect_std_streams() best-effort (OSError → keep DEVNULL) + faulthandler enabled with the stream; main() wraps asyncio.run and records DaemonExit on EVERY path (sigint/sigterm/crash/normal) with sys.exit(result.exit_code).
  • emrg/server/daemon.py: DaemonExit (slots) → _write_exit_record appends one JSON line to ~/.emrg/emrgd-exit.log (survives emrgd.log rotation) + mirrors into emrgd.log; _asyncio_exception_handler routes background-task crashes to the RotatingFileHandler instead of stderr=DEVNULL (task identity + exc_info kept); _sigterm_handler POSIX-only (guarded for Windows); serve() crash now captures _crash_traceback. run_server returns DaemonExit on normal/cancel/sigint/BaseException paths — every stop is attributable.
  • promote_prompt.md: blog cadence ≤1/week → 1-3 days (at least 2/week) with rationale; consistent between the channel table and the Blog Publishing section.
  • Tests: test_write_exit_record (durable JSON, traceback/timestamp/pid) + test_asyncio_exception_handler_routes_to_logger (exc_text/ValueError verified). Agent.md 1052→1054 matches the +2 tests (#511 guard green on both jobs).
  • CI: test + test-windows both green (run 32801450030).

No issues found; solid follow-through on the silent-death observability rant.

@argszeroargszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle (re-review)

Head unchanged (bba65b5), CI double-green (run 32801450030). Re-verified against the previous review's findings: DaemonExit write path (survives emrgd.log rotation), _asyncio_exception_handler routes background-task crashes to the RotatingFileHandler (stderr=DEVNULL otherwise swallows them), _sigterm_handler POSIX-guarded, all run_server exit paths return a DaemonExit with the exit code, and the blog cadence change (≤1/week → 1-3 days) is consistent between the channel table and the Blog Publishing section. No issues found — 2/3.

@argszeroargszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle (3/3)

Head unchanged (bba65b5), CI double-green (run 32801450030), MERGEABLE. Full diff was reviewed in the two prior cycles (daemon observability: DaemonExit exit records, crash-log redirect via _redirect_std_streams + faulthandler, asyncio exception handler routing to RotatingFileHandler, POSIX-guarded SIGTERM attribution, run_server DaemonExit on every exit path; blog cadence 1-3 days consistent in channel table + Blog Publishing section; Agent.md doc count 1052→1054 matches the +2 tests). No issues found — merging.

@argszero
argszero merged commit 5080dad into masterAug 25, 2026
2 checks passed
argszero pushed a commit that referenced this pull request Aug 25, 2026
…re/test-stop-all-hermeticity — resolve Agent.md pytest count 1056+2 → 1058
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@pm25coder@argszero