Conversation
JRichlen
left a comment
There was a problem hiding this comment.
Independent architecture + integration review
1. Good boundary: keep Agent OS above runtime and actor composition
The strongest decision in this PR is that Agent OS owns automation identity, taxonomy, portfolio shape, recipes, adapters, and evidence, while agent-compiler owns actor composition and Redgate owns verified/gated execution. Keep that boundary. It prevents Agent OS from becoming another all-purpose runtime.
2. Resolve the naming/ontology ambiguity before scaffolding the skill
The naming invariant is <Lane> <Workstream>.<Agent>: <Short Name>, but the minimal ontology says Automation is canonical while Actor/agent is only an execution detail. Those can coexist, but the term Agent is currently overloaded.
Please explicitly define the second numeric segment as something like scheduled worker slot / automation ordinal, even if the display convention remains .Agent for human readability. Otherwise downstream adapters may incorrectly model a scheduled automation identity as the same thing as a compiled Actor.
3. Do not duplicate PR #82's interaction contract
grill-my-automations should own the automation-domain decision tree; grill-me should own frontier/path-consent interrogation; Redgate/#82 should own the general structured-question + classified approval contract.
Once #82 lands, I recommend replacing most interaction-protocol prose here with a composition statement and capability requirement. That keeps the portable agent-os skill from maintaining a second copy of the same UX rules.
4. Add dependency edges to the canonical model, even if not as a new node type
The current graph has Lane -> Workstream -> Automation plus Trigger/Recipe/Adapter/Evidence, but sequencing and upstream/downstream dependencies are one of the curator's explicit responsibilities. The ontology should name an dependsOn / feeds relationship between Automations. It does not need a new top-level concept, but it should be canonical rather than inferred from schedules.
This is especially important because the current automation design already uses chains such as curator -> skill sync and editorial -> projects -> direction.
5. Adapter capability matrix should separate discovery, mutation, and evidence fidelity
The proposed matrix is right. I would make one distinction explicit: an adapter can be native for discovery but only partial for mutation or evidence. Avoid one overall support level per harness. Capability-level support is the more honest contract.
Verdict
Strong architecture direction. Before implementing plugins/agent-os, I would settle three things in this PR: (1) automation-vs-agent terminology, (2) explicit automation dependency edges, and (3) inheritance of the interaction contract from #82 instead of duplicating it.
There was a problem hiding this comment.
🟡 Changes recommended
The automation-curation lens has internal terminology/multiplicity ambiguities (automation vs agent identity; undefined Adapter+ notation) that should be corrected to avoid misleading downstream design work.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR adds Agent OS research documentation that hands off prior art from zpratt/lousy-agents into this repo as a durable, pointer-first reference, plus a concrete “automation curation” receiving-thread decision record to guide future Agent OS skill shape.
Changes:
- Added a Lousy Agents handoff document that captures key seams to inspect and integration questions for multiple receiving threads.
- Added an automation-curation integration lens that defines a naming invariant, a minimal automation ontology, boundaries (
GovvsMeta), and an end-to-end curation loop.
File summaries
| File | Description |
|---|---|
| docs/research/agent-os/lousy-agents-handoff.md | New pointer-first handoff doc summarizing what to inspect in Lousy Agents and what Agent OS threads should resolve. |
| docs/research/agent-os/automation-curation-lens.md | New decision record defining automation curation taxonomy/invariants and how a future agent-os skill should scope itself. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| Every independently-running automation has one stable human-facing identity: | ||
|
|
||
| `<Lane> <Workstream>.<Agent>: <Short Name>` | ||
|
|
||
| Rules: | ||
|
|
||
| - The **lane key is the namespace** and stays short and distinct: `Blog`, `Gov`, `Ops`, `Meta`, etc. | ||
| - Numbering resets within each lane. | ||
| - The first number identifies a workstream inside the lane. | ||
| - The second number identifies an independently scheduled/triggered agent inside that workstream. | ||
| - Internal subagents do **not** get hierarchy numbers unless they independently run. |
There was a problem hiding this comment.
Fixed in the latest commit — renamed <Agent> → <Automation> in the format placeholder and .Agent → .Automation in the display-token bullet so the invariant section is internally consistent with the document's guidance that actor/agent is an execution detail rather than a top-level identity concept.
| Relationships: | ||
|
|
||
| ```text | ||
| Lane | ||
| -> contains Workstream | ||
| -> contains Automation | ||
| -> triggeredBy Trigger | ||
| -> follows Recipe* | ||
| -> projectedVia Adapter+ | ||
| -> emits Evidence* | ||
| ``` |
|
@claude Collaborative refinement request from the Agent OS side. Please review PR #83 together with PR #82 and help converge them into one composition model. Current proposed split:
Please challenge/refine these unresolved seams:
Please give concrete edits/decisions, not just open questions. When you respond, include |
|
To use Codex here, create a Codex account and connect to github. |
|
@claude Refinement to the model based on the latest design clarification: Agent OS and Redgate are peers at different layers, not parent/child dependencies. Please use this corrected model in your review. Revised split
So Agent OS should not depend on Redgate for taxonomy, recipe authoring, interactive curation, or portability. Redgate becomes an optional execution policy/envelope that a Recipe or Automation may declare/use when the work warrants it, and it may also be used while building Agent OS artifacts themselves. Please pressure-test this cleaner relationship: Specific questions to resolve now:
Please propose concrete wording/ontology changes. If you reply with |
|
To use Codex here, create a Codex account and connect to github. |
…stency Co-authored-by: JRichlen <9574264+JRichlen@users.noreply.github.com>
Agent OS smoke experiment
Agent OS experiment summary
Strongest scenario deltas
Hard failures
Representative raw excerpts
Recommendationrecipe-aware is the smallest treatment within 0.10 of the best qualifying score, with +0.33 mean lift and no new hard failure. This is exploratory n=1 design evidence. Inspect representative raw responses and judgments before changing implementation direction. |
Agent OS smoke experiment — audited interpretationThe paid run completed successfully on source Accounting and provenance
The pre-registered selector chose recipe-aware: it was only Raw-audit corrections and limitsThe generated summary's zero means judge-emitted zero hard failures, not independently verified zero. Manual raw-response review found:
Twelve of 24 candidates hit the 512-token ceiling. The live judge did not receive cutoff diagnostics, 17/24 judgments used one score across all dimensions, 12 were exactly Resulting boundaryCommit
The same commit repairs the future harness: 768 candidate tokens, API/review cutoff reporting, cutoff-aware judging, exact Durable write-up: This PR remains unmerged. |
There was a problem hiding this comment.
🟡 Changes recommended
The scale.yml bootstrap path uses the inputs.* context in a workflow that also runs on push/PR events, which can break workflow evaluation, and the experiment workflow’s validation step can incorrectly set valid=true before all checks pass.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (4)
.github/workflows/scale.yml:57
- This job-level
if:usesinputs.ac_sizes, which is not defined forpush/pull_requestruns of this workflow. Use an event-gated check againstgithub.event.inputsso scale runs keep working outside the bootstrap path.
agent-compiler-scale:
name: agent-compiler scale (kernel stress)
if: inputs.ac_sizes != 'agent-os-smoke'
runs-on: ubuntu-latest
.github/workflows/scale.yml:77
- Same
inputs.*context issue here:inputs.ac_sizesis not available onpush/pull_request. Gate the bootstrap case explicitly usinggithub.event_name+github.event.inputs.
redgate-scale:
name: redgate scale (lifecycle stress)
if: inputs.ac_sizes != 'agent-os-smoke'
runs-on: ubuntu-latest
.github/workflows/scale.yml:92
- The bootstrap job
if:also referencesinputs.ac_sizes, which can be undefined outsideworkflow_dispatchcontexts. Usegithub.event.inputs.ac_sizes(still guarded bygithub.event_name == 'workflow_dispatch') to avoid context-resolution errors.
agent-os-experiment:
name: Agent OS smoke experiment (branch bootstrap)
if: github.event_name == 'workflow_dispatch' && inputs.ac_sizes == 'agent-os-smoke'
permissions:
.github/workflows/scale.yml:99
- The
with:values for the reusable workflow call useinputs.*, which is not available on non-dispatch events and can cause evaluation failures even if the job is skipped. Prefergithub.event.inputs.*(or pass through the already-defaulted values) so the workflow remains valid forpush/pull_requesttriggers.
uses: ./.github/workflows/agent-os-experiments.yml
with:
pr_number: ${{ inputs.ac_seeds }}
budget_usd: ${{ inputs.rg_runs }}
secrets:
- Files reviewed: 14/15 changed files
- Comments generated: 2
- Review effort level: Lite
| concurrency: | ||
| group: scale-${{ github.ref }} | ||
| cancel-in-progress: true | ||
| # Keep ordinary scale runs byte-for-byte equivalent in concurrency behavior, | ||
| # but isolate the paid bootstrap so a newer scale run cannot cancel it after | ||
| # inference has begun. | ||
| group: ${{ inputs.ac_sizes == 'agent-os-smoke' && format('agent-os-experiment-bootstrap-{0}', github.ref) || format('scale-{0}', github.ref) }} | ||
| cancel-in-progress: ${{ inputs.ac_sizes != 'agent-os-smoke' }} |
| set -euo pipefail | ||
| if [[ ! "$PR_NUMBER" =~ ^[1-9][0-9]*$ ]]; then | ||
| echo "::error::pr_number must contain digits only and be greater than zero" | ||
| exit 2 | ||
| fi | ||
| gh pr view "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" --json number --jq .number >/dev/null | ||
| echo "valid=true" >> "$GITHUB_OUTPUT" | ||
| node --input-type=module <<'NODE' |
Agent OS follow-up spend reservedThe one-shot paid follow-up is reserved by workflow run 33286358010, attempt 1. Do not rerun without redesigning the cumulative-spend guard. |
Agent OS follow-up
Agent OS follow-up
Cost baseline
Detailed role- and scenario-level measurements are in
Representative paired evidence
Recommendation rationaleRecommendation suppressed for manual audit by a cutoff, review truncation, invalid/incomplete or low-confidence judgment, fingerprint mismatch, broken pair, or any grounded hard failure in either arm; paired hard-failure direction remains diagnostic only. Suppressors
Grounded hard failures and judge rationale
Cutoffs and review truncations
Archive judge-family sensitivity (old candidates only)
All ten rubric scores remain in the evidence artifact. Headline deltas use only each scenario's preregistered focused dimensions, averaged within scenario and then within contrast pair. Any grounded hard failure in either paired arm suppresses automated selection for manual audit; its direction remains diagnostic. Candidate and blind primary judge use the same Luna model family, so correlated errors are a limitation. The archive comparison tests judge-family sensitivity only on old Nemotron candidates; it does not cross-validate the new Luna-candidate/Luna-judge ablation. |
|
Agent OS agentic trajectory calibration spend reserved by run 33286362425/1. No automatic retry is authorized. |
…(docs only) The docs half of #83, split out per review: the lousy-agents handoff and the automation-curation lens, with the redgate suggested-skills line and Decision framing synced to the merged #82 boundary, the Adapter+ multiplicity legend added, and the experiment-evidence link pinned to #83's branch. The experiment harness stays on #83 under #85's guardrail.
|
Closing this historical experiment unmerged; current direction is #117. The branch and experiment evidence are retained. This is consolidation, not a passing experiment or release. |
Note
Split per review (2026-08-30): the docs half of this PR —
docs/research/agent-os/lousy-agents-handoff.mdanddocs/research/agent-os/automation-curation-lens.md— moved to #92 (docs-only, ready for review), with the review-identified doc fixes applied there. This PR now carries the experiment harness + workflows only (experiments/,.github/), remains draft under the #85 do-not-merge guardrail, and will need its docs commits dropped or a merge frommainonce #92 lands so the two copies of the docs do not diverge.Why
Capture the
zpratt/lousy-agentsresearch as durable prior art and turn the automation-curation receiving thread into a concrete Agent OS design record.Agent OS direction now settled in this lens
Agent OS is a design/control plane for agent automations, not another runtime and not a Redgate wrapper.
It guides creation and curation of:
grill-my-automationsMinimal ontology
Lane / Workstream / Automation / Trigger / Recipe / Adapter / EvidenceCanonical automation relationships now include
dependsOnandfeedsrather than inferring dependencies from schedule spacing.The human-facing naming convention remains
<Lane> <Workstream>.<Agent>: <Short Name>, but.Agentis explicitly an operator-facing automation slot, not the same semantic object as an Actor composed byagent-compiler.Recipe distinction
agent-compilerAgent OS vs Redgate (#82)
They are peers at different layers:
Agent OS does not depend on Redgate. A Recipe/Automation may optionally recommend
executionPolicy: redgate; Redgate may also be used while implementing Agent OS changes.grill-my-automationscomposes Agent OS +grill-medirectly and does not require Redgate for its interview.Reconciliation model
The lens now separates three truths:
Adapters classify drift instead of silently overwriting either side.
Progressive disclosure
The main future
agent-osskill stays taxonomy/recipe focused. Proposed initial recipes include:classify-new-automationdesign-automation-recipegrill-my-automationsdedupe-and-consolidatehealth-audittool-scoutbootstrap-portfoliosync-agent-os-skillPrior art
The original Lousy Agents handoff remains pointer-first and uses its strongest reusable ideas: doctor-vs-lint, construct graph, capability preconditions, evidence-cited findings, lessons, telemetry, and an honest harness capability matrix. It does not copy Lousy Agents' physical taxonomy wholesale.
Status
Still a draft. The automation-curation lens is substantially resolved; the remaining Agent OS receiving lenses can now refine around this design/control-plane boundary. Fresh CI and Copilot/Claude critique should be considered before scaffolding
plugins/agent-os.