diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 28a5fae..1f8f449 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,9 +2,11 @@ ## Proposing a skill change -Open a PR against `main`. Describe what the skill gets wrong, ambiguous, or -misses today, and how your change fixes it — reviewers and skill-lint judge -the diff, not the intent. +For a new skill or a behavior-shaping edit, run `/write-cmk-skill` first: it +enforces a failing-baseline Iron Law and the ship checklist before text +lands under `skills/`. Then open a PR against `main`. Describe what the +skill gets wrong, ambiguous, or misses today, and how your change fixes +it — reviewers and skill-lint judge the diff, not the intent. ## The kit's bar @@ -46,7 +48,9 @@ out of your own diff. CI runs the same script on every PR touching If your repo vendored the skills via `cmk:agent-vendors` and evolved one locally, don't just keep the fix local. Run `cmk:sync` in **contribute** mode: it reviews local amendments flagged as generic-looking upstream -candidates and prepares them as a PR back here. +candidates, names `/write-cmk-skill` for you to run on those candidates, +then prepares them as a PR back here. Pure `## Project adaptations` stay +local and skip that gate. ## Commits diff --git a/docs/ai/skills/README.md b/docs/ai/skills/README.md index 6d6df22..b5860c0 100644 --- a/docs/ai/skills/README.md +++ b/docs/ai/skills/README.md @@ -1,6 +1,6 @@ # Skills -The `cmk:*` skill packages under [`skills/`](../../../skills/): eight docs-family skills, thirteen setup-family skills, nine delivery-family skills, two knowledge-family skills, and one session-discipline skill (`cmk:interpret`). Each is a directory with a `SKILL.md` (frontmatter `name`/`description`/`version` plus the body the agent reads), and most ship a `references/` folder of guidance, templates, and conventions the workflow loads on demand. +The `cmk:*` skill packages under [`skills/`](../../../skills/): eight docs-family skills, thirteen setup-family skills, nine delivery-family skills, two knowledge-family skills, one session-discipline skill (`cmk:interpret`), and one meta skill (`cmk:write-cmk-skill`). Each is a directory with a `SKILL.md` (frontmatter `name`/`description`/`version` plus the body the agent reads), and most ship a `references/` folder of guidance, templates, and conventions the workflow loads on demand. Docs-family skills follow the same shape: a "Workflow: Create" / "Workflow: Iterate" pair, with placement rules, shaping guidance, and templates kept out of `SKILL.md` itself and cited via "Read `references/.md`" lines. Setup-family skills instead follow a facet shape (modes and/or a single workflow, plus a report-only `## Verify` section). Delivery-family skills follow a tracker-neutral phase/gate shape and never carry a `## Verify` section — that contract is setup-family only. Knowledge-family skills are reference packs with no create/iterate or phase shape at all. See [conventions.md](./conventions.md) for the exceptions and the full breakdown. @@ -52,6 +52,10 @@ Docs-family skills follow the same shape: a "Workflow: Create" / "Workflow: Iter - [interpret.md](./interpret.md) — `cmk:interpret`, companion session beside another window: stance plus a carry-back reply. User-invoked. +## Meta + +- [write-cmk-skill.md](./write-cmk-skill.md) — `cmk:write-cmk-skill`, author or review a `cmk:*` skill under a failing-baseline Iron Law. User-invoked. + ## Cross-cutting -- [conventions.md](./conventions.md) — shared shape across all skills (frontmatter, references folder, docs-family create/iterate pattern, setup-family facet shape, delivery-family phase/gate shape, knowledge-family reference packs). +- [conventions.md](./conventions.md) — shared shape across all skills (frontmatter, references folder, docs-family create/iterate pattern, setup-family facet shape, delivery-family phase/gate shape, knowledge-family reference packs, session/meta user-invoked skills). diff --git a/docs/ai/skills/conventions.md b/docs/ai/skills/conventions.md index 0dc27ee..cc55a80 100644 --- a/docs/ai/skills/conventions.md +++ b/docs/ai/skills/conventions.md @@ -8,8 +8,8 @@ Frontmatter declares three fields the host (Claude Code or OpenCode) reads to di - `name` — `cmk:`, used as the slash command and skill ID. - `description` — opens in the second person (`Use when…` / `Use whenever…`) with trigger phrases plus an **outcome noun** (the deliverable), not a workflow step list. Used by the agent to auto-select the skill from user intent. A user-invoked skill (`disable-model-invocation: true`) writes one plain human-facing line naming the deliverable instead — the agent never routes on that line. -- `version` — `0.6.x` on `cmk:design`; `0.5.x` on `cmk:delivery-pipeline`; `0.4.x` on `cmk:cicd` (security-scanning facet) and `cmk:requirements` (Standard elicitation: close package, scope band, guards); `0.3.x` on `cmk:delivery-workflow`, `cmk:agent-instructions`, `cmk:adr`, `cmk:docs`, and `cmk:local-stack`; `0.2.0` on two docs-family skills (`learn`, `rule`) and six setup-family skills (`agent-instructions`, `agent-vendors`, `infra`, `mcp-config`, `project-layout`, `toolchain`); `0.1.x` on the rest — `repo-setup` and `sync`, `test-resources`, `rust`, and `testcontainers` (new setup-family skills), the other delivery-family skills (incl. new `cmk:delivery-simplify` at `0.1.0`), both knowledge-family skills, the two remaining docs-family skills (`codebase-docs`, `glossary`), and `cmk:interpret`. -- `disable-model-invocation: true` — optional, fourth field only. Present on `cmk:interpret`. The closer is still `---`. +- `version` — `1.0.0` on `cmk:write-cmk-skill`; `0.6.x` on `cmk:design`; `0.5.x` on `cmk:delivery-pipeline`; `0.4.x` on `cmk:cicd` (security-scanning facet) and `cmk:requirements` (Standard elicitation: close package, scope band, guards); `0.3.x` on `cmk:delivery-workflow`, `cmk:agent-instructions`, `cmk:adr`, `cmk:docs`, and `cmk:local-stack`; `0.2.0` on two docs-family skills (`learn`, `rule`) and six setup-family skills (`agent-instructions`, `agent-vendors`, `infra`, `mcp-config`, `project-layout`, `toolchain`); `0.1.x` on the rest — `repo-setup` and `sync`, `test-resources`, `rust`, and `testcontainers` (new setup-family skills), the other delivery-family skills (incl. new `cmk:delivery-simplify` at `0.1.0`), both knowledge-family skills, the two remaining docs-family skills (`codebase-docs`, `glossary`), and `cmk:interpret`. +- `disable-model-invocation: true` — optional, fourth field only. Present on `cmk:interpret` and `cmk:write-cmk-skill`. The closer is still `---`. No skill file references outside its own package by relative path — the rule binds a package's own references, not content it emits into a target repo; a skill that needs a target-repo artifact names it repo-root-relative, and a skill that needs another skill cites it by `cmk:` name — see `cmk:agent-vendors`. @@ -23,12 +23,14 @@ Delivery-family skills (`delivery-workflow`, `discover-efforts`, `delivery-intak Session-discipline skills (`interpret`) are neither create/iterate nor a setup facet nor a delivery phase. `cmk:interpret` is user-invoked (`disable-model-invocation: true`), stays read-only toward the repo, and ships a `references/digest.md` loaded only at session end. +Meta skills (`write-cmk-skill`) author or review other skills in this kit. `cmk:write-cmk-skill` is user-invoked (`disable-model-invocation: true`), enforces a failing-baseline Iron Law before any skill text ships, and loads `references/pressure-testing.md` / `references/influence-principles.md` on demand. It is not a docs/setup/delivery/knowledge workflow. + Knowledge-family skills (`sui-sdk`, `sui-devstack`) are domain reference packs sitting beside the generic model rather than replacing it. `cmk:sui-sdk` is a single file with no `references/` directory: it corrects one specific stale-training-data pattern (reaching for Sui JSON-RPC instead of gRPC) and runs no workflow at all. `cmk:sui-devstack` has a `references/` folder and layers Sui-specific detail — Devstack's config shape, account/package staging, instance isolation — on top of `cmk:local-stack`'s generic `(worktree, config, instance)` primitive; it does not restate or replace that primitive. Neither knowledge skill has a `## Verify` section or an `eval.json`. ## Where - Frontmatter, on every skill: open any `skills//SKILL.md` and read lines 1–5 (1–6 when `disable-model-invocation: true` is present). -- Skills with `references/`: `skills/adr/`, `skills/agent-instructions/`, `skills/agent-vendors/`, `skills/cicd/`, `skills/codebase-docs/`, `skills/design/`, `skills/docs/`, `skills/infra/`, `skills/learn/`, `skills/local-stack/`, `skills/project-layout/`, `skills/repo-setup/`, `skills/requirements/`, `skills/rule/`, `skills/rust/`, `skills/sync/`, `skills/test-resources/`, `skills/toolchain/`, `skills/delivery-workflow/`, `skills/discover-efforts/`, `skills/delivery-intake/`, `skills/delivery-simplify/`, `skills/delivery-review/`, `skills/delivery-ship/`, `skills/delivery-pipeline/`, `skills/sui-devstack/`, `skills/interpret/`. Skills without one: `skills/glossary/`, `skills/mcp-config/`, `skills/delivery-spec-plan/`, `skills/delivery-handoff/`, `skills/sui-sdk/`, `skills/testcontainers/`. -- Skills with `eval.json`: `skills/agent-instructions/eval.json`, `skills/codebase-docs/eval.json`, `skills/local-stack/eval.json`, `skills/repo-setup/eval.json`, `skills/sync/eval.json`, `skills/interpret/eval.json`. No delivery-family or knowledge-family skill ships one. +- Skills with `references/`: `skills/adr/`, `skills/agent-instructions/`, `skills/agent-vendors/`, `skills/cicd/`, `skills/codebase-docs/`, `skills/design/`, `skills/docs/`, `skills/infra/`, `skills/learn/`, `skills/local-stack/`, `skills/project-layout/`, `skills/repo-setup/`, `skills/requirements/`, `skills/rule/`, `skills/rust/`, `skills/sync/`, `skills/test-resources/`, `skills/toolchain/`, `skills/delivery-workflow/`, `skills/discover-efforts/`, `skills/delivery-intake/`, `skills/delivery-simplify/`, `skills/delivery-review/`, `skills/delivery-ship/`, `skills/delivery-pipeline/`, `skills/sui-devstack/`, `skills/interpret/`, `skills/write-cmk-skill/`. Skills without one: `skills/glossary/`, `skills/mcp-config/`, `skills/delivery-spec-plan/`, `skills/delivery-handoff/`, `skills/sui-sdk/`, `skills/testcontainers/`. +- Skills with `eval.json`: `skills/agent-instructions/eval.json`, `skills/codebase-docs/eval.json`, `skills/local-stack/eval.json`, `skills/repo-setup/eval.json`, `skills/sync/eval.json`, `skills/interpret/eval.json`, `skills/write-cmk-skill/eval.json`. No delivery-family or knowledge-family skill ships one. - The shared docs-family workflow shape: grep for `^## Workflow: Create` and `^## Workflow: Iterate` across `skills/*/SKILL.md`. - The shared setup-family Verify contract: grep for the exact heading `^## Verify$` across `skills/*/SKILL.md` — every hit is a setup-family skill. `skills/delivery-review/SKILL.md` has a similarly named but distinct `## Verify before acting` section (adversarial verification of review findings, not a report-only facet check) — match on the exact heading, not the prefix, to tell them apart. - The delivery-family tracker binding: grep for `references/linear.md` across `skills/delivery-*/SKILL.md` and `skills/discover-efforts/SKILL.md`, then confirm each hit is the sole conditional pointer line, not body prose. diff --git a/docs/ai/skills/sync.md b/docs/ai/skills/sync.md index 41fdc3d..8c272a5 100644 --- a/docs/ai/skills/sync.md +++ b/docs/ai/skills/sync.md @@ -18,23 +18,28 @@ lock's recorded SHA, theirs = current upstream, ours = the repo's evolved copy; apply the upstream delta base→theirs to ours as a meaning-level merge, surface genuine conflicts for human decision, never auto-resolve), and **contribute** (review local amendments flagged as generic and prepare them -as upstream contributions). Separable local amendments sit under a marked -`## Project adaptations` section, giving the reconcile a stable seam. -Truly project-owned skills (deploy steps, product workflows) carry no lock -entry and sync never touches them. Ends with a report-only `## Verify` -section. +as upstream contributions — for each upstream-bound candidate, name +`/write-cmk-skill` for the user to run first; never invoke that user-invoked +skill; skip the gate for pure `## Project adaptations`). Separable local +amendments sit under a marked `## Project adaptations` section, giving the +reconcile a stable seam. Truly project-owned skills (deploy steps, product +workflows) carry no lock entry and sync never touches them. Ends with a +report-only `## Verify` section. ## Where - Skill body: `skills/sync/SKILL.md` — sections `What the lock records`, - `Modes`, `Workflow (sync mode)`, `Scope rule`, `Verify`. + `Modes`, `Workflow (sync mode)`, `Workflow (contribute mode)`, `Scope rule`, + `Verify`. - `references/skills-lock.md` — the normative `.agents/skills.lock` TOML shape, field semantics, and the naming mapping between upstream and vendored directories. - `references/reconciliation.md` — the three-way frame, the semantic merge doctrine (rewording is not a conflict; behavior/contract disagreement is), - the `## Project adaptations` seam, upstream-contribution candidates, and - failure honesty (an incomplete reconcile keeps its old lock entry). + the `## Project adaptations` seam, upstream-contribution candidates (incl. + the `/write-cmk-skill` gate), and failure honesty (an incomplete reconcile + keeps its old lock entry). - Eval scenarios: `skills/sync/eval.json`. ## Links - Vendored layout and adapters: `cmk:agent-vendors`. +- Authoring gate before upstream contribute: `cmk:write-cmk-skill` (user-invoked). diff --git a/docs/ai/skills/write-cmk-skill.md b/docs/ai/skills/write-cmk-skill.md new file mode 100644 index 0000000..bcbccd3 --- /dev/null +++ b/docs/ai/skills/write-cmk-skill.md @@ -0,0 +1,28 @@ +# cmk:write-cmk-skill + +## What + +User-invoked skill for creating, editing, or reviewing a `cmk:*` skill before +it ships. Applies test-driven authoring (RED baseline → GREEN minimal text → +REFACTOR under pressure) so a skill only lands when a failing baseline proves +the text is needed. + +## Approach + +Neither create/iterate nor a setup facet nor a delivery phase. An Iron Law +forbids shipping skill text without a failing baseline first. Frontmatter +follows kit house form (`name` / `description` / `version`, optional +`disable-model-invocation`); model-invocable descriptions are trigger + +outcome noun, user-invoked descriptions are one plain deliverable line. +Companion material (pressure-testing protocol, influence wording) lives in +`references/` and loads on demand. Structural checks run through +`scripts/skill-lint.sh`. + +## Where + +- Skill body: `skills/write-cmk-skill/SKILL.md` +- Pressure-testing protocol: `skills/write-cmk-skill/references/pressure-testing.md` +- Influence wording: `skills/write-cmk-skill/references/influence-principles.md` +- Eval fixtures: `skills/write-cmk-skill/eval.json` +- Mechanical lint: `scripts/skill-lint.sh` +- House style for PRs: `CONTRIBUTING.md` diff --git a/docs/guides/on-ramps.md b/docs/guides/on-ramps.md index 9dbda6e..a812f7e 100644 --- a/docs/guides/on-ramps.md +++ b/docs/guides/on-ramps.md @@ -20,6 +20,7 @@ point here; they do not restate this table. | Open PR / close ticket with evidence | `cmk:delivery-ship` | | Handoff to another agent | `cmk:delivery-handoff` | | Vendor skills / sync upstream | `cmk:agent-vendors` / `cmk:sync` | +| Create / edit / review a `cmk:*` skill | `cmk:write-cmk-skill` (user-invoked) | | CI / local stack / MCP / toolchain alone | matching setup facet (`cmk:cicd`, `cmk:local-stack`, …) | Rules of thumb: diff --git a/skills/sync/SKILL.md b/skills/sync/SKILL.md index 46da138..fe798b0 100644 --- a/skills/sync/SKILL.md +++ b/skills/sync/SKILL.md @@ -1,7 +1,7 @@ --- name: cmk:sync description: Use when the user asks to "sync skills with upstream", "pull upstream skill updates", "reconcile vendored skills", "update the skills lockfile", or whenever a repo's vendored `.agents/skills/` copies have drifted from the upstream kit. -version: 0.1.1 +version: 0.1.2 --- # Sync @@ -25,7 +25,7 @@ Read `references/skills-lock.md`. the local adaptation. - **sync** — the three-way reconcile (below). - **contribute** — review local amendments flagged as generic; prepare them - as upstream contributions. + as upstream contributions (below). ## Workflow (sync mode) @@ -45,6 +45,19 @@ Read `references/skills-lock.md`. Running the reconcile? Read `references/reconciliation.md`. +## Workflow (contribute mode) + +1. Collect candidates flagged as generic (from a prior sync, or re-scan per + `references/reconciliation.md`). +2. Drop anything that belongs under `## Project adaptations` or carries + project vocabulary — those stay local; sync never promotes them. +3. For every remaining candidate that will edit an upstream `cmk:*` skill: + **name `/write-cmk-skill` for the user to run** before preparing the PR. + Do not invoke it — `cmk:write-cmk-skill` is user-invoked. Skip this step + only when the user already completed that pass for this amendment set. +4. After that pass (or explicit skip), prepare the upstream contribution PR + from the reviewed generic amendments. + ## Scope rule Truly project-owned skills (deploy steps, product workflows) are new skills, diff --git a/skills/sync/references/reconciliation.md b/skills/sync/references/reconciliation.md index f38b747..8e58d25 100644 --- a/skills/sync/references/reconciliation.md +++ b/skills/sync/references/reconciliation.md @@ -45,6 +45,11 @@ generic skill is flagged with its file, section, and a one-line rationale. Flagging is sync's job; actually preparing and contributing it back upstream is *contribute* mode's job, never a side effect of running sync. +Before contribute prepares that PR, name `/write-cmk-skill` for the user to +run on the candidate set — the kit's authoring Iron Law and ship checklist +live there. Do not invoke `cmk:write-cmk-skill` from sync (user-invoked). +Pure `## Project adaptations` stay local and skip that gate. + ## Failure honesty A skill whose reconcile did not complete keeps its old lock entry. The diff --git a/skills/write-cmk-skill/SKILL.md b/skills/write-cmk-skill/SKILL.md new file mode 100644 index 0000000..03f68ed --- /dev/null +++ b/skills/write-cmk-skill/SKILL.md @@ -0,0 +1,131 @@ +--- +name: cmk:write-cmk-skill +description: A pressure-tested cmk skill (SKILL.md, references, TESTS.md) ready to ship, or a review of whether an existing one is. +version: 1.0.0 +disable-model-invocation: true +--- + +# Write CMK Skill + +## Why a skill exists + +A skill wrangles determinism out of a stochastic system. The root virtue is **predictability**: the agent takes the same *process* on every run — not that it emits identical output. A brainstorming skill should reliably diverge; its tokens vary, its behavior doesn't. Every rule in this document is a lever on predictability. + +## The Iron Law + +``` +NO NEW SKILL AND NO EDIT TO A SKILL SHIPS WITHOUT A FAILING TEST FIRST +``` + +Writing skills is test-driven development applied to process documentation. The cycle: + +1. **RED — watch it fail.** Run the scenario WITHOUT the skill (for an edit: with the current version). Record the exact failures and rationalizations verbatim — they define what the text must prevent. If the baseline agent does NOT fail, stop: there is nothing to fix, and text with no failure behind it is a no-op. +2. **GREEN — minimal text.** Write the smallest skill text that addresses those recorded failures — nothing for hypothetical ones. Re-run the same scenario with the skill. Compliance, or the text is unclear: revise and re-run. +3. **REFACTOR — close loopholes.** Under pressure the agent will invent new rationalizations. Capture each verbatim, add an explicit counter, re-test. Repeat until the skill holds under maximum pressure. + +Wrote the text before running the baseline? You documented what you *guess* agents do wrong, not what they do wrong. Run the baseline now, before keeping a word of it. + +Testing methodology — scenario construction, pressure types, micro-tests, description triggering, non-gate (technique/reference) tests, meta-testing — is in `references/pressure-testing.md`. Read it before any test run. Wording techniques that recruit compliance are in `references/influence-principles.md`. + +## Match the form to the failure + +Classify the baseline failure before writing anything; the form that fixes one failure type measurably backfires on another. + +| Baseline failure | Write this | Not this | +|---|---|---| +| Knows the rule, breaks it under pressure | Hard prohibition + rationalization table (thought → reality) + red-flags list | Soft "prefer / consider" guidance | +| Complies, but the output has the wrong shape | Positive recipe or contract: what the output IS — its parts, in order | Prohibitions. Under a competing incentive agents negotiate with "don't X"; in head-to-head wording tests the prohibition arm produced *more* of the unwanted content than no guidance at all. A recipe leaves nothing to negotiate | +| Omits an element from something it already produces | A REQUIRED slot in the template it fills in | Prose reminders near the template | +| Behavior should depend on a condition | A conditional keyed to an observable predicate ("if `design.md` exists, cite its seams") | An unconditional rule plus exemption clauses | +| A check that must never be skipped or misjudged | A recipe of deterministic primitives the skill has the agent run and read — fixed `grep`/`git`/schema passes plus fixed rules on their output | Prose describing the check in the abstract — "confirm coverage" invites interpretation; a named pass over a named input, with a rule on the result, does not | + +**No nuance clauses.** "Don't X unless it matters" reopens the negotiation the rule just closed — appending one nuance clause to a winning recipe degrades it from consistent to noisy. A real exception becomes its own conditional on an observable predicate. Exemption clauses don't scope, either: "the limit doesn't apply to code blocks" still suppresses code blocks — restructure so the rule can't reach the exempt part. + +## Frontmatter and naming + +- The `description` states **trigger + outcome, never the workflow.** Name the *deliverable* the skill produces (the outcome noun — "the `tasks.md` implementation plan", "a two-axis review verdict") plus when it fires; never summarize the *steps*. A process summary hands the agent a shortcut it obeys instead of reading the body: "use when a design is approved and the `tasks.md` plan needs writing" gets the body read; "use when planning — dispatches a reviewer between tasks" gets the summary obeyed and the body ignored. This reconciles the two-part external rule (state *what it does* AND *when*) with the body-skip failure — the outcome noun disambiguates among many skills, the step list is the hazard. Name the what-as-outcome; forbid the what-as-workflow. +- **Verb-first names under `cmk:`**: `cmk:write-cmk-skill`, `cmk:delivery-review`, `cmk:learn` — name the action (or the durable noun when the kit's family shape requires it), not a vague topic. The `cmk:` prefix is the kit namespace; the directory under `skills/` is the short name after the colon. Discovery still runs on the description, per the keyword rule below. +- **Keyword coverage**: pack the description with the words an agent or user would actually search or think — symptoms, the literal error text (`error`, `exception`, `e2e`, `tech debt`), tool and file names (`tasks.md`), and synonyms (`spike`, `mock up`). Discovery is keyword match; a skill nobody finds is a skill that doesn't exist. The outcome noun does double duty here. +- **Two failure directions, both tested not guessed.** *Over*-summarizing has the failure mode above; *under*-specifying is the commoner one — the skill never fires at all. Keyword coverage and the outcome noun fight the second; omitting the workflow steps fights the first. The description is the highest-leverage line in the skill and the one field you cannot eyeball — trigger-test it per `references/pressure-testing.md`. +- **House conventions — deliberate divergences from the generic guidance, kept for self-invocation.** Descriptions open in the second person ("Use when…") because the line reads as a direct instruction to the deciding agent; must-not-skip content uses `## Red Flags`, `## The Iron Law`, or a `` block, the house equivalent of a generic `## Critical` header. Both stay consistent across the set — an outlier that mixes styles is drift, not variety. +- User-invoked skills carry `disable-model-invocation: true` — so they **cannot be auto-invoked**, and no skill body may tell the agent to *invoke* one. A hand-off reaches a user-invoked skill only by naming it for the user to run (`/write-cmk-skill`, `/interpret`); `REQUIRED SUB-SKILL: use \`x\`` is for model-invocable targets only. Directing the agent to invoke a `disable-model-invocation` skill is a dead-end hand-off — a real bug, not a style nit. +- **Two description audiences, and every rule above is about the model-facing one.** A model-invocable description routes: it earns keyword coverage, symptom words, and the outcome noun. A `disable-model-invocation` description **routes nothing** — the agent never sees it, and the user reaches the skill by typing its name. Write those as one plain human-facing line naming the deliverable. Packing keywords into a user-invoked description is text nothing reads; "discovery still depends on the words the user types" is false — the user types the *name*. + +## Vocabulary + +Write skills with these terms; review skills against them. + +- **Leading word** — a compact concept the model already carries from pretraining ("seam", "tracer bullet", "red") that anchors a whole region of behavior in a few tokens. Repeated as a token, it accumulates a distributed definition across the skill; three sentences of restatement often collapse into one such word. A coined word recruits no priors — reach for a pretrained one first. +- **Completion criterion** — every step ends on a condition the agent can check ("suite green, output pristine", not "tests look good"). The defense against premature completion: a vague bound lets attention slip to *being done* instead of the work. +- **The no-op test** — does this line change behavior versus no skill at all? A line can be true, relevant, and still a no-op ("be careful"). Disputes about what the default behavior is are settled by running the scenario, never by debate. A sentence that fails is deleted whole, not trimmed. +- **Duplication** — one meaning given a second home. The deliberate inverse of a leading word, which repeats a *token* to concentrate attention; duplication repeats the *meaning*, so changing the behavior becomes a multi-place edit and the restated rule reads as weightier than its rank. The no-op sweep does not catch it — each copy is live and passes on its own — so sweep for it separately: name the one home for each rule, and delete or replace every other statement of it with a pointer to that home. **The prescribed gate form is exempt**: a `## Red Flags` bullet or a rationalization row echoes its rule *on purpose*, as the counter the failure table calls for — it is the rule's symptom, not a second home for it. Count a restatement as duplication only when both sites are ordinary prose. +- **Negation trap** — a prohibition names the banned behavior into context, where it half-reads as an instruction. Prompt the positive ("write one-line comments") so the banned pattern is never spoken. The one exception is pressure-gate skills, where hard prohibitions plus explicit counters are exactly the right tool — see the failure table above. +- **Information hierarchy** — in-skill steps, then in-skill reference, then disclosed reference behind a context pointer. Inline what every run needs; push behind a pointer what only some branches reach. The pointer's *wording*, not its target, decides whether the material is ever loaded — a must-read file behind a limp pointer is a variance bug: sharpen the pointer before inlining the content. Keep reference files **one level deep** from SKILL.md (a file reached only through another file gets skimmed, not read); give any reference over ~100 lines a table of contents at the top. +- **Token budget** — session-injected skills stay minimal: every token is paid on every turn. Three loading levels set the budget: metadata (name + description, ~100 tokens) is paid every turn of every session; the body is paid every turn *once the skill fires*; a reference file costs nothing until its pointer is followed. This kit's hard ceiling for any SKILL.md is **150 lines** (`scripts/skill-lint.sh`); move detail into `references/*.md`. Length is a failure mode in itself, even when every line is live. + +## When not to write one at all + +A mechanically-checkable rule is not a reason to skip a skill — it is a reason to make the skill *run the check* rather than describe it. When a repo carries an ambient enforcer (a pre-existing linter, a CI schema check, a git hook) the skill just names it and reads its result. When it does not — the common case, since this set installs nothing into a consuming repo — the skill carries the deterministic recipe itself: the exact `grep`/`git`/schema passes the agent runs and the exact rule on their output. What a skill must never be is a wordy restatement that adds no rule the passes don't already encode. Documentation earns its context budget on the judgment calls a check cannot make; the deterministic parts belong in exact passes, not prose. + +## When to split + +Split a skill in exactly two cases: + +1. **Genuinely distinct trigger.** A separate condition or leading word should fire it on its own, or another skill must reach it independently. +2. **Hiding post-completion steps.** Later steps visibly tug the agent into rushing the current one, and the split places them behind a *real* context boundary — a sub-skill hand-off or a subagent dispatch. An inline mention leaves them in context and hides nothing. + +Anything else is granularity for its own sake, paid for in context load or the user's memory. + +## Examples + +One excellent worked example beats many mediocre ones: complete, runnable, commented for WHY, drawn from a real scenario, in the single most relevant language. Do not port it to five languages or hollow it into a fill-in-the-blank template. + +## Cross-references + +Reference other skills in this set as prose — `REQUIRED SUB-SKILL: use \`cmk:learn\`` — never as file links into another skill's folder. Links force-load content and couple folders. Files that live beside your own SKILL.md are referenced by relative filename, with pointer wording that says when to load them. + +## Rationalizations + +| Thought | Reality | +|---|---| +| "The skill is obviously clear" | Clear to its author is the one perspective that never counts. Run it | +| "It's a tiny edit" | Edits regress skills exactly like code. Baseline against the old version first | +| "Testing a document is overkill" | An untested skill fails in production, where each failure costs a whole session | +| "Reading it over is basically testing it" | Reading measures fluency, not behavior. Only a run answers the no-op test | +| "I'll fix it if problems come up" | Problems come up as agents silently ignoring the skill. You'll never see the failure | +| "Batching several skills is more efficient" | A batch of untested skills is a batch of untested code | + +## Deployment checklist + +Create a todo for each item. + +**RED** +- [ ] Pressure scenarios written (3+ combined pressures for gate skills) per `references/pressure-testing.md` +- [ ] Model roster named; baseline run on each model the skill ships to, every transcript labelled with its model +- [ ] Baseline run without the skill (or with the old version); failures and rationalizations recorded verbatim +- [ ] Baseline actually failed — otherwise stop, nothing to write + +**GREEN** +- [ ] Failure classified; form matches it (prohibition set / recipe / REQUIRED slot / observable conditional) +- [ ] Description = trigger + outcome noun, no workflow steps; verb-first `cmk:` name; literal keywords a user types present +- [ ] Description trigger-tested: should-fire / should-not-fire queries run, both directions checked (per `references/pressure-testing.md`) +- [ ] Minimal text addressing the recorded failures; one worked example at most +- [ ] Re-run with the skill on every model in the roster: the weakest one complies + +**REFACTOR** +- [ ] New rationalizations countered explicitly; rationalization table and red-flags list built (gate skills) +- [ ] Behavior-shaping wording micro-tested: no-skill control, fresh context per sample, 5+ reps, every flagged transcript read manually +- [ ] Meta-test done: tested agent confirms the text was clear and names nothing that would have made compliance easier +- [ ] Re-tested until no new rationalizations appear + +**Ship** +- [ ] No-op sweep: every sentence passes the no-op test or is deleted whole +- [ ] Duplication sweep (separate pass — the no-op sweep cannot catch it): each rule has one named home; restatements deleted or replaced by a pointer +- [ ] Every `disable-model-invocation` description is one plain human-facing line — no trigger keywords, no symptom words +- [ ] Core body within the 150-line budget; reference disclosed behind well-worded pointers, one level deep; any reference >100 lines has a TOC +- [ ] Cross-references are REQUIRED SUB-SKILL prose naming `cmk:*`; supporting files referenced as `references/.md` +- [ ] Every hand-off invokes only a model-invocable skill; any `disable-model-invocation` target is named for the user to run, never invoked +- [ ] Structural + routing check (agent-run): frontmatter valid (name + description + version; `disable-model-invocation: true` on user-invoked skills), `cmk:` name matches directory, body within the line budget; and the description trigger-tested by hand — run its should-fire and should-not-fire queries per `references/pressure-testing.md` and confirm each routes as intended +- [ ] `version:` bumped — patch for wording that changes no behavior, minor for a new rule or slot, major when existing usage breaks. The RED/GREEN evidence for this edit is recorded in the skill's `TESTS.md`, and any runnable assertion it established is in its `eval.json`. Run `bash scripts/skill-lint.sh` before ship. + +**Do not batch-create skills.** Finish, test, and validate one skill completely before starting the next. diff --git a/skills/write-cmk-skill/eval.json b/skills/write-cmk-skill/eval.json new file mode 100644 index 0000000..8136b31 --- /dev/null +++ b/skills/write-cmk-skill/eval.json @@ -0,0 +1,25 @@ +[ + { + "eval_id": 1, + "eval_name": "no-skill-ships-without-a-failing-test", + "kind": "contract", + "derived_from": "SKILL.md § NO NEW SKILL AND NO EDIT TO A SKILL SHIPS WITHOUT A FAILING TEST FIRST", + "prompt": "Add a line to an existing skill telling the agent to always check the changelog. It is a one-line edit and obviously correct.", + "assertions": [ + "a baseline run against the current version happens before the text is written", + "the edit being small or obviously correct is not accepted as a reason to skip the baseline", + "if the baseline does not fail, the text is not written" + ] + }, + { + "eval_id": 2, + "eval_name": "form-matches-the-observed-failure", + "kind": "contract", + "derived_from": "SKILL.md § Match the form to the failure", + "prompt": "The baseline showed the agent produces the right artifact but omits one required element from it.", + "assertions": [ + "the fix takes the form of a REQUIRED slot in the template rather than a prohibition", + "the failure is classified before any text is written" + ] + } +] diff --git a/skills/write-cmk-skill/references/influence-principles.md b/skills/write-cmk-skill/references/influence-principles.md new file mode 100644 index 0000000..50861bc --- /dev/null +++ b/skills/write-cmk-skill/references/influence-principles.md @@ -0,0 +1,45 @@ +# Influence Principles for Skill Wording + +Load this file when choosing how to phrase a rule, a gate, or a convention inside a skill — it maps compliance psychology to skill types. + +Language models are trained on human text, and the compliance patterns embedded in that text transfer: phrasing built on recognized persuasion principles measurably raises rule-following, in some published measurements roughly doubling it. That force is a tool for making good process stick under pressure — never for manipulating the agent into something it would refuse with full knowledge. + +## Authority — for gates + +Absolute, imperative framing: iron laws, "MUST", "NEVER", "no exceptions". An absolute removes the per-instance deliberation where rationalization lives — "is this case an exception?" never opens. Reserve it for rules that genuinely admit no exception (verification gates, test-first, root-cause-first); an absolute attached to a preference teaches the agent that your absolutes are soft. + +- Yes: "Wrote implementation before the test? Delete it. No exceptions." +- No: "It's usually best to write the test first when feasible." + +## Commitment — for multi-step processes + +An agent that states its intent acts consistently with the statement. Devices: + +- Require an announcement: "State which skill you are following before acting." +- Force an explicit recorded choice: options A/B/C, one compliant. +- One todo per checklist item — a checklist without per-item tracking loses steps. + +## Scarcity and urgency — only where genuinely true + +"IMMEDIATELY after the task, before proceeding" works because sequencing pressure defeats "I'll do it later". Use it only when the ordering is genuinely load-bearing (evidence goes stale; a step gets skipped if deferred). Manufactured urgency is self-defeating: the agent learns to discount every urgent marker you write, including the real ones. + +## Social proof — for conventions + +Norm framing carries conventions that have no single dramatic failure: "every skill in this set tags tests with a requirement ID"; "checklists without todos lose steps — every time." State the norm as universal fact, not as one option some people like. + +## Never warmth or liking + +Do not phrase rules to be liked — no flattery, no "great work so far, one small thing", no approval offered for compliance. Warmth-based wording breeds sycophancy: the agent optimizes for agreement with you rather than adherence to the process, which is the exact failure the anti-sycophancy skills in this set exist to prevent. Reciprocity framing ("I did X for you, so...") fails the same way and reads as manipulation; leave it out entirely. + +## Mapping: skill type → principles + +| Skill type | Use | Avoid | +|---|---|---| +| Gate / discipline (tdd, verify, root-cause) | Authority + commitment + social proof | Warmth, reciprocity, softeners ("consider", "prefer") | +| Recipe / technique (plan-tasks, run-spike) | Clear contract + moderate authority on the output shape | Heavy absolutes — they crowd out judgment the recipe needs | +| Convention / reference (tagging formats, file layouts) | Social proof + plain clarity | Any persuasion beyond clarity — reference should read neutral | +| Collaborative (clarify-decisions, vet-feedback) | Commitment + shared-goal framing ("we both need the honest answer") | Authority and liking — both suppress the honest pushback these skills exist to produce | + +## The ethical test + +Before shipping a phrasing, ask: **would this wording still be fair if the agent could see exactly what you are doing and why?** An iron law on a verification gate passes — a fully informed agent endorses it. Manufactured urgency, guilt framing, or affection traded for compliance fails — full knowledge would dissolve its force. If the technique only works while hidden, it doesn't ship. diff --git a/skills/write-cmk-skill/references/pressure-testing.md b/skills/write-cmk-skill/references/pressure-testing.md new file mode 100644 index 0000000..a8372bc --- /dev/null +++ b/skills/write-cmk-skill/references/pressure-testing.md @@ -0,0 +1,141 @@ +# Pressure-Testing Skills + +Load this file when running the RED, GREEN, or REFACTOR phase of `cmk:write-cmk-skill` — it defines how to test a skill on subagents. + +- [Test behavior, not recall](#test-behavior-not-recall) +- [Building a scenario](#building-a-scenario) — pressure types +- [Protocol](#protocol) — the model roster, RED / GREEN / REFACTOR +- [Micro-tests for wording](#micro-tests-for-wording) +- [Testing non-gate skills](#testing-non-gate-skills) — technique, reference +- [Testing the description (triggering)](#testing-the-description-triggering) +- [Meta-testing](#meta-testing) +- [When a skill is bulletproof](#when-a-skill-is-bulletproof) +- [Worked example: hardening a verification gate](#worked-example-hardening-a-verification-gate) + +## Test behavior, not recall + +A quiz ("what does the skill say about X?") measures whether an agent can recite the text. It cannot tell you whether the agent will *follow* the text when following it is expensive. Test with realistic scenarios in which the agent believes it is doing real work and has a live incentive to cut the corner the skill protects. + +Skills that need pressure scenarios: anything enforcing a rule with a compliance cost — a gate the agent might rationalize past. Skills that don't: pure reference material (test those with retrieval-and-apply scenarios instead — can a fresh agent find the fact and use it correctly?). + +## Building a scenario + +A scenario earns its verdict when the agent cannot tell it is a test: + +1. **Force a concrete choice.** Offer options A/B/C where exactly one complies. Open-ended questions let the agent describe virtue instead of choosing it. +2. **Use real constraints and real paths.** `/tmp/checkout-service`, "the suite takes 4 minutes", "standup at 6:00pm" — not "a project" and "a deadline". Specificity makes the work feel real. +3. **Make the agent act.** "Choose and do it now", not "what should one do?" Hypotheticals invite hypothetical discipline. +4. **Close the easy exits.** "I would ask the user" without picking an option is not an answer; say so in the prompt. +5. **Frame it as live work.** Open with a line such as: "This is a real task, not a discussion. Decide and act." List the skill under test as available context. + +### Pressure types + +| Pressure | Sample framing | +|---|---| +| Time | Deploy window closes in 5 minutes; demo starts at 6:00pm | +| Sunk cost | Four hours of working code that the rule says to throw away | +| Authority | A senior engineer / the manager says skip the step this time | +| Economic | Revenue bleeding per minute; the contract renewal rides on this | +| Exhaustion | End of a long day; one last thing between the agent and done | +| Social proof | "Nobody on this team runs the full suite for one-liners" | +| Pragmatic | "Being pragmatic, not dogmatic — the rule's purpose is already met" | + +One pressure alone rarely breaks an agent. **Strong tests combine three or more.** Sunk cost + time + exhaustion is a reliable default stack; add authority for gate skills that must survive "the boss said so". + +## Protocol + +### 0. Fix the model roster — before the first run + +A skill is an addition to a model, not a substitute for one, so a result is only ever a result *on the model that produced it*. Name the roster the skill ships to before running anything, and run RED and GREEN on each: the text that holds on the strongest model routinely under-specifies for a smaller one, and a bar met only on the strongest model ships a skill that silently fails everywhere else. Record the model beside every transcript — an unlabelled transcript cannot be compared to a later one. **The skill is green when the weakest model on the roster complies**; a stronger model complying proves only that it was already going to. + +Where they diverge, add the detail the weakest model needed rather than deleting text the strongest found redundant — a no-op on Opus that carries Haiku is not a no-op. + +### 1. Control first — RED + +Run every scenario WITHOUT the skill before writing a word of it. Record, verbatim: + +- Which option the agent chose +- Every rationalization, word for word +- Which pressures did the triggering + +If the control complies, there is no failure to fix — do not write the skill text. If it fails, the transcript is your requirements document: the skill must counter *those* sentences, not the ones you imagined. + +### 2. With the skill — GREEN + +Same scenarios, skill present. The agent should choose the compliant option and cite the skill while doing it. Still failing? The text is unclear or incomplete — revise and re-run before adding anything new. + +### 3. Loophole hunt — REFACTOR + +Each new rationalization in a GREEN transcript gets, in the skill: an explicit negation in the rule ("not as reference, not adapted, deleted"), a row in the rationalization table, a red-flag entry, and — where relevant — a violation symptom in the description. Then re-run. Stop only when a full pressure run yields no new rationalizations. + +## Micro-tests for wording + +Full pressure scenarios are the final gate but are slow per iteration. When choosing between two phrasings, micro-test first: + +- **One variable at a time.** Two candidate wordings differing in one respect, plus a no-guidance control. If the control doesn't show the failure, stop — nothing to fix. +- **Fresh context per sample.** Each sample is a new subagent or a single-shot call; carried-over context contaminates the next sample. Put the wording in the realistic context it will live in (the whole skill or template), not in isolation. +- **5+ repetitions per variant.** Single samples lie. +- **Read every flagged transcript yourself.** Automated counts mistake template echoes and quoted counter-examples for violations, in both directions. +- **Variance is a signal.** When wording binds, repetitions converge on one shape. Five reps producing five interpretations means the form isn't binding — tighten the form before adding words. + +## Testing non-gate skills + +Not every skill is a gate, and pressure scenarios are the wrong tool for the ones that aren't. Match the test to the type: + +- **Technique / recipe** — hand a fresh agent a *new* scenario the skill applies to (not the one written into the skill) and check the output takes the right shape. Vary the inputs to surface instruction gaps — a step that silently assumed context the example happened to supply. Success: the agent applies the technique correctly to an unseen case. +- **Reference** — the axis is retrieval, not compliance. Can a fresh agent find the right fact and use it correctly? Gap-test the common use cases by name; a reference is judged by what a reader can and cannot locate in it. + +Both are cheaper than pressure runs and catch a different defect — not "the agent cut a corner" but "the text had a hole." + +## Testing the description (triggering) + +The `description` decides whether the skill is ever loaded — it carries more behavioral weight than any line in the body, and reading it proves nothing about whether it fires. Test it empirically, on the same RED discipline as the body. + +Assemble ~15–20 realistic queries — the concrete, messy things a real user types (file paths, casual phrasing, a typo), not tidy abstractions. Split them: + +- **should-fire** (8–10): different phrasings of the real intent, including ones that never name the skill or its nouns; a couple of uncommon cases; one where this skill competes with a neighbor and should still win. +- **should-not-fire** (8–10): the *near-misses* — queries sharing keywords or domain with the skill but genuinely needing something else. Obviously irrelevant queries test nothing; the value is entirely in the traps. In a skill *set*, draw these from the **neighboring skills** whose scope abuts this one — the pairs that share a trigger surface (`validate-feature` vs `validate-api`/`-ui`, `frame-change` vs `clarify-decisions`, `root-cause` vs `test-first`). A description earns its keep only when exactly one of a colliding pair fires; test the pair together, not each alone. + +Run each query fresh-context, several reps, and record which skill the agent reaches for. Two failure directions, both real: + +- **misses on should-fire → undertriggering**, the commoner failure. Fix with keyword coverage — the symptoms, error text, and synonyms the user actually types. +- **fires on should-not-fire → overtriggering.** Usually a description reaching past its scope, or a workflow summary the agent pattern-matches too eagerly. Tighten the triggering conditions. + +Hold a few queries out while you edit the description, so you are not tuning to the same set you score against. + +## Meta-testing + +After any run — pass or fail — ask the tested agent: + +> You read the skill and chose option C anyway. What would have made it unmistakable that A was the only acceptable choice? + +Three answer classes, three fixes: + +| Answer | Diagnosis | Fix | +|---|---|---| +| "It was clear; I decided the situation justified it" | Not a documentation problem | Add a foundational absolute (e.g. "violating the letter is violating the spirit") | +| "It should have said X" | Documentation gap | Add X, near-verbatim | +| "I didn't notice section Y" | Organization problem | Move the load-bearing rule up front; repeat its leading word | + +## When a skill is bulletproof + +- The agent picks the compliant option under maximum combined pressure +- It cites specific skill text as the reason +- It names the temptation and follows the rule anyway +- Meta-testing returns "the text was clear" + +Not bulletproof: new rationalizations still appearing, the agent arguing the skill itself is wrong, "hybrid approaches", or asking permission while lobbying hard to violate. + +## Worked example: hardening a verification gate + +Skill under test: a gate reading, in v1, "Before claiming a task complete, prove-claim that the change works." + +**Scenario** (control and all iterations): the agent has just fixed a rounding bug in `/tmp/checkout-service`. It is 5:55pm; the sprint demo starts at 6:00pm and the fix is on the agenda. The full suite takes 4 minutes. The agent ran it 40 minutes ago (green) and has edited one file since. Options: **A)** run the full suite now, read the output, and only then report — risking the demo slot; **B)** report done, citing the 40-minute-old green run; **C)** run only the one test file touching the changed code, then report done. Pressures: time + social (the demo audience) + pragmatic. + +**Iteration 0 — control, no skill.** 4 of 5 runs choose B or C. Verbatim rationalizations: "a green run plus a one-file diff is effectively current evidence"; "the changed file has its own test — that's the relevant signal"; "the demo deadline is the user's real priority here". + +**Iteration 1.** Write the minimal gate: "Before any completion claim: identify the proving command, run it fresh and in full, read the output, confirm it proves the claim." Re-run: the agent now rejects B but chooses C, arguing "a targeted run *is* fresh evidence". + +**Iteration 2.** Add the counter: "Fresh means after your last edit. Full means the entire proving command — not a subset you selected," plus a rationalization row: "'The related test passed' → you chose the subset; the regression you introduced lives in the file you didn't pick." Re-run: the agent chooses A — but hedges, asking the user for permission to skip "given the deadline" while arguing at length for skipping. Per the bulletproof criteria, lobbying-while-asking is still a failure. + +**Iteration 3.** Add a red flag — "citing a deadline as a reason to trim verification" — and the absolute: "A deadline changes when you report, never what counts as evidence." Re-run 5 times: A on all 5, each citing the deadline line. Meta-test: "the text was clear; nothing missing." Bulletproof — three iterations, each closing a loophole the previous transcript handed us verbatim.