feat: add CLI tools backend

[ian-pascoe]

Summary

• add cliTools as a typed, shell-free Caplets backend for curated CLI actions
• add caplets author cli to generate reviewable CLI Caplet manifests from repo workflows and git/gh/package templates
• document the new backend, add bundled CLI examples, regenerate schemas, and add a minor changeset

Validation

• pre-commit hook: lint-staged passed
• pre-push hook: pnpm verify passed via temporary local pnpm shim because pnpm is not installed on PATH in this shell
  • format:check
  • lint
  • typecheck
  • schema:check
  • test (18 files, 233 tests)
  • benchmark:check
  • build

Summary by CodeRabbit

• New Features

  • Added a new cliTools backend for typed, shell-free CLI actions with per-action timeouts, output-size limits, input schema validation, and JSON output parsing.
  • Added example Caplets exposing curated GitHub and repository CLI actions.
  • New authoring command to generate reviewable CLI Caplet manifests.

• Documentation

  • Expanded README, docs, and schemas with CLI tools semantics, examples, and config guidance.

• Tests

  • Added unit and integration tests covering authoring, discovery, execution, errors, and config validation.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: d3cc0603-d731-4b90-a8af-a9124ba7ce94

📥 Commits

Reviewing files that changed from the base of the PR and between 2f49b9e and cf85c5d.

📒 Files selected for processing (6)

• README.md
• caplets/repo-cli/CAPLET.md
• src/cli-tools.ts
• src/cli/author.ts
• test/author-cli.test.ts
• test/cli-tools.test.ts

---

📝 Walkthrough

Walkthrough

This PR introduces a new cliTools Caplets backend enabling typed, shell-free CLI tool execution through MCP. It adds schema/config support, a CliToolsManager runtime, registry/runtime wiring, caplets author cli authoring, example caplets, README/docs/plan updates, and tests.

Changes

CLI Tools Backend Feature

Layer / File(s)	Summary
Configuration and schema definitions schemas/caplet.schema.json, schemas/caplets-config.schema.json, src/caplet-files.ts, src/config.ts	Adds cliTools to JSON schemas and TypeScript config types, defines per-action command/args/env/cwd/timeouts/output/annotations, integrates cliTools into frontmatter/config validation, path normalization, and JSON-schema patching.
CliToolsManager core implementation src/cli-tools.ts	New CliToolsManager implements tool availability checks, listing/search/get, input validation, $input interpolation, command-path resolution (PATH + PATHEXT), spawn-based execution with shell: false, timeout and max-output enforcement, JSON output parsing, and error mapping.
Runtime and registry integration src/runtime.ts, src/registry.ts, src/tools.ts, src/cli/inspection.ts, src/capability-description.mjs	Wires the CLI manager into CapletsRuntime, registry lookups/enumeration, backend detail reporting, and routes tool RPCs (check_backend, list_tools, get_tool, call_tool) to CliToolsManager.
CLI Caplet authoring command src/cli.ts, src/cli/author.ts	Adds authorCliCaplet and caplets author cli to generate YAML Caplet manifests from repositories (git/gh/package templates), detect package manager and scripts, and emit sorted action entries.
Example caplet definitions caplets/repo-cli/CAPLET.md, caplets/github-cli/CAPLET.md	Adds example Caplets demonstrating safe read-oriented CLI actions (git, gh, package scripts) and annotations.
Documentation, tests, and metadata README.md, docs/plans/2026-05-14-cli-tools-backend.md, src/generated-tool-input-schema.mjs, docs/benchmarks/coding-agent.md, .changeset/curated-cli-tools.md, test/cli-tools.test.ts, test/author-cli.test.ts, test/config.test.ts, test/registry.test.ts, test/runtime.test.ts, test/openapi.test.ts	Updates README and plans doc to document cliTools semantics; adjusts benchmark numbers; adds comprehensive tests for authoring, config/schema output, registry/runtime integration, and CliToolsManager execution/limits/validation/error handling.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• spiritledsoftware/caplets#18: Overlaps runtime wiring in CapletsRuntime used by the CLI backend integration.
• spiritledsoftware/caplets#16: Related CLI command/commander wiring that also modifies src/cli.ts.
• spiritledsoftware/caplets#21: Similar backend plumbing changes (adds/handles another backend type) and updates to backendFor/registry patterns.

Poem

🐰 I hop, I write a YAML song,

Commands kept tidy, safe, and strong,
No shells to fear, just args that run,
JSON parsed and timeouts done,
Caplets call — the chores are done.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat: add CLI tools backend' is concise, clear, and directly describes the primary feature addition: introducing a new backend type for CLI tools to the Caplets system.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/cli-tools-backend

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint skipped: no ESLint configuration detected in root package.json. To enable, add eslint to devDependencies.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 7

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@caplets/repo-cli/CAPLET.md`:
- Around line 32-34: The CAPLET entry marks package_test as read-only via the
annotations.readOnlyHint flag, which is misleading because "pnpm run test" can
perform writes; remove the readOnlyHint annotation (or set
annotations.readOnlyHint to false) from the package_test/caplet definition so
the test caplet no longer claims read-only behavior and accurately reflects
potential side effects (look for the keys timeoutMs, annotations, readOnlyHint,
and package_test in the CAPLET.md diff).
In `@docs/benchmarks/coding-agent.md`:
- Around line 17-24: The benchmark doc's deterministic section uses 8400 and
2100 while the README uses canonical values 8358 and 2090; update every
occurrence in the "Serialized payload bytes", "Approx. tokens", "Candidate set
before discovery" and the "Deterministic Results" paragraph to use 8358
(serialized top-level bytes) and 2090 (tokens) and adjust any dependent phrasing
(e.g., "from 32090 bytes to 8400 bytes" → "from 32090 bytes to 8358 bytes" and
the token line) so the numbers and any derived percentages match the README's
canonical set.
In `@src/cli-tools.ts`:
- Around line 379-380: The condition that decides whether to treat command as a
path only checks for "/" so Windows-style separators are missed; update the
check in the command resolution branch (the line using isAbsolute(command) ||
command.includes("/")) to also consider backslashes (e.g., isAbsolute(command)
|| command.includes("/") || command.includes("\\") or use a regex like /[\/\\]/)
so that assertExecutable(command) is invoked for Windows-style relative paths as
well.
- Around line 31-40: The check currently evaluates templates with empty input
and fails actions that use runtime $input values; modify the loop over
actionsFor(config) so you detect if cwd or action.command contains "$input" (or
the specific "$input." token) and skip the existence check and
resolveCommandPath for those templated values; keep the existing behavior for
non-templated strings (use interpolateString + existsSync and call
resolveCommandPath), and continue to throw CapletsError("CONFIG_INVALID", ...)
only when a non-templated cwd is missing or when resolveCommandPath fails for a
non-templated command.
In `@src/cli/author.ts`:
- Around line 35-36: The CLI currently always sets include =
parseInclude(options.include) which applies the default group(s) so supplying
--command still yields additive templates; change the handler so that if
options.command is present AND options.include is undefined/empty, override
include to a single-group list representing the requested command template
(e.g., set include = [options.commandTemplateName] or call parseInclude with
that single value) before building the actions record; update the logic around
where include is used (the parseInclude call and the code that builds actions,
referenced by the include variable and options.command/options.include) so
--command produces only the single template group when --include was not
explicitly passed.
- Around line 161-162: The current logic sets annotations.readOnlyHint based on
script !== "build", which incorrectly marks most package scripts as read-only;
change this so readOnlyHint is false by default and only true for explicit,
known read-only scripts (or remove the hint entirely). Update the annotations
assignment that references readOnlyHint and script in the author CLI code to use
a whitelist (e.g., readOnlyScripts = new Set([...]) and set readOnlyHint:
readOnlyScripts.has(script)) or simply set readOnlyHint: false, ensuring the
change touches the annotations object where readOnlyHint is currently computed.
In `@test/cli-tools.test.ts`:
- Around line 41-52: Add a new assertion to the test that verifies the command
is run without a shell by passing a payload containing shell metacharacters
(e.g., characters like ; | && $() or >) into manager.callTool(caplet,
"echo_json", ...) and assert the returned structuredContent.json contains that
exact string unchanged; locate the test case around the existing it("spawns
commands without a shell and returns parsed JSON output") and extend it to call
manager.callTool again with a metacharacter-laden message and expect
result.structuredContent.json.message to strictly equal the input string,
ensuring callTool (and the CliToolsManager invocation) does not perform shell
interpretation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: defc6725-9c4e-412f-87a6-283a88ed3099

📥 Commits

Reviewing files that changed from the base of the PR and between 665ad58 and 2f49b9e.

📒 Files selected for processing (25)

• .changeset/curated-cli-tools.md
• README.md
• caplets/github-cli/CAPLET.md
• caplets/repo-cli/CAPLET.md
• docs/benchmarks/coding-agent.md
• docs/plans/2026-05-14-cli-tools-backend.md
• schemas/caplet.schema.json
• schemas/caplets-config.schema.json
• src/capability-description.mjs
• src/caplet-files.ts
• src/cli-tools.ts
• src/cli.ts
• src/cli/author.ts
• src/cli/inspection.ts
• src/config.ts
• src/generated-tool-input-schema.mjs
• src/registry.ts
• src/runtime.ts
• src/tools.ts
• test/author-cli.test.ts
• test/cli-tools.test.ts
• test/config.test.ts
• test/openapi.test.ts
• test/registry.test.ts
• test/runtime.test.ts

[coderabbitai]

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Align benchmark figures with README to avoid conflicting product claims.

This file now reports 8400/2100, while README still reports 8358/2090 for the same deterministic benchmark. Please keep one canonical precision/number set across both docs.

Also applies to: 41-41

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/benchmarks/coding-agent.md` around lines 17 - 24, The benchmark doc's
deterministic section uses 8400 and 2100 while the README uses canonical values
8358 and 2090; update every occurrence in the "Serialized payload bytes",
"Approx. tokens", "Candidate set before discovery" and the "Deterministic
Results" paragraph to use 8358 (serialized top-level bytes) and 2090 (tokens)
and adjust any dependent phrasing (e.g., "from 32090 bytes to 8400 bytes" →
"from 32090 bytes to 8358 bytes" and the token line) so the numbers and any
derived percentages match the README's canonical set.

[greptile-apps]

Greptile Summary

This PR introduces the cliTools backend — a typed, shell-free mechanism for spawning local CLI processes from Caplet config files — and adds a caplets author cli command that auto-generates reviewable CLI Caplet manifests by inspecting a repo's git, gh, and package manager context.

• src/cli-tools.ts: new CliToolsManager with input interpolation, $input-aware arg/env/cwd templating, output byte-limit enforcement, AbortController-based timeouts, and JSON stdout parsing. All three issues flagged in the previous review round (null required-field check, tolerateInvalidJson on non-zero exits, isAbortError name-only guard) are correctly resolved in this version.
• src/cli/author.ts: CLI authoring helper that detects package manager, reads package.json scripts, and renders a YAML Caplet manifest; wired into caplets author cli via src/cli.ts.
• Config/schema/runtime plumbing: src/config.ts, src/caplet-files.ts, src/registry.ts, src/runtime.ts, and src/tools.ts are all updated consistently to include cliTools alongside the existing MCP, OpenAPI, GraphQL, and HTTP backends.

Confidence Score: 5/5

Safe to merge. The new backend is well-contained, shell injection is prevented by spawning without a shell, and all three issues raised in the previous review round have been addressed.

The cliTools backend is correctly wired across config parsing, Zod validation, runtime, and tool dispatch. Input interpolation throws on missing or wrong-typed fields, the AbortError guard uses only error.name, non-zero exits tolerate invalid JSON, and null required fields are rejected. Test coverage spans byte limits, timeouts, and field selection. The single gap is an unguarded JSON.parse in the authoring helper, which only affects the caplets author cli convenience command.

src/cli/author.ts — packageActions reads package.json without a try/catch around JSON.parse, producing an uncontextualized SyntaxError if the file is malformed.

Important Files Changed

Filename	Overview
src/cli-tools.ts	Core CLI tools backend: shell-free process spawning with input interpolation, JSON output parsing, timeout/byte-limit guards, and clean abort handling. Previous review findings all correctly addressed.
src/cli/author.ts	CLI authoring helper that generates cliTools Caplet manifests. package.json is read and parsed without a try/catch, so a malformed package.json produces an uncontextualized SyntaxError.
src/config.ts	Adds full cliTools schema, Zod validation, duplicate-ID detection, path normalization, and project-config rejection. Well-integrated with the existing backend pattern.
src/caplet-files.ts	Extends Caplet file parsing to support the cliTools backend with schema, relative cwd normalization, and correct backend dispatch.
src/runtime.ts	Integrates CliToolsManager into CapletsRuntime: construction, registry updates, invalidation, and caplet lookup consistent with existing manager wiring.
src/tools.ts	Wires CliToolsManager into handleServerTool dispatch, including field selection support for CLI tool results.
test/cli-tools.test.ts	Comprehensive tests covering JSON output, non-zero exits with invalid JSON, input validation, byte limits, timeouts, field selection, and duplicate-ID rejection.

Sequence Diagram

sequenceDiagram
participant Agent as MCP Agent
participant RT as CapletsRuntime
participant TH as handleServerTool
participant CLM as CliToolsManager
participant SC as spawnCommand
Agent->>RT: "call_tool caplet=repo-cli tool=git_status"
RT->>TH: handleTool(serverId, request)
TH->>CLM: callTool(config, git_status, args)
CLM->>CLM: validateInput(action, args)
CLM->>CLM: resolveExecution interpolate $input resolve cwd/env
CLM->>SC: spawnCommand(execution, signal, elapsed)
SC->>SC: "spawn(command, args, shell=false)"
Note over SC: AbortController timeout + maxOutputBytes per chunk
SC-->>CLM: SpawnResult exitCode stdout stderr
CLM->>CLM: "parseStructuredResult parse JSON if output.type=json"
CLM-->>TH: CompatibilityCallToolResult
TH-->>Agent: content structuredContent isError

Loading

_{Reviews (2): Last reviewed commit: "fix: address cli tools review feedback" | Re-trigger Greptile}