Skip to content

feat(executor): add stdin payload transport for script steps - #253

Merged
Jason Robert (jrob5756) merged 2 commits into
mainfrom
feat/18-script-payload-transport
Jun 16, 2026
Merged

feat(executor): add stdin payload transport for script steps#253
Jason Robert (jrob5756) merged 2 commits into
mainfrom
feat/18-script-payload-transport

Conversation

@jrob5756

Copy link
Copy Markdown
Collaborator

Summary

Closes#18.

Adds a first-class, cross-platform stdin: field to type: script steps — a raw Jinja2 string template rendered against the workflow context and piped to the child process's stdin as UTF-8. This is the clean, argument-name-agnostic fix for the command-line-length fragility (notably Windows ARG_MAX) that previously forced workflow authors into per-flag temp-file workarounds (--evaluations-file).

- name: summarizetype: scriptcommand: python3args: ["scripts/analyze.py"]stdin: "{{ evaluator.output.evaluations | tojson }}"# JSON via the tojson filter

Why stdin (vs. temp files / arg-spillover)

  • No temp-file lifecycle to manage — nothing to clean up.
  • process.communicate(input=...) streams stdin in the background while draining stdout/stderr, so multi-MB payloads can't deadlock the pipe or hit ARG_MAX.
  • A raw string template is strictly more general than a dedicated stdin_json: JSON is {{ x | tojson }}, arbitrary text (diffs, CSV, prompts) is {{ x }}. Matches how command/args already work.

Behavior contract

  • Omitted (stdin: None) → child inherits the parent's stdin (unchanged legacy behavior).
  • Present (any string, including "") → stdin is piped; an explicit empty string sends immediate EOF.
  • stdinargs — both are passed when set; no precedence conflict.
  • Encoding is UTF-8 (consistent with PYTHONUTF8=1 and the existing stdout/stderr decode).

Changes

  • config/schema.py — new stdin field + a single script-exclusive validation guard (mirrors the status/output_template pattern; rejects stdin on all six non-script step types, including agent/human_gate).
  • executor/script.py — render + pipe stdin; new ScriptOutput.stdin_bytes.
  • engine/workflow.pyscript_completed event carries stdin_bytes for dashboard/JSONL observability (never the payload itself).
  • Docs/exampledocs/workflow-syntax.md, authoring skill reference, NG1 note in the original design doc; new examples/script-stdin.yaml registered in examples/README.md.

Acceptance criteria

  • Works uniformly on Windows/macOS/Linux (stdin pipe via asyncio communicate).
  • No dependency on argument names.
  • Clear precedence when both args and stdin are configured (orthogonal, documented).
  • Temp artifact lifecycle deterministic / cleaned up (N/A — no temp files).
  • Tests covering large payloads + length behavior.

Tests

  • 8 executor unit tests (round-trip, 2 MB payload, empty, omitted back-compat, Jinja render, tojson JSON handoff, stdin+args coexistence, UTF-8 byte count).
  • 9 schema tests (accepted on script; rejected on all 6 non-script types).
  • 2 engine integration tests (structured payload handoff; 1 MB payload + stdin_bytes event).
  • Full sweep: 1811 passed. Lint ✅, typecheck ✅, validate-examples ✅ (23 examples).

Add a `stdin:` field to `type: script` steps — a Jinja2 string template
rendered against the workflow context and piped to the child process's
stdin as UTF-8. Workflows can now hand large or structured payloads to
scripts without hitting command-line length limits (notably Windows
ARG_MAX), removing the need for argument-name-specific temp-file
workarounds.
- schema: new `stdin` field plus a script-exclusive validation guard that
rejects it on every non-script step type.
- executor: render and pipe stdin via `communicate(input=...)` (streamed in
the background, so multi-MB payloads can't deadlock); inherit the parent
stdin when omitted. Adds `ScriptOutput.stdin_bytes`.
- engine: surface `stdin_bytes` on the `script_completed` event.
- docs, authoring skill, and a new `examples/script-stdin.yaml`.
`stdin` and `args` are orthogonal; an explicit empty string pipes an
immediate EOF; omitting it preserves the legacy inherit behavior.
Refs #18
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apply review feedback on the new `stdin:` field:
- Wrap the strict UTF-8 encode of the stdin payload in ExecutionError so an
unpaired-surrogate payload (reachable via upstream JSON) surfaces a named
error with a suggestion instead of a bare UnicodeEncodeError.
- Correct "Windows ARG_MAX" terminology across code comments, schema docs,
the authoring skill, and the example — ARG_MAX is POSIX (and larger);
Windows caps the command line at ~32 KB. Clarify that routing through
stdin (not `communicate`) is what avoids the argv length limit.
- Tighten the `stdin_bytes` docstring (submitted, not consumed, bytes) and
add it to the `_execute_script` Returns docstring.
- Clarify the schema guard comment (sole rejection path for non-script types).
- Add failure-mode tests: bidirectional multi-MB no-deadlock guard, child
early-exit/BrokenPipe tolerance, timeout while writing stdin, invalid UTF-8
-> ExecutionError, and undefined-variable render -> TemplateError.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jrob5756
Jason Robert (jrob5756) merged commit 5141989 into mainJun 16, 2026
9 checks passed
@jrob5756
Jason Robert (jrob5756) deleted the feat/18-script-payload-transport branch June 16, 2026 21:37
Jason Robert (jrob5756) added a commit that referenced this pull request Jun 16, 2026
Resolve conflicts from #252 (context_tier), #253 (script stdin), and #245
(dashboard Stop/Kill checkpoint):
- schema.py / workflow-syntax.md: keep both new runtime fields
(runtime.checkpoint + runtime.default_context_tier).
- workflow.py: keep both new method sets (periodic checkpoint helpers +
handle_dashboard_stop). Fix a semantic interaction: handle_dashboard_stop
used `_last_checkpoint_path is not None` as its idempotency guard, but
periodic checkpoints (#244) now also set `_last_checkpoint_path`, which
would wrongly skip the dashboard-stop save + workflow_failed emit. Switched
it to a dedicated `_dashboard_stop_handled` flag and added regression tests.
- AGENTS.md: merge the checkpoint.py descriptions for both features.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert (jrob5756) added a commit that referenced this pull request Jun 17, 2026
Resolve conflicts from #252 (context_tier), #253 (script stdin), #254/#245
(dashboard-stop checkpoint), and #255 (periodic checkpoints):
- cli/run.py: keep both the validator event cases and the new
checkpoint_save_failed console case in the on_event chain.
- AGENTS.md / examples/README.md: keep both features' sections.
- Dashboard bundle: rebuilt from merged frontend source (one bundle reflects
main's ErrorBanner/Header/store changes and the validator events/handlers).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jason Robert (jrob5756) added a commit that referenced this pull request Jun 17, 2026
…1.0.0 (#261)
Document the 0.1.19 release and unblock the context_tier feature.
Changelog (0.1.19):
- Added: validator block (#220), periodic/milestone checkpoints (#255),
script stdin transport (#253), experimental provider tier (#241),
claude-agent-sdk provider (#104).
- Fixed: nested explicit input projection (#239).
- Corrected a stale human_gate note: explicit-mode
`<gate>.output.additional_input.<field>` now resolves via #239.
Dependency:
- Bump github-copilot-sdk floor >=0.3.0 -> >=1.0.0 (re-locked to 1.0.1).
0.3.0's create_session() has no context_tier kwarg, so the context_tier
feature (#251) raised a hard ProviderError on the pinned floor. Verified
on 1.0.1: context_tier works and the full suite (excl. real_api) is green.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: first-class cross-platform payload transport for script steps

1 participant

@jrob5756