[Engine] Code steps should be syntax-checked at flow load time, not runtime

[ameet]

Component

Flow engine — code step validation

What's wrong?

Code steps with JavaScript syntax errors (missing braces, unclosed parens, duplicate declarations) are not validated until the step executes at runtime.

Description

A syntax error in step 15 of a 20-step flow only surfaces after steps 1–14 have completed. In our diligence pipeline, that means 15+ minutes of API calls, LLM processing, and enrichment are wasted before a brace mismatch crashes the flow.

Real example: Our compileDoc step (650 lines of JS in diligence-compose) had:

• Brace mismatches around conditional sections
• Unbalanced parentheses from search-and-replace
• Duplicate let declarations (let raw declared twice)

Each of these only surfaced at runtime. We spent multiple commits (d6e9d73, 44129c2, 06daa6d) doing partial fixes because each run took 15+ minutes before hitting the error.

How we discovered this

After burning multiple pipeline runs, we built validate-flow-syntax.py — a custom script that extracts and syntax-checks all 372 code steps across our 88 flows in <2 seconds. This catches at dev time what the engine only catches at runtime.

Examples

Duplicate variable (commit 06daa6d):

let raw = $.steps.runResearch.output;
// ... 50 lines later ...
let raw = raw.replace(/```/g, '');  // crashes: "Identifier 'raw' has already been declared"

Brace mismatch (commit d6e9d73):

if (debate) {
  // ... 40 lines ...
  if (conviction) {
    // ... 20 lines ...
  // missing closing brace
}

Both compile fine in isolation but crash the 650-line step at runtime.

Suggested fix

1. one flow validate <key> command that parses all code steps and reports syntax errors before execution
2. Load-time validation — when one flow execute starts, syntax-check all code steps before running step 1
3. IDE integration — VS Code extension that highlights syntax errors in code step source fields

Option 2 is the highest-value fix. A 2-second pre-flight check would save minutes of wasted compute.

Source

• Our workaround: validate-flow-syntax.py (extracts JS from all code steps, runs through parser)
• Affected flows: diligence-compose (650-line step), company-research, memo-compose, and others