fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance - #22

Merged
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes
Jun 26, 2026
Merged

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance#22
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes

Conversation

@Rchari1

Copy link
Copy Markdown
Member

Findings + fixes from an autonomous QA campaign on the chat→solve path (drove the real opencode backend headlessly with a matrix of user requests; full notes in vault qa-20260620-amicode-chat-solves.md). Stacked on #14; base rchari/template-plot-pulse. Two real bugs fixed, two limitations mitigated by guidance.

F1 (BLOCKER) — chat hung on a permission prompt for every solve

The instruction-injection fix (#12) made the agent read the bundled template at an absolute path outside the session dir + write /tmp/amicode-work. opencode defaults external_directory to ask, so with no interactive approver the turn hangs forever (headless) and nags on every solve (GUI) — the "trouble running gates" symptom returning. It only worked before because the GUI popup was clicked (always-remembered).
Fix:buildOpencodeConfigContent injects permission: { bash, edit, webfetch, external_directory: allow }.
Verified: baseline X-gate chat → authors solve.jl → launches amico-runF=0.99994.

F4 — multi-qubit requests: agent hallucinated an API and crashed

A CNOT request → the agent didn't decline; it hand-rolled TransmonSystem(ω=[ω1,ω2], g=…) (a multi-qubit API that doesn't exist) → TypeError: keyword ω expected Float64, got Vector{Float64}, while claiming "solve launched" with a tidy params table.
Fix: AGENTS.md now scopes the build to single-qubit only and tells the agent to decline multi-qubit gates.
Verified: re-asking for a CNOT now returns "…not supported in this build. TransmonSystem models a single qubit only… If you have a single-qubit gate (X, H, T, …) I can help" — no crash, no solve launched.

F3 — long gates were silently under-resolved

T=30 ns with the default N=50 (0.6 ns/step) → objective plateaus → F=0.914 reported as success.
Fix: AGENTS.md guidance to keep ~5–10 timesteps/ns and scale N with gate time.
Verified: asking for a 30 ns gate now → agent authors N=200 (~6.7 steps/ns).

F2 — levels ≥5 is init/iteration-sensitive (soft)

levels=5 stalled (f≈69) on one seed with the default max_iter=60, but converged to F=0.99999 when the agent bumped max_iter=200. Not broken — the template default is too low for 5 levels, and the agent already compensates. AGENTS.md adds regime guidance (prefer 3–4 levels; warn on 5+). Deeper physics fixes (energy-shift / DRAG warm-start / fixed seed) deferred.

What's already good (no change needed)

Correct gate translation (H/Y/Z), sane defaults on underspecified prompts, and genuinely smart parameter adaptation — fast 4 ns gate → raised N + drive_max + max_iter (F≈1.0); 5 levels → raised max_iter (F=0.99999). Template gates X/Y/Z/H + tight drive bound all converge F > 0.999.

Tests

40 extension tests green; new guards on the permission block + the single-qubit scoping + regime guidance.

🤖 Generated with Claude Code

@Rchari1

Copy link
Copy Markdown
MemberAuthor

@jack-champagne

Rchari1and others added 2 commits June 20, 2026 21:44
…rectory hang)
The instruction-injection fix made the agent read the bundled template at an
absolute path outside the session cwd + write /tmp/amicode-work scratch.
opencode defaults external_directory to 'ask' → the turn hangs forever with no
interactive approver (headless) and nags on every solve (GUI). buildOpencodeConfigContent
now injects permission:{bash,edit,webfetch,external_directory: allow} so the
controlled solve workflow proceeds without prompts.
Found via headless chat QA (qa-20260620): baseline X-gate chat stalled on the
external_directory ask; with the permission block the agent reads the template,
mkdirs scratch, authors solve.jl, and launches amico-run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2/F3/F4)
QA (qa-20260620) found the agent over-reaches on out-of-scope requests:
- F4: a 2-qubit (CNOT) request → the agent HALLUCINATES a multi-qubit API
(TransmonSystem(ω=[..],g=..)) → TypeError crash, while claiming 'solve launched'.
- F3: a 30 ns gate with the default N=50 is under-resolved → F=0.914 reported as success.
- F2: levels=5 stalls (integrator stiffness) at f~69.
AGENTS.md now: declares the build single-qubit only and tells the agent to
decline multi-qubit gates instead of authoring a crashing script; and gives
regime guidance (avoid 5+ levels; keep ~5-10 timesteps/ns so longer gates scale
N up; bump max_iter for harder cases). Guard tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jack-champagne

Copy link
Copy Markdown
Member

Correct root-cause: external_directory is one of the few opencode permissions defaulting to "ask", so headless it hangs / nags per solve. Fix is in the right place. Two trims worth making here:

  • [important, in-PR] external_directory: "allow" is a blanket grant to the whole filesystem outside the workspace. Scope it to the roots actually used (/tmp/amicode-work + the bundled template dir) — opencode supports per-path patterns. As-is it's broader than a single-purpose assistant needs.
  • [minor] webfetch: "allow" and edit: "allow" are redundant (both default to allow), and webfetch grants network egress the solve flow never uses. Drop webfetch; the load-bearing line is external_directory alone.

Note this doesn't make chat→solve work end-to-end on its own — the #11 author/run path mismatch was still live here and is reconciled in #23 (now validated).

@Rchari1
Rchari1 changed the base branch from rchari/template-plot-pulse to mainJune 26, 2026 20:11

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The external_directory fix is correctly diagnosed and lands the blocker. Three changes (inline): drop the unused webfetch allow; scope external_directory to the two roots the agent actually touches (or document the trust boundary if the schema won't scope); and verify the nested permission block deep-merges rather than replacing the user's permission config (+ a test).

bash/edit are redundant-as-defaults but harmless — leave them; scoping bash is fiddly given the compound mkdir … && nohup amico-run … launch.

AGENTS.md guidance is solid and the N / steps-per-ns numbers are consistent — two prose fixes: "do not hand-roll a coupled multi-transmon system (it will not construct)" — MultiTransmonSystem does exist and would construct; say "out of scope for this single-lab build." And "the integrator stiffens" for levels ≥5 is loose — it's conditioning/leakage/cost.

Tests are content/string guards — they assert the keys/words are emitted, not that opencode honors them, and nothing covers the permission merge.

permission: {
bash: "allow",
edit: "allow",
webfetch: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] webfetch isn't used by the solve workflow (it never fetches a URL) and already defaults to allow, so this line only forces egress back on for a user who'd restricted it globally. The new test omits it too. Drop it:

Suggested change
webfetch: "allow",

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped in 4942a2bwebfetch is gone (and edit/bash left as you suggested).

bash: "allow",
edit: "allow",
webfetch: "allow",
external_directory: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] This is the line that fixes the hang — correct, external_directory defaults to "ask" in 1.17.x. But "allow" grants the agent's file tools the entire filesystem outside the session cwd. The agent itself needs only two roots: it reads the bundled template (opts.templateSrc) and writes scratch to /tmp/amicode-work (amico-run's writes to ~/.amico/runs / ~/.amico/julia are the subprocess's, not the agent's). If opencode 1.17.3 supports path-scoped external_directory, scope it:

external_directory: {
"/tmp/amicode-work/**": "allow",
"<extension templates dir>/**": "allow",
},

If it only takes a flat string, keep "allow" but document the trust boundary.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scoped in 4942a2b. external_directory is now an object keyed to the two roots the agent's file tools touch — the bundled templates dir (path.dirname(templatePath) + the exact template file) and /tmp/amicode-work (plus /private/tmp/amicode-work for macOS's /tmp→/private/tmp symlink). amico-run's own ~/.amico writes are the subprocess's, not the agent's, so they're not granted. The path-scoped object form is accepted by 1.17.3 — verified via opencode debug config (resolved config preserves the per-path keys). Threading templatePath through buildOpencodeConfigContent so the dir is computed at runtime (correct for both dev and the installed VSIX).

return JSON.stringify({
$schema: "https://opencode.ai/config.json",
instructions: [agentsPath],
permission: {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] Verify the merge for this nested object. The doc-comment says opencode merges the injected config over the user's — but that was only confirmed for top-level model/provider and the instructions array (#12). If opencode shallow-replaces rather than deep-merges, injecting permission here wipes any other permission keys the user set globally (doom_loop, per-tool patterns). Nothing tests this — confirm against 1.17.3 and add a test that the user's other permission keys survive.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified — opencode 1.17.3 deep-merges the nested permission object; it does not shallow-replace. Tested with opencode debug config: a global permission.doom_loop:"deny" survives alongside our injected external_directory keys. Added an integration test (opencode_config.test.ts → 'opencode permission merge (1.17.3)') that runs the vendored binary's debug config with a global config + our injected content and asserts the user's key survives; skipIf the binary is absent. So injecting permission adds keys without clobbering the user's other permission settings.

…drop webfetch, verify deep-merge
- Scope `external_directory` (least privilege) to the two roots the agent's file
tools actually touch: the bundled templates dir (reads the solve template) and
/tmp/amicode-work (writes solve.jl). Path-scoped object form verified accepted
by opencode 1.17.3 via `opencode debug config`. Includes /private/tmp/** for
macOS's /tmp symlink. amico-run's own ~/.amico writes are the subprocess's, not
the agent's, so they need no grant.
- Drop `webfetch` (unused by the solve flow; already defaults to allow). Keep
bash/edit per Jack (redundant-as-defaults but harmless; bash launch not worth
scoping).
- Verify the permission merge: opencode 1.17.3 DEEP-merges the injected
`permission` object over the user's global config (a global permission.doom_loop
survives). Added an integration test (runs the vendored binary's `debug config`,
skipped when absent) so this can't regress to a clobber.
- AGENTS.md prose fixes: multi-qubit is "out of scope for this single-lab build"
(MultiTransmonSystem does exist/construct), and levels≥5 degrades via
conditioning/leakage/cost, not "integrator stiffens".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Rchari1

Copy link
Copy Markdown
MemberAuthor

Thanks @jack-champagne — all addressed in 4942a2b:

  • webfetch dropped; bash/edit kept.
  • external_directory scoped to the template dir + /tmp/amicode-work (+ /private/tmp for the macOS symlink); path-scoped object form verified accepted by 1.17.3 via opencode debug config.
  • deep-merge verified — opencode 1.17.3 deep-merges permission (a global doom_loop survives alongside our injected keys). Added an integration test that runs the vendored binary's debug config and asserts the user's key survives (skips when the binary's absent).
  • AGENTS.md: multi-qubit reworded to "out of scope for this single-lab build" (MultiTransmonSystem does construct); levels≥5 reworded to conditioning/leakage/cost.

44 tests green (incl. the new merge test). One note: the runtime enforcement of the scoped globs (vs schema-acceptance, which is verified) I'll smoke-test on the dev host with a live solve before this merges — if a path canonicalization edge re-introduces the prompt, the documented-allow fallback you sanctioned is a 2-line change. Re-requesting review.

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three addressed in 4942a2b — verified against the diff:

  • webfetch dropped; bash/edit left at allow as agreed.
  • external_directory scoped to the template file + its dir + /tmp/amicode-work (and /private/tmp for the macOS symlink). Blanket allow gone, amico-run's own ~/.amico writes correctly ungranted.
  • deep-merge confirmed empirically via opencode debug config + integration test.

One follow-up, not blocking: the merge test is describe.skipIf(!existsSync(OC_BIN)) — it silently skips when the vendored binary is absent, so in CI before fetch:opencode it's green-without-verifying (same false-green class as #25). Sequence that test job behind fetch:opencode or the guard is decorative.

Good to merge.

@jack-champagne
jack-champagne merged commit 4249162 into mainJun 26, 2026
3 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Rchari1@jack-champagne
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance - #22

Merged
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes
Jun 26, 2026
Merged

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance#22
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes

Conversation

@Rchari1

Copy link
Copy Markdown
Member

Findings + fixes from an autonomous QA campaign on the chat→solve path (drove the real opencode backend headlessly with a matrix of user requests; full notes in vault qa-20260620-amicode-chat-solves.md). Stacked on #14; base rchari/template-plot-pulse. Two real bugs fixed, two limitations mitigated by guidance.

F1 (BLOCKER) — chat hung on a permission prompt for every solve

The instruction-injection fix (#12) made the agent read the bundled template at an absolute path outside the session dir + write /tmp/amicode-work. opencode defaults external_directory to ask, so with no interactive approver the turn hangs forever (headless) and nags on every solve (GUI) — the "trouble running gates" symptom returning. It only worked before because the GUI popup was clicked (always-remembered).
Fix:buildOpencodeConfigContent injects permission: { bash, edit, webfetch, external_directory: allow }.
Verified: baseline X-gate chat → authors solve.jl → launches amico-runF=0.99994.

F4 — multi-qubit requests: agent hallucinated an API and crashed

A CNOT request → the agent didn't decline; it hand-rolled TransmonSystem(ω=[ω1,ω2], g=…) (a multi-qubit API that doesn't exist) → TypeError: keyword ω expected Float64, got Vector{Float64}, while claiming "solve launched" with a tidy params table.
Fix: AGENTS.md now scopes the build to single-qubit only and tells the agent to decline multi-qubit gates.
Verified: re-asking for a CNOT now returns "…not supported in this build. TransmonSystem models a single qubit only… If you have a single-qubit gate (X, H, T, …) I can help" — no crash, no solve launched.

F3 — long gates were silently under-resolved

T=30 ns with the default N=50 (0.6 ns/step) → objective plateaus → F=0.914 reported as success.
Fix: AGENTS.md guidance to keep ~5–10 timesteps/ns and scale N with gate time.
Verified: asking for a 30 ns gate now → agent authors N=200 (~6.7 steps/ns).

F2 — levels ≥5 is init/iteration-sensitive (soft)

levels=5 stalled (f≈69) on one seed with the default max_iter=60, but converged to F=0.99999 when the agent bumped max_iter=200. Not broken — the template default is too low for 5 levels, and the agent already compensates. AGENTS.md adds regime guidance (prefer 3–4 levels; warn on 5+). Deeper physics fixes (energy-shift / DRAG warm-start / fixed seed) deferred.

What's already good (no change needed)

Correct gate translation (H/Y/Z), sane defaults on underspecified prompts, and genuinely smart parameter adaptation — fast 4 ns gate → raised N + drive_max + max_iter (F≈1.0); 5 levels → raised max_iter (F=0.99999). Template gates X/Y/Z/H + tight drive bound all converge F > 0.999.

Tests

40 extension tests green; new guards on the permission block + the single-qubit scoping + regime guidance.

🤖 Generated with Claude Code

@Rchari1

Copy link
Copy Markdown
MemberAuthor

@jack-champagne

Rchari1and others added 2 commits June 20, 2026 21:44
…rectory hang)
The instruction-injection fix made the agent read the bundled template at an
absolute path outside the session cwd + write /tmp/amicode-work scratch.
opencode defaults external_directory to 'ask' → the turn hangs forever with no
interactive approver (headless) and nags on every solve (GUI). buildOpencodeConfigContent
now injects permission:{bash,edit,webfetch,external_directory: allow} so the
controlled solve workflow proceeds without prompts.
Found via headless chat QA (qa-20260620): baseline X-gate chat stalled on the
external_directory ask; with the permission block the agent reads the template,
mkdirs scratch, authors solve.jl, and launches amico-run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2/F3/F4)
QA (qa-20260620) found the agent over-reaches on out-of-scope requests:
- F4: a 2-qubit (CNOT) request → the agent HALLUCINATES a multi-qubit API
(TransmonSystem(ω=[..],g=..)) → TypeError crash, while claiming 'solve launched'.
- F3: a 30 ns gate with the default N=50 is under-resolved → F=0.914 reported as success.
- F2: levels=5 stalls (integrator stiffness) at f~69.
AGENTS.md now: declares the build single-qubit only and tells the agent to
decline multi-qubit gates instead of authoring a crashing script; and gives
regime guidance (avoid 5+ levels; keep ~5-10 timesteps/ns so longer gates scale
N up; bump max_iter for harder cases). Guard tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jack-champagne

Copy link
Copy Markdown
Member

Correct root-cause: external_directory is one of the few opencode permissions defaulting to "ask", so headless it hangs / nags per solve. Fix is in the right place. Two trims worth making here:

  • [important, in-PR] external_directory: "allow" is a blanket grant to the whole filesystem outside the workspace. Scope it to the roots actually used (/tmp/amicode-work + the bundled template dir) — opencode supports per-path patterns. As-is it's broader than a single-purpose assistant needs.
  • [minor] webfetch: "allow" and edit: "allow" are redundant (both default to allow), and webfetch grants network egress the solve flow never uses. Drop webfetch; the load-bearing line is external_directory alone.

Note this doesn't make chat→solve work end-to-end on its own — the #11 author/run path mismatch was still live here and is reconciled in #23 (now validated).

@Rchari1
Rchari1 changed the base branch from rchari/template-plot-pulse to mainJune 26, 2026 20:11

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The external_directory fix is correctly diagnosed and lands the blocker. Three changes (inline): drop the unused webfetch allow; scope external_directory to the two roots the agent actually touches (or document the trust boundary if the schema won't scope); and verify the nested permission block deep-merges rather than replacing the user's permission config (+ a test).

bash/edit are redundant-as-defaults but harmless — leave them; scoping bash is fiddly given the compound mkdir … && nohup amico-run … launch.

AGENTS.md guidance is solid and the N / steps-per-ns numbers are consistent — two prose fixes: "do not hand-roll a coupled multi-transmon system (it will not construct)" — MultiTransmonSystem does exist and would construct; say "out of scope for this single-lab build." And "the integrator stiffens" for levels ≥5 is loose — it's conditioning/leakage/cost.

Tests are content/string guards — they assert the keys/words are emitted, not that opencode honors them, and nothing covers the permission merge.

permission: {
bash: "allow",
edit: "allow",
webfetch: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] webfetch isn't used by the solve workflow (it never fetches a URL) and already defaults to allow, so this line only forces egress back on for a user who'd restricted it globally. The new test omits it too. Drop it:

Suggested change
webfetch: "allow",

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped in 4942a2bwebfetch is gone (and edit/bash left as you suggested).

bash: "allow",
edit: "allow",
webfetch: "allow",
external_directory: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] This is the line that fixes the hang — correct, external_directory defaults to "ask" in 1.17.x. But "allow" grants the agent's file tools the entire filesystem outside the session cwd. The agent itself needs only two roots: it reads the bundled template (opts.templateSrc) and writes scratch to /tmp/amicode-work (amico-run's writes to ~/.amico/runs / ~/.amico/julia are the subprocess's, not the agent's). If opencode 1.17.3 supports path-scoped external_directory, scope it:

external_directory: {
"/tmp/amicode-work/**": "allow",
"<extension templates dir>/**": "allow",
},

If it only takes a flat string, keep "allow" but document the trust boundary.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scoped in 4942a2b. external_directory is now an object keyed to the two roots the agent's file tools touch — the bundled templates dir (path.dirname(templatePath) + the exact template file) and /tmp/amicode-work (plus /private/tmp/amicode-work for macOS's /tmp→/private/tmp symlink). amico-run's own ~/.amico writes are the subprocess's, not the agent's, so they're not granted. The path-scoped object form is accepted by 1.17.3 — verified via opencode debug config (resolved config preserves the per-path keys). Threading templatePath through buildOpencodeConfigContent so the dir is computed at runtime (correct for both dev and the installed VSIX).

return JSON.stringify({
$schema: "https://opencode.ai/config.json",
instructions: [agentsPath],
permission: {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] Verify the merge for this nested object. The doc-comment says opencode merges the injected config over the user's — but that was only confirmed for top-level model/provider and the instructions array (#12). If opencode shallow-replaces rather than deep-merges, injecting permission here wipes any other permission keys the user set globally (doom_loop, per-tool patterns). Nothing tests this — confirm against 1.17.3 and add a test that the user's other permission keys survive.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified — opencode 1.17.3 deep-merges the nested permission object; it does not shallow-replace. Tested with opencode debug config: a global permission.doom_loop:"deny" survives alongside our injected external_directory keys. Added an integration test (opencode_config.test.ts → 'opencode permission merge (1.17.3)') that runs the vendored binary's debug config with a global config + our injected content and asserts the user's key survives; skipIf the binary is absent. So injecting permission adds keys without clobbering the user's other permission settings.

…drop webfetch, verify deep-merge
- Scope `external_directory` (least privilege) to the two roots the agent's file
tools actually touch: the bundled templates dir (reads the solve template) and
/tmp/amicode-work (writes solve.jl). Path-scoped object form verified accepted
by opencode 1.17.3 via `opencode debug config`. Includes /private/tmp/** for
macOS's /tmp symlink. amico-run's own ~/.amico writes are the subprocess's, not
the agent's, so they need no grant.
- Drop `webfetch` (unused by the solve flow; already defaults to allow). Keep
bash/edit per Jack (redundant-as-defaults but harmless; bash launch not worth
scoping).
- Verify the permission merge: opencode 1.17.3 DEEP-merges the injected
`permission` object over the user's global config (a global permission.doom_loop
survives). Added an integration test (runs the vendored binary's `debug config`,
skipped when absent) so this can't regress to a clobber.
- AGENTS.md prose fixes: multi-qubit is "out of scope for this single-lab build"
(MultiTransmonSystem does exist/construct), and levels≥5 degrades via
conditioning/leakage/cost, not "integrator stiffens".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Rchari1

Copy link
Copy Markdown
MemberAuthor

Thanks @jack-champagne — all addressed in 4942a2b:

  • webfetch dropped; bash/edit kept.
  • external_directory scoped to the template dir + /tmp/amicode-work (+ /private/tmp for the macOS symlink); path-scoped object form verified accepted by 1.17.3 via opencode debug config.
  • deep-merge verified — opencode 1.17.3 deep-merges permission (a global doom_loop survives alongside our injected keys). Added an integration test that runs the vendored binary's debug config and asserts the user's key survives (skips when the binary's absent).
  • AGENTS.md: multi-qubit reworded to "out of scope for this single-lab build" (MultiTransmonSystem does construct); levels≥5 reworded to conditioning/leakage/cost.

44 tests green (incl. the new merge test). One note: the runtime enforcement of the scoped globs (vs schema-acceptance, which is verified) I'll smoke-test on the dev host with a live solve before this merges — if a path canonicalization edge re-introduces the prompt, the documented-allow fallback you sanctioned is a 2-line change. Re-requesting review.

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three addressed in 4942a2b — verified against the diff:

  • webfetch dropped; bash/edit left at allow as agreed.
  • external_directory scoped to the template file + its dir + /tmp/amicode-work (and /private/tmp for the macOS symlink). Blanket allow gone, amico-run's own ~/.amico writes correctly ungranted.
  • deep-merge confirmed empirically via opencode debug config + integration test.

One follow-up, not blocking: the merge test is describe.skipIf(!existsSync(OC_BIN)) — it silently skips when the vendored binary is absent, so in CI before fetch:opencode it's green-without-verifying (same false-green class as #25). Sequence that test job behind fetch:opencode or the guard is decorative.

Good to merge.

@jack-champagne
jack-champagne merged commit 4249162 into mainJun 26, 2026
3 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Rchari1@jack-champagne
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance - #22

Merged
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes
Jun 26, 2026
Merged

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance#22
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes

Conversation

@Rchari1

Copy link
Copy Markdown
Member

Findings + fixes from an autonomous QA campaign on the chat→solve path (drove the real opencode backend headlessly with a matrix of user requests; full notes in vault qa-20260620-amicode-chat-solves.md). Stacked on #14; base rchari/template-plot-pulse. Two real bugs fixed, two limitations mitigated by guidance.

F1 (BLOCKER) — chat hung on a permission prompt for every solve

The instruction-injection fix (#12) made the agent read the bundled template at an absolute path outside the session dir + write /tmp/amicode-work. opencode defaults external_directory to ask, so with no interactive approver the turn hangs forever (headless) and nags on every solve (GUI) — the "trouble running gates" symptom returning. It only worked before because the GUI popup was clicked (always-remembered).
Fix:buildOpencodeConfigContent injects permission: { bash, edit, webfetch, external_directory: allow }.
Verified: baseline X-gate chat → authors solve.jl → launches amico-runF=0.99994.

F4 — multi-qubit requests: agent hallucinated an API and crashed

A CNOT request → the agent didn't decline; it hand-rolled TransmonSystem(ω=[ω1,ω2], g=…) (a multi-qubit API that doesn't exist) → TypeError: keyword ω expected Float64, got Vector{Float64}, while claiming "solve launched" with a tidy params table.
Fix: AGENTS.md now scopes the build to single-qubit only and tells the agent to decline multi-qubit gates.
Verified: re-asking for a CNOT now returns "…not supported in this build. TransmonSystem models a single qubit only… If you have a single-qubit gate (X, H, T, …) I can help" — no crash, no solve launched.

F3 — long gates were silently under-resolved

T=30 ns with the default N=50 (0.6 ns/step) → objective plateaus → F=0.914 reported as success.
Fix: AGENTS.md guidance to keep ~5–10 timesteps/ns and scale N with gate time.
Verified: asking for a 30 ns gate now → agent authors N=200 (~6.7 steps/ns).

F2 — levels ≥5 is init/iteration-sensitive (soft)

levels=5 stalled (f≈69) on one seed with the default max_iter=60, but converged to F=0.99999 when the agent bumped max_iter=200. Not broken — the template default is too low for 5 levels, and the agent already compensates. AGENTS.md adds regime guidance (prefer 3–4 levels; warn on 5+). Deeper physics fixes (energy-shift / DRAG warm-start / fixed seed) deferred.

What's already good (no change needed)

Correct gate translation (H/Y/Z), sane defaults on underspecified prompts, and genuinely smart parameter adaptation — fast 4 ns gate → raised N + drive_max + max_iter (F≈1.0); 5 levels → raised max_iter (F=0.99999). Template gates X/Y/Z/H + tight drive bound all converge F > 0.999.

Tests

40 extension tests green; new guards on the permission block + the single-qubit scoping + regime guidance.

🤖 Generated with Claude Code

@Rchari1

Copy link
Copy Markdown
MemberAuthor

@jack-champagne

Rchari1and others added 2 commits June 20, 2026 21:44
…rectory hang)
The instruction-injection fix made the agent read the bundled template at an
absolute path outside the session cwd + write /tmp/amicode-work scratch.
opencode defaults external_directory to 'ask' → the turn hangs forever with no
interactive approver (headless) and nags on every solve (GUI). buildOpencodeConfigContent
now injects permission:{bash,edit,webfetch,external_directory: allow} so the
controlled solve workflow proceeds without prompts.
Found via headless chat QA (qa-20260620): baseline X-gate chat stalled on the
external_directory ask; with the permission block the agent reads the template,
mkdirs scratch, authors solve.jl, and launches amico-run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2/F3/F4)
QA (qa-20260620) found the agent over-reaches on out-of-scope requests:
- F4: a 2-qubit (CNOT) request → the agent HALLUCINATES a multi-qubit API
(TransmonSystem(ω=[..],g=..)) → TypeError crash, while claiming 'solve launched'.
- F3: a 30 ns gate with the default N=50 is under-resolved → F=0.914 reported as success.
- F2: levels=5 stalls (integrator stiffness) at f~69.
AGENTS.md now: declares the build single-qubit only and tells the agent to
decline multi-qubit gates instead of authoring a crashing script; and gives
regime guidance (avoid 5+ levels; keep ~5-10 timesteps/ns so longer gates scale
N up; bump max_iter for harder cases). Guard tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jack-champagne

Copy link
Copy Markdown
Member

Correct root-cause: external_directory is one of the few opencode permissions defaulting to "ask", so headless it hangs / nags per solve. Fix is in the right place. Two trims worth making here:

  • [important, in-PR] external_directory: "allow" is a blanket grant to the whole filesystem outside the workspace. Scope it to the roots actually used (/tmp/amicode-work + the bundled template dir) — opencode supports per-path patterns. As-is it's broader than a single-purpose assistant needs.
  • [minor] webfetch: "allow" and edit: "allow" are redundant (both default to allow), and webfetch grants network egress the solve flow never uses. Drop webfetch; the load-bearing line is external_directory alone.

Note this doesn't make chat→solve work end-to-end on its own — the #11 author/run path mismatch was still live here and is reconciled in #23 (now validated).

@Rchari1
Rchari1 changed the base branch from rchari/template-plot-pulse to mainJune 26, 2026 20:11

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The external_directory fix is correctly diagnosed and lands the blocker. Three changes (inline): drop the unused webfetch allow; scope external_directory to the two roots the agent actually touches (or document the trust boundary if the schema won't scope); and verify the nested permission block deep-merges rather than replacing the user's permission config (+ a test).

bash/edit are redundant-as-defaults but harmless — leave them; scoping bash is fiddly given the compound mkdir … && nohup amico-run … launch.

AGENTS.md guidance is solid and the N / steps-per-ns numbers are consistent — two prose fixes: "do not hand-roll a coupled multi-transmon system (it will not construct)" — MultiTransmonSystem does exist and would construct; say "out of scope for this single-lab build." And "the integrator stiffens" for levels ≥5 is loose — it's conditioning/leakage/cost.

Tests are content/string guards — they assert the keys/words are emitted, not that opencode honors them, and nothing covers the permission merge.

permission: {
bash: "allow",
edit: "allow",
webfetch: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] webfetch isn't used by the solve workflow (it never fetches a URL) and already defaults to allow, so this line only forces egress back on for a user who'd restricted it globally. The new test omits it too. Drop it:

Suggested change
webfetch: "allow",

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped in 4942a2bwebfetch is gone (and edit/bash left as you suggested).

bash: "allow",
edit: "allow",
webfetch: "allow",
external_directory: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] This is the line that fixes the hang — correct, external_directory defaults to "ask" in 1.17.x. But "allow" grants the agent's file tools the entire filesystem outside the session cwd. The agent itself needs only two roots: it reads the bundled template (opts.templateSrc) and writes scratch to /tmp/amicode-work (amico-run's writes to ~/.amico/runs / ~/.amico/julia are the subprocess's, not the agent's). If opencode 1.17.3 supports path-scoped external_directory, scope it:

external_directory: {
"/tmp/amicode-work/**": "allow",
"<extension templates dir>/**": "allow",
},

If it only takes a flat string, keep "allow" but document the trust boundary.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scoped in 4942a2b. external_directory is now an object keyed to the two roots the agent's file tools touch — the bundled templates dir (path.dirname(templatePath) + the exact template file) and /tmp/amicode-work (plus /private/tmp/amicode-work for macOS's /tmp→/private/tmp symlink). amico-run's own ~/.amico writes are the subprocess's, not the agent's, so they're not granted. The path-scoped object form is accepted by 1.17.3 — verified via opencode debug config (resolved config preserves the per-path keys). Threading templatePath through buildOpencodeConfigContent so the dir is computed at runtime (correct for both dev and the installed VSIX).

return JSON.stringify({
$schema: "https://opencode.ai/config.json",
instructions: [agentsPath],
permission: {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] Verify the merge for this nested object. The doc-comment says opencode merges the injected config over the user's — but that was only confirmed for top-level model/provider and the instructions array (#12). If opencode shallow-replaces rather than deep-merges, injecting permission here wipes any other permission keys the user set globally (doom_loop, per-tool patterns). Nothing tests this — confirm against 1.17.3 and add a test that the user's other permission keys survive.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified — opencode 1.17.3 deep-merges the nested permission object; it does not shallow-replace. Tested with opencode debug config: a global permission.doom_loop:"deny" survives alongside our injected external_directory keys. Added an integration test (opencode_config.test.ts → 'opencode permission merge (1.17.3)') that runs the vendored binary's debug config with a global config + our injected content and asserts the user's key survives; skipIf the binary is absent. So injecting permission adds keys without clobbering the user's other permission settings.

…drop webfetch, verify deep-merge
- Scope `external_directory` (least privilege) to the two roots the agent's file
tools actually touch: the bundled templates dir (reads the solve template) and
/tmp/amicode-work (writes solve.jl). Path-scoped object form verified accepted
by opencode 1.17.3 via `opencode debug config`. Includes /private/tmp/** for
macOS's /tmp symlink. amico-run's own ~/.amico writes are the subprocess's, not
the agent's, so they need no grant.
- Drop `webfetch` (unused by the solve flow; already defaults to allow). Keep
bash/edit per Jack (redundant-as-defaults but harmless; bash launch not worth
scoping).
- Verify the permission merge: opencode 1.17.3 DEEP-merges the injected
`permission` object over the user's global config (a global permission.doom_loop
survives). Added an integration test (runs the vendored binary's `debug config`,
skipped when absent) so this can't regress to a clobber.
- AGENTS.md prose fixes: multi-qubit is "out of scope for this single-lab build"
(MultiTransmonSystem does exist/construct), and levels≥5 degrades via
conditioning/leakage/cost, not "integrator stiffens".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Rchari1

Copy link
Copy Markdown
MemberAuthor

Thanks @jack-champagne — all addressed in 4942a2b:

  • webfetch dropped; bash/edit kept.
  • external_directory scoped to the template dir + /tmp/amicode-work (+ /private/tmp for the macOS symlink); path-scoped object form verified accepted by 1.17.3 via opencode debug config.
  • deep-merge verified — opencode 1.17.3 deep-merges permission (a global doom_loop survives alongside our injected keys). Added an integration test that runs the vendored binary's debug config and asserts the user's key survives (skips when the binary's absent).
  • AGENTS.md: multi-qubit reworded to "out of scope for this single-lab build" (MultiTransmonSystem does construct); levels≥5 reworded to conditioning/leakage/cost.

44 tests green (incl. the new merge test). One note: the runtime enforcement of the scoped globs (vs schema-acceptance, which is verified) I'll smoke-test on the dev host with a live solve before this merges — if a path canonicalization edge re-introduces the prompt, the documented-allow fallback you sanctioned is a 2-line change. Re-requesting review.

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three addressed in 4942a2b — verified against the diff:

  • webfetch dropped; bash/edit left at allow as agreed.
  • external_directory scoped to the template file + its dir + /tmp/amicode-work (and /private/tmp for the macOS symlink). Blanket allow gone, amico-run's own ~/.amico writes correctly ungranted.
  • deep-merge confirmed empirically via opencode debug config + integration test.

One follow-up, not blocking: the merge test is describe.skipIf(!existsSync(OC_BIN)) — it silently skips when the vendored binary is absent, so in CI before fetch:opencode it's green-without-verifying (same false-green class as #25). Sequence that test job behind fetch:opencode or the guard is decorative.

Good to merge.

@jack-champagne
jack-champagne merged commit 4249162 into mainJun 26, 2026
3 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Rchari1@jack-champagne
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance - #22

Merged
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes
Jun 26, 2026
Merged

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance#22
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes

Conversation

@Rchari1

Copy link
Copy Markdown
Member

Findings + fixes from an autonomous QA campaign on the chat→solve path (drove the real opencode backend headlessly with a matrix of user requests; full notes in vault qa-20260620-amicode-chat-solves.md). Stacked on #14; base rchari/template-plot-pulse. Two real bugs fixed, two limitations mitigated by guidance.

F1 (BLOCKER) — chat hung on a permission prompt for every solve

The instruction-injection fix (#12) made the agent read the bundled template at an absolute path outside the session dir + write /tmp/amicode-work. opencode defaults external_directory to ask, so with no interactive approver the turn hangs forever (headless) and nags on every solve (GUI) — the "trouble running gates" symptom returning. It only worked before because the GUI popup was clicked (always-remembered).
Fix:buildOpencodeConfigContent injects permission: { bash, edit, webfetch, external_directory: allow }.
Verified: baseline X-gate chat → authors solve.jl → launches amico-runF=0.99994.

F4 — multi-qubit requests: agent hallucinated an API and crashed

A CNOT request → the agent didn't decline; it hand-rolled TransmonSystem(ω=[ω1,ω2], g=…) (a multi-qubit API that doesn't exist) → TypeError: keyword ω expected Float64, got Vector{Float64}, while claiming "solve launched" with a tidy params table.
Fix: AGENTS.md now scopes the build to single-qubit only and tells the agent to decline multi-qubit gates.
Verified: re-asking for a CNOT now returns "…not supported in this build. TransmonSystem models a single qubit only… If you have a single-qubit gate (X, H, T, …) I can help" — no crash, no solve launched.

F3 — long gates were silently under-resolved

T=30 ns with the default N=50 (0.6 ns/step) → objective plateaus → F=0.914 reported as success.
Fix: AGENTS.md guidance to keep ~5–10 timesteps/ns and scale N with gate time.
Verified: asking for a 30 ns gate now → agent authors N=200 (~6.7 steps/ns).

F2 — levels ≥5 is init/iteration-sensitive (soft)

levels=5 stalled (f≈69) on one seed with the default max_iter=60, but converged to F=0.99999 when the agent bumped max_iter=200. Not broken — the template default is too low for 5 levels, and the agent already compensates. AGENTS.md adds regime guidance (prefer 3–4 levels; warn on 5+). Deeper physics fixes (energy-shift / DRAG warm-start / fixed seed) deferred.

What's already good (no change needed)

Correct gate translation (H/Y/Z), sane defaults on underspecified prompts, and genuinely smart parameter adaptation — fast 4 ns gate → raised N + drive_max + max_iter (F≈1.0); 5 levels → raised max_iter (F=0.99999). Template gates X/Y/Z/H + tight drive bound all converge F > 0.999.

Tests

40 extension tests green; new guards on the permission block + the single-qubit scoping + regime guidance.

🤖 Generated with Claude Code

@Rchari1

Copy link
Copy Markdown
MemberAuthor

@jack-champagne

Rchari1and others added 2 commits June 20, 2026 21:44
…rectory hang)
The instruction-injection fix made the agent read the bundled template at an
absolute path outside the session cwd + write /tmp/amicode-work scratch.
opencode defaults external_directory to 'ask' → the turn hangs forever with no
interactive approver (headless) and nags on every solve (GUI). buildOpencodeConfigContent
now injects permission:{bash,edit,webfetch,external_directory: allow} so the
controlled solve workflow proceeds without prompts.
Found via headless chat QA (qa-20260620): baseline X-gate chat stalled on the
external_directory ask; with the permission block the agent reads the template,
mkdirs scratch, authors solve.jl, and launches amico-run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2/F3/F4)
QA (qa-20260620) found the agent over-reaches on out-of-scope requests:
- F4: a 2-qubit (CNOT) request → the agent HALLUCINATES a multi-qubit API
(TransmonSystem(ω=[..],g=..)) → TypeError crash, while claiming 'solve launched'.
- F3: a 30 ns gate with the default N=50 is under-resolved → F=0.914 reported as success.
- F2: levels=5 stalls (integrator stiffness) at f~69.
AGENTS.md now: declares the build single-qubit only and tells the agent to
decline multi-qubit gates instead of authoring a crashing script; and gives
regime guidance (avoid 5+ levels; keep ~5-10 timesteps/ns so longer gates scale
N up; bump max_iter for harder cases). Guard tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jack-champagne

Copy link
Copy Markdown
Member

Correct root-cause: external_directory is one of the few opencode permissions defaulting to "ask", so headless it hangs / nags per solve. Fix is in the right place. Two trims worth making here:

  • [important, in-PR] external_directory: "allow" is a blanket grant to the whole filesystem outside the workspace. Scope it to the roots actually used (/tmp/amicode-work + the bundled template dir) — opencode supports per-path patterns. As-is it's broader than a single-purpose assistant needs.
  • [minor] webfetch: "allow" and edit: "allow" are redundant (both default to allow), and webfetch grants network egress the solve flow never uses. Drop webfetch; the load-bearing line is external_directory alone.

Note this doesn't make chat→solve work end-to-end on its own — the #11 author/run path mismatch was still live here and is reconciled in #23 (now validated).

@Rchari1
Rchari1 changed the base branch from rchari/template-plot-pulse to mainJune 26, 2026 20:11

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The external_directory fix is correctly diagnosed and lands the blocker. Three changes (inline): drop the unused webfetch allow; scope external_directory to the two roots the agent actually touches (or document the trust boundary if the schema won't scope); and verify the nested permission block deep-merges rather than replacing the user's permission config (+ a test).

bash/edit are redundant-as-defaults but harmless — leave them; scoping bash is fiddly given the compound mkdir … && nohup amico-run … launch.

AGENTS.md guidance is solid and the N / steps-per-ns numbers are consistent — two prose fixes: "do not hand-roll a coupled multi-transmon system (it will not construct)" — MultiTransmonSystem does exist and would construct; say "out of scope for this single-lab build." And "the integrator stiffens" for levels ≥5 is loose — it's conditioning/leakage/cost.

Tests are content/string guards — they assert the keys/words are emitted, not that opencode honors them, and nothing covers the permission merge.

permission: {
bash: "allow",
edit: "allow",
webfetch: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] webfetch isn't used by the solve workflow (it never fetches a URL) and already defaults to allow, so this line only forces egress back on for a user who'd restricted it globally. The new test omits it too. Drop it:

Suggested change
webfetch: "allow",

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped in 4942a2bwebfetch is gone (and edit/bash left as you suggested).

bash: "allow",
edit: "allow",
webfetch: "allow",
external_directory: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] This is the line that fixes the hang — correct, external_directory defaults to "ask" in 1.17.x. But "allow" grants the agent's file tools the entire filesystem outside the session cwd. The agent itself needs only two roots: it reads the bundled template (opts.templateSrc) and writes scratch to /tmp/amicode-work (amico-run's writes to ~/.amico/runs / ~/.amico/julia are the subprocess's, not the agent's). If opencode 1.17.3 supports path-scoped external_directory, scope it:

external_directory: {
"/tmp/amicode-work/**": "allow",
"<extension templates dir>/**": "allow",
},

If it only takes a flat string, keep "allow" but document the trust boundary.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scoped in 4942a2b. external_directory is now an object keyed to the two roots the agent's file tools touch — the bundled templates dir (path.dirname(templatePath) + the exact template file) and /tmp/amicode-work (plus /private/tmp/amicode-work for macOS's /tmp→/private/tmp symlink). amico-run's own ~/.amico writes are the subprocess's, not the agent's, so they're not granted. The path-scoped object form is accepted by 1.17.3 — verified via opencode debug config (resolved config preserves the per-path keys). Threading templatePath through buildOpencodeConfigContent so the dir is computed at runtime (correct for both dev and the installed VSIX).

return JSON.stringify({
$schema: "https://opencode.ai/config.json",
instructions: [agentsPath],
permission: {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] Verify the merge for this nested object. The doc-comment says opencode merges the injected config over the user's — but that was only confirmed for top-level model/provider and the instructions array (#12). If opencode shallow-replaces rather than deep-merges, injecting permission here wipes any other permission keys the user set globally (doom_loop, per-tool patterns). Nothing tests this — confirm against 1.17.3 and add a test that the user's other permission keys survive.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified — opencode 1.17.3 deep-merges the nested permission object; it does not shallow-replace. Tested with opencode debug config: a global permission.doom_loop:"deny" survives alongside our injected external_directory keys. Added an integration test (opencode_config.test.ts → 'opencode permission merge (1.17.3)') that runs the vendored binary's debug config with a global config + our injected content and asserts the user's key survives; skipIf the binary is absent. So injecting permission adds keys without clobbering the user's other permission settings.

…drop webfetch, verify deep-merge
- Scope `external_directory` (least privilege) to the two roots the agent's file
tools actually touch: the bundled templates dir (reads the solve template) and
/tmp/amicode-work (writes solve.jl). Path-scoped object form verified accepted
by opencode 1.17.3 via `opencode debug config`. Includes /private/tmp/** for
macOS's /tmp symlink. amico-run's own ~/.amico writes are the subprocess's, not
the agent's, so they need no grant.
- Drop `webfetch` (unused by the solve flow; already defaults to allow). Keep
bash/edit per Jack (redundant-as-defaults but harmless; bash launch not worth
scoping).
- Verify the permission merge: opencode 1.17.3 DEEP-merges the injected
`permission` object over the user's global config (a global permission.doom_loop
survives). Added an integration test (runs the vendored binary's `debug config`,
skipped when absent) so this can't regress to a clobber.
- AGENTS.md prose fixes: multi-qubit is "out of scope for this single-lab build"
(MultiTransmonSystem does exist/construct), and levels≥5 degrades via
conditioning/leakage/cost, not "integrator stiffens".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Rchari1

Copy link
Copy Markdown
MemberAuthor

Thanks @jack-champagne — all addressed in 4942a2b:

  • webfetch dropped; bash/edit kept.
  • external_directory scoped to the template dir + /tmp/amicode-work (+ /private/tmp for the macOS symlink); path-scoped object form verified accepted by 1.17.3 via opencode debug config.
  • deep-merge verified — opencode 1.17.3 deep-merges permission (a global doom_loop survives alongside our injected keys). Added an integration test that runs the vendored binary's debug config and asserts the user's key survives (skips when the binary's absent).
  • AGENTS.md: multi-qubit reworded to "out of scope for this single-lab build" (MultiTransmonSystem does construct); levels≥5 reworded to conditioning/leakage/cost.

44 tests green (incl. the new merge test). One note: the runtime enforcement of the scoped globs (vs schema-acceptance, which is verified) I'll smoke-test on the dev host with a live solve before this merges — if a path canonicalization edge re-introduces the prompt, the documented-allow fallback you sanctioned is a 2-line change. Re-requesting review.

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three addressed in 4942a2b — verified against the diff:

  • webfetch dropped; bash/edit left at allow as agreed.
  • external_directory scoped to the template file + its dir + /tmp/amicode-work (and /private/tmp for the macOS symlink). Blanket allow gone, amico-run's own ~/.amico writes correctly ungranted.
  • deep-merge confirmed empirically via opencode debug config + integration test.

One follow-up, not blocking: the merge test is describe.skipIf(!existsSync(OC_BIN)) — it silently skips when the vendored binary is absent, so in CI before fetch:opencode it's green-without-verifying (same false-green class as #25). Sequence that test job behind fetch:opencode or the guard is decorative.

Good to merge.

@jack-champagne
jack-champagne merged commit 4249162 into mainJun 26, 2026
3 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Rchari1@jack-champagne
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance - #22

Merged
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes
Jun 26, 2026
Merged

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance#22
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes

Conversation

@Rchari1

Copy link
Copy Markdown
Member

Findings + fixes from an autonomous QA campaign on the chat→solve path (drove the real opencode backend headlessly with a matrix of user requests; full notes in vault qa-20260620-amicode-chat-solves.md). Stacked on #14; base rchari/template-plot-pulse. Two real bugs fixed, two limitations mitigated by guidance.

F1 (BLOCKER) — chat hung on a permission prompt for every solve

The instruction-injection fix (#12) made the agent read the bundled template at an absolute path outside the session dir + write /tmp/amicode-work. opencode defaults external_directory to ask, so with no interactive approver the turn hangs forever (headless) and nags on every solve (GUI) — the "trouble running gates" symptom returning. It only worked before because the GUI popup was clicked (always-remembered).
Fix:buildOpencodeConfigContent injects permission: { bash, edit, webfetch, external_directory: allow }.
Verified: baseline X-gate chat → authors solve.jl → launches amico-runF=0.99994.

F4 — multi-qubit requests: agent hallucinated an API and crashed

A CNOT request → the agent didn't decline; it hand-rolled TransmonSystem(ω=[ω1,ω2], g=…) (a multi-qubit API that doesn't exist) → TypeError: keyword ω expected Float64, got Vector{Float64}, while claiming "solve launched" with a tidy params table.
Fix: AGENTS.md now scopes the build to single-qubit only and tells the agent to decline multi-qubit gates.
Verified: re-asking for a CNOT now returns "…not supported in this build. TransmonSystem models a single qubit only… If you have a single-qubit gate (X, H, T, …) I can help" — no crash, no solve launched.

F3 — long gates were silently under-resolved

T=30 ns with the default N=50 (0.6 ns/step) → objective plateaus → F=0.914 reported as success.
Fix: AGENTS.md guidance to keep ~5–10 timesteps/ns and scale N with gate time.
Verified: asking for a 30 ns gate now → agent authors N=200 (~6.7 steps/ns).

F2 — levels ≥5 is init/iteration-sensitive (soft)

levels=5 stalled (f≈69) on one seed with the default max_iter=60, but converged to F=0.99999 when the agent bumped max_iter=200. Not broken — the template default is too low for 5 levels, and the agent already compensates. AGENTS.md adds regime guidance (prefer 3–4 levels; warn on 5+). Deeper physics fixes (energy-shift / DRAG warm-start / fixed seed) deferred.

What's already good (no change needed)

Correct gate translation (H/Y/Z), sane defaults on underspecified prompts, and genuinely smart parameter adaptation — fast 4 ns gate → raised N + drive_max + max_iter (F≈1.0); 5 levels → raised max_iter (F=0.99999). Template gates X/Y/Z/H + tight drive bound all converge F > 0.999.

Tests

40 extension tests green; new guards on the permission block + the single-qubit scoping + regime guidance.

🤖 Generated with Claude Code

@Rchari1

Copy link
Copy Markdown
MemberAuthor

@jack-champagne

Rchari1and others added 2 commits June 20, 2026 21:44
…rectory hang)
The instruction-injection fix made the agent read the bundled template at an
absolute path outside the session cwd + write /tmp/amicode-work scratch.
opencode defaults external_directory to 'ask' → the turn hangs forever with no
interactive approver (headless) and nags on every solve (GUI). buildOpencodeConfigContent
now injects permission:{bash,edit,webfetch,external_directory: allow} so the
controlled solve workflow proceeds without prompts.
Found via headless chat QA (qa-20260620): baseline X-gate chat stalled on the
external_directory ask; with the permission block the agent reads the template,
mkdirs scratch, authors solve.jl, and launches amico-run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2/F3/F4)
QA (qa-20260620) found the agent over-reaches on out-of-scope requests:
- F4: a 2-qubit (CNOT) request → the agent HALLUCINATES a multi-qubit API
(TransmonSystem(ω=[..],g=..)) → TypeError crash, while claiming 'solve launched'.
- F3: a 30 ns gate with the default N=50 is under-resolved → F=0.914 reported as success.
- F2: levels=5 stalls (integrator stiffness) at f~69.
AGENTS.md now: declares the build single-qubit only and tells the agent to
decline multi-qubit gates instead of authoring a crashing script; and gives
regime guidance (avoid 5+ levels; keep ~5-10 timesteps/ns so longer gates scale
N up; bump max_iter for harder cases). Guard tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jack-champagne

Copy link
Copy Markdown
Member

Correct root-cause: external_directory is one of the few opencode permissions defaulting to "ask", so headless it hangs / nags per solve. Fix is in the right place. Two trims worth making here:

  • [important, in-PR] external_directory: "allow" is a blanket grant to the whole filesystem outside the workspace. Scope it to the roots actually used (/tmp/amicode-work + the bundled template dir) — opencode supports per-path patterns. As-is it's broader than a single-purpose assistant needs.
  • [minor] webfetch: "allow" and edit: "allow" are redundant (both default to allow), and webfetch grants network egress the solve flow never uses. Drop webfetch; the load-bearing line is external_directory alone.

Note this doesn't make chat→solve work end-to-end on its own — the #11 author/run path mismatch was still live here and is reconciled in #23 (now validated).

@Rchari1
Rchari1 changed the base branch from rchari/template-plot-pulse to mainJune 26, 2026 20:11

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The external_directory fix is correctly diagnosed and lands the blocker. Three changes (inline): drop the unused webfetch allow; scope external_directory to the two roots the agent actually touches (or document the trust boundary if the schema won't scope); and verify the nested permission block deep-merges rather than replacing the user's permission config (+ a test).

bash/edit are redundant-as-defaults but harmless — leave them; scoping bash is fiddly given the compound mkdir … && nohup amico-run … launch.

AGENTS.md guidance is solid and the N / steps-per-ns numbers are consistent — two prose fixes: "do not hand-roll a coupled multi-transmon system (it will not construct)" — MultiTransmonSystem does exist and would construct; say "out of scope for this single-lab build." And "the integrator stiffens" for levels ≥5 is loose — it's conditioning/leakage/cost.

Tests are content/string guards — they assert the keys/words are emitted, not that opencode honors them, and nothing covers the permission merge.

permission: {
bash: "allow",
edit: "allow",
webfetch: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] webfetch isn't used by the solve workflow (it never fetches a URL) and already defaults to allow, so this line only forces egress back on for a user who'd restricted it globally. The new test omits it too. Drop it:

Suggested change
webfetch: "allow",

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped in 4942a2bwebfetch is gone (and edit/bash left as you suggested).

bash: "allow",
edit: "allow",
webfetch: "allow",
external_directory: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] This is the line that fixes the hang — correct, external_directory defaults to "ask" in 1.17.x. But "allow" grants the agent's file tools the entire filesystem outside the session cwd. The agent itself needs only two roots: it reads the bundled template (opts.templateSrc) and writes scratch to /tmp/amicode-work (amico-run's writes to ~/.amico/runs / ~/.amico/julia are the subprocess's, not the agent's). If opencode 1.17.3 supports path-scoped external_directory, scope it:

external_directory: {
"/tmp/amicode-work/**": "allow",
"<extension templates dir>/**": "allow",
},

If it only takes a flat string, keep "allow" but document the trust boundary.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scoped in 4942a2b. external_directory is now an object keyed to the two roots the agent's file tools touch — the bundled templates dir (path.dirname(templatePath) + the exact template file) and /tmp/amicode-work (plus /private/tmp/amicode-work for macOS's /tmp→/private/tmp symlink). amico-run's own ~/.amico writes are the subprocess's, not the agent's, so they're not granted. The path-scoped object form is accepted by 1.17.3 — verified via opencode debug config (resolved config preserves the per-path keys). Threading templatePath through buildOpencodeConfigContent so the dir is computed at runtime (correct for both dev and the installed VSIX).

return JSON.stringify({
$schema: "https://opencode.ai/config.json",
instructions: [agentsPath],
permission: {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] Verify the merge for this nested object. The doc-comment says opencode merges the injected config over the user's — but that was only confirmed for top-level model/provider and the instructions array (#12). If opencode shallow-replaces rather than deep-merges, injecting permission here wipes any other permission keys the user set globally (doom_loop, per-tool patterns). Nothing tests this — confirm against 1.17.3 and add a test that the user's other permission keys survive.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified — opencode 1.17.3 deep-merges the nested permission object; it does not shallow-replace. Tested with opencode debug config: a global permission.doom_loop:"deny" survives alongside our injected external_directory keys. Added an integration test (opencode_config.test.ts → 'opencode permission merge (1.17.3)') that runs the vendored binary's debug config with a global config + our injected content and asserts the user's key survives; skipIf the binary is absent. So injecting permission adds keys without clobbering the user's other permission settings.

…drop webfetch, verify deep-merge
- Scope `external_directory` (least privilege) to the two roots the agent's file
tools actually touch: the bundled templates dir (reads the solve template) and
/tmp/amicode-work (writes solve.jl). Path-scoped object form verified accepted
by opencode 1.17.3 via `opencode debug config`. Includes /private/tmp/** for
macOS's /tmp symlink. amico-run's own ~/.amico writes are the subprocess's, not
the agent's, so they need no grant.
- Drop `webfetch` (unused by the solve flow; already defaults to allow). Keep
bash/edit per Jack (redundant-as-defaults but harmless; bash launch not worth
scoping).
- Verify the permission merge: opencode 1.17.3 DEEP-merges the injected
`permission` object over the user's global config (a global permission.doom_loop
survives). Added an integration test (runs the vendored binary's `debug config`,
skipped when absent) so this can't regress to a clobber.
- AGENTS.md prose fixes: multi-qubit is "out of scope for this single-lab build"
(MultiTransmonSystem does exist/construct), and levels≥5 degrades via
conditioning/leakage/cost, not "integrator stiffens".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Rchari1

Copy link
Copy Markdown
MemberAuthor

Thanks @jack-champagne — all addressed in 4942a2b:

  • webfetch dropped; bash/edit kept.
  • external_directory scoped to the template dir + /tmp/amicode-work (+ /private/tmp for the macOS symlink); path-scoped object form verified accepted by 1.17.3 via opencode debug config.
  • deep-merge verified — opencode 1.17.3 deep-merges permission (a global doom_loop survives alongside our injected keys). Added an integration test that runs the vendored binary's debug config and asserts the user's key survives (skips when the binary's absent).
  • AGENTS.md: multi-qubit reworded to "out of scope for this single-lab build" (MultiTransmonSystem does construct); levels≥5 reworded to conditioning/leakage/cost.

44 tests green (incl. the new merge test). One note: the runtime enforcement of the scoped globs (vs schema-acceptance, which is verified) I'll smoke-test on the dev host with a live solve before this merges — if a path canonicalization edge re-introduces the prompt, the documented-allow fallback you sanctioned is a 2-line change. Re-requesting review.

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three addressed in 4942a2b — verified against the diff:

  • webfetch dropped; bash/edit left at allow as agreed.
  • external_directory scoped to the template file + its dir + /tmp/amicode-work (and /private/tmp for the macOS symlink). Blanket allow gone, amico-run's own ~/.amico writes correctly ungranted.
  • deep-merge confirmed empirically via opencode debug config + integration test.

One follow-up, not blocking: the merge test is describe.skipIf(!existsSync(OC_BIN)) — it silently skips when the vendored binary is absent, so in CI before fetch:opencode it's green-without-verifying (same false-green class as #25). Sequence that test job behind fetch:opencode or the guard is decorative.

Good to merge.

@jack-champagne
jack-champagne merged commit 4249162 into mainJun 26, 2026
3 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Rchari1@jack-champagne
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance - #22

Merged
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes
Jun 26, 2026
Merged

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance#22
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes

Conversation

@Rchari1

Copy link
Copy Markdown
Member

Findings + fixes from an autonomous QA campaign on the chat→solve path (drove the real opencode backend headlessly with a matrix of user requests; full notes in vault qa-20260620-amicode-chat-solves.md). Stacked on #14; base rchari/template-plot-pulse. Two real bugs fixed, two limitations mitigated by guidance.

F1 (BLOCKER) — chat hung on a permission prompt for every solve

The instruction-injection fix (#12) made the agent read the bundled template at an absolute path outside the session dir + write /tmp/amicode-work. opencode defaults external_directory to ask, so with no interactive approver the turn hangs forever (headless) and nags on every solve (GUI) — the "trouble running gates" symptom returning. It only worked before because the GUI popup was clicked (always-remembered).
Fix:buildOpencodeConfigContent injects permission: { bash, edit, webfetch, external_directory: allow }.
Verified: baseline X-gate chat → authors solve.jl → launches amico-runF=0.99994.

F4 — multi-qubit requests: agent hallucinated an API and crashed

A CNOT request → the agent didn't decline; it hand-rolled TransmonSystem(ω=[ω1,ω2], g=…) (a multi-qubit API that doesn't exist) → TypeError: keyword ω expected Float64, got Vector{Float64}, while claiming "solve launched" with a tidy params table.
Fix: AGENTS.md now scopes the build to single-qubit only and tells the agent to decline multi-qubit gates.
Verified: re-asking for a CNOT now returns "…not supported in this build. TransmonSystem models a single qubit only… If you have a single-qubit gate (X, H, T, …) I can help" — no crash, no solve launched.

F3 — long gates were silently under-resolved

T=30 ns with the default N=50 (0.6 ns/step) → objective plateaus → F=0.914 reported as success.
Fix: AGENTS.md guidance to keep ~5–10 timesteps/ns and scale N with gate time.
Verified: asking for a 30 ns gate now → agent authors N=200 (~6.7 steps/ns).

F2 — levels ≥5 is init/iteration-sensitive (soft)

levels=5 stalled (f≈69) on one seed with the default max_iter=60, but converged to F=0.99999 when the agent bumped max_iter=200. Not broken — the template default is too low for 5 levels, and the agent already compensates. AGENTS.md adds regime guidance (prefer 3–4 levels; warn on 5+). Deeper physics fixes (energy-shift / DRAG warm-start / fixed seed) deferred.

What's already good (no change needed)

Correct gate translation (H/Y/Z), sane defaults on underspecified prompts, and genuinely smart parameter adaptation — fast 4 ns gate → raised N + drive_max + max_iter (F≈1.0); 5 levels → raised max_iter (F=0.99999). Template gates X/Y/Z/H + tight drive bound all converge F > 0.999.

Tests

40 extension tests green; new guards on the permission block + the single-qubit scoping + regime guidance.

🤖 Generated with Claude Code

@Rchari1

Copy link
Copy Markdown
MemberAuthor

@jack-champagne

Rchari1and others added 2 commits June 20, 2026 21:44
…rectory hang)
The instruction-injection fix made the agent read the bundled template at an
absolute path outside the session cwd + write /tmp/amicode-work scratch.
opencode defaults external_directory to 'ask' → the turn hangs forever with no
interactive approver (headless) and nags on every solve (GUI). buildOpencodeConfigContent
now injects permission:{bash,edit,webfetch,external_directory: allow} so the
controlled solve workflow proceeds without prompts.
Found via headless chat QA (qa-20260620): baseline X-gate chat stalled on the
external_directory ask; with the permission block the agent reads the template,
mkdirs scratch, authors solve.jl, and launches amico-run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2/F3/F4)
QA (qa-20260620) found the agent over-reaches on out-of-scope requests:
- F4: a 2-qubit (CNOT) request → the agent HALLUCINATES a multi-qubit API
(TransmonSystem(ω=[..],g=..)) → TypeError crash, while claiming 'solve launched'.
- F3: a 30 ns gate with the default N=50 is under-resolved → F=0.914 reported as success.
- F2: levels=5 stalls (integrator stiffness) at f~69.
AGENTS.md now: declares the build single-qubit only and tells the agent to
decline multi-qubit gates instead of authoring a crashing script; and gives
regime guidance (avoid 5+ levels; keep ~5-10 timesteps/ns so longer gates scale
N up; bump max_iter for harder cases). Guard tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jack-champagne

Copy link
Copy Markdown
Member

Correct root-cause: external_directory is one of the few opencode permissions defaulting to "ask", so headless it hangs / nags per solve. Fix is in the right place. Two trims worth making here:

  • [important, in-PR] external_directory: "allow" is a blanket grant to the whole filesystem outside the workspace. Scope it to the roots actually used (/tmp/amicode-work + the bundled template dir) — opencode supports per-path patterns. As-is it's broader than a single-purpose assistant needs.
  • [minor] webfetch: "allow" and edit: "allow" are redundant (both default to allow), and webfetch grants network egress the solve flow never uses. Drop webfetch; the load-bearing line is external_directory alone.

Note this doesn't make chat→solve work end-to-end on its own — the #11 author/run path mismatch was still live here and is reconciled in #23 (now validated).

@Rchari1
Rchari1 changed the base branch from rchari/template-plot-pulse to mainJune 26, 2026 20:11

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The external_directory fix is correctly diagnosed and lands the blocker. Three changes (inline): drop the unused webfetch allow; scope external_directory to the two roots the agent actually touches (or document the trust boundary if the schema won't scope); and verify the nested permission block deep-merges rather than replacing the user's permission config (+ a test).

bash/edit are redundant-as-defaults but harmless — leave them; scoping bash is fiddly given the compound mkdir … && nohup amico-run … launch.

AGENTS.md guidance is solid and the N / steps-per-ns numbers are consistent — two prose fixes: "do not hand-roll a coupled multi-transmon system (it will not construct)" — MultiTransmonSystem does exist and would construct; say "out of scope for this single-lab build." And "the integrator stiffens" for levels ≥5 is loose — it's conditioning/leakage/cost.

Tests are content/string guards — they assert the keys/words are emitted, not that opencode honors them, and nothing covers the permission merge.

permission: {
bash: "allow",
edit: "allow",
webfetch: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] webfetch isn't used by the solve workflow (it never fetches a URL) and already defaults to allow, so this line only forces egress back on for a user who'd restricted it globally. The new test omits it too. Drop it:

Suggested change
webfetch: "allow",

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped in 4942a2bwebfetch is gone (and edit/bash left as you suggested).

bash: "allow",
edit: "allow",
webfetch: "allow",
external_directory: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] This is the line that fixes the hang — correct, external_directory defaults to "ask" in 1.17.x. But "allow" grants the agent's file tools the entire filesystem outside the session cwd. The agent itself needs only two roots: it reads the bundled template (opts.templateSrc) and writes scratch to /tmp/amicode-work (amico-run's writes to ~/.amico/runs / ~/.amico/julia are the subprocess's, not the agent's). If opencode 1.17.3 supports path-scoped external_directory, scope it:

external_directory: {
"/tmp/amicode-work/**": "allow",
"<extension templates dir>/**": "allow",
},

If it only takes a flat string, keep "allow" but document the trust boundary.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scoped in 4942a2b. external_directory is now an object keyed to the two roots the agent's file tools touch — the bundled templates dir (path.dirname(templatePath) + the exact template file) and /tmp/amicode-work (plus /private/tmp/amicode-work for macOS's /tmp→/private/tmp symlink). amico-run's own ~/.amico writes are the subprocess's, not the agent's, so they're not granted. The path-scoped object form is accepted by 1.17.3 — verified via opencode debug config (resolved config preserves the per-path keys). Threading templatePath through buildOpencodeConfigContent so the dir is computed at runtime (correct for both dev and the installed VSIX).

return JSON.stringify({
$schema: "https://opencode.ai/config.json",
instructions: [agentsPath],
permission: {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] Verify the merge for this nested object. The doc-comment says opencode merges the injected config over the user's — but that was only confirmed for top-level model/provider and the instructions array (#12). If opencode shallow-replaces rather than deep-merges, injecting permission here wipes any other permission keys the user set globally (doom_loop, per-tool patterns). Nothing tests this — confirm against 1.17.3 and add a test that the user's other permission keys survive.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified — opencode 1.17.3 deep-merges the nested permission object; it does not shallow-replace. Tested with opencode debug config: a global permission.doom_loop:"deny" survives alongside our injected external_directory keys. Added an integration test (opencode_config.test.ts → 'opencode permission merge (1.17.3)') that runs the vendored binary's debug config with a global config + our injected content and asserts the user's key survives; skipIf the binary is absent. So injecting permission adds keys without clobbering the user's other permission settings.

…drop webfetch, verify deep-merge
- Scope `external_directory` (least privilege) to the two roots the agent's file
tools actually touch: the bundled templates dir (reads the solve template) and
/tmp/amicode-work (writes solve.jl). Path-scoped object form verified accepted
by opencode 1.17.3 via `opencode debug config`. Includes /private/tmp/** for
macOS's /tmp symlink. amico-run's own ~/.amico writes are the subprocess's, not
the agent's, so they need no grant.
- Drop `webfetch` (unused by the solve flow; already defaults to allow). Keep
bash/edit per Jack (redundant-as-defaults but harmless; bash launch not worth
scoping).
- Verify the permission merge: opencode 1.17.3 DEEP-merges the injected
`permission` object over the user's global config (a global permission.doom_loop
survives). Added an integration test (runs the vendored binary's `debug config`,
skipped when absent) so this can't regress to a clobber.
- AGENTS.md prose fixes: multi-qubit is "out of scope for this single-lab build"
(MultiTransmonSystem does exist/construct), and levels≥5 degrades via
conditioning/leakage/cost, not "integrator stiffens".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Rchari1

Copy link
Copy Markdown
MemberAuthor

Thanks @jack-champagne — all addressed in 4942a2b:

  • webfetch dropped; bash/edit kept.
  • external_directory scoped to the template dir + /tmp/amicode-work (+ /private/tmp for the macOS symlink); path-scoped object form verified accepted by 1.17.3 via opencode debug config.
  • deep-merge verified — opencode 1.17.3 deep-merges permission (a global doom_loop survives alongside our injected keys). Added an integration test that runs the vendored binary's debug config and asserts the user's key survives (skips when the binary's absent).
  • AGENTS.md: multi-qubit reworded to "out of scope for this single-lab build" (MultiTransmonSystem does construct); levels≥5 reworded to conditioning/leakage/cost.

44 tests green (incl. the new merge test). One note: the runtime enforcement of the scoped globs (vs schema-acceptance, which is verified) I'll smoke-test on the dev host with a live solve before this merges — if a path canonicalization edge re-introduces the prompt, the documented-allow fallback you sanctioned is a 2-line change. Re-requesting review.

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three addressed in 4942a2b — verified against the diff:

  • webfetch dropped; bash/edit left at allow as agreed.
  • external_directory scoped to the template file + its dir + /tmp/amicode-work (and /private/tmp for the macOS symlink). Blanket allow gone, amico-run's own ~/.amico writes correctly ungranted.
  • deep-merge confirmed empirically via opencode debug config + integration test.

One follow-up, not blocking: the merge test is describe.skipIf(!existsSync(OC_BIN)) — it silently skips when the vendored binary is absent, so in CI before fetch:opencode it's green-without-verifying (same false-green class as #25). Sequence that test job behind fetch:opencode or the guard is decorative.

Good to merge.

@jack-champagne
jack-champagne merged commit 4249162 into mainJun 26, 2026
3 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Rchari1@jack-champagne
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance - #22

Merged
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes
Jun 26, 2026
Merged

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance#22
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes

Conversation

@Rchari1

Copy link
Copy Markdown
Member

Findings + fixes from an autonomous QA campaign on the chat→solve path (drove the real opencode backend headlessly with a matrix of user requests; full notes in vault qa-20260620-amicode-chat-solves.md). Stacked on #14; base rchari/template-plot-pulse. Two real bugs fixed, two limitations mitigated by guidance.

F1 (BLOCKER) — chat hung on a permission prompt for every solve

The instruction-injection fix (#12) made the agent read the bundled template at an absolute path outside the session dir + write /tmp/amicode-work. opencode defaults external_directory to ask, so with no interactive approver the turn hangs forever (headless) and nags on every solve (GUI) — the "trouble running gates" symptom returning. It only worked before because the GUI popup was clicked (always-remembered).
Fix:buildOpencodeConfigContent injects permission: { bash, edit, webfetch, external_directory: allow }.
Verified: baseline X-gate chat → authors solve.jl → launches amico-runF=0.99994.

F4 — multi-qubit requests: agent hallucinated an API and crashed

A CNOT request → the agent didn't decline; it hand-rolled TransmonSystem(ω=[ω1,ω2], g=…) (a multi-qubit API that doesn't exist) → TypeError: keyword ω expected Float64, got Vector{Float64}, while claiming "solve launched" with a tidy params table.
Fix: AGENTS.md now scopes the build to single-qubit only and tells the agent to decline multi-qubit gates.
Verified: re-asking for a CNOT now returns "…not supported in this build. TransmonSystem models a single qubit only… If you have a single-qubit gate (X, H, T, …) I can help" — no crash, no solve launched.

F3 — long gates were silently under-resolved

T=30 ns with the default N=50 (0.6 ns/step) → objective plateaus → F=0.914 reported as success.
Fix: AGENTS.md guidance to keep ~5–10 timesteps/ns and scale N with gate time.
Verified: asking for a 30 ns gate now → agent authors N=200 (~6.7 steps/ns).

F2 — levels ≥5 is init/iteration-sensitive (soft)

levels=5 stalled (f≈69) on one seed with the default max_iter=60, but converged to F=0.99999 when the agent bumped max_iter=200. Not broken — the template default is too low for 5 levels, and the agent already compensates. AGENTS.md adds regime guidance (prefer 3–4 levels; warn on 5+). Deeper physics fixes (energy-shift / DRAG warm-start / fixed seed) deferred.

What's already good (no change needed)

Correct gate translation (H/Y/Z), sane defaults on underspecified prompts, and genuinely smart parameter adaptation — fast 4 ns gate → raised N + drive_max + max_iter (F≈1.0); 5 levels → raised max_iter (F=0.99999). Template gates X/Y/Z/H + tight drive bound all converge F > 0.999.

Tests

40 extension tests green; new guards on the permission block + the single-qubit scoping + regime guidance.

🤖 Generated with Claude Code

@Rchari1

Copy link
Copy Markdown
MemberAuthor

@jack-champagne

Rchari1and others added 2 commits June 20, 2026 21:44
…rectory hang)
The instruction-injection fix made the agent read the bundled template at an
absolute path outside the session cwd + write /tmp/amicode-work scratch.
opencode defaults external_directory to 'ask' → the turn hangs forever with no
interactive approver (headless) and nags on every solve (GUI). buildOpencodeConfigContent
now injects permission:{bash,edit,webfetch,external_directory: allow} so the
controlled solve workflow proceeds without prompts.
Found via headless chat QA (qa-20260620): baseline X-gate chat stalled on the
external_directory ask; with the permission block the agent reads the template,
mkdirs scratch, authors solve.jl, and launches amico-run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2/F3/F4)
QA (qa-20260620) found the agent over-reaches on out-of-scope requests:
- F4: a 2-qubit (CNOT) request → the agent HALLUCINATES a multi-qubit API
(TransmonSystem(ω=[..],g=..)) → TypeError crash, while claiming 'solve launched'.
- F3: a 30 ns gate with the default N=50 is under-resolved → F=0.914 reported as success.
- F2: levels=5 stalls (integrator stiffness) at f~69.
AGENTS.md now: declares the build single-qubit only and tells the agent to
decline multi-qubit gates instead of authoring a crashing script; and gives
regime guidance (avoid 5+ levels; keep ~5-10 timesteps/ns so longer gates scale
N up; bump max_iter for harder cases). Guard tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jack-champagne

Copy link
Copy Markdown
Member

Correct root-cause: external_directory is one of the few opencode permissions defaulting to "ask", so headless it hangs / nags per solve. Fix is in the right place. Two trims worth making here:

  • [important, in-PR] external_directory: "allow" is a blanket grant to the whole filesystem outside the workspace. Scope it to the roots actually used (/tmp/amicode-work + the bundled template dir) — opencode supports per-path patterns. As-is it's broader than a single-purpose assistant needs.
  • [minor] webfetch: "allow" and edit: "allow" are redundant (both default to allow), and webfetch grants network egress the solve flow never uses. Drop webfetch; the load-bearing line is external_directory alone.

Note this doesn't make chat→solve work end-to-end on its own — the #11 author/run path mismatch was still live here and is reconciled in #23 (now validated).

@Rchari1
Rchari1 changed the base branch from rchari/template-plot-pulse to mainJune 26, 2026 20:11

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The external_directory fix is correctly diagnosed and lands the blocker. Three changes (inline): drop the unused webfetch allow; scope external_directory to the two roots the agent actually touches (or document the trust boundary if the schema won't scope); and verify the nested permission block deep-merges rather than replacing the user's permission config (+ a test).

bash/edit are redundant-as-defaults but harmless — leave them; scoping bash is fiddly given the compound mkdir … && nohup amico-run … launch.

AGENTS.md guidance is solid and the N / steps-per-ns numbers are consistent — two prose fixes: "do not hand-roll a coupled multi-transmon system (it will not construct)" — MultiTransmonSystem does exist and would construct; say "out of scope for this single-lab build." And "the integrator stiffens" for levels ≥5 is loose — it's conditioning/leakage/cost.

Tests are content/string guards — they assert the keys/words are emitted, not that opencode honors them, and nothing covers the permission merge.

permission: {
bash: "allow",
edit: "allow",
webfetch: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] webfetch isn't used by the solve workflow (it never fetches a URL) and already defaults to allow, so this line only forces egress back on for a user who'd restricted it globally. The new test omits it too. Drop it:

Suggested change
webfetch: "allow",

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped in 4942a2bwebfetch is gone (and edit/bash left as you suggested).

bash: "allow",
edit: "allow",
webfetch: "allow",
external_directory: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] This is the line that fixes the hang — correct, external_directory defaults to "ask" in 1.17.x. But "allow" grants the agent's file tools the entire filesystem outside the session cwd. The agent itself needs only two roots: it reads the bundled template (opts.templateSrc) and writes scratch to /tmp/amicode-work (amico-run's writes to ~/.amico/runs / ~/.amico/julia are the subprocess's, not the agent's). If opencode 1.17.3 supports path-scoped external_directory, scope it:

external_directory: {
"/tmp/amicode-work/**": "allow",
"<extension templates dir>/**": "allow",
},

If it only takes a flat string, keep "allow" but document the trust boundary.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scoped in 4942a2b. external_directory is now an object keyed to the two roots the agent's file tools touch — the bundled templates dir (path.dirname(templatePath) + the exact template file) and /tmp/amicode-work (plus /private/tmp/amicode-work for macOS's /tmp→/private/tmp symlink). amico-run's own ~/.amico writes are the subprocess's, not the agent's, so they're not granted. The path-scoped object form is accepted by 1.17.3 — verified via opencode debug config (resolved config preserves the per-path keys). Threading templatePath through buildOpencodeConfigContent so the dir is computed at runtime (correct for both dev and the installed VSIX).

return JSON.stringify({
$schema: "https://opencode.ai/config.json",
instructions: [agentsPath],
permission: {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] Verify the merge for this nested object. The doc-comment says opencode merges the injected config over the user's — but that was only confirmed for top-level model/provider and the instructions array (#12). If opencode shallow-replaces rather than deep-merges, injecting permission here wipes any other permission keys the user set globally (doom_loop, per-tool patterns). Nothing tests this — confirm against 1.17.3 and add a test that the user's other permission keys survive.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified — opencode 1.17.3 deep-merges the nested permission object; it does not shallow-replace. Tested with opencode debug config: a global permission.doom_loop:"deny" survives alongside our injected external_directory keys. Added an integration test (opencode_config.test.ts → 'opencode permission merge (1.17.3)') that runs the vendored binary's debug config with a global config + our injected content and asserts the user's key survives; skipIf the binary is absent. So injecting permission adds keys without clobbering the user's other permission settings.

…drop webfetch, verify deep-merge
- Scope `external_directory` (least privilege) to the two roots the agent's file
tools actually touch: the bundled templates dir (reads the solve template) and
/tmp/amicode-work (writes solve.jl). Path-scoped object form verified accepted
by opencode 1.17.3 via `opencode debug config`. Includes /private/tmp/** for
macOS's /tmp symlink. amico-run's own ~/.amico writes are the subprocess's, not
the agent's, so they need no grant.
- Drop `webfetch` (unused by the solve flow; already defaults to allow). Keep
bash/edit per Jack (redundant-as-defaults but harmless; bash launch not worth
scoping).
- Verify the permission merge: opencode 1.17.3 DEEP-merges the injected
`permission` object over the user's global config (a global permission.doom_loop
survives). Added an integration test (runs the vendored binary's `debug config`,
skipped when absent) so this can't regress to a clobber.
- AGENTS.md prose fixes: multi-qubit is "out of scope for this single-lab build"
(MultiTransmonSystem does exist/construct), and levels≥5 degrades via
conditioning/leakage/cost, not "integrator stiffens".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Rchari1

Copy link
Copy Markdown
MemberAuthor

Thanks @jack-champagne — all addressed in 4942a2b:

  • webfetch dropped; bash/edit kept.
  • external_directory scoped to the template dir + /tmp/amicode-work (+ /private/tmp for the macOS symlink); path-scoped object form verified accepted by 1.17.3 via opencode debug config.
  • deep-merge verified — opencode 1.17.3 deep-merges permission (a global doom_loop survives alongside our injected keys). Added an integration test that runs the vendored binary's debug config and asserts the user's key survives (skips when the binary's absent).
  • AGENTS.md: multi-qubit reworded to "out of scope for this single-lab build" (MultiTransmonSystem does construct); levels≥5 reworded to conditioning/leakage/cost.

44 tests green (incl. the new merge test). One note: the runtime enforcement of the scoped globs (vs schema-acceptance, which is verified) I'll smoke-test on the dev host with a live solve before this merges — if a path canonicalization edge re-introduces the prompt, the documented-allow fallback you sanctioned is a 2-line change. Re-requesting review.

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three addressed in 4942a2b — verified against the diff:

  • webfetch dropped; bash/edit left at allow as agreed.
  • external_directory scoped to the template file + its dir + /tmp/amicode-work (and /private/tmp for the macOS symlink). Blanket allow gone, amico-run's own ~/.amico writes correctly ungranted.
  • deep-merge confirmed empirically via opencode debug config + integration test.

One follow-up, not blocking: the merge test is describe.skipIf(!existsSync(OC_BIN)) — it silently skips when the vendored binary is absent, so in CI before fetch:opencode it's green-without-verifying (same false-green class as #25). Sequence that test job behind fetch:opencode or the guard is decorative.

Good to merge.

@jack-champagne
jack-champagne merged commit 4249162 into mainJun 26, 2026
3 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Rchari1@jack-champagne
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance - #22

Merged
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes
Jun 26, 2026
Merged

fix: chat→solve QA — permission-hang blocker + multi-qubit/regime guidance#22
jack-champagne merged 3 commits into
mainfrom
rchari/qa-chat-fixes

Conversation

@Rchari1

Copy link
Copy Markdown
Member

Findings + fixes from an autonomous QA campaign on the chat→solve path (drove the real opencode backend headlessly with a matrix of user requests; full notes in vault qa-20260620-amicode-chat-solves.md). Stacked on #14; base rchari/template-plot-pulse. Two real bugs fixed, two limitations mitigated by guidance.

F1 (BLOCKER) — chat hung on a permission prompt for every solve

The instruction-injection fix (#12) made the agent read the bundled template at an absolute path outside the session dir + write /tmp/amicode-work. opencode defaults external_directory to ask, so with no interactive approver the turn hangs forever (headless) and nags on every solve (GUI) — the "trouble running gates" symptom returning. It only worked before because the GUI popup was clicked (always-remembered).
Fix:buildOpencodeConfigContent injects permission: { bash, edit, webfetch, external_directory: allow }.
Verified: baseline X-gate chat → authors solve.jl → launches amico-runF=0.99994.

F4 — multi-qubit requests: agent hallucinated an API and crashed

A CNOT request → the agent didn't decline; it hand-rolled TransmonSystem(ω=[ω1,ω2], g=…) (a multi-qubit API that doesn't exist) → TypeError: keyword ω expected Float64, got Vector{Float64}, while claiming "solve launched" with a tidy params table.
Fix: AGENTS.md now scopes the build to single-qubit only and tells the agent to decline multi-qubit gates.
Verified: re-asking for a CNOT now returns "…not supported in this build. TransmonSystem models a single qubit only… If you have a single-qubit gate (X, H, T, …) I can help" — no crash, no solve launched.

F3 — long gates were silently under-resolved

T=30 ns with the default N=50 (0.6 ns/step) → objective plateaus → F=0.914 reported as success.
Fix: AGENTS.md guidance to keep ~5–10 timesteps/ns and scale N with gate time.
Verified: asking for a 30 ns gate now → agent authors N=200 (~6.7 steps/ns).

F2 — levels ≥5 is init/iteration-sensitive (soft)

levels=5 stalled (f≈69) on one seed with the default max_iter=60, but converged to F=0.99999 when the agent bumped max_iter=200. Not broken — the template default is too low for 5 levels, and the agent already compensates. AGENTS.md adds regime guidance (prefer 3–4 levels; warn on 5+). Deeper physics fixes (energy-shift / DRAG warm-start / fixed seed) deferred.

What's already good (no change needed)

Correct gate translation (H/Y/Z), sane defaults on underspecified prompts, and genuinely smart parameter adaptation — fast 4 ns gate → raised N + drive_max + max_iter (F≈1.0); 5 levels → raised max_iter (F=0.99999). Template gates X/Y/Z/H + tight drive bound all converge F > 0.999.

Tests

40 extension tests green; new guards on the permission block + the single-qubit scoping + regime guidance.

🤖 Generated with Claude Code

@Rchari1

Copy link
Copy Markdown
MemberAuthor

@jack-champagne

Rchari1and others added 2 commits June 20, 2026 21:44
…rectory hang)
The instruction-injection fix made the agent read the bundled template at an
absolute path outside the session cwd + write /tmp/amicode-work scratch.
opencode defaults external_directory to 'ask' → the turn hangs forever with no
interactive approver (headless) and nags on every solve (GUI). buildOpencodeConfigContent
now injects permission:{bash,edit,webfetch,external_directory: allow} so the
controlled solve workflow proceeds without prompts.
Found via headless chat QA (qa-20260620): baseline X-gate chat stalled on the
external_directory ask; with the permission block the agent reads the template,
mkdirs scratch, authors solve.jl, and launches amico-run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2/F3/F4)
QA (qa-20260620) found the agent over-reaches on out-of-scope requests:
- F4: a 2-qubit (CNOT) request → the agent HALLUCINATES a multi-qubit API
(TransmonSystem(ω=[..],g=..)) → TypeError crash, while claiming 'solve launched'.
- F3: a 30 ns gate with the default N=50 is under-resolved → F=0.914 reported as success.
- F2: levels=5 stalls (integrator stiffness) at f~69.
AGENTS.md now: declares the build single-qubit only and tells the agent to
decline multi-qubit gates instead of authoring a crashing script; and gives
regime guidance (avoid 5+ levels; keep ~5-10 timesteps/ns so longer gates scale
N up; bump max_iter for harder cases). Guard tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jack-champagne

Copy link
Copy Markdown
Member

Correct root-cause: external_directory is one of the few opencode permissions defaulting to "ask", so headless it hangs / nags per solve. Fix is in the right place. Two trims worth making here:

  • [important, in-PR] external_directory: "allow" is a blanket grant to the whole filesystem outside the workspace. Scope it to the roots actually used (/tmp/amicode-work + the bundled template dir) — opencode supports per-path patterns. As-is it's broader than a single-purpose assistant needs.
  • [minor] webfetch: "allow" and edit: "allow" are redundant (both default to allow), and webfetch grants network egress the solve flow never uses. Drop webfetch; the load-bearing line is external_directory alone.

Note this doesn't make chat→solve work end-to-end on its own — the #11 author/run path mismatch was still live here and is reconciled in #23 (now validated).

@Rchari1
Rchari1 changed the base branch from rchari/template-plot-pulse to mainJune 26, 2026 20:11

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The external_directory fix is correctly diagnosed and lands the blocker. Three changes (inline): drop the unused webfetch allow; scope external_directory to the two roots the agent actually touches (or document the trust boundary if the schema won't scope); and verify the nested permission block deep-merges rather than replacing the user's permission config (+ a test).

bash/edit are redundant-as-defaults but harmless — leave them; scoping bash is fiddly given the compound mkdir … && nohup amico-run … launch.

AGENTS.md guidance is solid and the N / steps-per-ns numbers are consistent — two prose fixes: "do not hand-roll a coupled multi-transmon system (it will not construct)" — MultiTransmonSystem does exist and would construct; say "out of scope for this single-lab build." And "the integrator stiffens" for levels ≥5 is loose — it's conditioning/leakage/cost.

Tests are content/string guards — they assert the keys/words are emitted, not that opencode honors them, and nothing covers the permission merge.

permission: {
bash: "allow",
edit: "allow",
webfetch: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] webfetch isn't used by the solve workflow (it never fetches a URL) and already defaults to allow, so this line only forces egress back on for a user who'd restricted it globally. The new test omits it too. Drop it:

Suggested change
webfetch: "allow",

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped in 4942a2bwebfetch is gone (and edit/bash left as you suggested).

bash: "allow",
edit: "allow",
webfetch: "allow",
external_directory: "allow",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] This is the line that fixes the hang — correct, external_directory defaults to "ask" in 1.17.x. But "allow" grants the agent's file tools the entire filesystem outside the session cwd. The agent itself needs only two roots: it reads the bundled template (opts.templateSrc) and writes scratch to /tmp/amicode-work (amico-run's writes to ~/.amico/runs / ~/.amico/julia are the subprocess's, not the agent's). If opencode 1.17.3 supports path-scoped external_directory, scope it:

external_directory: {
"/tmp/amicode-work/**": "allow",
"<extension templates dir>/**": "allow",
},

If it only takes a flat string, keep "allow" but document the trust boundary.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scoped in 4942a2b. external_directory is now an object keyed to the two roots the agent's file tools touch — the bundled templates dir (path.dirname(templatePath) + the exact template file) and /tmp/amicode-work (plus /private/tmp/amicode-work for macOS's /tmp→/private/tmp symlink). amico-run's own ~/.amico writes are the subprocess's, not the agent's, so they're not granted. The path-scoped object form is accepted by 1.17.3 — verified via opencode debug config (resolved config preserves the per-path keys). Threading templatePath through buildOpencodeConfigContent so the dir is computed at runtime (correct for both dev and the installed VSIX).

return JSON.stringify({
$schema: "https://opencode.ai/config.json",
instructions: [agentsPath],
permission: {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[important] Verify the merge for this nested object. The doc-comment says opencode merges the injected config over the user's — but that was only confirmed for top-level model/provider and the instructions array (#12). If opencode shallow-replaces rather than deep-merges, injecting permission here wipes any other permission keys the user set globally (doom_loop, per-tool patterns). Nothing tests this — confirm against 1.17.3 and add a test that the user's other permission keys survive.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified — opencode 1.17.3 deep-merges the nested permission object; it does not shallow-replace. Tested with opencode debug config: a global permission.doom_loop:"deny" survives alongside our injected external_directory keys. Added an integration test (opencode_config.test.ts → 'opencode permission merge (1.17.3)') that runs the vendored binary's debug config with a global config + our injected content and asserts the user's key survives; skipIf the binary is absent. So injecting permission adds keys without clobbering the user's other permission settings.

…drop webfetch, verify deep-merge
- Scope `external_directory` (least privilege) to the two roots the agent's file
tools actually touch: the bundled templates dir (reads the solve template) and
/tmp/amicode-work (writes solve.jl). Path-scoped object form verified accepted
by opencode 1.17.3 via `opencode debug config`. Includes /private/tmp/** for
macOS's /tmp symlink. amico-run's own ~/.amico writes are the subprocess's, not
the agent's, so they need no grant.
- Drop `webfetch` (unused by the solve flow; already defaults to allow). Keep
bash/edit per Jack (redundant-as-defaults but harmless; bash launch not worth
scoping).
- Verify the permission merge: opencode 1.17.3 DEEP-merges the injected
`permission` object over the user's global config (a global permission.doom_loop
survives). Added an integration test (runs the vendored binary's `debug config`,
skipped when absent) so this can't regress to a clobber.
- AGENTS.md prose fixes: multi-qubit is "out of scope for this single-lab build"
(MultiTransmonSystem does exist/construct), and levels≥5 degrades via
conditioning/leakage/cost, not "integrator stiffens".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Rchari1

Copy link
Copy Markdown
MemberAuthor

Thanks @jack-champagne — all addressed in 4942a2b:

  • webfetch dropped; bash/edit kept.
  • external_directory scoped to the template dir + /tmp/amicode-work (+ /private/tmp for the macOS symlink); path-scoped object form verified accepted by 1.17.3 via opencode debug config.
  • deep-merge verified — opencode 1.17.3 deep-merges permission (a global doom_loop survives alongside our injected keys). Added an integration test that runs the vendored binary's debug config and asserts the user's key survives (skips when the binary's absent).
  • AGENTS.md: multi-qubit reworded to "out of scope for this single-lab build" (MultiTransmonSystem does construct); levels≥5 reworded to conditioning/leakage/cost.

44 tests green (incl. the new merge test). One note: the runtime enforcement of the scoped globs (vs schema-acceptance, which is verified) I'll smoke-test on the dev host with a live solve before this merges — if a path canonicalization edge re-introduces the prompt, the documented-allow fallback you sanctioned is a 2-line change. Re-requesting review.

@jack-champagnejack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three addressed in 4942a2b — verified against the diff:

  • webfetch dropped; bash/edit left at allow as agreed.
  • external_directory scoped to the template file + its dir + /tmp/amicode-work (and /private/tmp for the macOS symlink). Blanket allow gone, amico-run's own ~/.amico writes correctly ungranted.
  • deep-merge confirmed empirically via opencode debug config + integration test.

One follow-up, not blocking: the merge test is describe.skipIf(!existsSync(OC_BIN)) — it silently skips when the vendored binary is absent, so in CI before fetch:opencode it's green-without-verifying (same false-green class as #25). Sequence that test job behind fetch:opencode or the guard is decorative.

Good to merge.

@jack-champagne
jack-champagne merged commit 4249162 into mainJun 26, 2026
3 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Rchari1@jack-champagne