Skip to content

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts - #12

Merged
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts
May 25, 2026
Merged

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts#12
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

Two unrelated but small changes bundled in one PR (happy to split if preferred).

1. fix(local_manager): tolerate cross-loop shutdown of cleanup task

Symptom: At interpreter exit, processes that drive the sandbox manager via a per-worker asyncio loop print

WARNING: SandboxService: error stopping manager: Event loop is closed
ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-7' coro=<LocalSandboxManager._cleanup_loop()>>

Cause: LocalSandboxManager._cleanup_task is created on whatever asyncio loop first calls start() — in practice a per-worker thread loop. When that worker thread tears its loop down (typical finally: loop.close() pattern) before atexit-driven SandboxService.shutdown_all reaches us, manager.stop() running on a different loop tries to cancel() + await the task. The task's owner loop is gone, so:

  • await self._cleanup_task raises RuntimeError: Event loop is closed
  • the task remains in pending state → asyncio's Task.__del__ prints the second line on GC

Fix: detect a closed owner loop and (a) suppress the asyncio pending-task warning via _log_destroy_pending, (b) skip the cancel/await dance entirely (the cancellation can't be driven to completion anyway). Tolerate RuntimeError in the live-loop branch as belt-and-suspenders.

Reproduction (no eval framework needed):

importasyncio, threadingfromms_enclave.sandbox.manager.local_managerimportLocalSandboxManagerasyncdefsetup_only():
mgr=LocalSandboxManager()
awaitmgr.start()
returnmgrout= []
defworker():
loop=asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
out.append(loop.run_until_complete(setup_only()))
finally:
loop.close()
t=threading.Thread(target=worker); t.start(); t.join()
main_loop=asyncio.new_event_loop()
asyncio.set_event_loop(main_loop)
main_loop.run_until_complete(out[0].stop()) # raised "Event loop is closed" beforemain_loop.close()

Before this PR: prints Event loop is closed + Task was destroyed but it is pending.
After this PR: clean exit.

End-to-end verified through evalscope's SWE-bench Pro × external codex pipeline (3 samples, ~10 min) — both noise lines gone.

2. feat(docker_sandbox): expose extra_hosts on DockerSandboxConfig

Adds a Dict[str, str] field forwarded verbatim to docker-py's extra_hosts kwarg, so containers can be launched with custom /etc/hosts entries. The motivating use case is {"host.docker.internal": "host-gateway"} to reach host services from inside the container on Linux, where that alias is not auto-provided (Docker Desktop adds it automatically on macOS/Windows but not on plain dockerd Linux).

Documented on the field; no behavioural change when left as the default empty dict.

Test plan

  • Repro snippet above goes from noisy → clean
  • evalscope SWE-bench Pro × codex run (3 samples) — clean shutdown
  • Existing ms-enclave tests pass (please run in CI)

LocalSandboxManager._cleanup_task is bound to whatever asyncio loop first
called start() — typically a per-worker thread loop. When that worker
thread tears down its loop before SandboxService.shutdown_all runs at
process exit, manager.stop() on the main loop hits:
RuntimeError: Event loop is closed
(when awaiting the cancellation) and asyncio then prints:
Task was destroyed but it is pending!
on GC of the orphaned cleanup task.
Detect a closed owner loop and:
- suppress the asyncio pending-task warning via _log_destroy_pending
- skip the cancel/await dance entirely (the task's loop is gone, so
the cancellation cannot be driven to completion anyway)
Otherwise behave exactly as before. Tolerate RuntimeError in the live-loop
branch too as belt-and-suspenders for any other ordering race.
Adds a Dict[str, str] field forwarded verbatim to docker-py's
extra_hosts kwarg, so containers can be launched with custom /etc/hosts
entries (e.g. {"host.docker.internal": "host-gateway"} to reach
host services from inside the container on Linux, where that alias is
not auto-provided).
CopilotAI review requested due to automatic review settings May 25, 2026 07:21

@gemini-code-assistgemini-code-assistBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the extra_hosts configuration for Docker sandboxes, allowing custom host-to-IP mappings in /etc/hosts. It also improves the shutdown logic in local_manager.py to prevent asyncio warnings when cleaning up tasks across different event loops. Feedback suggests further refining the task cleanup by checking if the task is already done and ensuring internal warning suppression is applied if a RuntimeError occurs during the await process.

Comment threadms_enclave/sandbox/manager/local_manager.py Outdated
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes two targeted improvements in ms_enclave’s sandbox runtime: (1) hardens LocalSandboxManager.stop() against interpreter-exit shutdown when the cleanup task’s owning event loop has already been closed, and (2) adds support for passing Docker extra_hosts entries via DockerSandboxConfig to improve host reachability from Linux containers.

Changes:

  • Make LocalSandboxManager.stop() tolerate cleanup-task shutdown when the task’s event loop is already closed.
  • Add extra_hosts: Dict[str, str] to DockerSandboxConfig.
  • Forward extra_hosts into Docker container creation in DockerSandbox.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
ms_enclave/sandbox/manager/local_manager.pyAdds cross-loop shutdown handling for the background cleanup task during manager stop.
ms_enclave/sandbox/model/config.pyExtends DockerSandboxConfig with an extra_hosts mapping field.
ms_enclave/sandbox/boxes/docker_sandbox.pyPasses extra_hosts through to docker-py’s container create() call.
Comments suppressed due to low confidence (1)

ms_enclave/sandbox/manager/local_manager.py:71

  • stop() writes to the private Task._log_destroy_pending attribute without any guard. Because this is an internal asyncio implementation detail, it may be missing or become read-only across Python versions/implementations, which would turn shutdown into an AttributeError. Consider using hasattr/setattr inside a try/except (AttributeError, TypeError) and falling back to a best-effort cancel() + dropping the reference when it cannot be modified.
 if task_loop is None or task_loop.is_closed():
self._cleanup_task._log_destroy_pending = False
else:

Comment threadms_enclave/sandbox/manager/local_manager.py
Comment threadms_enclave/sandbox/manager/local_manager.py
…stop
Address review feedback: short-circuit when the cleanup task is already
done, suppress destroy-pending warning on cross-loop RuntimeError, and
drop the task reference so a restarted manager starts clean.
@Yunnglin
Yunnglin merged commit 0aeedd9 into mainMay 25, 2026
1 check passed
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Royniel pushed a commit to Royniel/evalscope that referenced this pull request Jun 10, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Yunnglin
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts by Yunnglin · Pull Request #12 · modelscope/ms-enclave · GitHub
Skip to content

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts - #12

Merged
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts
May 25, 2026
Merged

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts#12
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

Two unrelated but small changes bundled in one PR (happy to split if preferred).

1. fix(local_manager): tolerate cross-loop shutdown of cleanup task

Symptom: At interpreter exit, processes that drive the sandbox manager via a per-worker asyncio loop print

WARNING: SandboxService: error stopping manager: Event loop is closed
ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-7' coro=<LocalSandboxManager._cleanup_loop()>>

Cause: LocalSandboxManager._cleanup_task is created on whatever asyncio loop first calls start() — in practice a per-worker thread loop. When that worker thread tears its loop down (typical finally: loop.close() pattern) before atexit-driven SandboxService.shutdown_all reaches us, manager.stop() running on a different loop tries to cancel() + await the task. The task's owner loop is gone, so:

  • await self._cleanup_task raises RuntimeError: Event loop is closed
  • the task remains in pending state → asyncio's Task.__del__ prints the second line on GC

Fix: detect a closed owner loop and (a) suppress the asyncio pending-task warning via _log_destroy_pending, (b) skip the cancel/await dance entirely (the cancellation can't be driven to completion anyway). Tolerate RuntimeError in the live-loop branch as belt-and-suspenders.

Reproduction (no eval framework needed):

importasyncio, threadingfromms_enclave.sandbox.manager.local_managerimportLocalSandboxManagerasyncdefsetup_only():
mgr=LocalSandboxManager()
awaitmgr.start()
returnmgrout= []
defworker():
loop=asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
out.append(loop.run_until_complete(setup_only()))
finally:
loop.close()
t=threading.Thread(target=worker); t.start(); t.join()
main_loop=asyncio.new_event_loop()
asyncio.set_event_loop(main_loop)
main_loop.run_until_complete(out[0].stop()) # raised "Event loop is closed" beforemain_loop.close()

Before this PR: prints Event loop is closed + Task was destroyed but it is pending.
After this PR: clean exit.

End-to-end verified through evalscope's SWE-bench Pro × external codex pipeline (3 samples, ~10 min) — both noise lines gone.

2. feat(docker_sandbox): expose extra_hosts on DockerSandboxConfig

Adds a Dict[str, str] field forwarded verbatim to docker-py's extra_hosts kwarg, so containers can be launched with custom /etc/hosts entries. The motivating use case is {"host.docker.internal": "host-gateway"} to reach host services from inside the container on Linux, where that alias is not auto-provided (Docker Desktop adds it automatically on macOS/Windows but not on plain dockerd Linux).

Documented on the field; no behavioural change when left as the default empty dict.

Test plan

  • Repro snippet above goes from noisy → clean
  • evalscope SWE-bench Pro × codex run (3 samples) — clean shutdown
  • Existing ms-enclave tests pass (please run in CI)

LocalSandboxManager._cleanup_task is bound to whatever asyncio loop first
called start() — typically a per-worker thread loop. When that worker
thread tears down its loop before SandboxService.shutdown_all runs at
process exit, manager.stop() on the main loop hits:
RuntimeError: Event loop is closed
(when awaiting the cancellation) and asyncio then prints:
Task was destroyed but it is pending!
on GC of the orphaned cleanup task.
Detect a closed owner loop and:
- suppress the asyncio pending-task warning via _log_destroy_pending
- skip the cancel/await dance entirely (the task's loop is gone, so
the cancellation cannot be driven to completion anyway)
Otherwise behave exactly as before. Tolerate RuntimeError in the live-loop
branch too as belt-and-suspenders for any other ordering race.
Adds a Dict[str, str] field forwarded verbatim to docker-py's
extra_hosts kwarg, so containers can be launched with custom /etc/hosts
entries (e.g. {"host.docker.internal": "host-gateway"} to reach
host services from inside the container on Linux, where that alias is
not auto-provided).
CopilotAI review requested due to automatic review settings May 25, 2026 07:21

@gemini-code-assistgemini-code-assistBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the extra_hosts configuration for Docker sandboxes, allowing custom host-to-IP mappings in /etc/hosts. It also improves the shutdown logic in local_manager.py to prevent asyncio warnings when cleaning up tasks across different event loops. Feedback suggests further refining the task cleanup by checking if the task is already done and ensuring internal warning suppression is applied if a RuntimeError occurs during the await process.

Comment threadms_enclave/sandbox/manager/local_manager.py Outdated
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes two targeted improvements in ms_enclave’s sandbox runtime: (1) hardens LocalSandboxManager.stop() against interpreter-exit shutdown when the cleanup task’s owning event loop has already been closed, and (2) adds support for passing Docker extra_hosts entries via DockerSandboxConfig to improve host reachability from Linux containers.

Changes:

  • Make LocalSandboxManager.stop() tolerate cleanup-task shutdown when the task’s event loop is already closed.
  • Add extra_hosts: Dict[str, str] to DockerSandboxConfig.
  • Forward extra_hosts into Docker container creation in DockerSandbox.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
ms_enclave/sandbox/manager/local_manager.pyAdds cross-loop shutdown handling for the background cleanup task during manager stop.
ms_enclave/sandbox/model/config.pyExtends DockerSandboxConfig with an extra_hosts mapping field.
ms_enclave/sandbox/boxes/docker_sandbox.pyPasses extra_hosts through to docker-py’s container create() call.
Comments suppressed due to low confidence (1)

ms_enclave/sandbox/manager/local_manager.py:71

  • stop() writes to the private Task._log_destroy_pending attribute without any guard. Because this is an internal asyncio implementation detail, it may be missing or become read-only across Python versions/implementations, which would turn shutdown into an AttributeError. Consider using hasattr/setattr inside a try/except (AttributeError, TypeError) and falling back to a best-effort cancel() + dropping the reference when it cannot be modified.
 if task_loop is None or task_loop.is_closed():
self._cleanup_task._log_destroy_pending = False
else:

Comment threadms_enclave/sandbox/manager/local_manager.py
Comment threadms_enclave/sandbox/manager/local_manager.py
…stop
Address review feedback: short-circuit when the cleanup task is already
done, suppress destroy-pending warning on cross-loop RuntimeError, and
drop the task reference so a restarted manager starts clean.
@Yunnglin
Yunnglin merged commit 0aeedd9 into mainMay 25, 2026
1 check passed
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Royniel pushed a commit to Royniel/evalscope that referenced this pull request Jun 10, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Yunnglin
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts by Yunnglin · Pull Request #12 · modelscope/ms-enclave · GitHub
Skip to content

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts - #12

Merged
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts
May 25, 2026
Merged

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts#12
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

Two unrelated but small changes bundled in one PR (happy to split if preferred).

1. fix(local_manager): tolerate cross-loop shutdown of cleanup task

Symptom: At interpreter exit, processes that drive the sandbox manager via a per-worker asyncio loop print

WARNING: SandboxService: error stopping manager: Event loop is closed
ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-7' coro=<LocalSandboxManager._cleanup_loop()>>

Cause: LocalSandboxManager._cleanup_task is created on whatever asyncio loop first calls start() — in practice a per-worker thread loop. When that worker thread tears its loop down (typical finally: loop.close() pattern) before atexit-driven SandboxService.shutdown_all reaches us, manager.stop() running on a different loop tries to cancel() + await the task. The task's owner loop is gone, so:

  • await self._cleanup_task raises RuntimeError: Event loop is closed
  • the task remains in pending state → asyncio's Task.__del__ prints the second line on GC

Fix: detect a closed owner loop and (a) suppress the asyncio pending-task warning via _log_destroy_pending, (b) skip the cancel/await dance entirely (the cancellation can't be driven to completion anyway). Tolerate RuntimeError in the live-loop branch as belt-and-suspenders.

Reproduction (no eval framework needed):

importasyncio, threadingfromms_enclave.sandbox.manager.local_managerimportLocalSandboxManagerasyncdefsetup_only():
mgr=LocalSandboxManager()
awaitmgr.start()
returnmgrout= []
defworker():
loop=asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
out.append(loop.run_until_complete(setup_only()))
finally:
loop.close()
t=threading.Thread(target=worker); t.start(); t.join()
main_loop=asyncio.new_event_loop()
asyncio.set_event_loop(main_loop)
main_loop.run_until_complete(out[0].stop()) # raised "Event loop is closed" beforemain_loop.close()

Before this PR: prints Event loop is closed + Task was destroyed but it is pending.
After this PR: clean exit.

End-to-end verified through evalscope's SWE-bench Pro × external codex pipeline (3 samples, ~10 min) — both noise lines gone.

2. feat(docker_sandbox): expose extra_hosts on DockerSandboxConfig

Adds a Dict[str, str] field forwarded verbatim to docker-py's extra_hosts kwarg, so containers can be launched with custom /etc/hosts entries. The motivating use case is {"host.docker.internal": "host-gateway"} to reach host services from inside the container on Linux, where that alias is not auto-provided (Docker Desktop adds it automatically on macOS/Windows but not on plain dockerd Linux).

Documented on the field; no behavioural change when left as the default empty dict.

Test plan

  • Repro snippet above goes from noisy → clean
  • evalscope SWE-bench Pro × codex run (3 samples) — clean shutdown
  • Existing ms-enclave tests pass (please run in CI)

LocalSandboxManager._cleanup_task is bound to whatever asyncio loop first
called start() — typically a per-worker thread loop. When that worker
thread tears down its loop before SandboxService.shutdown_all runs at
process exit, manager.stop() on the main loop hits:
RuntimeError: Event loop is closed
(when awaiting the cancellation) and asyncio then prints:
Task was destroyed but it is pending!
on GC of the orphaned cleanup task.
Detect a closed owner loop and:
- suppress the asyncio pending-task warning via _log_destroy_pending
- skip the cancel/await dance entirely (the task's loop is gone, so
the cancellation cannot be driven to completion anyway)
Otherwise behave exactly as before. Tolerate RuntimeError in the live-loop
branch too as belt-and-suspenders for any other ordering race.
Adds a Dict[str, str] field forwarded verbatim to docker-py's
extra_hosts kwarg, so containers can be launched with custom /etc/hosts
entries (e.g. {"host.docker.internal": "host-gateway"} to reach
host services from inside the container on Linux, where that alias is
not auto-provided).
CopilotAI review requested due to automatic review settings May 25, 2026 07:21

@gemini-code-assistgemini-code-assistBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the extra_hosts configuration for Docker sandboxes, allowing custom host-to-IP mappings in /etc/hosts. It also improves the shutdown logic in local_manager.py to prevent asyncio warnings when cleaning up tasks across different event loops. Feedback suggests further refining the task cleanup by checking if the task is already done and ensuring internal warning suppression is applied if a RuntimeError occurs during the await process.

Comment threadms_enclave/sandbox/manager/local_manager.py Outdated
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes two targeted improvements in ms_enclave’s sandbox runtime: (1) hardens LocalSandboxManager.stop() against interpreter-exit shutdown when the cleanup task’s owning event loop has already been closed, and (2) adds support for passing Docker extra_hosts entries via DockerSandboxConfig to improve host reachability from Linux containers.

Changes:

  • Make LocalSandboxManager.stop() tolerate cleanup-task shutdown when the task’s event loop is already closed.
  • Add extra_hosts: Dict[str, str] to DockerSandboxConfig.
  • Forward extra_hosts into Docker container creation in DockerSandbox.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
ms_enclave/sandbox/manager/local_manager.pyAdds cross-loop shutdown handling for the background cleanup task during manager stop.
ms_enclave/sandbox/model/config.pyExtends DockerSandboxConfig with an extra_hosts mapping field.
ms_enclave/sandbox/boxes/docker_sandbox.pyPasses extra_hosts through to docker-py’s container create() call.
Comments suppressed due to low confidence (1)

ms_enclave/sandbox/manager/local_manager.py:71

  • stop() writes to the private Task._log_destroy_pending attribute without any guard. Because this is an internal asyncio implementation detail, it may be missing or become read-only across Python versions/implementations, which would turn shutdown into an AttributeError. Consider using hasattr/setattr inside a try/except (AttributeError, TypeError) and falling back to a best-effort cancel() + dropping the reference when it cannot be modified.
 if task_loop is None or task_loop.is_closed():
self._cleanup_task._log_destroy_pending = False
else:

Comment threadms_enclave/sandbox/manager/local_manager.py
Comment threadms_enclave/sandbox/manager/local_manager.py
…stop
Address review feedback: short-circuit when the cleanup task is already
done, suppress destroy-pending warning on cross-loop RuntimeError, and
drop the task reference so a restarted manager starts clean.
@Yunnglin
Yunnglin merged commit 0aeedd9 into mainMay 25, 2026
1 check passed
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Royniel pushed a commit to Royniel/evalscope that referenced this pull request Jun 10, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Yunnglin
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts by Yunnglin · Pull Request #12 · modelscope/ms-enclave · GitHub
Skip to content

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts - #12

Merged
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts
May 25, 2026
Merged

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts#12
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

Two unrelated but small changes bundled in one PR (happy to split if preferred).

1. fix(local_manager): tolerate cross-loop shutdown of cleanup task

Symptom: At interpreter exit, processes that drive the sandbox manager via a per-worker asyncio loop print

WARNING: SandboxService: error stopping manager: Event loop is closed
ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-7' coro=<LocalSandboxManager._cleanup_loop()>>

Cause: LocalSandboxManager._cleanup_task is created on whatever asyncio loop first calls start() — in practice a per-worker thread loop. When that worker thread tears its loop down (typical finally: loop.close() pattern) before atexit-driven SandboxService.shutdown_all reaches us, manager.stop() running on a different loop tries to cancel() + await the task. The task's owner loop is gone, so:

  • await self._cleanup_task raises RuntimeError: Event loop is closed
  • the task remains in pending state → asyncio's Task.__del__ prints the second line on GC

Fix: detect a closed owner loop and (a) suppress the asyncio pending-task warning via _log_destroy_pending, (b) skip the cancel/await dance entirely (the cancellation can't be driven to completion anyway). Tolerate RuntimeError in the live-loop branch as belt-and-suspenders.

Reproduction (no eval framework needed):

importasyncio, threadingfromms_enclave.sandbox.manager.local_managerimportLocalSandboxManagerasyncdefsetup_only():
mgr=LocalSandboxManager()
awaitmgr.start()
returnmgrout= []
defworker():
loop=asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
out.append(loop.run_until_complete(setup_only()))
finally:
loop.close()
t=threading.Thread(target=worker); t.start(); t.join()
main_loop=asyncio.new_event_loop()
asyncio.set_event_loop(main_loop)
main_loop.run_until_complete(out[0].stop()) # raised "Event loop is closed" beforemain_loop.close()

Before this PR: prints Event loop is closed + Task was destroyed but it is pending.
After this PR: clean exit.

End-to-end verified through evalscope's SWE-bench Pro × external codex pipeline (3 samples, ~10 min) — both noise lines gone.

2. feat(docker_sandbox): expose extra_hosts on DockerSandboxConfig

Adds a Dict[str, str] field forwarded verbatim to docker-py's extra_hosts kwarg, so containers can be launched with custom /etc/hosts entries. The motivating use case is {"host.docker.internal": "host-gateway"} to reach host services from inside the container on Linux, where that alias is not auto-provided (Docker Desktop adds it automatically on macOS/Windows but not on plain dockerd Linux).

Documented on the field; no behavioural change when left as the default empty dict.

Test plan

  • Repro snippet above goes from noisy → clean
  • evalscope SWE-bench Pro × codex run (3 samples) — clean shutdown
  • Existing ms-enclave tests pass (please run in CI)

LocalSandboxManager._cleanup_task is bound to whatever asyncio loop first
called start() — typically a per-worker thread loop. When that worker
thread tears down its loop before SandboxService.shutdown_all runs at
process exit, manager.stop() on the main loop hits:
RuntimeError: Event loop is closed
(when awaiting the cancellation) and asyncio then prints:
Task was destroyed but it is pending!
on GC of the orphaned cleanup task.
Detect a closed owner loop and:
- suppress the asyncio pending-task warning via _log_destroy_pending
- skip the cancel/await dance entirely (the task's loop is gone, so
the cancellation cannot be driven to completion anyway)
Otherwise behave exactly as before. Tolerate RuntimeError in the live-loop
branch too as belt-and-suspenders for any other ordering race.
Adds a Dict[str, str] field forwarded verbatim to docker-py's
extra_hosts kwarg, so containers can be launched with custom /etc/hosts
entries (e.g. {"host.docker.internal": "host-gateway"} to reach
host services from inside the container on Linux, where that alias is
not auto-provided).
CopilotAI review requested due to automatic review settings May 25, 2026 07:21

@gemini-code-assistgemini-code-assistBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the extra_hosts configuration for Docker sandboxes, allowing custom host-to-IP mappings in /etc/hosts. It also improves the shutdown logic in local_manager.py to prevent asyncio warnings when cleaning up tasks across different event loops. Feedback suggests further refining the task cleanup by checking if the task is already done and ensuring internal warning suppression is applied if a RuntimeError occurs during the await process.

Comment threadms_enclave/sandbox/manager/local_manager.py Outdated
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes two targeted improvements in ms_enclave’s sandbox runtime: (1) hardens LocalSandboxManager.stop() against interpreter-exit shutdown when the cleanup task’s owning event loop has already been closed, and (2) adds support for passing Docker extra_hosts entries via DockerSandboxConfig to improve host reachability from Linux containers.

Changes:

  • Make LocalSandboxManager.stop() tolerate cleanup-task shutdown when the task’s event loop is already closed.
  • Add extra_hosts: Dict[str, str] to DockerSandboxConfig.
  • Forward extra_hosts into Docker container creation in DockerSandbox.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
ms_enclave/sandbox/manager/local_manager.pyAdds cross-loop shutdown handling for the background cleanup task during manager stop.
ms_enclave/sandbox/model/config.pyExtends DockerSandboxConfig with an extra_hosts mapping field.
ms_enclave/sandbox/boxes/docker_sandbox.pyPasses extra_hosts through to docker-py’s container create() call.
Comments suppressed due to low confidence (1)

ms_enclave/sandbox/manager/local_manager.py:71

  • stop() writes to the private Task._log_destroy_pending attribute without any guard. Because this is an internal asyncio implementation detail, it may be missing or become read-only across Python versions/implementations, which would turn shutdown into an AttributeError. Consider using hasattr/setattr inside a try/except (AttributeError, TypeError) and falling back to a best-effort cancel() + dropping the reference when it cannot be modified.
 if task_loop is None or task_loop.is_closed():
self._cleanup_task._log_destroy_pending = False
else:

Comment threadms_enclave/sandbox/manager/local_manager.py
Comment threadms_enclave/sandbox/manager/local_manager.py
…stop
Address review feedback: short-circuit when the cleanup task is already
done, suppress destroy-pending warning on cross-loop RuntimeError, and
drop the task reference so a restarted manager starts clean.
@Yunnglin
Yunnglin merged commit 0aeedd9 into mainMay 25, 2026
1 check passed
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Royniel pushed a commit to Royniel/evalscope that referenced this pull request Jun 10, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Yunnglin
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts by Yunnglin · Pull Request #12 · modelscope/ms-enclave · GitHub
Skip to content

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts - #12

Merged
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts
May 25, 2026
Merged

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts#12
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

Two unrelated but small changes bundled in one PR (happy to split if preferred).

1. fix(local_manager): tolerate cross-loop shutdown of cleanup task

Symptom: At interpreter exit, processes that drive the sandbox manager via a per-worker asyncio loop print

WARNING: SandboxService: error stopping manager: Event loop is closed
ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-7' coro=<LocalSandboxManager._cleanup_loop()>>

Cause: LocalSandboxManager._cleanup_task is created on whatever asyncio loop first calls start() — in practice a per-worker thread loop. When that worker thread tears its loop down (typical finally: loop.close() pattern) before atexit-driven SandboxService.shutdown_all reaches us, manager.stop() running on a different loop tries to cancel() + await the task. The task's owner loop is gone, so:

  • await self._cleanup_task raises RuntimeError: Event loop is closed
  • the task remains in pending state → asyncio's Task.__del__ prints the second line on GC

Fix: detect a closed owner loop and (a) suppress the asyncio pending-task warning via _log_destroy_pending, (b) skip the cancel/await dance entirely (the cancellation can't be driven to completion anyway). Tolerate RuntimeError in the live-loop branch as belt-and-suspenders.

Reproduction (no eval framework needed):

importasyncio, threadingfromms_enclave.sandbox.manager.local_managerimportLocalSandboxManagerasyncdefsetup_only():
mgr=LocalSandboxManager()
awaitmgr.start()
returnmgrout= []
defworker():
loop=asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
out.append(loop.run_until_complete(setup_only()))
finally:
loop.close()
t=threading.Thread(target=worker); t.start(); t.join()
main_loop=asyncio.new_event_loop()
asyncio.set_event_loop(main_loop)
main_loop.run_until_complete(out[0].stop()) # raised "Event loop is closed" beforemain_loop.close()

Before this PR: prints Event loop is closed + Task was destroyed but it is pending.
After this PR: clean exit.

End-to-end verified through evalscope's SWE-bench Pro × external codex pipeline (3 samples, ~10 min) — both noise lines gone.

2. feat(docker_sandbox): expose extra_hosts on DockerSandboxConfig

Adds a Dict[str, str] field forwarded verbatim to docker-py's extra_hosts kwarg, so containers can be launched with custom /etc/hosts entries. The motivating use case is {"host.docker.internal": "host-gateway"} to reach host services from inside the container on Linux, where that alias is not auto-provided (Docker Desktop adds it automatically on macOS/Windows but not on plain dockerd Linux).

Documented on the field; no behavioural change when left as the default empty dict.

Test plan

  • Repro snippet above goes from noisy → clean
  • evalscope SWE-bench Pro × codex run (3 samples) — clean shutdown
  • Existing ms-enclave tests pass (please run in CI)

LocalSandboxManager._cleanup_task is bound to whatever asyncio loop first
called start() — typically a per-worker thread loop. When that worker
thread tears down its loop before SandboxService.shutdown_all runs at
process exit, manager.stop() on the main loop hits:
RuntimeError: Event loop is closed
(when awaiting the cancellation) and asyncio then prints:
Task was destroyed but it is pending!
on GC of the orphaned cleanup task.
Detect a closed owner loop and:
- suppress the asyncio pending-task warning via _log_destroy_pending
- skip the cancel/await dance entirely (the task's loop is gone, so
the cancellation cannot be driven to completion anyway)
Otherwise behave exactly as before. Tolerate RuntimeError in the live-loop
branch too as belt-and-suspenders for any other ordering race.
Adds a Dict[str, str] field forwarded verbatim to docker-py's
extra_hosts kwarg, so containers can be launched with custom /etc/hosts
entries (e.g. {"host.docker.internal": "host-gateway"} to reach
host services from inside the container on Linux, where that alias is
not auto-provided).
CopilotAI review requested due to automatic review settings May 25, 2026 07:21

@gemini-code-assistgemini-code-assistBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the extra_hosts configuration for Docker sandboxes, allowing custom host-to-IP mappings in /etc/hosts. It also improves the shutdown logic in local_manager.py to prevent asyncio warnings when cleaning up tasks across different event loops. Feedback suggests further refining the task cleanup by checking if the task is already done and ensuring internal warning suppression is applied if a RuntimeError occurs during the await process.

Comment threadms_enclave/sandbox/manager/local_manager.py Outdated
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes two targeted improvements in ms_enclave’s sandbox runtime: (1) hardens LocalSandboxManager.stop() against interpreter-exit shutdown when the cleanup task’s owning event loop has already been closed, and (2) adds support for passing Docker extra_hosts entries via DockerSandboxConfig to improve host reachability from Linux containers.

Changes:

  • Make LocalSandboxManager.stop() tolerate cleanup-task shutdown when the task’s event loop is already closed.
  • Add extra_hosts: Dict[str, str] to DockerSandboxConfig.
  • Forward extra_hosts into Docker container creation in DockerSandbox.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
ms_enclave/sandbox/manager/local_manager.pyAdds cross-loop shutdown handling for the background cleanup task during manager stop.
ms_enclave/sandbox/model/config.pyExtends DockerSandboxConfig with an extra_hosts mapping field.
ms_enclave/sandbox/boxes/docker_sandbox.pyPasses extra_hosts through to docker-py’s container create() call.
Comments suppressed due to low confidence (1)

ms_enclave/sandbox/manager/local_manager.py:71

  • stop() writes to the private Task._log_destroy_pending attribute without any guard. Because this is an internal asyncio implementation detail, it may be missing or become read-only across Python versions/implementations, which would turn shutdown into an AttributeError. Consider using hasattr/setattr inside a try/except (AttributeError, TypeError) and falling back to a best-effort cancel() + dropping the reference when it cannot be modified.
 if task_loop is None or task_loop.is_closed():
self._cleanup_task._log_destroy_pending = False
else:

Comment threadms_enclave/sandbox/manager/local_manager.py
Comment threadms_enclave/sandbox/manager/local_manager.py
…stop
Address review feedback: short-circuit when the cleanup task is already
done, suppress destroy-pending warning on cross-loop RuntimeError, and
drop the task reference so a restarted manager starts clean.
@Yunnglin
Yunnglin merged commit 0aeedd9 into mainMay 25, 2026
1 check passed
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Royniel pushed a commit to Royniel/evalscope that referenced this pull request Jun 10, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Yunnglin
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts by Yunnglin · Pull Request #12 · modelscope/ms-enclave · GitHub
Skip to content

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts - #12

Merged
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts
May 25, 2026
Merged

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts#12
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

Two unrelated but small changes bundled in one PR (happy to split if preferred).

1. fix(local_manager): tolerate cross-loop shutdown of cleanup task

Symptom: At interpreter exit, processes that drive the sandbox manager via a per-worker asyncio loop print

WARNING: SandboxService: error stopping manager: Event loop is closed
ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-7' coro=<LocalSandboxManager._cleanup_loop()>>

Cause: LocalSandboxManager._cleanup_task is created on whatever asyncio loop first calls start() — in practice a per-worker thread loop. When that worker thread tears its loop down (typical finally: loop.close() pattern) before atexit-driven SandboxService.shutdown_all reaches us, manager.stop() running on a different loop tries to cancel() + await the task. The task's owner loop is gone, so:

  • await self._cleanup_task raises RuntimeError: Event loop is closed
  • the task remains in pending state → asyncio's Task.__del__ prints the second line on GC

Fix: detect a closed owner loop and (a) suppress the asyncio pending-task warning via _log_destroy_pending, (b) skip the cancel/await dance entirely (the cancellation can't be driven to completion anyway). Tolerate RuntimeError in the live-loop branch as belt-and-suspenders.

Reproduction (no eval framework needed):

importasyncio, threadingfromms_enclave.sandbox.manager.local_managerimportLocalSandboxManagerasyncdefsetup_only():
mgr=LocalSandboxManager()
awaitmgr.start()
returnmgrout= []
defworker():
loop=asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
out.append(loop.run_until_complete(setup_only()))
finally:
loop.close()
t=threading.Thread(target=worker); t.start(); t.join()
main_loop=asyncio.new_event_loop()
asyncio.set_event_loop(main_loop)
main_loop.run_until_complete(out[0].stop()) # raised "Event loop is closed" beforemain_loop.close()

Before this PR: prints Event loop is closed + Task was destroyed but it is pending.
After this PR: clean exit.

End-to-end verified through evalscope's SWE-bench Pro × external codex pipeline (3 samples, ~10 min) — both noise lines gone.

2. feat(docker_sandbox): expose extra_hosts on DockerSandboxConfig

Adds a Dict[str, str] field forwarded verbatim to docker-py's extra_hosts kwarg, so containers can be launched with custom /etc/hosts entries. The motivating use case is {"host.docker.internal": "host-gateway"} to reach host services from inside the container on Linux, where that alias is not auto-provided (Docker Desktop adds it automatically on macOS/Windows but not on plain dockerd Linux).

Documented on the field; no behavioural change when left as the default empty dict.

Test plan

  • Repro snippet above goes from noisy → clean
  • evalscope SWE-bench Pro × codex run (3 samples) — clean shutdown
  • Existing ms-enclave tests pass (please run in CI)

LocalSandboxManager._cleanup_task is bound to whatever asyncio loop first
called start() — typically a per-worker thread loop. When that worker
thread tears down its loop before SandboxService.shutdown_all runs at
process exit, manager.stop() on the main loop hits:
RuntimeError: Event loop is closed
(when awaiting the cancellation) and asyncio then prints:
Task was destroyed but it is pending!
on GC of the orphaned cleanup task.
Detect a closed owner loop and:
- suppress the asyncio pending-task warning via _log_destroy_pending
- skip the cancel/await dance entirely (the task's loop is gone, so
the cancellation cannot be driven to completion anyway)
Otherwise behave exactly as before. Tolerate RuntimeError in the live-loop
branch too as belt-and-suspenders for any other ordering race.
Adds a Dict[str, str] field forwarded verbatim to docker-py's
extra_hosts kwarg, so containers can be launched with custom /etc/hosts
entries (e.g. {"host.docker.internal": "host-gateway"} to reach
host services from inside the container on Linux, where that alias is
not auto-provided).
CopilotAI review requested due to automatic review settings May 25, 2026 07:21

@gemini-code-assistgemini-code-assistBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the extra_hosts configuration for Docker sandboxes, allowing custom host-to-IP mappings in /etc/hosts. It also improves the shutdown logic in local_manager.py to prevent asyncio warnings when cleaning up tasks across different event loops. Feedback suggests further refining the task cleanup by checking if the task is already done and ensuring internal warning suppression is applied if a RuntimeError occurs during the await process.

Comment threadms_enclave/sandbox/manager/local_manager.py Outdated
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes two targeted improvements in ms_enclave’s sandbox runtime: (1) hardens LocalSandboxManager.stop() against interpreter-exit shutdown when the cleanup task’s owning event loop has already been closed, and (2) adds support for passing Docker extra_hosts entries via DockerSandboxConfig to improve host reachability from Linux containers.

Changes:

  • Make LocalSandboxManager.stop() tolerate cleanup-task shutdown when the task’s event loop is already closed.
  • Add extra_hosts: Dict[str, str] to DockerSandboxConfig.
  • Forward extra_hosts into Docker container creation in DockerSandbox.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
ms_enclave/sandbox/manager/local_manager.pyAdds cross-loop shutdown handling for the background cleanup task during manager stop.
ms_enclave/sandbox/model/config.pyExtends DockerSandboxConfig with an extra_hosts mapping field.
ms_enclave/sandbox/boxes/docker_sandbox.pyPasses extra_hosts through to docker-py’s container create() call.
Comments suppressed due to low confidence (1)

ms_enclave/sandbox/manager/local_manager.py:71

  • stop() writes to the private Task._log_destroy_pending attribute without any guard. Because this is an internal asyncio implementation detail, it may be missing or become read-only across Python versions/implementations, which would turn shutdown into an AttributeError. Consider using hasattr/setattr inside a try/except (AttributeError, TypeError) and falling back to a best-effort cancel() + dropping the reference when it cannot be modified.
 if task_loop is None or task_loop.is_closed():
self._cleanup_task._log_destroy_pending = False
else:

Comment threadms_enclave/sandbox/manager/local_manager.py
Comment threadms_enclave/sandbox/manager/local_manager.py
…stop
Address review feedback: short-circuit when the cleanup task is already
done, suppress destroy-pending warning on cross-loop RuntimeError, and
drop the task reference so a restarted manager starts clean.
@Yunnglin
Yunnglin merged commit 0aeedd9 into mainMay 25, 2026
1 check passed
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Royniel pushed a commit to Royniel/evalscope that referenced this pull request Jun 10, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Yunnglin
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts by Yunnglin · Pull Request #12 · modelscope/ms-enclave · GitHub
Skip to content

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts - #12

Merged
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts
May 25, 2026
Merged

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts#12
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

Two unrelated but small changes bundled in one PR (happy to split if preferred).

1. fix(local_manager): tolerate cross-loop shutdown of cleanup task

Symptom: At interpreter exit, processes that drive the sandbox manager via a per-worker asyncio loop print

WARNING: SandboxService: error stopping manager: Event loop is closed
ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-7' coro=<LocalSandboxManager._cleanup_loop()>>

Cause: LocalSandboxManager._cleanup_task is created on whatever asyncio loop first calls start() — in practice a per-worker thread loop. When that worker thread tears its loop down (typical finally: loop.close() pattern) before atexit-driven SandboxService.shutdown_all reaches us, manager.stop() running on a different loop tries to cancel() + await the task. The task's owner loop is gone, so:

  • await self._cleanup_task raises RuntimeError: Event loop is closed
  • the task remains in pending state → asyncio's Task.__del__ prints the second line on GC

Fix: detect a closed owner loop and (a) suppress the asyncio pending-task warning via _log_destroy_pending, (b) skip the cancel/await dance entirely (the cancellation can't be driven to completion anyway). Tolerate RuntimeError in the live-loop branch as belt-and-suspenders.

Reproduction (no eval framework needed):

importasyncio, threadingfromms_enclave.sandbox.manager.local_managerimportLocalSandboxManagerasyncdefsetup_only():
mgr=LocalSandboxManager()
awaitmgr.start()
returnmgrout= []
defworker():
loop=asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
out.append(loop.run_until_complete(setup_only()))
finally:
loop.close()
t=threading.Thread(target=worker); t.start(); t.join()
main_loop=asyncio.new_event_loop()
asyncio.set_event_loop(main_loop)
main_loop.run_until_complete(out[0].stop()) # raised "Event loop is closed" beforemain_loop.close()

Before this PR: prints Event loop is closed + Task was destroyed but it is pending.
After this PR: clean exit.

End-to-end verified through evalscope's SWE-bench Pro × external codex pipeline (3 samples, ~10 min) — both noise lines gone.

2. feat(docker_sandbox): expose extra_hosts on DockerSandboxConfig

Adds a Dict[str, str] field forwarded verbatim to docker-py's extra_hosts kwarg, so containers can be launched with custom /etc/hosts entries. The motivating use case is {"host.docker.internal": "host-gateway"} to reach host services from inside the container on Linux, where that alias is not auto-provided (Docker Desktop adds it automatically on macOS/Windows but not on plain dockerd Linux).

Documented on the field; no behavioural change when left as the default empty dict.

Test plan

  • Repro snippet above goes from noisy → clean
  • evalscope SWE-bench Pro × codex run (3 samples) — clean shutdown
  • Existing ms-enclave tests pass (please run in CI)

LocalSandboxManager._cleanup_task is bound to whatever asyncio loop first
called start() — typically a per-worker thread loop. When that worker
thread tears down its loop before SandboxService.shutdown_all runs at
process exit, manager.stop() on the main loop hits:
RuntimeError: Event loop is closed
(when awaiting the cancellation) and asyncio then prints:
Task was destroyed but it is pending!
on GC of the orphaned cleanup task.
Detect a closed owner loop and:
- suppress the asyncio pending-task warning via _log_destroy_pending
- skip the cancel/await dance entirely (the task's loop is gone, so
the cancellation cannot be driven to completion anyway)
Otherwise behave exactly as before. Tolerate RuntimeError in the live-loop
branch too as belt-and-suspenders for any other ordering race.
Adds a Dict[str, str] field forwarded verbatim to docker-py's
extra_hosts kwarg, so containers can be launched with custom /etc/hosts
entries (e.g. {"host.docker.internal": "host-gateway"} to reach
host services from inside the container on Linux, where that alias is
not auto-provided).
CopilotAI review requested due to automatic review settings May 25, 2026 07:21

@gemini-code-assistgemini-code-assistBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the extra_hosts configuration for Docker sandboxes, allowing custom host-to-IP mappings in /etc/hosts. It also improves the shutdown logic in local_manager.py to prevent asyncio warnings when cleaning up tasks across different event loops. Feedback suggests further refining the task cleanup by checking if the task is already done and ensuring internal warning suppression is applied if a RuntimeError occurs during the await process.

Comment threadms_enclave/sandbox/manager/local_manager.py Outdated
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes two targeted improvements in ms_enclave’s sandbox runtime: (1) hardens LocalSandboxManager.stop() against interpreter-exit shutdown when the cleanup task’s owning event loop has already been closed, and (2) adds support for passing Docker extra_hosts entries via DockerSandboxConfig to improve host reachability from Linux containers.

Changes:

  • Make LocalSandboxManager.stop() tolerate cleanup-task shutdown when the task’s event loop is already closed.
  • Add extra_hosts: Dict[str, str] to DockerSandboxConfig.
  • Forward extra_hosts into Docker container creation in DockerSandbox.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
ms_enclave/sandbox/manager/local_manager.pyAdds cross-loop shutdown handling for the background cleanup task during manager stop.
ms_enclave/sandbox/model/config.pyExtends DockerSandboxConfig with an extra_hosts mapping field.
ms_enclave/sandbox/boxes/docker_sandbox.pyPasses extra_hosts through to docker-py’s container create() call.
Comments suppressed due to low confidence (1)

ms_enclave/sandbox/manager/local_manager.py:71

  • stop() writes to the private Task._log_destroy_pending attribute without any guard. Because this is an internal asyncio implementation detail, it may be missing or become read-only across Python versions/implementations, which would turn shutdown into an AttributeError. Consider using hasattr/setattr inside a try/except (AttributeError, TypeError) and falling back to a best-effort cancel() + dropping the reference when it cannot be modified.
 if task_loop is None or task_loop.is_closed():
self._cleanup_task._log_destroy_pending = False
else:

Comment threadms_enclave/sandbox/manager/local_manager.py
Comment threadms_enclave/sandbox/manager/local_manager.py
…stop
Address review feedback: short-circuit when the cleanup task is already
done, suppress destroy-pending warning on cross-loop RuntimeError, and
drop the task reference so a restarted manager starts clean.
@Yunnglin
Yunnglin merged commit 0aeedd9 into mainMay 25, 2026
1 check passed
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Royniel pushed a commit to Royniel/evalscope that referenced this pull request Jun 10, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Yunnglin
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts by Yunnglin · Pull Request #12 · modelscope/ms-enclave · GitHub
Skip to content

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts - #12

Merged
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts
May 25, 2026
Merged

fix: tolerate cross-loop shutdown + add DockerSandbox extra_hosts#12
Yunnglin merged 3 commits into
mainfrom
fix/shutdown-and-extra-hosts

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

Two unrelated but small changes bundled in one PR (happy to split if preferred).

1. fix(local_manager): tolerate cross-loop shutdown of cleanup task

Symptom: At interpreter exit, processes that drive the sandbox manager via a per-worker asyncio loop print

WARNING: SandboxService: error stopping manager: Event loop is closed
ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-7' coro=<LocalSandboxManager._cleanup_loop()>>

Cause: LocalSandboxManager._cleanup_task is created on whatever asyncio loop first calls start() — in practice a per-worker thread loop. When that worker thread tears its loop down (typical finally: loop.close() pattern) before atexit-driven SandboxService.shutdown_all reaches us, manager.stop() running on a different loop tries to cancel() + await the task. The task's owner loop is gone, so:

  • await self._cleanup_task raises RuntimeError: Event loop is closed
  • the task remains in pending state → asyncio's Task.__del__ prints the second line on GC

Fix: detect a closed owner loop and (a) suppress the asyncio pending-task warning via _log_destroy_pending, (b) skip the cancel/await dance entirely (the cancellation can't be driven to completion anyway). Tolerate RuntimeError in the live-loop branch as belt-and-suspenders.

Reproduction (no eval framework needed):

importasyncio, threadingfromms_enclave.sandbox.manager.local_managerimportLocalSandboxManagerasyncdefsetup_only():
mgr=LocalSandboxManager()
awaitmgr.start()
returnmgrout= []
defworker():
loop=asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
out.append(loop.run_until_complete(setup_only()))
finally:
loop.close()
t=threading.Thread(target=worker); t.start(); t.join()
main_loop=asyncio.new_event_loop()
asyncio.set_event_loop(main_loop)
main_loop.run_until_complete(out[0].stop()) # raised "Event loop is closed" beforemain_loop.close()

Before this PR: prints Event loop is closed + Task was destroyed but it is pending.
After this PR: clean exit.

End-to-end verified through evalscope's SWE-bench Pro × external codex pipeline (3 samples, ~10 min) — both noise lines gone.

2. feat(docker_sandbox): expose extra_hosts on DockerSandboxConfig

Adds a Dict[str, str] field forwarded verbatim to docker-py's extra_hosts kwarg, so containers can be launched with custom /etc/hosts entries. The motivating use case is {"host.docker.internal": "host-gateway"} to reach host services from inside the container on Linux, where that alias is not auto-provided (Docker Desktop adds it automatically on macOS/Windows but not on plain dockerd Linux).

Documented on the field; no behavioural change when left as the default empty dict.

Test plan

  • Repro snippet above goes from noisy → clean
  • evalscope SWE-bench Pro × codex run (3 samples) — clean shutdown
  • Existing ms-enclave tests pass (please run in CI)

LocalSandboxManager._cleanup_task is bound to whatever asyncio loop first
called start() — typically a per-worker thread loop. When that worker
thread tears down its loop before SandboxService.shutdown_all runs at
process exit, manager.stop() on the main loop hits:
RuntimeError: Event loop is closed
(when awaiting the cancellation) and asyncio then prints:
Task was destroyed but it is pending!
on GC of the orphaned cleanup task.
Detect a closed owner loop and:
- suppress the asyncio pending-task warning via _log_destroy_pending
- skip the cancel/await dance entirely (the task's loop is gone, so
the cancellation cannot be driven to completion anyway)
Otherwise behave exactly as before. Tolerate RuntimeError in the live-loop
branch too as belt-and-suspenders for any other ordering race.
Adds a Dict[str, str] field forwarded verbatim to docker-py's
extra_hosts kwarg, so containers can be launched with custom /etc/hosts
entries (e.g. {"host.docker.internal": "host-gateway"} to reach
host services from inside the container on Linux, where that alias is
not auto-provided).
CopilotAI review requested due to automatic review settings May 25, 2026 07:21

@gemini-code-assistgemini-code-assistBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the extra_hosts configuration for Docker sandboxes, allowing custom host-to-IP mappings in /etc/hosts. It also improves the shutdown logic in local_manager.py to prevent asyncio warnings when cleaning up tasks across different event loops. Feedback suggests further refining the task cleanup by checking if the task is already done and ensuring internal warning suppression is applied if a RuntimeError occurs during the await process.

Comment threadms_enclave/sandbox/manager/local_manager.py Outdated
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes two targeted improvements in ms_enclave’s sandbox runtime: (1) hardens LocalSandboxManager.stop() against interpreter-exit shutdown when the cleanup task’s owning event loop has already been closed, and (2) adds support for passing Docker extra_hosts entries via DockerSandboxConfig to improve host reachability from Linux containers.

Changes:

  • Make LocalSandboxManager.stop() tolerate cleanup-task shutdown when the task’s event loop is already closed.
  • Add extra_hosts: Dict[str, str] to DockerSandboxConfig.
  • Forward extra_hosts into Docker container creation in DockerSandbox.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

FileDescription
ms_enclave/sandbox/manager/local_manager.pyAdds cross-loop shutdown handling for the background cleanup task during manager stop.
ms_enclave/sandbox/model/config.pyExtends DockerSandboxConfig with an extra_hosts mapping field.
ms_enclave/sandbox/boxes/docker_sandbox.pyPasses extra_hosts through to docker-py’s container create() call.
Comments suppressed due to low confidence (1)

ms_enclave/sandbox/manager/local_manager.py:71

  • stop() writes to the private Task._log_destroy_pending attribute without any guard. Because this is an internal asyncio implementation detail, it may be missing or become read-only across Python versions/implementations, which would turn shutdown into an AttributeError. Consider using hasattr/setattr inside a try/except (AttributeError, TypeError) and falling back to a best-effort cancel() + dropping the reference when it cannot be modified.
 if task_loop is None or task_loop.is_closed():
self._cleanup_task._log_destroy_pending = False
else:

Comment threadms_enclave/sandbox/manager/local_manager.py
Comment threadms_enclave/sandbox/manager/local_manager.py
…stop
Address review feedback: short-circuit when the cleanup task is already
done, suppress destroy-pending warning on cross-loop RuntimeError, and
drop the task reference so a restarted manager starts clean.
@Yunnglin
Yunnglin merged commit 0aeedd9 into mainMay 25, 2026
1 check passed
Yunnglin added a commit to modelscope/evalscope that referenced this pull request May 25, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Royniel pushed a commit to Royniel/evalscope that referenced this pull request Jun 10, 2026
* feat(external-agent): P0 bridge for third-party CLI agents
Add a reverse-proxy bridge that exposes an Anthropic /v1/messages
endpoint and routes traffic to EvalScope's Model.generate_async,
letting external agent CLIs (claude-code, mock) drive any
DefaultDataAdapter-based benchmark while their LLM calls flow back
through the bridge and into a structured Trajectory.
* HTTP bridge with non-streaming + synthesized SSE Anthropic route
* Per-loop ModelProxyServer keyed on id(loop) with auto-shutdown via
AsyncioLoopRunner.register_close_callback (no fresh port per sample)
* Runner protocol + MockAgentRunner / ClaudeCodeRunner; AgentEnvironment
gains an env= kwarg honored by local + enclave backends
* TaskConfig.external_agent typed as Optional[Any] and validated
lazily so config import does not pull the runner stack
* IDEALAB credentials read from EVALSCOPE_IDEALAB_TOKEN (via .env)
* rewise p0
* update p0
* update p0
* update registry
* update p1
* update web view
* simplify web view
* openai sse proxy bridge
* update webui refactor
* refactor(web): warm-cream light theme + per-theme score vars
- Light theme reworked from cool-slate translation to warm-cream Console:
surface ladder (#faf9f5/#f0ebe1/#fff/#f5f0e7), solid hex hairlines
(#e6dfd8/#d6cdbe/#c1b6a3) replacing translucent violet, warm-ink shadow
tints, warm-grey text ladder. Documented in DESIGN.md.
- Score gradient gains per-theme CSS vars (--score-fg-s/-l/-bg-a-mul) so
yellow mid-tones stay legible on cream without forking the brand HSL.
scoreColor/scoreBg switch to modern CSS color function syntax.
- Chart palette forks per-theme RGB to restore data-bearing punch on cream.
- ScoreRing stroke bumped to 6px (mini)/8px (summary) per spec; ScoreChip
switches to outline style to sidestep yellow-fill legibility on cream.
- Bulk inline-style → Tailwind className across reports/single chat/ui;
CSS-var refs preserved so theme switching is unaffected.
- scoreColor import consolidated to @/utils/colorScale (drop Table re-export).
* openai responses api bridge + codex runner
Adds the third bridge route POST /openai/v1/responses + the OpenAI
Responses SSE event sequence (pre-resolve mode, mirroring inspect_ai)
so codex v0.133+ (which dropped chat completions) can drive
external-agent evaluations through evalscope.
* translate_responses.py: full coverage of codex/OpenAI input items —
message / function_call / function_call_output / reasoning /
custom_tool_call(_output) / computer_call(_output) plus opaque
placeholder rendering for built-in tool items (web_search_call,
mcp_call, mcp_list_tools, file_search_call, code_interpreter_call,
image_generation_call, local_shell_call). item_reference + unknown
types log WARN and skip (forward-compat, no raise).
* sse_responses.py: pre-resolve synthesizer for the full Responses
event sequence (created → in_progress → output_item.added/done × N
→ completed), strictly monotonic sequence_number, 32-char
function_call_arguments chunking matching inspect_ai. Upstream error
path emits OpenAI SDK ResponseErrorEvent shape (flat
type/code/message/param/sequence_number).
* server.py: route registration + handlers. Shared _auth_check_openai
and _prepare_sse_response helpers extracted; anthropic 401 shape
preserved for claude-code compatibility.
* trace_recorder.py: record_responses_turn reusing _record_turn via
new messages_key parameter; _ingest_initial_user_message filters
type=='message' to avoid mis-ingesting function_call_output as user
prompts; _extract_responses_tool_results WARNs once when an output
appears mid-input[] instead of at the tail.
* CodexRunner: positional prompt (avoids ms_enclave stdin gap),
-c model_providers.evalscope.* config injection, npm install
fallback for codex CLI. Setup + run + answer extraction via
--output-last-message.
Tests: 23 new mock cases (event_sequence × 3 spy oracle, round_trip
× 5, tool_calls_streaming × 2, multi_turn × 1, instructions_merge × 3,
previous_response_id_ignored × 1, extended_item_types × 6, error_paths
× 2) + opt-in real codex × qwen3-max e2e (test_codex_runner). All 47
mock tests + lint clean. SWE-bench Pro 1 sample × codex × qwen3-max
verified end-to-end on enclave: 37 turns (35 exec_command +
1 apply_patch + final stop), 828K input / 9.5K output tokens, rc=0,
bridge ERROR/WARN = 0.
* update codex bridge
* external-agent: review cleanup, trace fixes, agent docs reshuffle
Runners
- CodexRunner: hardcode sandbox=workspace-write + always-bypass-approvals;
drop output-last-message kwarg (constant). Remaining knobs: model_name,
extra_args, extra_config, home_override, auto_install, install_timeout_s,
node_setup_url, npm_package. Default kwargs={} now works end-to-end.
- ClaudeCodeRunner: rename install_node -> auto_install for API parity
with CodexRunner.
- Both: expose install_timeout_s (default 600s codex / 300s claude-code)
so slow CDNs / apt mirrors don't need a source edit. Class _INSTALL_TIMEOUT_S
kept as the default sentinel for callers that already passed it.
Trace recorder
- _ingest_initial_messages: capture the FULL initial setup on the first
responses-API turn (top-level instructions + every system/developer item
+ every user item, in document order), not just the first user message.
Fixes the SWE-bench Pro symptom where codex put the task in `instructions`
and the recorded transcript only contained <environment_context>.
- _extract_responses_tool_results: scan the whole input[] and dedup against
already-recorded ChatMessageTool entries, replacing the tail-only
shortcut. codex re-sends every (function_call, function_call_output)
pair interleaved through input[], not clumped at the tail, so the old
logic logged 100+ WARNs per multi-turn run and re-emitted prior tool
results as new events.
Tests
- Four new BridgeTraceRecorder cases for the responses path: instructions
-> ChatMessageSystem, multi-user ingest, no re-ingest on later turns,
empty-user-content edge case.
- Delete tests/agent/external/test_real_claude_code.py and test_streaming.py
(subsumed by qwen_via_claude_code + walking_skeleton). Rename
test_responses_instructions_merge.py -> test_responses_input_assembly.py
to match what it actually covers.
Docs
- Split docs/{en,zh}/user_guides/agent.md into a subdir with index.md
(overview) + native.md (AgentLoop / Strategy / Environment) + bridge.md
(external CLI bridge: claude-code, codex). Cross-link from sandbox /
parameters / visualization / READMEs / index.
Scripts
- Delete scripts/run_codex_qwen.py, run_swebenchpro_codex_qwen.py,
run_swebenchpro_qwen_cc.py. All three were dev-time one-shot triggers
with no in-tree references; their code paths are now fully covered by
tests/benchmark/test_agent_loop.py (TestSWEBenchProExternalClaudeCode,
TestSWEBenchProExternalCodex). pytest -v -s runs the same end-to-end.
Verified
- 51 mock tests + lint pass
- SWE-bench Pro x codex x qwen3-max (limit=3, 10min) clean exit
- ms-enclave cross-loop shutdown noise gone after upstream PR
(modelscope/ms-enclave#12)
* update codex bridge
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Yunnglin