feat(chat): preview agent media on web and mobile - #5047

Open
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2
Open

feat(chat): preview agent media on web and mobile#5047
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2

Conversation

@gabrielelpidio

@gabrielelpidiogabrielelpidio commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Problem

Agent-produced images and saved browser evidence are currently shown as paths instead of useful previews. The previous implementation in #4872 depended on the V1 orchestration layer removed by #2829.

What changed

  • add a durable orchestration V2 image_view turn item and project Codex imageView / saved imageGeneration events into it
  • render V2 image outputs and markdown image/video paths in web and mobile chat
  • resolve workspace media, browser artifacts, and V2 thread images through short-lived signed asset URLs
  • persist preview_snapshot evidence either to a unique server artifact (save: true) or a validated workspace-relative PNG (savePath)
  • harden external-file serving and screenshot writes against traversal and symlink swaps

Codex has a native image event and now emits the first-class V2 item. Other providers still get provider-independent markdown media rendering until their adapters expose an equivalent native event.

Important

This PR is intentionally stacked on #2829. TODO: retarget/rebase it to main after #2829 merges.

Verification

  • 8 focused test files, 111 tests passed
  • contracts, shared, web, and mobile typechecks passed
  • real V2 Codex turn persisted a completed image_view item and rendered the same projection in web and iOS
  • web and iOS both rendered the first-class V2 image output and a relative markdown workspace image; image expansion was also exercised

The server typecheck currently reaches the existing #2829 error in untouched apps/server/src/mcp/toolkits/worktree/registration.test.ts (ServerConfig | WorkspacePaths missing from the expected test context). #2829's own Check and Test jobs are already failing at this base head.

Screenshots

Web — inline previews

web-media-previews-v2

Web — expanded preview

web-media-preview-expanded-v2

Mobile — iOS

mobile-media-previews-v2

Reimplements the media-preview work from #4872 against orchestration V2.

Model: GPT-5.6-Sol
Harness: T3 Code (Codex)


Note

High Risk
Touches signed asset serving, symlink-safe file I/O, and new asset claim types—security-sensitive paths that must stay fail-closed; broad surface across server, contracts, web, and mobile.

Overview
Adds inline image and video previews in web and mobile chat instead of raw paths, reimplemented on orchestration V2 after the V1 layer removal.

Orchestration & feed: Introduces a durable V2 image_view turn item (Codex imageView / saved imageGeneration). Completed items surface as dedicated image-output timeline/feed rows; failed ones stay in the work log.

Markdown media: New MarkdownMedia (web + mobile) resolves markdown img/video via resolveMarkdownMediaSource—direct URLs, thread workspace files, browser-artifact, or thread-image—and loads them through signed asset URLs with loading/error UI and image expand.

Assets & server: Extends AssetResource with browser-artifact and thread-image; video workspace previews use exact-file claims. resolveAsset can return streamed open-file responses. noFollowFile hardens reads/writes and screenshot persistence against traversal/symlinks.

Preview MCP:preview_snapshot can save: true (server artifact + savedScreenshotPath) or savePath (validated workspace PNG); tool hints updated accordingly.

Mobile markdown:renderImage / context lets native selectable markdown delegate image nodes to the same media pipeline.

Reviewed by Cursor Bugbot for commit 0387ef2. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add inline media preview for agent images and videos in web and mobile chat

  • Adds image_view as a new turn item type in the orchestration V2 contract, projected from Codex image/imageGeneration items and emitted as image-output timeline/feed entries in both web and mobile.
  • Web chat renders image-output rows as clickable thumbnail previews via ImageOutputTimelineRow, and markdown img/video tags are now rendered through a new MarkdownMedia component that resolves thread-scoped asset URLs.
  • Mobile chat adds MarkdownMedia and ThreadImageOutput components with loading, unavailable, and tap-to-expand states; custom image rendering is injected into markdown via a new renderImage prop and React context.
  • The preview_snapshot MCP tool now accepts optional save (boolean) or savePath parameters to persist screenshots either to the server's browser-artifacts store or a workspace-relative path, with symlink traversal protection.
  • New AssetResource variants (browser-artifact, thread-image) are added to contracts, with corresponding resolveAsset and issueAssetUrl logic backed by no-follow file opening on Linux, macOS, and other platforms.
  • Risk: resolveAsset now returns an open-file variant with a ReadStream in addition to the existing file variant; callers that exhaustively switch on ResolvedAsset must handle the new case.

Macroscope summarized 0387ef2.

juliusmarmingeand others added 30 commits April 17, 2026 17:29
Co-authored-by: codex <codex@users.noreply.github.com>
- Initialize provider as unchecked in a pending state
- Update initial probe message to reflect session-local status
- Type the runtime effect with `Scope`
- Build the ACP session runtime without wrapping it in `Effect.scoped`
- Use strict TurnId and ProviderItemId parsing in Codex session routing
- Decode in-memory stdio chunks in streaming mode to avoid split UTF-8 corruption
- Transfer session-owned scopes into adapter state
- Ensure runtime scopes close on stop and startup failure
- Add regression coverage for scoped lifecycle cleanup
- Close the managed native event logger when the adapter layer tears down
- Make session runtime close idempotent with an atomic closed flag
- Add coverage for flushing thread native logs on shutdown
- Use codex app-server snapshots for auth, models, and skills
- Remove legacy CLI/config discovery paths and related helpers
- Update tests for the new provider status flow
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
- Document the target orchestration graph, IDs, lifecycles, and capability model
- Add Codex app-server probe fixtures and update the probe test harness
- Introduce orchestration v2 service interfaces and error types
- Add replay runtime, fixtures, and integration coverage
- Update shared contracts and probe transcripts
Co-authored-by: codex <codex@users.noreply.github.com>
- Add Codex adapter and replay harness wiring
- Introduce in-memory orchestration projections and provider registry
- Expand orchestration contracts for turn and runtime events
Co-authored-by: codex <codex@users.noreply.github.com>
- Add context transfer IDs, schemas, and projections
- Support cheap fork creation and Codex native fork rollback
- Cover fork idempotency and replay behavior in tests
- Track remaining projection, context transfer, rollback, capability, and subagent work
- Clarify current V2 baseline and debugger-only follow-ups
- Map fork and merge-back turns into stored handoffs and transfer resolutions
- Add shell snapshot projection support plus coverage tests
- Update replay fixtures and web contracts for the new turn flow
Co-authored-by: codex <codex@users.noreply.github.com>
- Move Codex replay recording into `apps/server`
- Add Claude Agent SDK replay fixtures and test harness
- Update orchestration-v2 fixture scenarios and docs
- Move Claude provider runtime logic into its own module
- Share the SDK query runner between live and replay paths
- Add replay driver error wrapping for unexpected failures
Port orchestration V2 provider adapter wiring to the provider-instance driver registry.
Co-authored-by: codex <codex@users.noreply.github.com>
- persist the selected model on run records
- surface run model selection in the debug UI
- update replay fixtures and contracts for the new field
- Record Claude SDK transcripts across multiple prompts and restart/query modes
- Add approval and tool-call replay coverage for new orchestration fixtures
- Update Claude adapter testkit to model open/prompt/permission frames
- Derive Claude SDK query options from runtime policy
- Add read-only replay fixture and policy mapping tests
- Reuse shared approval-policy fixtures across orchestrator tests
Co-authored-by: codex <codex@users.noreply.github.com>
- add active steering and interrupt-restart replay fixtures
- update Claude adapter/orchestrator turn handling for steering
- refresh replay and integration test coverage
- add interrupt and mid-tool replay fixtures for Claude and Codex
- log Claude Agent SDK protocol frames to native event traces
- project Codex commandExecution start events into orchestration updates
- Map Cursor SDK agents and runs to V2 thread and turn lifecycles
- Update MCP capability, tool, and testing guidance for SDK-based injection

@cursorcursorBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 084f6c7. Configure here.

Comment threadapps/server/src/assets/AssetAccess.ts
@macroscopeapp

macroscopeappBot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

This PR introduces a new feature for previewing agent media in chat, including new turn item types, asset resources, and security-sensitive file handling code (symlink protection, path traversal prevention). The scope includes orchestration changes and new UI components across web and mobile—changes that warrant human review.

You can customize Macroscope's approvability policy. Learn more.

@maria-rcks
maria-rcksforce-pushed the t3code/codex-turn-mapping branch from a543fd4 to 378615bCompareAugust 3, 2026 16:01
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 5 times, most recently from 22bd872 to a27c1ccCompareAugust 10, 2026 17:05
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 6 times, most recently from 519c42a to 4c55679CompareAugust 17, 2026 10:28
@andybergon

Copy link
Copy Markdown

Fresh reproduction after #6433 merged:

  • Server: T3 Code Nightly 0.0.34-nightly.20260824.1172; Android client version not captured.
  • A Codex turn returned three valid PNGs through the image-view tool.
  • Desktop rendered all three in the work stream.
  • Mobile did not expose them as tappable or downloadable images, and the completed turn’s final message could not preserve them as normal attachments.
  • Current source routes normal attachments and workspace Markdown images to the full-screen viewer, but image-view results remain trapped in tool output.

This confirms that the first-class agent image-output path covered by this PR is still needed after #6433.

@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 13 times, most recently from ceea97b to d2f1f51CompareSeptember 2, 2026 18:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@gabrielelpidio@andybergon@juliusmarminge@maria-rcks@mwolson@PixPMusic@nsxdavid@Yusuf007R
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

feat(chat): preview agent media on web and mobile - #5047

Open
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2
Open

feat(chat): preview agent media on web and mobile#5047
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2

Conversation

@gabrielelpidio

@gabrielelpidiogabrielelpidio commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Problem

Agent-produced images and saved browser evidence are currently shown as paths instead of useful previews. The previous implementation in #4872 depended on the V1 orchestration layer removed by #2829.

What changed

  • add a durable orchestration V2 image_view turn item and project Codex imageView / saved imageGeneration events into it
  • render V2 image outputs and markdown image/video paths in web and mobile chat
  • resolve workspace media, browser artifacts, and V2 thread images through short-lived signed asset URLs
  • persist preview_snapshot evidence either to a unique server artifact (save: true) or a validated workspace-relative PNG (savePath)
  • harden external-file serving and screenshot writes against traversal and symlink swaps

Codex has a native image event and now emits the first-class V2 item. Other providers still get provider-independent markdown media rendering until their adapters expose an equivalent native event.

Important

This PR is intentionally stacked on #2829. TODO: retarget/rebase it to main after #2829 merges.

Verification

  • 8 focused test files, 111 tests passed
  • contracts, shared, web, and mobile typechecks passed
  • real V2 Codex turn persisted a completed image_view item and rendered the same projection in web and iOS
  • web and iOS both rendered the first-class V2 image output and a relative markdown workspace image; image expansion was also exercised

The server typecheck currently reaches the existing #2829 error in untouched apps/server/src/mcp/toolkits/worktree/registration.test.ts (ServerConfig | WorkspacePaths missing from the expected test context). #2829's own Check and Test jobs are already failing at this base head.

Screenshots

Web — inline previews

web-media-previews-v2

Web — expanded preview

web-media-preview-expanded-v2

Mobile — iOS

mobile-media-previews-v2

Reimplements the media-preview work from #4872 against orchestration V2.

Model: GPT-5.6-Sol
Harness: T3 Code (Codex)


Note

High Risk
Touches signed asset serving, symlink-safe file I/O, and new asset claim types—security-sensitive paths that must stay fail-closed; broad surface across server, contracts, web, and mobile.

Overview
Adds inline image and video previews in web and mobile chat instead of raw paths, reimplemented on orchestration V2 after the V1 layer removal.

Orchestration & feed: Introduces a durable V2 image_view turn item (Codex imageView / saved imageGeneration). Completed items surface as dedicated image-output timeline/feed rows; failed ones stay in the work log.

Markdown media: New MarkdownMedia (web + mobile) resolves markdown img/video via resolveMarkdownMediaSource—direct URLs, thread workspace files, browser-artifact, or thread-image—and loads them through signed asset URLs with loading/error UI and image expand.

Assets & server: Extends AssetResource with browser-artifact and thread-image; video workspace previews use exact-file claims. resolveAsset can return streamed open-file responses. noFollowFile hardens reads/writes and screenshot persistence against traversal/symlinks.

Preview MCP:preview_snapshot can save: true (server artifact + savedScreenshotPath) or savePath (validated workspace PNG); tool hints updated accordingly.

Mobile markdown:renderImage / context lets native selectable markdown delegate image nodes to the same media pipeline.

Reviewed by Cursor Bugbot for commit 0387ef2. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add inline media preview for agent images and videos in web and mobile chat

  • Adds image_view as a new turn item type in the orchestration V2 contract, projected from Codex image/imageGeneration items and emitted as image-output timeline/feed entries in both web and mobile.
  • Web chat renders image-output rows as clickable thumbnail previews via ImageOutputTimelineRow, and markdown img/video tags are now rendered through a new MarkdownMedia component that resolves thread-scoped asset URLs.
  • Mobile chat adds MarkdownMedia and ThreadImageOutput components with loading, unavailable, and tap-to-expand states; custom image rendering is injected into markdown via a new renderImage prop and React context.
  • The preview_snapshot MCP tool now accepts optional save (boolean) or savePath parameters to persist screenshots either to the server's browser-artifacts store or a workspace-relative path, with symlink traversal protection.
  • New AssetResource variants (browser-artifact, thread-image) are added to contracts, with corresponding resolveAsset and issueAssetUrl logic backed by no-follow file opening on Linux, macOS, and other platforms.
  • Risk: resolveAsset now returns an open-file variant with a ReadStream in addition to the existing file variant; callers that exhaustively switch on ResolvedAsset must handle the new case.

Macroscope summarized 0387ef2.

juliusmarmingeand others added 30 commits April 17, 2026 17:29
Co-authored-by: codex <codex@users.noreply.github.com>
- Initialize provider as unchecked in a pending state
- Update initial probe message to reflect session-local status
- Type the runtime effect with `Scope`
- Build the ACP session runtime without wrapping it in `Effect.scoped`
- Use strict TurnId and ProviderItemId parsing in Codex session routing
- Decode in-memory stdio chunks in streaming mode to avoid split UTF-8 corruption
- Transfer session-owned scopes into adapter state
- Ensure runtime scopes close on stop and startup failure
- Add regression coverage for scoped lifecycle cleanup
- Close the managed native event logger when the adapter layer tears down
- Make session runtime close idempotent with an atomic closed flag
- Add coverage for flushing thread native logs on shutdown
- Use codex app-server snapshots for auth, models, and skills
- Remove legacy CLI/config discovery paths and related helpers
- Update tests for the new provider status flow
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
- Document the target orchestration graph, IDs, lifecycles, and capability model
- Add Codex app-server probe fixtures and update the probe test harness
- Introduce orchestration v2 service interfaces and error types
- Add replay runtime, fixtures, and integration coverage
- Update shared contracts and probe transcripts
Co-authored-by: codex <codex@users.noreply.github.com>
- Add Codex adapter and replay harness wiring
- Introduce in-memory orchestration projections and provider registry
- Expand orchestration contracts for turn and runtime events
Co-authored-by: codex <codex@users.noreply.github.com>
- Add context transfer IDs, schemas, and projections
- Support cheap fork creation and Codex native fork rollback
- Cover fork idempotency and replay behavior in tests
- Track remaining projection, context transfer, rollback, capability, and subagent work
- Clarify current V2 baseline and debugger-only follow-ups
- Map fork and merge-back turns into stored handoffs and transfer resolutions
- Add shell snapshot projection support plus coverage tests
- Update replay fixtures and web contracts for the new turn flow
Co-authored-by: codex <codex@users.noreply.github.com>
- Move Codex replay recording into `apps/server`
- Add Claude Agent SDK replay fixtures and test harness
- Update orchestration-v2 fixture scenarios and docs
- Move Claude provider runtime logic into its own module
- Share the SDK query runner between live and replay paths
- Add replay driver error wrapping for unexpected failures
Port orchestration V2 provider adapter wiring to the provider-instance driver registry.
Co-authored-by: codex <codex@users.noreply.github.com>
- persist the selected model on run records
- surface run model selection in the debug UI
- update replay fixtures and contracts for the new field
- Record Claude SDK transcripts across multiple prompts and restart/query modes
- Add approval and tool-call replay coverage for new orchestration fixtures
- Update Claude adapter testkit to model open/prompt/permission frames
- Derive Claude SDK query options from runtime policy
- Add read-only replay fixture and policy mapping tests
- Reuse shared approval-policy fixtures across orchestrator tests
Co-authored-by: codex <codex@users.noreply.github.com>
- add active steering and interrupt-restart replay fixtures
- update Claude adapter/orchestrator turn handling for steering
- refresh replay and integration test coverage
- add interrupt and mid-tool replay fixtures for Claude and Codex
- log Claude Agent SDK protocol frames to native event traces
- project Codex commandExecution start events into orchestration updates
- Map Cursor SDK agents and runs to V2 thread and turn lifecycles
- Update MCP capability, tool, and testing guidance for SDK-based injection

@cursorcursorBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 084f6c7. Configure here.

Comment threadapps/server/src/assets/AssetAccess.ts
@macroscopeapp

macroscopeappBot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

This PR introduces a new feature for previewing agent media in chat, including new turn item types, asset resources, and security-sensitive file handling code (symlink protection, path traversal prevention). The scope includes orchestration changes and new UI components across web and mobile—changes that warrant human review.

You can customize Macroscope's approvability policy. Learn more.

@maria-rcks
maria-rcksforce-pushed the t3code/codex-turn-mapping branch from a543fd4 to 378615bCompareAugust 3, 2026 16:01
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 5 times, most recently from 22bd872 to a27c1ccCompareAugust 10, 2026 17:05
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 6 times, most recently from 519c42a to 4c55679CompareAugust 17, 2026 10:28
@andybergon

Copy link
Copy Markdown

Fresh reproduction after #6433 merged:

  • Server: T3 Code Nightly 0.0.34-nightly.20260824.1172; Android client version not captured.
  • A Codex turn returned three valid PNGs through the image-view tool.
  • Desktop rendered all three in the work stream.
  • Mobile did not expose them as tappable or downloadable images, and the completed turn’s final message could not preserve them as normal attachments.
  • Current source routes normal attachments and workspace Markdown images to the full-screen viewer, but image-view results remain trapped in tool output.

This confirms that the first-class agent image-output path covered by this PR is still needed after #6433.

@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 13 times, most recently from ceea97b to d2f1f51CompareSeptember 2, 2026 18:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@gabrielelpidio@andybergon@juliusmarminge@maria-rcks@mwolson@PixPMusic@nsxdavid@Yusuf007R
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(chat): preview agent media on web and mobile - #5047

Open
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2
Open

feat(chat): preview agent media on web and mobile#5047
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2

Conversation

@gabrielelpidio

@gabrielelpidiogabrielelpidio commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Problem

Agent-produced images and saved browser evidence are currently shown as paths instead of useful previews. The previous implementation in #4872 depended on the V1 orchestration layer removed by #2829.

What changed

  • add a durable orchestration V2 image_view turn item and project Codex imageView / saved imageGeneration events into it
  • render V2 image outputs and markdown image/video paths in web and mobile chat
  • resolve workspace media, browser artifacts, and V2 thread images through short-lived signed asset URLs
  • persist preview_snapshot evidence either to a unique server artifact (save: true) or a validated workspace-relative PNG (savePath)
  • harden external-file serving and screenshot writes against traversal and symlink swaps

Codex has a native image event and now emits the first-class V2 item. Other providers still get provider-independent markdown media rendering until their adapters expose an equivalent native event.

Important

This PR is intentionally stacked on #2829. TODO: retarget/rebase it to main after #2829 merges.

Verification

  • 8 focused test files, 111 tests passed
  • contracts, shared, web, and mobile typechecks passed
  • real V2 Codex turn persisted a completed image_view item and rendered the same projection in web and iOS
  • web and iOS both rendered the first-class V2 image output and a relative markdown workspace image; image expansion was also exercised

The server typecheck currently reaches the existing #2829 error in untouched apps/server/src/mcp/toolkits/worktree/registration.test.ts (ServerConfig | WorkspacePaths missing from the expected test context). #2829's own Check and Test jobs are already failing at this base head.

Screenshots

Web — inline previews

web-media-previews-v2

Web — expanded preview

web-media-preview-expanded-v2

Mobile — iOS

mobile-media-previews-v2

Reimplements the media-preview work from #4872 against orchestration V2.

Model: GPT-5.6-Sol
Harness: T3 Code (Codex)


Note

High Risk
Touches signed asset serving, symlink-safe file I/O, and new asset claim types—security-sensitive paths that must stay fail-closed; broad surface across server, contracts, web, and mobile.

Overview
Adds inline image and video previews in web and mobile chat instead of raw paths, reimplemented on orchestration V2 after the V1 layer removal.

Orchestration & feed: Introduces a durable V2 image_view turn item (Codex imageView / saved imageGeneration). Completed items surface as dedicated image-output timeline/feed rows; failed ones stay in the work log.

Markdown media: New MarkdownMedia (web + mobile) resolves markdown img/video via resolveMarkdownMediaSource—direct URLs, thread workspace files, browser-artifact, or thread-image—and loads them through signed asset URLs with loading/error UI and image expand.

Assets & server: Extends AssetResource with browser-artifact and thread-image; video workspace previews use exact-file claims. resolveAsset can return streamed open-file responses. noFollowFile hardens reads/writes and screenshot persistence against traversal/symlinks.

Preview MCP:preview_snapshot can save: true (server artifact + savedScreenshotPath) or savePath (validated workspace PNG); tool hints updated accordingly.

Mobile markdown:renderImage / context lets native selectable markdown delegate image nodes to the same media pipeline.

Reviewed by Cursor Bugbot for commit 0387ef2. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add inline media preview for agent images and videos in web and mobile chat

  • Adds image_view as a new turn item type in the orchestration V2 contract, projected from Codex image/imageGeneration items and emitted as image-output timeline/feed entries in both web and mobile.
  • Web chat renders image-output rows as clickable thumbnail previews via ImageOutputTimelineRow, and markdown img/video tags are now rendered through a new MarkdownMedia component that resolves thread-scoped asset URLs.
  • Mobile chat adds MarkdownMedia and ThreadImageOutput components with loading, unavailable, and tap-to-expand states; custom image rendering is injected into markdown via a new renderImage prop and React context.
  • The preview_snapshot MCP tool now accepts optional save (boolean) or savePath parameters to persist screenshots either to the server's browser-artifacts store or a workspace-relative path, with symlink traversal protection.
  • New AssetResource variants (browser-artifact, thread-image) are added to contracts, with corresponding resolveAsset and issueAssetUrl logic backed by no-follow file opening on Linux, macOS, and other platforms.
  • Risk: resolveAsset now returns an open-file variant with a ReadStream in addition to the existing file variant; callers that exhaustively switch on ResolvedAsset must handle the new case.

Macroscope summarized 0387ef2.

juliusmarmingeand others added 30 commits April 17, 2026 17:29
Co-authored-by: codex <codex@users.noreply.github.com>
- Initialize provider as unchecked in a pending state
- Update initial probe message to reflect session-local status
- Type the runtime effect with `Scope`
- Build the ACP session runtime without wrapping it in `Effect.scoped`
- Use strict TurnId and ProviderItemId parsing in Codex session routing
- Decode in-memory stdio chunks in streaming mode to avoid split UTF-8 corruption
- Transfer session-owned scopes into adapter state
- Ensure runtime scopes close on stop and startup failure
- Add regression coverage for scoped lifecycle cleanup
- Close the managed native event logger when the adapter layer tears down
- Make session runtime close idempotent with an atomic closed flag
- Add coverage for flushing thread native logs on shutdown
- Use codex app-server snapshots for auth, models, and skills
- Remove legacy CLI/config discovery paths and related helpers
- Update tests for the new provider status flow
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
- Document the target orchestration graph, IDs, lifecycles, and capability model
- Add Codex app-server probe fixtures and update the probe test harness
- Introduce orchestration v2 service interfaces and error types
- Add replay runtime, fixtures, and integration coverage
- Update shared contracts and probe transcripts
Co-authored-by: codex <codex@users.noreply.github.com>
- Add Codex adapter and replay harness wiring
- Introduce in-memory orchestration projections and provider registry
- Expand orchestration contracts for turn and runtime events
Co-authored-by: codex <codex@users.noreply.github.com>
- Add context transfer IDs, schemas, and projections
- Support cheap fork creation and Codex native fork rollback
- Cover fork idempotency and replay behavior in tests
- Track remaining projection, context transfer, rollback, capability, and subagent work
- Clarify current V2 baseline and debugger-only follow-ups
- Map fork and merge-back turns into stored handoffs and transfer resolutions
- Add shell snapshot projection support plus coverage tests
- Update replay fixtures and web contracts for the new turn flow
Co-authored-by: codex <codex@users.noreply.github.com>
- Move Codex replay recording into `apps/server`
- Add Claude Agent SDK replay fixtures and test harness
- Update orchestration-v2 fixture scenarios and docs
- Move Claude provider runtime logic into its own module
- Share the SDK query runner between live and replay paths
- Add replay driver error wrapping for unexpected failures
Port orchestration V2 provider adapter wiring to the provider-instance driver registry.
Co-authored-by: codex <codex@users.noreply.github.com>
- persist the selected model on run records
- surface run model selection in the debug UI
- update replay fixtures and contracts for the new field
- Record Claude SDK transcripts across multiple prompts and restart/query modes
- Add approval and tool-call replay coverage for new orchestration fixtures
- Update Claude adapter testkit to model open/prompt/permission frames
- Derive Claude SDK query options from runtime policy
- Add read-only replay fixture and policy mapping tests
- Reuse shared approval-policy fixtures across orchestrator tests
Co-authored-by: codex <codex@users.noreply.github.com>
- add active steering and interrupt-restart replay fixtures
- update Claude adapter/orchestrator turn handling for steering
- refresh replay and integration test coverage
- add interrupt and mid-tool replay fixtures for Claude and Codex
- log Claude Agent SDK protocol frames to native event traces
- project Codex commandExecution start events into orchestration updates
- Map Cursor SDK agents and runs to V2 thread and turn lifecycles
- Update MCP capability, tool, and testing guidance for SDK-based injection

@cursorcursorBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 084f6c7. Configure here.

Comment threadapps/server/src/assets/AssetAccess.ts
@macroscopeapp

macroscopeappBot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

This PR introduces a new feature for previewing agent media in chat, including new turn item types, asset resources, and security-sensitive file handling code (symlink protection, path traversal prevention). The scope includes orchestration changes and new UI components across web and mobile—changes that warrant human review.

You can customize Macroscope's approvability policy. Learn more.

@maria-rcks
maria-rcksforce-pushed the t3code/codex-turn-mapping branch from a543fd4 to 378615bCompareAugust 3, 2026 16:01
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 5 times, most recently from 22bd872 to a27c1ccCompareAugust 10, 2026 17:05
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 6 times, most recently from 519c42a to 4c55679CompareAugust 17, 2026 10:28
@andybergon

Copy link
Copy Markdown

Fresh reproduction after #6433 merged:

  • Server: T3 Code Nightly 0.0.34-nightly.20260824.1172; Android client version not captured.
  • A Codex turn returned three valid PNGs through the image-view tool.
  • Desktop rendered all three in the work stream.
  • Mobile did not expose them as tappable or downloadable images, and the completed turn’s final message could not preserve them as normal attachments.
  • Current source routes normal attachments and workspace Markdown images to the full-screen viewer, but image-view results remain trapped in tool output.

This confirms that the first-class agent image-output path covered by this PR is still needed after #6433.

@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 13 times, most recently from ceea97b to d2f1f51CompareSeptember 2, 2026 18:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@gabrielelpidio@andybergon@juliusmarminge@maria-rcks@mwolson@PixPMusic@nsxdavid@Yusuf007R
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(chat): preview agent media on web and mobile - #5047

Open
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2
Open

feat(chat): preview agent media on web and mobile#5047
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2

Conversation

@gabrielelpidio

@gabrielelpidiogabrielelpidio commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Problem

Agent-produced images and saved browser evidence are currently shown as paths instead of useful previews. The previous implementation in #4872 depended on the V1 orchestration layer removed by #2829.

What changed

  • add a durable orchestration V2 image_view turn item and project Codex imageView / saved imageGeneration events into it
  • render V2 image outputs and markdown image/video paths in web and mobile chat
  • resolve workspace media, browser artifacts, and V2 thread images through short-lived signed asset URLs
  • persist preview_snapshot evidence either to a unique server artifact (save: true) or a validated workspace-relative PNG (savePath)
  • harden external-file serving and screenshot writes against traversal and symlink swaps

Codex has a native image event and now emits the first-class V2 item. Other providers still get provider-independent markdown media rendering until their adapters expose an equivalent native event.

Important

This PR is intentionally stacked on #2829. TODO: retarget/rebase it to main after #2829 merges.

Verification

  • 8 focused test files, 111 tests passed
  • contracts, shared, web, and mobile typechecks passed
  • real V2 Codex turn persisted a completed image_view item and rendered the same projection in web and iOS
  • web and iOS both rendered the first-class V2 image output and a relative markdown workspace image; image expansion was also exercised

The server typecheck currently reaches the existing #2829 error in untouched apps/server/src/mcp/toolkits/worktree/registration.test.ts (ServerConfig | WorkspacePaths missing from the expected test context). #2829's own Check and Test jobs are already failing at this base head.

Screenshots

Web — inline previews

web-media-previews-v2

Web — expanded preview

web-media-preview-expanded-v2

Mobile — iOS

mobile-media-previews-v2

Reimplements the media-preview work from #4872 against orchestration V2.

Model: GPT-5.6-Sol
Harness: T3 Code (Codex)


Note

High Risk
Touches signed asset serving, symlink-safe file I/O, and new asset claim types—security-sensitive paths that must stay fail-closed; broad surface across server, contracts, web, and mobile.

Overview
Adds inline image and video previews in web and mobile chat instead of raw paths, reimplemented on orchestration V2 after the V1 layer removal.

Orchestration & feed: Introduces a durable V2 image_view turn item (Codex imageView / saved imageGeneration). Completed items surface as dedicated image-output timeline/feed rows; failed ones stay in the work log.

Markdown media: New MarkdownMedia (web + mobile) resolves markdown img/video via resolveMarkdownMediaSource—direct URLs, thread workspace files, browser-artifact, or thread-image—and loads them through signed asset URLs with loading/error UI and image expand.

Assets & server: Extends AssetResource with browser-artifact and thread-image; video workspace previews use exact-file claims. resolveAsset can return streamed open-file responses. noFollowFile hardens reads/writes and screenshot persistence against traversal/symlinks.

Preview MCP:preview_snapshot can save: true (server artifact + savedScreenshotPath) or savePath (validated workspace PNG); tool hints updated accordingly.

Mobile markdown:renderImage / context lets native selectable markdown delegate image nodes to the same media pipeline.

Reviewed by Cursor Bugbot for commit 0387ef2. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add inline media preview for agent images and videos in web and mobile chat

  • Adds image_view as a new turn item type in the orchestration V2 contract, projected from Codex image/imageGeneration items and emitted as image-output timeline/feed entries in both web and mobile.
  • Web chat renders image-output rows as clickable thumbnail previews via ImageOutputTimelineRow, and markdown img/video tags are now rendered through a new MarkdownMedia component that resolves thread-scoped asset URLs.
  • Mobile chat adds MarkdownMedia and ThreadImageOutput components with loading, unavailable, and tap-to-expand states; custom image rendering is injected into markdown via a new renderImage prop and React context.
  • The preview_snapshot MCP tool now accepts optional save (boolean) or savePath parameters to persist screenshots either to the server's browser-artifacts store or a workspace-relative path, with symlink traversal protection.
  • New AssetResource variants (browser-artifact, thread-image) are added to contracts, with corresponding resolveAsset and issueAssetUrl logic backed by no-follow file opening on Linux, macOS, and other platforms.
  • Risk: resolveAsset now returns an open-file variant with a ReadStream in addition to the existing file variant; callers that exhaustively switch on ResolvedAsset must handle the new case.

Macroscope summarized 0387ef2.

juliusmarmingeand others added 30 commits April 17, 2026 17:29
Co-authored-by: codex <codex@users.noreply.github.com>
- Initialize provider as unchecked in a pending state
- Update initial probe message to reflect session-local status
- Type the runtime effect with `Scope`
- Build the ACP session runtime without wrapping it in `Effect.scoped`
- Use strict TurnId and ProviderItemId parsing in Codex session routing
- Decode in-memory stdio chunks in streaming mode to avoid split UTF-8 corruption
- Transfer session-owned scopes into adapter state
- Ensure runtime scopes close on stop and startup failure
- Add regression coverage for scoped lifecycle cleanup
- Close the managed native event logger when the adapter layer tears down
- Make session runtime close idempotent with an atomic closed flag
- Add coverage for flushing thread native logs on shutdown
- Use codex app-server snapshots for auth, models, and skills
- Remove legacy CLI/config discovery paths and related helpers
- Update tests for the new provider status flow
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
- Document the target orchestration graph, IDs, lifecycles, and capability model
- Add Codex app-server probe fixtures and update the probe test harness
- Introduce orchestration v2 service interfaces and error types
- Add replay runtime, fixtures, and integration coverage
- Update shared contracts and probe transcripts
Co-authored-by: codex <codex@users.noreply.github.com>
- Add Codex adapter and replay harness wiring
- Introduce in-memory orchestration projections and provider registry
- Expand orchestration contracts for turn and runtime events
Co-authored-by: codex <codex@users.noreply.github.com>
- Add context transfer IDs, schemas, and projections
- Support cheap fork creation and Codex native fork rollback
- Cover fork idempotency and replay behavior in tests
- Track remaining projection, context transfer, rollback, capability, and subagent work
- Clarify current V2 baseline and debugger-only follow-ups
- Map fork and merge-back turns into stored handoffs and transfer resolutions
- Add shell snapshot projection support plus coverage tests
- Update replay fixtures and web contracts for the new turn flow
Co-authored-by: codex <codex@users.noreply.github.com>
- Move Codex replay recording into `apps/server`
- Add Claude Agent SDK replay fixtures and test harness
- Update orchestration-v2 fixture scenarios and docs
- Move Claude provider runtime logic into its own module
- Share the SDK query runner between live and replay paths
- Add replay driver error wrapping for unexpected failures
Port orchestration V2 provider adapter wiring to the provider-instance driver registry.
Co-authored-by: codex <codex@users.noreply.github.com>
- persist the selected model on run records
- surface run model selection in the debug UI
- update replay fixtures and contracts for the new field
- Record Claude SDK transcripts across multiple prompts and restart/query modes
- Add approval and tool-call replay coverage for new orchestration fixtures
- Update Claude adapter testkit to model open/prompt/permission frames
- Derive Claude SDK query options from runtime policy
- Add read-only replay fixture and policy mapping tests
- Reuse shared approval-policy fixtures across orchestrator tests
Co-authored-by: codex <codex@users.noreply.github.com>
- add active steering and interrupt-restart replay fixtures
- update Claude adapter/orchestrator turn handling for steering
- refresh replay and integration test coverage
- add interrupt and mid-tool replay fixtures for Claude and Codex
- log Claude Agent SDK protocol frames to native event traces
- project Codex commandExecution start events into orchestration updates
- Map Cursor SDK agents and runs to V2 thread and turn lifecycles
- Update MCP capability, tool, and testing guidance for SDK-based injection

@cursorcursorBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 084f6c7. Configure here.

Comment threadapps/server/src/assets/AssetAccess.ts
@macroscopeapp

macroscopeappBot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

This PR introduces a new feature for previewing agent media in chat, including new turn item types, asset resources, and security-sensitive file handling code (symlink protection, path traversal prevention). The scope includes orchestration changes and new UI components across web and mobile—changes that warrant human review.

You can customize Macroscope's approvability policy. Learn more.

@maria-rcks
maria-rcksforce-pushed the t3code/codex-turn-mapping branch from a543fd4 to 378615bCompareAugust 3, 2026 16:01
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 5 times, most recently from 22bd872 to a27c1ccCompareAugust 10, 2026 17:05
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 6 times, most recently from 519c42a to 4c55679CompareAugust 17, 2026 10:28
@andybergon

Copy link
Copy Markdown

Fresh reproduction after #6433 merged:

  • Server: T3 Code Nightly 0.0.34-nightly.20260824.1172; Android client version not captured.
  • A Codex turn returned three valid PNGs through the image-view tool.
  • Desktop rendered all three in the work stream.
  • Mobile did not expose them as tappable or downloadable images, and the completed turn’s final message could not preserve them as normal attachments.
  • Current source routes normal attachments and workspace Markdown images to the full-screen viewer, but image-view results remain trapped in tool output.

This confirms that the first-class agent image-output path covered by this PR is still needed after #6433.

@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 13 times, most recently from ceea97b to d2f1f51CompareSeptember 2, 2026 18:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@gabrielelpidio@andybergon@juliusmarminge@maria-rcks@mwolson@PixPMusic@nsxdavid@Yusuf007R
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

feat(chat): preview agent media on web and mobile - #5047

Open
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2
Open

feat(chat): preview agent media on web and mobile#5047
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2

Conversation

@gabrielelpidio

@gabrielelpidiogabrielelpidio commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Problem

Agent-produced images and saved browser evidence are currently shown as paths instead of useful previews. The previous implementation in #4872 depended on the V1 orchestration layer removed by #2829.

What changed

  • add a durable orchestration V2 image_view turn item and project Codex imageView / saved imageGeneration events into it
  • render V2 image outputs and markdown image/video paths in web and mobile chat
  • resolve workspace media, browser artifacts, and V2 thread images through short-lived signed asset URLs
  • persist preview_snapshot evidence either to a unique server artifact (save: true) or a validated workspace-relative PNG (savePath)
  • harden external-file serving and screenshot writes against traversal and symlink swaps

Codex has a native image event and now emits the first-class V2 item. Other providers still get provider-independent markdown media rendering until their adapters expose an equivalent native event.

Important

This PR is intentionally stacked on #2829. TODO: retarget/rebase it to main after #2829 merges.

Verification

  • 8 focused test files, 111 tests passed
  • contracts, shared, web, and mobile typechecks passed
  • real V2 Codex turn persisted a completed image_view item and rendered the same projection in web and iOS
  • web and iOS both rendered the first-class V2 image output and a relative markdown workspace image; image expansion was also exercised

The server typecheck currently reaches the existing #2829 error in untouched apps/server/src/mcp/toolkits/worktree/registration.test.ts (ServerConfig | WorkspacePaths missing from the expected test context). #2829's own Check and Test jobs are already failing at this base head.

Screenshots

Web — inline previews

web-media-previews-v2

Web — expanded preview

web-media-preview-expanded-v2

Mobile — iOS

mobile-media-previews-v2

Reimplements the media-preview work from #4872 against orchestration V2.

Model: GPT-5.6-Sol
Harness: T3 Code (Codex)


Note

High Risk
Touches signed asset serving, symlink-safe file I/O, and new asset claim types—security-sensitive paths that must stay fail-closed; broad surface across server, contracts, web, and mobile.

Overview
Adds inline image and video previews in web and mobile chat instead of raw paths, reimplemented on orchestration V2 after the V1 layer removal.

Orchestration & feed: Introduces a durable V2 image_view turn item (Codex imageView / saved imageGeneration). Completed items surface as dedicated image-output timeline/feed rows; failed ones stay in the work log.

Markdown media: New MarkdownMedia (web + mobile) resolves markdown img/video via resolveMarkdownMediaSource—direct URLs, thread workspace files, browser-artifact, or thread-image—and loads them through signed asset URLs with loading/error UI and image expand.

Assets & server: Extends AssetResource with browser-artifact and thread-image; video workspace previews use exact-file claims. resolveAsset can return streamed open-file responses. noFollowFile hardens reads/writes and screenshot persistence against traversal/symlinks.

Preview MCP:preview_snapshot can save: true (server artifact + savedScreenshotPath) or savePath (validated workspace PNG); tool hints updated accordingly.

Mobile markdown:renderImage / context lets native selectable markdown delegate image nodes to the same media pipeline.

Reviewed by Cursor Bugbot for commit 0387ef2. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add inline media preview for agent images and videos in web and mobile chat

  • Adds image_view as a new turn item type in the orchestration V2 contract, projected from Codex image/imageGeneration items and emitted as image-output timeline/feed entries in both web and mobile.
  • Web chat renders image-output rows as clickable thumbnail previews via ImageOutputTimelineRow, and markdown img/video tags are now rendered through a new MarkdownMedia component that resolves thread-scoped asset URLs.
  • Mobile chat adds MarkdownMedia and ThreadImageOutput components with loading, unavailable, and tap-to-expand states; custom image rendering is injected into markdown via a new renderImage prop and React context.
  • The preview_snapshot MCP tool now accepts optional save (boolean) or savePath parameters to persist screenshots either to the server's browser-artifacts store or a workspace-relative path, with symlink traversal protection.
  • New AssetResource variants (browser-artifact, thread-image) are added to contracts, with corresponding resolveAsset and issueAssetUrl logic backed by no-follow file opening on Linux, macOS, and other platforms.
  • Risk: resolveAsset now returns an open-file variant with a ReadStream in addition to the existing file variant; callers that exhaustively switch on ResolvedAsset must handle the new case.

Macroscope summarized 0387ef2.

juliusmarmingeand others added 30 commits April 17, 2026 17:29
Co-authored-by: codex <codex@users.noreply.github.com>
- Initialize provider as unchecked in a pending state
- Update initial probe message to reflect session-local status
- Type the runtime effect with `Scope`
- Build the ACP session runtime without wrapping it in `Effect.scoped`
- Use strict TurnId and ProviderItemId parsing in Codex session routing
- Decode in-memory stdio chunks in streaming mode to avoid split UTF-8 corruption
- Transfer session-owned scopes into adapter state
- Ensure runtime scopes close on stop and startup failure
- Add regression coverage for scoped lifecycle cleanup
- Close the managed native event logger when the adapter layer tears down
- Make session runtime close idempotent with an atomic closed flag
- Add coverage for flushing thread native logs on shutdown
- Use codex app-server snapshots for auth, models, and skills
- Remove legacy CLI/config discovery paths and related helpers
- Update tests for the new provider status flow
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
- Document the target orchestration graph, IDs, lifecycles, and capability model
- Add Codex app-server probe fixtures and update the probe test harness
- Introduce orchestration v2 service interfaces and error types
- Add replay runtime, fixtures, and integration coverage
- Update shared contracts and probe transcripts
Co-authored-by: codex <codex@users.noreply.github.com>
- Add Codex adapter and replay harness wiring
- Introduce in-memory orchestration projections and provider registry
- Expand orchestration contracts for turn and runtime events
Co-authored-by: codex <codex@users.noreply.github.com>
- Add context transfer IDs, schemas, and projections
- Support cheap fork creation and Codex native fork rollback
- Cover fork idempotency and replay behavior in tests
- Track remaining projection, context transfer, rollback, capability, and subagent work
- Clarify current V2 baseline and debugger-only follow-ups
- Map fork and merge-back turns into stored handoffs and transfer resolutions
- Add shell snapshot projection support plus coverage tests
- Update replay fixtures and web contracts for the new turn flow
Co-authored-by: codex <codex@users.noreply.github.com>
- Move Codex replay recording into `apps/server`
- Add Claude Agent SDK replay fixtures and test harness
- Update orchestration-v2 fixture scenarios and docs
- Move Claude provider runtime logic into its own module
- Share the SDK query runner between live and replay paths
- Add replay driver error wrapping for unexpected failures
Port orchestration V2 provider adapter wiring to the provider-instance driver registry.
Co-authored-by: codex <codex@users.noreply.github.com>
- persist the selected model on run records
- surface run model selection in the debug UI
- update replay fixtures and contracts for the new field
- Record Claude SDK transcripts across multiple prompts and restart/query modes
- Add approval and tool-call replay coverage for new orchestration fixtures
- Update Claude adapter testkit to model open/prompt/permission frames
- Derive Claude SDK query options from runtime policy
- Add read-only replay fixture and policy mapping tests
- Reuse shared approval-policy fixtures across orchestrator tests
Co-authored-by: codex <codex@users.noreply.github.com>
- add active steering and interrupt-restart replay fixtures
- update Claude adapter/orchestrator turn handling for steering
- refresh replay and integration test coverage
- add interrupt and mid-tool replay fixtures for Claude and Codex
- log Claude Agent SDK protocol frames to native event traces
- project Codex commandExecution start events into orchestration updates
- Map Cursor SDK agents and runs to V2 thread and turn lifecycles
- Update MCP capability, tool, and testing guidance for SDK-based injection

@cursorcursorBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 084f6c7. Configure here.

Comment threadapps/server/src/assets/AssetAccess.ts
@macroscopeapp

macroscopeappBot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

This PR introduces a new feature for previewing agent media in chat, including new turn item types, asset resources, and security-sensitive file handling code (symlink protection, path traversal prevention). The scope includes orchestration changes and new UI components across web and mobile—changes that warrant human review.

You can customize Macroscope's approvability policy. Learn more.

@maria-rcks
maria-rcksforce-pushed the t3code/codex-turn-mapping branch from a543fd4 to 378615bCompareAugust 3, 2026 16:01
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 5 times, most recently from 22bd872 to a27c1ccCompareAugust 10, 2026 17:05
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 6 times, most recently from 519c42a to 4c55679CompareAugust 17, 2026 10:28
@andybergon

Copy link
Copy Markdown

Fresh reproduction after #6433 merged:

  • Server: T3 Code Nightly 0.0.34-nightly.20260824.1172; Android client version not captured.
  • A Codex turn returned three valid PNGs through the image-view tool.
  • Desktop rendered all three in the work stream.
  • Mobile did not expose them as tappable or downloadable images, and the completed turn’s final message could not preserve them as normal attachments.
  • Current source routes normal attachments and workspace Markdown images to the full-screen viewer, but image-view results remain trapped in tool output.

This confirms that the first-class agent image-output path covered by this PR is still needed after #6433.

@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 13 times, most recently from ceea97b to d2f1f51CompareSeptember 2, 2026 18:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@gabrielelpidio@andybergon@juliusmarminge@maria-rcks@mwolson@PixPMusic@nsxdavid@Yusuf007R
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(chat): preview agent media on web and mobile - #5047

Open
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2
Open

feat(chat): preview agent media on web and mobile#5047
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2

Conversation

@gabrielelpidio

@gabrielelpidiogabrielelpidio commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Problem

Agent-produced images and saved browser evidence are currently shown as paths instead of useful previews. The previous implementation in #4872 depended on the V1 orchestration layer removed by #2829.

What changed

  • add a durable orchestration V2 image_view turn item and project Codex imageView / saved imageGeneration events into it
  • render V2 image outputs and markdown image/video paths in web and mobile chat
  • resolve workspace media, browser artifacts, and V2 thread images through short-lived signed asset URLs
  • persist preview_snapshot evidence either to a unique server artifact (save: true) or a validated workspace-relative PNG (savePath)
  • harden external-file serving and screenshot writes against traversal and symlink swaps

Codex has a native image event and now emits the first-class V2 item. Other providers still get provider-independent markdown media rendering until their adapters expose an equivalent native event.

Important

This PR is intentionally stacked on #2829. TODO: retarget/rebase it to main after #2829 merges.

Verification

  • 8 focused test files, 111 tests passed
  • contracts, shared, web, and mobile typechecks passed
  • real V2 Codex turn persisted a completed image_view item and rendered the same projection in web and iOS
  • web and iOS both rendered the first-class V2 image output and a relative markdown workspace image; image expansion was also exercised

The server typecheck currently reaches the existing #2829 error in untouched apps/server/src/mcp/toolkits/worktree/registration.test.ts (ServerConfig | WorkspacePaths missing from the expected test context). #2829's own Check and Test jobs are already failing at this base head.

Screenshots

Web — inline previews

web-media-previews-v2

Web — expanded preview

web-media-preview-expanded-v2

Mobile — iOS

mobile-media-previews-v2

Reimplements the media-preview work from #4872 against orchestration V2.

Model: GPT-5.6-Sol
Harness: T3 Code (Codex)


Note

High Risk
Touches signed asset serving, symlink-safe file I/O, and new asset claim types—security-sensitive paths that must stay fail-closed; broad surface across server, contracts, web, and mobile.

Overview
Adds inline image and video previews in web and mobile chat instead of raw paths, reimplemented on orchestration V2 after the V1 layer removal.

Orchestration & feed: Introduces a durable V2 image_view turn item (Codex imageView / saved imageGeneration). Completed items surface as dedicated image-output timeline/feed rows; failed ones stay in the work log.

Markdown media: New MarkdownMedia (web + mobile) resolves markdown img/video via resolveMarkdownMediaSource—direct URLs, thread workspace files, browser-artifact, or thread-image—and loads them through signed asset URLs with loading/error UI and image expand.

Assets & server: Extends AssetResource with browser-artifact and thread-image; video workspace previews use exact-file claims. resolveAsset can return streamed open-file responses. noFollowFile hardens reads/writes and screenshot persistence against traversal/symlinks.

Preview MCP:preview_snapshot can save: true (server artifact + savedScreenshotPath) or savePath (validated workspace PNG); tool hints updated accordingly.

Mobile markdown:renderImage / context lets native selectable markdown delegate image nodes to the same media pipeline.

Reviewed by Cursor Bugbot for commit 0387ef2. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add inline media preview for agent images and videos in web and mobile chat

  • Adds image_view as a new turn item type in the orchestration V2 contract, projected from Codex image/imageGeneration items and emitted as image-output timeline/feed entries in both web and mobile.
  • Web chat renders image-output rows as clickable thumbnail previews via ImageOutputTimelineRow, and markdown img/video tags are now rendered through a new MarkdownMedia component that resolves thread-scoped asset URLs.
  • Mobile chat adds MarkdownMedia and ThreadImageOutput components with loading, unavailable, and tap-to-expand states; custom image rendering is injected into markdown via a new renderImage prop and React context.
  • The preview_snapshot MCP tool now accepts optional save (boolean) or savePath parameters to persist screenshots either to the server's browser-artifacts store or a workspace-relative path, with symlink traversal protection.
  • New AssetResource variants (browser-artifact, thread-image) are added to contracts, with corresponding resolveAsset and issueAssetUrl logic backed by no-follow file opening on Linux, macOS, and other platforms.
  • Risk: resolveAsset now returns an open-file variant with a ReadStream in addition to the existing file variant; callers that exhaustively switch on ResolvedAsset must handle the new case.

Macroscope summarized 0387ef2.

juliusmarmingeand others added 30 commits April 17, 2026 17:29
Co-authored-by: codex <codex@users.noreply.github.com>
- Initialize provider as unchecked in a pending state
- Update initial probe message to reflect session-local status
- Type the runtime effect with `Scope`
- Build the ACP session runtime without wrapping it in `Effect.scoped`
- Use strict TurnId and ProviderItemId parsing in Codex session routing
- Decode in-memory stdio chunks in streaming mode to avoid split UTF-8 corruption
- Transfer session-owned scopes into adapter state
- Ensure runtime scopes close on stop and startup failure
- Add regression coverage for scoped lifecycle cleanup
- Close the managed native event logger when the adapter layer tears down
- Make session runtime close idempotent with an atomic closed flag
- Add coverage for flushing thread native logs on shutdown
- Use codex app-server snapshots for auth, models, and skills
- Remove legacy CLI/config discovery paths and related helpers
- Update tests for the new provider status flow
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
- Document the target orchestration graph, IDs, lifecycles, and capability model
- Add Codex app-server probe fixtures and update the probe test harness
- Introduce orchestration v2 service interfaces and error types
- Add replay runtime, fixtures, and integration coverage
- Update shared contracts and probe transcripts
Co-authored-by: codex <codex@users.noreply.github.com>
- Add Codex adapter and replay harness wiring
- Introduce in-memory orchestration projections and provider registry
- Expand orchestration contracts for turn and runtime events
Co-authored-by: codex <codex@users.noreply.github.com>
- Add context transfer IDs, schemas, and projections
- Support cheap fork creation and Codex native fork rollback
- Cover fork idempotency and replay behavior in tests
- Track remaining projection, context transfer, rollback, capability, and subagent work
- Clarify current V2 baseline and debugger-only follow-ups
- Map fork and merge-back turns into stored handoffs and transfer resolutions
- Add shell snapshot projection support plus coverage tests
- Update replay fixtures and web contracts for the new turn flow
Co-authored-by: codex <codex@users.noreply.github.com>
- Move Codex replay recording into `apps/server`
- Add Claude Agent SDK replay fixtures and test harness
- Update orchestration-v2 fixture scenarios and docs
- Move Claude provider runtime logic into its own module
- Share the SDK query runner between live and replay paths
- Add replay driver error wrapping for unexpected failures
Port orchestration V2 provider adapter wiring to the provider-instance driver registry.
Co-authored-by: codex <codex@users.noreply.github.com>
- persist the selected model on run records
- surface run model selection in the debug UI
- update replay fixtures and contracts for the new field
- Record Claude SDK transcripts across multiple prompts and restart/query modes
- Add approval and tool-call replay coverage for new orchestration fixtures
- Update Claude adapter testkit to model open/prompt/permission frames
- Derive Claude SDK query options from runtime policy
- Add read-only replay fixture and policy mapping tests
- Reuse shared approval-policy fixtures across orchestrator tests
Co-authored-by: codex <codex@users.noreply.github.com>
- add active steering and interrupt-restart replay fixtures
- update Claude adapter/orchestrator turn handling for steering
- refresh replay and integration test coverage
- add interrupt and mid-tool replay fixtures for Claude and Codex
- log Claude Agent SDK protocol frames to native event traces
- project Codex commandExecution start events into orchestration updates
- Map Cursor SDK agents and runs to V2 thread and turn lifecycles
- Update MCP capability, tool, and testing guidance for SDK-based injection

@cursorcursorBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 084f6c7. Configure here.

Comment threadapps/server/src/assets/AssetAccess.ts
@macroscopeapp

macroscopeappBot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

This PR introduces a new feature for previewing agent media in chat, including new turn item types, asset resources, and security-sensitive file handling code (symlink protection, path traversal prevention). The scope includes orchestration changes and new UI components across web and mobile—changes that warrant human review.

You can customize Macroscope's approvability policy. Learn more.

@maria-rcks
maria-rcksforce-pushed the t3code/codex-turn-mapping branch from a543fd4 to 378615bCompareAugust 3, 2026 16:01
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 5 times, most recently from 22bd872 to a27c1ccCompareAugust 10, 2026 17:05
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 6 times, most recently from 519c42a to 4c55679CompareAugust 17, 2026 10:28
@andybergon

Copy link
Copy Markdown

Fresh reproduction after #6433 merged:

  • Server: T3 Code Nightly 0.0.34-nightly.20260824.1172; Android client version not captured.
  • A Codex turn returned three valid PNGs through the image-view tool.
  • Desktop rendered all three in the work stream.
  • Mobile did not expose them as tappable or downloadable images, and the completed turn’s final message could not preserve them as normal attachments.
  • Current source routes normal attachments and workspace Markdown images to the full-screen viewer, but image-view results remain trapped in tool output.

This confirms that the first-class agent image-output path covered by this PR is still needed after #6433.

@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 13 times, most recently from ceea97b to d2f1f51CompareSeptember 2, 2026 18:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@gabrielelpidio@andybergon@juliusmarminge@maria-rcks@mwolson@PixPMusic@nsxdavid@Yusuf007R
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(chat): preview agent media on web and mobile - #5047

Open
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2
Open

feat(chat): preview agent media on web and mobile#5047
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2

Conversation

@gabrielelpidio

@gabrielelpidiogabrielelpidio commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Problem

Agent-produced images and saved browser evidence are currently shown as paths instead of useful previews. The previous implementation in #4872 depended on the V1 orchestration layer removed by #2829.

What changed

  • add a durable orchestration V2 image_view turn item and project Codex imageView / saved imageGeneration events into it
  • render V2 image outputs and markdown image/video paths in web and mobile chat
  • resolve workspace media, browser artifacts, and V2 thread images through short-lived signed asset URLs
  • persist preview_snapshot evidence either to a unique server artifact (save: true) or a validated workspace-relative PNG (savePath)
  • harden external-file serving and screenshot writes against traversal and symlink swaps

Codex has a native image event and now emits the first-class V2 item. Other providers still get provider-independent markdown media rendering until their adapters expose an equivalent native event.

Important

This PR is intentionally stacked on #2829. TODO: retarget/rebase it to main after #2829 merges.

Verification

  • 8 focused test files, 111 tests passed
  • contracts, shared, web, and mobile typechecks passed
  • real V2 Codex turn persisted a completed image_view item and rendered the same projection in web and iOS
  • web and iOS both rendered the first-class V2 image output and a relative markdown workspace image; image expansion was also exercised

The server typecheck currently reaches the existing #2829 error in untouched apps/server/src/mcp/toolkits/worktree/registration.test.ts (ServerConfig | WorkspacePaths missing from the expected test context). #2829's own Check and Test jobs are already failing at this base head.

Screenshots

Web — inline previews

web-media-previews-v2

Web — expanded preview

web-media-preview-expanded-v2

Mobile — iOS

mobile-media-previews-v2

Reimplements the media-preview work from #4872 against orchestration V2.

Model: GPT-5.6-Sol
Harness: T3 Code (Codex)


Note

High Risk
Touches signed asset serving, symlink-safe file I/O, and new asset claim types—security-sensitive paths that must stay fail-closed; broad surface across server, contracts, web, and mobile.

Overview
Adds inline image and video previews in web and mobile chat instead of raw paths, reimplemented on orchestration V2 after the V1 layer removal.

Orchestration & feed: Introduces a durable V2 image_view turn item (Codex imageView / saved imageGeneration). Completed items surface as dedicated image-output timeline/feed rows; failed ones stay in the work log.

Markdown media: New MarkdownMedia (web + mobile) resolves markdown img/video via resolveMarkdownMediaSource—direct URLs, thread workspace files, browser-artifact, or thread-image—and loads them through signed asset URLs with loading/error UI and image expand.

Assets & server: Extends AssetResource with browser-artifact and thread-image; video workspace previews use exact-file claims. resolveAsset can return streamed open-file responses. noFollowFile hardens reads/writes and screenshot persistence against traversal/symlinks.

Preview MCP:preview_snapshot can save: true (server artifact + savedScreenshotPath) or savePath (validated workspace PNG); tool hints updated accordingly.

Mobile markdown:renderImage / context lets native selectable markdown delegate image nodes to the same media pipeline.

Reviewed by Cursor Bugbot for commit 0387ef2. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add inline media preview for agent images and videos in web and mobile chat

  • Adds image_view as a new turn item type in the orchestration V2 contract, projected from Codex image/imageGeneration items and emitted as image-output timeline/feed entries in both web and mobile.
  • Web chat renders image-output rows as clickable thumbnail previews via ImageOutputTimelineRow, and markdown img/video tags are now rendered through a new MarkdownMedia component that resolves thread-scoped asset URLs.
  • Mobile chat adds MarkdownMedia and ThreadImageOutput components with loading, unavailable, and tap-to-expand states; custom image rendering is injected into markdown via a new renderImage prop and React context.
  • The preview_snapshot MCP tool now accepts optional save (boolean) or savePath parameters to persist screenshots either to the server's browser-artifacts store or a workspace-relative path, with symlink traversal protection.
  • New AssetResource variants (browser-artifact, thread-image) are added to contracts, with corresponding resolveAsset and issueAssetUrl logic backed by no-follow file opening on Linux, macOS, and other platforms.
  • Risk: resolveAsset now returns an open-file variant with a ReadStream in addition to the existing file variant; callers that exhaustively switch on ResolvedAsset must handle the new case.

Macroscope summarized 0387ef2.

juliusmarmingeand others added 30 commits April 17, 2026 17:29
Co-authored-by: codex <codex@users.noreply.github.com>
- Initialize provider as unchecked in a pending state
- Update initial probe message to reflect session-local status
- Type the runtime effect with `Scope`
- Build the ACP session runtime without wrapping it in `Effect.scoped`
- Use strict TurnId and ProviderItemId parsing in Codex session routing
- Decode in-memory stdio chunks in streaming mode to avoid split UTF-8 corruption
- Transfer session-owned scopes into adapter state
- Ensure runtime scopes close on stop and startup failure
- Add regression coverage for scoped lifecycle cleanup
- Close the managed native event logger when the adapter layer tears down
- Make session runtime close idempotent with an atomic closed flag
- Add coverage for flushing thread native logs on shutdown
- Use codex app-server snapshots for auth, models, and skills
- Remove legacy CLI/config discovery paths and related helpers
- Update tests for the new provider status flow
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
- Document the target orchestration graph, IDs, lifecycles, and capability model
- Add Codex app-server probe fixtures and update the probe test harness
- Introduce orchestration v2 service interfaces and error types
- Add replay runtime, fixtures, and integration coverage
- Update shared contracts and probe transcripts
Co-authored-by: codex <codex@users.noreply.github.com>
- Add Codex adapter and replay harness wiring
- Introduce in-memory orchestration projections and provider registry
- Expand orchestration contracts for turn and runtime events
Co-authored-by: codex <codex@users.noreply.github.com>
- Add context transfer IDs, schemas, and projections
- Support cheap fork creation and Codex native fork rollback
- Cover fork idempotency and replay behavior in tests
- Track remaining projection, context transfer, rollback, capability, and subagent work
- Clarify current V2 baseline and debugger-only follow-ups
- Map fork and merge-back turns into stored handoffs and transfer resolutions
- Add shell snapshot projection support plus coverage tests
- Update replay fixtures and web contracts for the new turn flow
Co-authored-by: codex <codex@users.noreply.github.com>
- Move Codex replay recording into `apps/server`
- Add Claude Agent SDK replay fixtures and test harness
- Update orchestration-v2 fixture scenarios and docs
- Move Claude provider runtime logic into its own module
- Share the SDK query runner between live and replay paths
- Add replay driver error wrapping for unexpected failures
Port orchestration V2 provider adapter wiring to the provider-instance driver registry.
Co-authored-by: codex <codex@users.noreply.github.com>
- persist the selected model on run records
- surface run model selection in the debug UI
- update replay fixtures and contracts for the new field
- Record Claude SDK transcripts across multiple prompts and restart/query modes
- Add approval and tool-call replay coverage for new orchestration fixtures
- Update Claude adapter testkit to model open/prompt/permission frames
- Derive Claude SDK query options from runtime policy
- Add read-only replay fixture and policy mapping tests
- Reuse shared approval-policy fixtures across orchestrator tests
Co-authored-by: codex <codex@users.noreply.github.com>
- add active steering and interrupt-restart replay fixtures
- update Claude adapter/orchestrator turn handling for steering
- refresh replay and integration test coverage
- add interrupt and mid-tool replay fixtures for Claude and Codex
- log Claude Agent SDK protocol frames to native event traces
- project Codex commandExecution start events into orchestration updates
- Map Cursor SDK agents and runs to V2 thread and turn lifecycles
- Update MCP capability, tool, and testing guidance for SDK-based injection

@cursorcursorBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 084f6c7. Configure here.

Comment threadapps/server/src/assets/AssetAccess.ts
@macroscopeapp

macroscopeappBot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

This PR introduces a new feature for previewing agent media in chat, including new turn item types, asset resources, and security-sensitive file handling code (symlink protection, path traversal prevention). The scope includes orchestration changes and new UI components across web and mobile—changes that warrant human review.

You can customize Macroscope's approvability policy. Learn more.

@maria-rcks
maria-rcksforce-pushed the t3code/codex-turn-mapping branch from a543fd4 to 378615bCompareAugust 3, 2026 16:01
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 5 times, most recently from 22bd872 to a27c1ccCompareAugust 10, 2026 17:05
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 6 times, most recently from 519c42a to 4c55679CompareAugust 17, 2026 10:28
@andybergon

Copy link
Copy Markdown

Fresh reproduction after #6433 merged:

  • Server: T3 Code Nightly 0.0.34-nightly.20260824.1172; Android client version not captured.
  • A Codex turn returned three valid PNGs through the image-view tool.
  • Desktop rendered all three in the work stream.
  • Mobile did not expose them as tappable or downloadable images, and the completed turn’s final message could not preserve them as normal attachments.
  • Current source routes normal attachments and workspace Markdown images to the full-screen viewer, but image-view results remain trapped in tool output.

This confirms that the first-class agent image-output path covered by this PR is still needed after #6433.

@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 13 times, most recently from ceea97b to d2f1f51CompareSeptember 2, 2026 18:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@gabrielelpidio@andybergon@juliusmarminge@maria-rcks@mwolson@PixPMusic@nsxdavid@Yusuf007R
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

feat(chat): preview agent media on web and mobile - #5047

Open
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2
Open

feat(chat): preview agent media on web and mobile#5047
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2

Conversation

@gabrielelpidio

@gabrielelpidiogabrielelpidio commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Problem

Agent-produced images and saved browser evidence are currently shown as paths instead of useful previews. The previous implementation in #4872 depended on the V1 orchestration layer removed by #2829.

What changed

  • add a durable orchestration V2 image_view turn item and project Codex imageView / saved imageGeneration events into it
  • render V2 image outputs and markdown image/video paths in web and mobile chat
  • resolve workspace media, browser artifacts, and V2 thread images through short-lived signed asset URLs
  • persist preview_snapshot evidence either to a unique server artifact (save: true) or a validated workspace-relative PNG (savePath)
  • harden external-file serving and screenshot writes against traversal and symlink swaps

Codex has a native image event and now emits the first-class V2 item. Other providers still get provider-independent markdown media rendering until their adapters expose an equivalent native event.

Important

This PR is intentionally stacked on #2829. TODO: retarget/rebase it to main after #2829 merges.

Verification

  • 8 focused test files, 111 tests passed
  • contracts, shared, web, and mobile typechecks passed
  • real V2 Codex turn persisted a completed image_view item and rendered the same projection in web and iOS
  • web and iOS both rendered the first-class V2 image output and a relative markdown workspace image; image expansion was also exercised

The server typecheck currently reaches the existing #2829 error in untouched apps/server/src/mcp/toolkits/worktree/registration.test.ts (ServerConfig | WorkspacePaths missing from the expected test context). #2829's own Check and Test jobs are already failing at this base head.

Screenshots

Web — inline previews

web-media-previews-v2

Web — expanded preview

web-media-preview-expanded-v2

Mobile — iOS

mobile-media-previews-v2

Reimplements the media-preview work from #4872 against orchestration V2.

Model: GPT-5.6-Sol
Harness: T3 Code (Codex)


Note

High Risk
Touches signed asset serving, symlink-safe file I/O, and new asset claim types—security-sensitive paths that must stay fail-closed; broad surface across server, contracts, web, and mobile.

Overview
Adds inline image and video previews in web and mobile chat instead of raw paths, reimplemented on orchestration V2 after the V1 layer removal.

Orchestration & feed: Introduces a durable V2 image_view turn item (Codex imageView / saved imageGeneration). Completed items surface as dedicated image-output timeline/feed rows; failed ones stay in the work log.

Markdown media: New MarkdownMedia (web + mobile) resolves markdown img/video via resolveMarkdownMediaSource—direct URLs, thread workspace files, browser-artifact, or thread-image—and loads them through signed asset URLs with loading/error UI and image expand.

Assets & server: Extends AssetResource with browser-artifact and thread-image; video workspace previews use exact-file claims. resolveAsset can return streamed open-file responses. noFollowFile hardens reads/writes and screenshot persistence against traversal/symlinks.

Preview MCP:preview_snapshot can save: true (server artifact + savedScreenshotPath) or savePath (validated workspace PNG); tool hints updated accordingly.

Mobile markdown:renderImage / context lets native selectable markdown delegate image nodes to the same media pipeline.

Reviewed by Cursor Bugbot for commit 0387ef2. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add inline media preview for agent images and videos in web and mobile chat

  • Adds image_view as a new turn item type in the orchestration V2 contract, projected from Codex image/imageGeneration items and emitted as image-output timeline/feed entries in both web and mobile.
  • Web chat renders image-output rows as clickable thumbnail previews via ImageOutputTimelineRow, and markdown img/video tags are now rendered through a new MarkdownMedia component that resolves thread-scoped asset URLs.
  • Mobile chat adds MarkdownMedia and ThreadImageOutput components with loading, unavailable, and tap-to-expand states; custom image rendering is injected into markdown via a new renderImage prop and React context.
  • The preview_snapshot MCP tool now accepts optional save (boolean) or savePath parameters to persist screenshots either to the server's browser-artifacts store or a workspace-relative path, with symlink traversal protection.
  • New AssetResource variants (browser-artifact, thread-image) are added to contracts, with corresponding resolveAsset and issueAssetUrl logic backed by no-follow file opening on Linux, macOS, and other platforms.
  • Risk: resolveAsset now returns an open-file variant with a ReadStream in addition to the existing file variant; callers that exhaustively switch on ResolvedAsset must handle the new case.

Macroscope summarized 0387ef2.

juliusmarmingeand others added 30 commits April 17, 2026 17:29
Co-authored-by: codex <codex@users.noreply.github.com>
- Initialize provider as unchecked in a pending state
- Update initial probe message to reflect session-local status
- Type the runtime effect with `Scope`
- Build the ACP session runtime without wrapping it in `Effect.scoped`
- Use strict TurnId and ProviderItemId parsing in Codex session routing
- Decode in-memory stdio chunks in streaming mode to avoid split UTF-8 corruption
- Transfer session-owned scopes into adapter state
- Ensure runtime scopes close on stop and startup failure
- Add regression coverage for scoped lifecycle cleanup
- Close the managed native event logger when the adapter layer tears down
- Make session runtime close idempotent with an atomic closed flag
- Add coverage for flushing thread native logs on shutdown
- Use codex app-server snapshots for auth, models, and skills
- Remove legacy CLI/config discovery paths and related helpers
- Update tests for the new provider status flow
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
- Document the target orchestration graph, IDs, lifecycles, and capability model
- Add Codex app-server probe fixtures and update the probe test harness
- Introduce orchestration v2 service interfaces and error types
- Add replay runtime, fixtures, and integration coverage
- Update shared contracts and probe transcripts
Co-authored-by: codex <codex@users.noreply.github.com>
- Add Codex adapter and replay harness wiring
- Introduce in-memory orchestration projections and provider registry
- Expand orchestration contracts for turn and runtime events
Co-authored-by: codex <codex@users.noreply.github.com>
- Add context transfer IDs, schemas, and projections
- Support cheap fork creation and Codex native fork rollback
- Cover fork idempotency and replay behavior in tests
- Track remaining projection, context transfer, rollback, capability, and subagent work
- Clarify current V2 baseline and debugger-only follow-ups
- Map fork and merge-back turns into stored handoffs and transfer resolutions
- Add shell snapshot projection support plus coverage tests
- Update replay fixtures and web contracts for the new turn flow
Co-authored-by: codex <codex@users.noreply.github.com>
- Move Codex replay recording into `apps/server`
- Add Claude Agent SDK replay fixtures and test harness
- Update orchestration-v2 fixture scenarios and docs
- Move Claude provider runtime logic into its own module
- Share the SDK query runner between live and replay paths
- Add replay driver error wrapping for unexpected failures
Port orchestration V2 provider adapter wiring to the provider-instance driver registry.
Co-authored-by: codex <codex@users.noreply.github.com>
- persist the selected model on run records
- surface run model selection in the debug UI
- update replay fixtures and contracts for the new field
- Record Claude SDK transcripts across multiple prompts and restart/query modes
- Add approval and tool-call replay coverage for new orchestration fixtures
- Update Claude adapter testkit to model open/prompt/permission frames
- Derive Claude SDK query options from runtime policy
- Add read-only replay fixture and policy mapping tests
- Reuse shared approval-policy fixtures across orchestrator tests
Co-authored-by: codex <codex@users.noreply.github.com>
- add active steering and interrupt-restart replay fixtures
- update Claude adapter/orchestrator turn handling for steering
- refresh replay and integration test coverage
- add interrupt and mid-tool replay fixtures for Claude and Codex
- log Claude Agent SDK protocol frames to native event traces
- project Codex commandExecution start events into orchestration updates
- Map Cursor SDK agents and runs to V2 thread and turn lifecycles
- Update MCP capability, tool, and testing guidance for SDK-based injection

@cursorcursorBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 084f6c7. Configure here.

Comment threadapps/server/src/assets/AssetAccess.ts
@macroscopeapp

macroscopeappBot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

This PR introduces a new feature for previewing agent media in chat, including new turn item types, asset resources, and security-sensitive file handling code (symlink protection, path traversal prevention). The scope includes orchestration changes and new UI components across web and mobile—changes that warrant human review.

You can customize Macroscope's approvability policy. Learn more.

@maria-rcks
maria-rcksforce-pushed the t3code/codex-turn-mapping branch from a543fd4 to 378615bCompareAugust 3, 2026 16:01
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 5 times, most recently from 22bd872 to a27c1ccCompareAugust 10, 2026 17:05
@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 6 times, most recently from 519c42a to 4c55679CompareAugust 17, 2026 10:28
@andybergon

Copy link
Copy Markdown

Fresh reproduction after #6433 merged:

  • Server: T3 Code Nightly 0.0.34-nightly.20260824.1172; Android client version not captured.
  • A Codex turn returned three valid PNGs through the image-view tool.
  • Desktop rendered all three in the work stream.
  • Mobile did not expose them as tappable or downloadable images, and the completed turn’s final message could not preserve them as normal attachments.
  • Current source routes normal attachments and workspace Markdown images to the full-screen viewer, but image-view results remain trapped in tool output.

This confirms that the first-class agent image-output path covered by this PR is still needed after #6433.

@juliusmarminge
juliusmarmingeforce-pushed the t3code/codex-turn-mapping branch 13 times, most recently from ceea97b to d2f1f51CompareSeptember 2, 2026 18:07
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL1,000+ changed lines (additions + deletions).vouch:trustedPR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@gabrielelpidio@andybergon@juliusmarminge@maria-rcks@mwolson@PixPMusic@nsxdavid@Yusuf007R