feat(files): let the agent read HEIC photos

[waleedlatif1]

Summary

• iPhone photos reach the model as HEIC, which no vision model accepts — the Claude Messages API takes JPEG, PNG, GIF and WebP only. The agent saw nothing and reported the image as empty.
• 75 HEIC files are already in production, 64 of them in a single workspace uploaded over the last two days.
• sharp cannot cover this: its prebuilt libvips ships libheif with AV1 but not HEVC (sharp.format.heif.input.fileSuffix is ['.avif']), because HEVC is patent-encumbered and excluded from the prebuilt binaries. A real iPhone photo fails with "Security limit exceeded".
• Adds heic-convert (a WebAssembly build of libheif, ~1.05M weekly downloads) as a fallback decoder, and routes the image path on the effective MIME type so a phone upload stored as application/octet-stream is no longer read as an opaque binary the model never sees.

Design

Decoder selection is capability-based, not brand-based. sharp is always tried first; the WASM decoder runs only on bytes sharp could not read. The container brand cannot identify the codec anyway — mif1 carries either HEVC or AV1 — so choosing from it would push AV1 files down the slow path. This mirrors how PhotoPrism layers libvips over libheif.

Running libheif in WASM rather than natively also keeps a historically CVE-prone parser inside a sandbox, which matters more for multi-tenant SaaS than it does for the self-hosted photo apps (Immich, PhotoPrism, Nextcloud) that link it natively.

An undecodable image is now reported as unavailable rather than "too large", and HEIF that neither decoder can read is no longer passed through — handing the model bytes it cannot decode is what produced the confidently-empty description in the first place.

Verified against real files

Sample	Brand	sharp	heic-convert
iPhone photo	heic (HEVC)	❌ "Security limit exceeded"	✅ 2.99MB → 3992×2992 JPEG, ~950ms
libheif example	mif1 (AV1)	✅ decodes natively	not invoked

Type of Change

• Bug fix

Testing

Unit tests for container detection, including that AV1-coded and non-HEIF formats are classified correctly. Decode verified end-to-end against both samples above. 408 tests, typecheck, and check:api-validation pass.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Aug 6, 2026 10:46pm

[cursor]

PR Summary

Medium Risk
Changes copilot file-read vision prep (CPU-heavy transcoding, new WASM dependency) on a user-facing agent path; logic is bounded by existing size limits and null-on-failure handling.

Overview
Copilot VFS vision now treats iPhone HEIC/HEIF photos as real images instead of opaque binaries or empty vision input.

When sharp cannot read metadata (typical for HEVC-coded HEIC), the reader sniffs ISO-BMFF ftyp brands via isHeifContainer and falls back to transcodeHeicToJpeg (heic-convert / libheif WASM), then continues resize/compression. Sharp stays first for formats it handles (e.g. AV1 in mif1).

Image routing uses resolveEffectiveMimeType on the filename so uploads stored as application/octet-stream still go through the image path. Passthrough and re-encode decisions gate on MODEL_SUPPORTED_IMAGE_MIME_TYPES so unsupported formats are re-encoded to JPEG/WebP rather than passed through as undecodable bytes. Failure messaging shifts from “too large” to unavailable when decode or the 5MB vision budget fails.

Adds heic-convert and unit tests for container detection and the transcode null path on invalid bytes.

^{Reviewed by Cursor Bugbot for commit d0889a8. Configure here.}

[greptile-apps]

Greptile Summary

The PR enables Copilot VFS reads of HEIC/HEIF images by trying Sharp first and falling back to a WASM decoder when necessary.

• Resolves effective MIME types from filenames before choosing the image-read path.
• Re-encodes formats unsupported by vision models to JPEG or WebP.
• Detects HEIF through both major and compatible ISO-BMFF brands.
• Adds heic-convert, its type definitions, and focused HEIF detection tests.

Confidence Score: 3/5

The PR does not yet appear safe to merge because tenant-controlled HEIF decoding remains able to consume unbounded in-process memory or CPU.

The complete workspace object can still reach heic-convert before any byte, pixel, timeout, or cancellation bound, leaving the previously reported cross-tenant resource-exhaustion path outstanding.

Files Needing Attention: apps/sim/lib/copilot/vfs/file-reader.ts and apps/sim/lib/uploads/server/heic.ts

Important Files Changed

Filename	Overview
apps/sim/lib/copilot/vfs/file-reader.ts	Routes filename-resolved image types through vision preparation, adds HEIF fallback decoding, and re-encodes unsupported model formats.
apps/sim/lib/uploads/server/heic.ts	Adds HEIF brand detection and WASM transcoding; decoding remains unbounded as previously reported.
apps/sim/lib/uploads/server/heic.test.ts	Covers major and compatible HEIF brands, box boundaries, malformed inputs, and decoder failure.
apps/sim/package.json	Adds pinned runtime and type dependencies for heic-convert.
bun.lock	Locks heic-convert and its HEIF, JPEG, PNG, and WASM transitive dependencies.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Workspace file record] --> B[Resolve effective MIME]
B --> C{Image type?}
C -- No --> D[Existing non-image path]
C -- Yes --> E[Download file buffer]
E --> F[Try Sharp metadata]
F --> G{Decoded?}
G -- No --> H{HEIF container?}
H -- Yes --> I[Transcode with heic-convert]
I --> J[Read JPEG metadata]
H -- No --> K[Reject or supported passthrough]
G -- Yes --> L{Supported format and within limits?}
J --> L
L -- Yes --> M[Pass image to vision model]
L -- No --> N[Resize or re-encode to JPEG/WebP]
N --> M

Loading

_{Reviews (3): Last reviewed commit: "fix(files): read HEIF compatible brands,..." | Re-trigger Greptile}

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 6188438. Configure here.}

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit d0889a8. Configure here.}

[waleedlatif1]

Addressed the resource-exhaustion point in 4fd5ec7 rather than pushing back on it — it was correct. Nothing upstream bounded what could reach the decoder: uploads allow 100MB (validation.ts:15) and prepareImageForVision runs sharp with limitInputPixels: false, so a tenant-controlled file could spend unbounded CPU and memory on a single-threaded WASM decode during one read.

transcodeHeicToJpeg now refuses input above a 20MB ceiling and returns null, which the existing "[Image unavailable]" path already handles. 20MB is generous headroom over any phone photo — a 12MP iPhone HEIC runs 1-4MB — while bounding what one read can cost.

Being explicit about what that does and does not cover: it bounds file size, not pixel count. A small file declaring enormous dimensions is a separate vector, and it stays bounded by libheif's own security limits during parse — those are demonstrably active here, since the sharp attempt on a real iPhone HEIC fails with libheif's Security limit exceeded before any decode happens. There is no timeout or cancellation on the decode itself; the byte ceiling is what bounds its duration.