[BOT ISSUE] Anthropic: multimodal inputs (base64 images, PDFs) not processed into Attachments

[braintrust-bot]

Summary

The Anthropic integration logs base64-encoded multimodal input data (images, PDFs, documents) as raw strings in span inputs. The OpenAI and Google GenAI wrappers both convert equivalent binary data to Braintrust Attachment objects, which enables efficient storage and visual preview in the Braintrust UI.

When a user sends an image or PDF to the Anthropic Messages API, the full base64 payload (often hundreds of KB to several MB) is written directly into the span input field. This bloats span storage and provides no visual preview, unlike the other two major provider wrappers.

What is missing

The _get_input_from_kwargs function in py/src/braintrust/integrations/anthropic/tracing.py (line 417) passes messages through without processing multimodal content blocks. Anthropic multimodal content blocks include:

# Image input (base64)
{"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": "<huge base64 string>"}}

# Document input (base64 PDF)
{"type": "document", "source": {"type": "base64", "media_type": "application/pdf", "data": "<huge base64 string>"}}

These should be converted to braintrust.logger.Attachment objects (the same pattern used by OpenAI and Google GenAI wrappers), replacing the raw base64 data with a reference to efficiently stored binary content.

Comparison with other wrappers

Provider	Binary input handling	Implementation
OpenAI	Converts data URLs to Attachment objects	_process_attachments_in_input() in oai.py (line 107)
Google GenAI	Converts bytes to Attachment objects	Attachment(data=data, filename=..., content_type=...) in google_genai/tracing.py (line 85)
Anthropic	Logs raw base64 as-is	_get_input_from_kwargs() in anthropic/tracing.py (line 417) — no attachment processing

Braintrust docs status

unclear — The Anthropic integration page mentions multimodal content and attachments briefly but refers to a separate advanced tracing guide. No explicit statement about whether base64 inputs are converted to Attachments.

Upstream sources

• Anthropic Messages API — Vision: https://docs.anthropic.com/en/docs/build-with-claude/vision
• Anthropic Messages API — PDF support: https://docs.anthropic.com/en/docs/build-with-claude/pdf-support
• Anthropic content block types include image (with base64 source) and document (with base64 source)
• These are stable, GA features in the Anthropic Python SDK

Local files inspected

• py/src/braintrust/integrations/anthropic/tracing.py — _get_input_from_kwargs() (line 417) passes messages through without processing binary content; _start_span() (line 436) logs raw input directly; zero references to Attachment anywhere in the file
• py/src/braintrust/integrations/anthropic/_utils.py — no attachment processing
• py/src/braintrust/integrations/anthropic/test_anthropic.py — no tests for multimodal/image/PDF inputs
• py/src/braintrust/oai.py — _process_attachments_in_input() (line 107) demonstrates the expected pattern for converting data URLs to Attachment objects
• py/src/braintrust/integrations/google_genai/tracing.py — Attachment(data=data, ...) (line 85) demonstrates the same pattern for binary content

[AbhiPrasad]

released with https://github.com/braintrustdata/braintrust-sdk-python/releases/tag/py-sdk-v0.13.0