[bot] Add Groq Python SDK integration for chat completions, audio, and built-in tools instrumentation (6,267,274 weekly downloads)

[braintrust-bot]

Summary

Add native Braintrust instrumentation for the official Groq Python SDK (groq). The integration must support package-level setup, auto_instrument(), and manual wrap_groq() instrumentation for Groq and AsyncGroq clients.

Using Groq through openai.OpenAI(base_url="https://api.groq.com/openai/v1") is not an equivalent substitute: the OpenAI integration currently records metadata.provider = "openai", while Groq pricing attribution requires metadata.provider = "groq".

The implementation should use the TypeScript Groq integration as behavioral prior art while following the current Braintrust instrumentation specification as the source of truth.

Supported versions and surfaces

Support groq >= 1.0.0, with explicit matrix coverage for the latest pinned version and 1.0.0.

Instrument both sync and async forms of:

• client.chat.completions.create()
• client.embeddings.create()
• client.audio.transcriptions.create()
• client.audio.translations.create()
• client.audio.speech.create()

Span model and naming

Each provider API call produces exactly one llm span, including streaming calls.

Use stable operation names:

• groq.chat.completions.create
• groq.embeddings.create
• groq.audio.transcriptions.create
• groq.audio.translations.create
• groq.audio.speech.create

All spans created by the integration must include:

{
 "context": {
 "span_origin": {
 "instrumentation": { "name": "groq-auto" }
 }
 }
}

Provider exceptions must propagate unchanged, be recorded in the top-level error field, and still end the span.

Chat completions

Canonical payloads

Groq does not have a dedicated Braintrust UI normalizer, so chat payloads must use OpenAI Chat Completions format:

• input: the ordered messages array itself, not { "messages": [...] }
• output: the choices array, preserving index, finish_reason, and message
• do not flatten output to only choices[0].message.content

Tool calls must remain OpenAI-shaped. function.arguments must remain a JSON-encoded string through streaming accumulation; malformed JSON must pass through unchanged rather than being parsed or dropped.

Metadata

Every chat span must include:

• metadata.provider = "groq"
• metadata.model, preferring the model returned by Groq over the requested model

Only capture specification-approved request configuration fields when supplied and JSON-serializable:

• temperature
• top_p
• max_tokens
• frequency_penalty
• presence_penalty
• stop
• response_format
• tools
• tool_choice
• parallel_tool_calls
• max_tool_calls if Groq adds/supports it

Omit metadata.tools when no tools were supplied; do not emit an empty array. Preserve provider-native built-in tool declarations and serializable configuration without inventing function schemas.

Do not capture currently unsanctioned request fields such as max_completion_tokens, seed, n, stream, stream_options, service_tier, reasoning_format, or search_settings.

Metrics

Capture only canonical metrics reported by Groq:

• prompt_tokens
• completion_tokens
• tokens
• completion_reasoning_tokens when reported
• prompt_cached_tokens when Groq reports a canonical cache-read token count
• time_to_first_token for every streaming call, measured by Braintrust from request initiation to the first received chunk

For streaming, capture usage from the terminal chunk, including the x_groq.usage fallback when needed.

Do not emit Groq server timing fields (queue_time, prompt_time, completion_time, total_time, or seconds) or provider-specific cache fields as metrics unless they are first added to the specification.

Streaming

A stream must preserve the provider iterator/async-iterator interface. The span remains open until exhaustion, close, context-manager exit, or stream error. Accumulated output must be structurally equivalent to non-streaming choices, including multi-choice content and fragmented tool calls.

Vision and inline media

Recursively materialize supported inline OpenAI-style content parts as Braintrust attachments at the original media leaf. In particular, convert whole-value base64 data URLs under image_url.url; preserve remote URLs and leave malformed or unknown media unchanged. Attachment conversion failures must not affect the provider call or prevent span export.

Groq Compound and built-in tools

Groq Compound calls still go through chat.completions.create() as one request/response. They therefore produce exactly one llm span and no synthetic child tool spans.

Built-in tool declarations supplied in the request belong in metadata.tools with their provider-native type and JSON-serializable configuration preserved.

Groq's provider-executed choices[].message.executed_tools results currently have no sanctioned representation in the instrumentation specification. Do not capture them until the specification defines one. This is not a reason to fabricate child tool spans.

Reasoning

Capture completion_reasoning_tokens when reported. Reasoning content placement for OpenAI-chat-shaped third-party provider output is not currently defined consistently by the specification; do not capture Groq's provider-specific message.reasoning field until the specification resolves that shape.

Embeddings

Follow the embedding feature specification:

• one llm span per request, not per vector
• metadata.provider = "groq"
• resolved metadata.model
• canonical input: { "inputs": [{ "content": ... }] }, preserving scalar/batch order
• compact output: { "count": <number of embeddings> }
• never log vectors, vector prefixes, hashes, norms, or provider-native embedding output
• emit prompt_tokens and tokens when reported; omit completion_tokens

Audio and attachments

Transcription and translation

• Attach inline/local input audio at the original file input leaf.
• Preserve remote url inputs as URLs; do not fetch them only for tracing.
• Log transcript/translation text or returned structured transcript data as output.
• Include metadata.provider = "groq" and the resolved/requested model.
• Groq's x_groq.usage.seconds is not a sanctioned metric and must be omitted.

Speech

• Log input text as normal request input.
• Log generated audio as a Braintrust attachment in output, never in metadata.
• Preserve MIME type and byte size when available.
• Attachment conversion failures must preserve usable output and must not affect the provider response.

Repository plumbing and tests

Acceptance requires:

• py/src/braintrust/integrations/groq/ with public GroqIntegration and wrap_groq() exports
• registration in py/src/braintrust/integrations/__init__.py
• auto_instrument(groq=True) registration plus subprocess import-order/idempotence coverage
• [tool.braintrust.matrix].groq with explicit latest and 1.0.0 pins
• [tool.braintrust.cassette-dirs].groq and [tool.braintrust.vendor-packages] entries
• parametrized test_groq(<version>) nox session
• VCR-backed provider coverage where real Groq payload shape matters, including sync/async, streaming/non-streaming, tool calls, embeddings, and audio where practical
• focused local coverage for attachment fallback and provider error propagation

Backend validation

Validate that emitted Groq span identity and metrics are usable by the Braintrust backend:

• metadata.provider = "groq" is accepted and preserved
• Groq model IDs are preserved exactly as returned
• canonical token fields are eligible for estimated-cost lookup
• representative Groq model IDs (including llama-3.1-8b-instant, openai/gpt-oss-*, and groq/compound*) are recognized by the backend model registry where pricing/model metadata exists
• no backend or UI normalizer is assumed for Groq payload rendering

[AbhiPrasad]

Spec compliance review

Automated review of the requirements stated in this issue against the Braintrust instrumentation specification (braintrust-spec → skills/instrumentation-spec/references/instrumentation-guide.md and references/features/). This reviews the issue text, not any implementation.

Verdict: UNDERSPECIFIED — needs revision before implementation. One central design error (Groq Compound framed as agentic), one closed-list metadata violation, and a large set of MUST requirements the issue never states.
Category: completion API (chat completions + audio). Groq Compound built-in tools are also a completion API, not an agentic surface — see below.
Already implemented in this repo: no. There is no py/src/braintrust/integrations/groq/, no wrap_groq(), no test_groq nox session, no [tool.braintrust.matrix] entry, and no versioning.py entry. The only groq match outside cassette YAML is an incidental model-name string in py/src/braintrust/integrations/pydantic_ai/tracing.py. The issue's "No coverage in any instrumentation layer" section is accurate.

---

Correction to the base_url reachability claim

The issue says:

While Groq exposes an OpenAI-compatible endpoint that can be used via openai.OpenAI(base_url="https://api.groq.com/openai/v1") + wrap_openai() […]

This works mechanically — the gateway pattern is already exercised for OpenRouter in py/src/braintrust/integrations/openai/test_openai_openrouter_gateway.py — but it is spec-non-conformant today. _filter_metadata hardcodes the provider with no base_url detection:

# py/src/braintrust/integrations/openai/tracing.py:166
metadata: dict[str, Any] = {"provider": "openai"}

So every Groq-via-wrap_openai span reports metadata.provider = "openai". That violates features/token-and-cost-metrics.md §"Span identity", which requires metadata.provider to be "the provider, gateway, or reseller whose pricing applies to the call", and notes: "metadata.provider is required even when a model name appears globally unique. Resellers and gateways can sell the same model at different prices." Cost attribution for Groq-hosted models is therefore wrong on that path.

This strengthens the case for a native integration, and the issue should say so explicitly rather than presenting the base_url route as an adequate-but-inconvenient alternative.

---

Conflicts with the spec

1. Groq Compound built-in tools are framed as an agentic surface. This is the issue's central design error.

The issue says:

Additionally, Groq Compound's built-in tools (web_search, code_interpreter, browser_automation, wolfram_alpha, visit_website) are Groq-specific agentic execution surfaces where the model autonomously calls server-side tools and handles the agentic loop internally.

and:

Built-in tools are passed via the tools parameter with special tool identifiers. The model autonomously calls these tools on Groq's servers and returns the complete result. This is an agentic execution surface unique to Groq that has no equivalent in the OpenAI SDK.

This reads as a request for a task parent span with child llm and tool spans. That is wrong. These calls go through client.chat.completions.create(), which is a single request-response call, so per instrumentation-guide.md §"Completion APIs" and §"One span per API call":

Each discrete API call MUST produce exactly one llm span.

and:

If the model returns tool calls, the completion API span captures them in its output, but does NOT execute them or create tool spans.

The agentic span tree in §"Agentic API Instrumentation" applies when the SDK/framework runs the loop (Who executes tools? SDK / framework in the comparison table in §"Completion APIs vs Agentic APIs"). Here the loop runs entirely server-side inside one HTTP request; the Python client is a plain completion client. Fabricating child tool spans for executions the SDK never performed would invent span boundaries, timings, and parentage that the SDK cannot observe.

Correction: rewrite this section to state that Compound calls produce exactly one llm span, identical in shape to any other chat.completions.create() call. The request-side declarations are already covered by §"Provider-native tool types":

Some providers offer built-in tools that are not user-defined functions (e.g. Anthropic's computer_use, text_editor, bash; OpenAI's web_search, file_search). These MUST remain in metadata.tools if the user supplied them, since they affect the model's behavior. Preserve the provider-native type and JSON-serializable configuration for these tools; do not provide fake function names, input schemas, or output schemas.

So {"type": "browser_search"}, {"type": "code_interpreter"}, etc. go into metadata.tools with their native type preserved. The result side is a separate, genuinely blocking problem — see "Blocked on spec changes" below.

2. The proposed metadata key set exceeds the spec's closed allowlist.

The issue says:

chat.completions.create() parameters relevant for span metadata: model, temperature, max_tokens, max_completion_tokens, top_p, frequency_penalty, presence_penalty, seed, tools, tool_choice, response_format, stop, n.

instrumentation-guide.md §"Metadata" is a closed list:

Instrumentation MAY include only the following LLM request configuration fields in metadata when they are present and JSON-serializable: temperature, top_p, max_tokens, frequency_penalty, presence_penalty, stop, and response_format.

[…] Any additional metadata field requires a specification update before SDKs emit it.

Plus model and provider (both MUST, same section) and the tool fields from §"Available tool definitions" (tools, tool_choice, parallel_tool_calls, max_tool_calls) and §"Prompt metadata" (prompt).

max_completion_tokens, seed, and n are not sanctioned. Emitting them violates §"Data capturing policy":

Instrumentation MUST NOT capture any data or fields unless this guide explicitly requires or allows them. This applies even when the backend can store arbitrary JSON or a provider/framework SDK exposes additional data.

Correction: drop max_completion_tokens, seed, and n from the issue, or land a spec PR adding them first.

Note for whoever implements this: the existing openai and openrouter integrations already emit well beyond the allowlist — service_tier, store, web_search_options, verbosity, moderation in _CHAT_COMPLETION_METADATA_PARAMS (py/src/braintrust/integrations/openai/tracing.py:50), and transforms, route, plugins, models, service_tier in _CHAT_REQUEST_KEYS (py/src/braintrust/integrations/openrouter/tracing.py:56). That is pre-existing drift that predates the data-capturing policy. Do not treat it as precedent for a new integration. The same reasoning bars Groq-specific keys such as reasoning_format, search_settings, and service_tier.

3. Streaming must not be represented as metadata.

stream and stream_options are not in the §"Metadata" allowlist. Streaming is expressed through metrics.time_to_first_token (§"Streaming"), not a metadata flag. The openrouter integration's inclusion of stream/stream_options in _CHAT_REQUEST_KEYS is the same pre-existing drift as above.

4. The metric set must be explicitly bounded against x_groq.usage.

The issue says:

Response shape is OpenAI-compatible: ChatCompletion has choices, usage (prompt_tokens, completion_tokens, total_tokens), model, id — standard span metrics extraction should be straightforward, similar to the existing OpenAI integration.

That is accurate but under-bounded. Groq responses additionally carry x_groq.usage with queue_time, prompt_time, completion_time, and total_time (and seconds on transcriptions). None of these are in the metrics list. instrumentation-guide.md §"Metrics Reference" is unambiguous:

SDKs MUST NOT add metrics beyond the keys listed in this guide. Add new metric keys to this specification before emitting them.

features/token-and-cost-metrics.md §"Canonical metrics" is the same closed set. Correction: add an explicit non-goal to the issue — "x_groq.usage timing fields (queue_time, prompt_time, completion_time, total_time, seconds) MUST NOT be emitted as metrics keys." This is an easy and tempting mistake for a provider that volunteers server-side timing data, so it belongs in the issue text, not in a reviewer's head.

---

Missing requirements

1. The output payload format is never specified — and Groq is named in the spec by name.

instrumentation-guide.md §"Canonical payload format":

For all other providers (Mistral, Cohere, Groq, Fireworks, Together, xAI, Perplexity, custom providers, etc.), instrumentation MUST convert payloads into the OpenAI Chat Completions format so that the UI can render them.

Groq is not one of the three providers (OpenAI, Anthropic, Google) with a built-in UI normalizer, so there is no provider-native escape hatch. §"Output capture" requires the output to be an array of choice objects, not an extracted content string:

[
 {
 "index": 0,
 "finish_reason": "stop",
 "message": { "role": "assistant", "content": "..." }
 }
]

Per the field table in §"Tool calls in completion API output": index SHOULD be present, finish_reason MUST, message MUST.

Because Groq's SDK is Stainless-generated and already OpenAI-shaped, satisfying this is simply "log response.choices verbatim as the array." That is exactly what the closest prior art does — py/src/braintrust/integrations/mistral/tracing.py:1058 and py/src/braintrust/integrations/openrouter/tracing.py:553 both log choices directly. The issue should state this requirement rather than leaving "OpenAI-compatible" to be interpreted as "extract choices[0].message.content", which is the failure mode this spec section exists to prevent.

2. The input payload format is never specified.

§"Input capture" → §"Default: OpenAI Chat Completions format": an ordered array of OpenAI-style message objects is the input (not wrapped in {"messages": [...]}). Trivial for Groq, but it needs to be written down.

3. metadata.provider is never mentioned.

MUST on every LLM span (§"Metadata"; features/token-and-cost-metrics.md §"Span identity"). Set provider = "groq". Also from §"Metadata":

The model field SHOULD use the model string from the API response (which may include a version suffix) rather than the string the user passed in the request.

4. Tool-call payload shape is never specified.

§"Tool calls in completion API output", tool call object field table:

function.arguments | string | MUST | The arguments as a JSON-encoded string (not a JSON object). Example: "{\"location\": \"Paris\"}". The string MAY be malformed JSON if the model produced invalid output — in which case it MUST be passed through as-is rather than dropped.

The issue says only:

Tool/function calling: Standard OpenAI-style tool calling with tools parameter and tool_calls in responses.

Groq already returns arguments as a string, so the real requirement is "do not json.loads() it while accumulating streamed tool_calls deltas, and do not drop malformed JSON" — the classic streaming-accumulator bug. Spell it out.

5. metadata.tools / tool_choice handling is never specified.

§"Available tool definitions": tool definitions are request configuration, so they MUST be in metadata.tools and MUST NOT appear in the input messages array. If no tools were passed, metadata.tools MUST be omitted — "do NOT emit an empty array". §"Tool request controls" requires tool_choice to be normalized to the OpenAI vocabulary ("auto", "none", "required", {"type": "function", "function": {"name": "..."}}), and parallel_tool_calls / max_tool_calls preserved when supplied. §"Provider-native tool types" additionally forbids logging executable handlers or other non-JSON-serializable runtime objects.

6. time_to_first_token is omitted.

The issue says:

The integration must handle streaming span lifecycle (start span on call, accumulate chunks, finalize on stream exhaustion).

§"Streaming" → §"Additional metrics":

time_to_first_token | number | Seconds from request initiation to the first chunk received

This metric MUST be captured for all streaming calls. It is measured by the SDK, not reported by the provider.

Also from §"Accumulation": the accumulated input/output MUST be structurally identical to the non-streaming call — i.e. the same choices array — and "The user MUST NOT need to do anything different to get complete logged data for streamed responses."

7. Streaming token capture is omitted.

§"Token counts from stream metadata" requires token metrics to be captured from stream usage metadata. Groq-specific detail worth pinning in the issue: Groq returns usage in the terminal chunk under x_groq.usage and does not require stream_options={"include_usage": True} the way OpenAI does. Only the canonical prompt_tokens / completion_tokens / total_tokens values from that object may be mapped (to prompt_tokens / completion_tokens / tokens); the sibling timing fields must be discarded per conflict #4.

8. Reasoning-token metrics are omitted.

Groq's reasoning models (qwen3, openai/gpt-oss-*, deepseek-r1 distills) report usage.completion_tokens_details.reasoning_tokens. §"Reasoning Models" → §"Additional metrics" and the §"Metrics Reference" table require completion_reasoning_tokens when the provider reports it. features/token-and-cost-metrics.md marks it "MUST when reported". Missing values MUST be omitted, never fabricated as zero.

9. The audio surface is incomplete, and attachment handling is entirely absent.

The issue lists only:

| client.audio.transcriptions.create() | Speech-to-text via Whisper models | Transcription |

Groq also ships client.audio.translations.create() and client.audio.speech.create() (PlayAI TTS models). All three need coverage, and all three have attachment obligations under §"Multimodal / Attachments":

When an instrumented request or response contains inline binary media (images, PDFs, audio, video, etc.), SDKs MUST replace the raw media with Braintrust attachment references inside the log row's input or output payload.

Per §"Output payloads":

• Speech-to-text and OCR: log input media or documents as attachments. Log transcripts, pages, detected text, and structured extraction results as text/JSON.
• Text-to-speech: log input text as normal request input and log generated audio as an attachment.

and §"Multimodal API surfaces":

| Audio transcription / speech generation | Speech-to-text attaches input audio and logs transcript output. Text-to-speech logs text input and attaches generated audio output. |

Generated audio MUST live in output, not metadata ("Generated media MUST be logged in the output payload, not in metadata"). The replacement object shape is defined in §"Attachment reference" (type / content_type / filename / key, with key a UUID v4), and the upload/retry/partial-replacement mechanics are in features/attachments.md. Critically, per §"Multimodal / Attachments": if conversion or upload fails, instrumentation MUST preserve the original payload and MUST NOT throw an exception that prevents span export.

Direct prior art to follow: SpeechWrapper, _AudioFileWrapper, TranscriptionWrapper, and TranslationWrapper at py/src/braintrust/integrations/openai/tracing.py:1643-1874, wired through AudioSpeechPatcher / AudioTranscriptionsPatcher / AudioTranslationsPatcher at py/src/braintrust/integrations/openai/patchers.py:192-277. Note that TranscriptionWrapper there also handles the streaming transcription event protocol (transcript.text.delta / transcript.text.done) including TTFT.

10. Multimodal chat input attachments are omitted.

The issue mentions vision in passing:

| client.chat.completions.create() | Chat completions with tool calling, vision, JSON mode |

but never states the requirement. Groq vision accepts base64 data URIs at image_url.url. §"Input payloads" requires recursive scanning and conversion of data-URL strings, raw base64 with a MIME-supplying field, and bytes/file-like objects, with the reference placed at the exact leaf that held the data. features/attachments.md §"OpenAI" adds the important guard: only replace when the entire text value is a data URI. When a MIME type cannot be determined, leave the value unchanged.

11. Error handling is unspecified.

Failed calls must record the provider exception on the span's top-level error field (§"Span Data Model") and still end the span. See _log_error_and_end_span in py/src/braintrust/integrations/utils.py, used by the openrouter / mistral / cohere integrations.

12. context.span_origin provenance is omitted.

§"Context":

Braintrust-created spans MUST also include span-origin provenance. Omit fields whose values are unknown.

context.span_origin.instrumentation.name identifies the module that created the span, and the same section notes that provider SDKs "are not the span origin and SHOULD continue to appear in metadata.provider". Prior art: py/src/braintrust/integrations/openrouter/tracing.py:21 sets _INSTRUMENTATION = "openrouter-auto" and threads it via internal["instrumentation"] in its start_span shim. Groq should use groq-auto.

Related: because the SDK emits braintrust.-prefixed attributes, features/filter-ai-spans.md is satisfied automatically. Worth one line in the issue so nobody invents a groq.* attribute prefix — that is not in the allowlist (gen_ai., braintrust., llm., ai., traceloop.) and such spans would be silently discarded when BRAINTRUST_FILTER_AI_SPANS=true.

13. Span naming is unspecified.

§"Span naming": names SHOULD reflect provider and operation; SDKs MAY define names for additional providers. Prior art uses "Chat Completion" (py/src/braintrust/integrations/openai/tracing.py:576). Pick one and record it in the issue so it does not get decided ad hoc in review.

14. Repo plumbing is listed as absent but not required.

The issue's "No coverage in any instrumentation layer" section correctly enumerates what is missing but does not turn it into acceptance criteria. Per CLAUDE.md, a new integration needs: a [tool.braintrust.matrix].groq block with an explicit latest pin plus older versions, a [tool.braintrust.cassette-dirs] entry (so py/scripts/check-stale-cassettes.py can detect orphans), a min_version gate in the integration class and py/src/braintrust/integrations/versioning.py, registration in py/src/braintrust/integrations/__init__.py and auto_instrument(), and a parametrized test_groq(<version>) session in py/noxfile.py. VCR-backed cassettes should cover sync × async, streaming × non-streaming, tool calls, reasoning models, vision input, and each audio endpoint. The issue asserts

All methods have async variants on AsyncGroq with identical signatures.

without making async coverage a requirement.

---

Blocked on spec changes (braintrustdata/braintrust-spec)

1. Provider-executed (server-side) tool results have no sanctioned representation anywhere. This blocks the most differentiated Groq feature.

Groq Compound returns the server-side execution trace on the assistant message (choices[].message.executed_tools — search result sets, code-interpreter stdout, browser navigation steps). The spec covers the request side (§"Provider-native tool types" keeps built-in tools in metadata.tools) and forbids synthesizing child tool spans in a completion API (§"Completion APIs"), but defines no output location for results of tools the provider executed inside a single completion call.

Under §"Data capturing policy" —

Instrumentation MUST NOT capture any data or fields unless this guide explicitly requires or allows them.

— that data currently MUST NOT be captured at all. The net effect: a Groq Compound span would show the final assistant text and nothing about the searches, code executions, or browsing that produced it. That is the single most valuable observability signal for the feature, and it is unrepresentable.

This is not Groq-specific. The same gap applies to OpenAI's web_search / file_search output items and Anthropic's server-side tools. It needs a spec decision — most plausibly an allowance to preserve provider-native executed-tool blocks within the message object of the choice, since the output is already required to be OpenAI-Chat-Completions-shaped for Groq and there is no UI normalizer to lean on.

Until that spec change lands, the Compound portion of this issue cannot be implemented conformantly. The chat-completions and audio portions are unblocked.

2. Reasoning content has no defined home in the choices-array format.

§"Reasoning Models" → §"Output structure" shows a Responses-API-shaped array:

[
 { "type": "reasoning", "summary": [{ "type": "summary_text", "text": "..." }] },
 { "type": "message", "role": "assistant", "status": "completed", "content": [...] }
]

That is not the OpenAI Chat Completions choices array that §"Output capture" mandates for Groq. Groq surfaces reasoning at choices[].message.reasoning when reasoning_format="parsed", or inline in <think> tags when reasoning_format="raw". The spec does not say whether message.reasoning may be preserved on the choice object, and it does not sanction reasoning_format or reasoning_effort as metadata keys (reasoning_effort appears in this repo's openai allowlist at py/src/braintrust/integrations/openai/tracing.py:50 but is absent from the spec's §"Metadata" list). Needs a ruling before the issue can state a requirement. Note that completion_reasoning_tokens (the metric) is not blocked — only the reasoning content placement and the reasoning_format metadata key are.

3. Audio transcription/translation spans have no usable usage metric.

Groq bills audio by audio-seconds and reports x_groq.usage.seconds; there are no token counts on these responses. The closed metric list in §"Metrics Reference" and features/token-and-cost-metrics.md §"Canonical metrics" has no duration or seconds key, and prompt_audio_tokens does not apply (it is defined as a subset of prompt_tokens, which transcription responses do not report). Consequence: conformant Groq audio spans will carry zero usage metrics and no cost attribution. Flagging for spec owners; it does not block implementation, it just bounds what those spans can contain.

4. Model-ID normalization for reseller-hosted third-party models is undefined.

provider: "groq" is correct for pricing attribution, but metadata.model will be values like llama-3.3-70b-versatile or openai/gpt-oss-120b — the latter carrying a vendor prefix the pricing registry may key differently. features/token-and-cost-metrics.md §"Cost computation semantics" notes registry lookup "may still be model-keyed in some backend paths" but defines no normalization rule for reseller-hosted model IDs. OpenRouter solves this ad hoc via _parse_openrouter_model_string (py/src/braintrust/integrations/openrouter/tracing.py:171); there is no general rule to follow.

---

Suggested edits to this issue

• Rewrite the "Built-in tools / Groq Compound" section. Remove the "agentic execution surface" / "handles the agentic loop internally" framing. Replace with: Compound calls go through chat.completions.create() and MUST produce exactly one llm span with no children; built-in tool declarations are preserved provider-native in metadata.tools per §"Provider-native tool types"; capture of executed_tools results is blocked pending a braintrust-spec change (link the gap).
• Add an explicit payload-format section: input = OpenAI Chat Completions messages array; output = the choices array with index / finish_reason / message preserved (not a flattened content string). Cite that the spec names Groq explicitly in §"Canonical payload format". Reference mistral/tracing.py:1058 and openrouter/tracing.py:553 as the pattern.
• Add metadata.provider = "groq" and metadata.model-from-response as MUST requirements.
• Trim the metadata list to temperature, top_p, max_tokens, frequency_penalty, presence_penalty, stop, response_format, plus tools / tool_choice / parallel_tool_calls / max_tool_calls. Explicitly strike max_completion_tokens, seed, n, stream, stream_options, and pre-emptively strike service_tier, reasoning_format, and search_settings.
• Add a metrics section: prompt_tokens, completion_tokens, tokens; completion_reasoning_tokens when reported; time_to_first_token for all streaming calls. Add the explicit non-goal: x_groq.usage timing fields (queue_time, prompt_time, completion_time, total_time, seconds) MUST NOT be emitted as metrics.
• Add a tool-call requirement: function.arguments stays a JSON-encoded string through streaming accumulation; malformed JSON is passed through as-is, never dropped or reparsed.
• Expand the audio section from one endpoint to three — audio.transcriptions.create(), audio.translations.create(), audio.speech.create() — and add the attachment requirements: attach input audio for STT, attach generated audio in output (never metadata) for TTS. Cite openai/tracing.py:1643-1874 as the reference implementation.
• Add multimodal chat-input attachment conversion for base64 image_url data URIs, with the whole-value-only guard from features/attachments.md §"OpenAI".
• Add sections for error capture (top-level error, span still ends, never swallow) and context.span_origin (instrumentation = "groq-auto", per openrouter/tracing.py:21), plus a note that no groq.* attribute prefix may be invented (it would be dropped by BRAINTRUST_FILTER_AI_SPANS).
• Fix the span name — propose "Chat Completion", matching openai/tracing.py:576.
• Correct the base_url claim to note it emits provider: "openai" (openai/tracing.py:166) and is therefore non-conformant for cost attribution — this is a stronger argument for the native integration than "it's a distinct client class".
• Promote the repo-plumbing list to acceptance criteria: [tool.braintrust.matrix].groq pins, [tool.braintrust.cassette-dirs] entry, versioning.py min-version, integrations/__init__.py + auto_instrument() registration, test_groq(<version>) nox session, and VCR cassettes covering sync × async, streaming × non-streaming, tool calls, reasoning, vision, and each audio endpoint.