[LiteLLM] `_get_content` encodes audio `inline_data` as `audio_url` which is not a valid LiteLLM content type

[benjamin-callonnec]

Summary

When an ADK agent backed by LiteLlm receives a message containing an audio inline_data part, the _get_content function in lite_llm.py encodes it as {"type": "audio_url", "audio_url": {"url": "data:audio/...;base64,..."}}. This content type does not exist in the LiteLLM / OpenAI message schema. As a result, the audio data is silently dropped by the provider and the model receives an empty message.

Environment

• google-adk version: 1.31.0
• litellm version: 1.82.6
• Provider: any LiteLLM-compatible endpoint (tested with a custom OpenAI-compatible gateway)
• Python version: 3.11

Steps to Reproduce

1. Create an Agent backed by LiteLlm:

from google.adk.agents import Agent
from google.adk.models.lite_llm import LiteLlm

agent = Agent(
 model=LiteLlm(model="openai/gpt-4o-audio-preview"),
 instruction="You are a helpful assistant.",
)

2. Send a message containing an audio inline_data part (e.g. via the /run HTTP endpoint or runner.run_async):

message = {
 "role": "user",
 "parts": [
 {"text": "What is said in this audio?"},
 {
 "inline_data": {
 "mime_type": "audio/mp4",
 "data": "<base64-encoded audio bytes>",
 }
 },
 ],
}

3. Observe that the model response indicates no audio was received (e.g. "It looks like you forgot to attach the audio file").

Root Cause

The wrong mapping

In lite_llm.py, a lookup table maps MIME type prefixes to LiteLLM content type strings:

# lite_llm.py ~L102
_MEDIA_URL_CONTENT_TYPE_BY_MAJOR_MIME_TYPE = {
 "image": "image_url",
 "video": "video_url",
 "audio": "audio_url", # ← does not exist in LiteLLM
}

How it is used

In _get_content (~L836), when an inline_data part is encountered, the code builds a data URI and dispatches on this mapping:

base64_string = base64.b64encode(part.inline_data.data).decode("utf-8")
data_uri = f"data:{mime_type};base64,{base64_string}"

url_content_type = _media_url_content_type(mime_type) # returns "audio_url"
if url_content_type:
 content_objects.append({
 "type": url_content_type, # "audio_url"
 url_content_type: {"url": data_uri}, # "audio_url": {"url": "data:audio/mp4;base64,..."}
 })

This produces:

{
 "type": "audio_url",
 "audio_url": { "url": "data:audio/mp4;base64,..." }
}

What LiteLLM actually expects

The LiteLLM ChatCompletionUserMessage schema defines the following audio content block (mirroring the OpenAI spec):

{
 "type": "input_audio",
 "input_audio": {
 "data": "<base64-encoded bytes (no data URI prefix)>",
 "format": "mp3" | "wav"
 }
}

There are two mismatches:

	ADK output	LiteLLM expectation
type field	"audio_url"	"input_audio"
Payload key	"audio_url"	"input_audio"
Data format	Full data URI (data:audio/...;base64,…)	Raw base64 string + separate "format" field

audio_url is not a recognised content type in the LiteLLM schema, so providers either reject the block or silently ignore it.

Expected Behaviour

Audio inline_data parts should be serialised as input_audio content blocks:

{
 "type": "input_audio",
 "input_audio": {
 "data": "<raw base64 string>",
 "format": "mp3"
 }
}

Suggested Fix

Replace the "audio" entry in _MEDIA_URL_CONTENT_TYPE_BY_MAJOR_MIME_TYPE with dedicated handling in _get_content, similar to how file content is handled separately:

# Remove "audio" from the URL mapping
_MEDIA_URL_CONTENT_TYPE_BY_MAJOR_MIME_TYPE = {
 "image": "image_url",
 "video": "video_url",
 # "audio" intentionally omitted — handled separately as input_audio
}

# In _get_content, add a branch before the url_content_type check:
elif mime_type.startswith("audio/"):
 # Extract format from mime type (e.g. "audio/mp3" -> "mp3", "audio/mp4" -> "mp4")
 audio_format = mime_type.split("/", 1)[1].split(";")[0]
 # Normalise: mp4/m4a containers are typically mp3-compatible; adjust as needed
 content_objects.append({
 "type": "input_audio",
 "input_audio": {
 "data": base64_string, # raw base64, no data URI prefix
 "format": audio_format,
 },
 })

References

• _get_content function: google/adk/models/lite_llm.py
• _MEDIA_URL_CONTENT_TYPE_BY_MAJOR_MIME_TYPE mapping: lite_llm.py ~L102
• LiteLLM ChatCompletionAudioObject schema: input_audio with data (base64) and format fields