Issues
Noticed two issues when running Vertex AI Live agents in with both Audio input and text input (audio out only).
1.) Using an architecture with a parent agent and sub agent where both have a callback context tool. Whenever the user transfers to an agent, if they then call a tool, the agent will respond N + 1 times where N is the number of transfers that have happened in the current conversation (e.g., root_agent -> sub_agent -> root_agent -> tool_call: Will respond 3 times). Depending on latency, the audio will fully play fully or the latter responses will cut off the audio of the earlier ones. The transcription will always output all of the responses in full.
2.) When reconnecting using SessionResumption, if an agent transfer has taken place, only the conversation history up to the agent transfer is sent. Since this is a user turn, the agent will then respond answering the query that caused the agent transfer. This only happens with SessionResumption and does not happen when manually resetting the session. Still happens after transferring back to the root agent.
Not sure if these are both Gemini model issues or ADK issues.
Code
Basic Code For replication is here. Would need to be hooked up to a front end. Running through routes that have been added to ADKs get_fast_api_app(). All built off the SSE example available on the ADK docs website (or was it looks to have been taken down now).
Agents
defget_root_agent():
returnLlmAgent(
name="root_agent",
model="gemini-live-2.5-flash", # also: "gemini-live-2.5-flash-preview-native-audio-09-2025"description="The root agent",
instruction="deafault instruction",
tools=[get_math_tool()]
)
defget_sub_agent():
returnLlmAgent(
name="sub_agent",
model="gemini-live-2.5-flash", # also: "gemini-live-2.5-flash-preview-native-audio-09-2025"description="Used whenever the user says they want to talk to the sub agent.",
instruction="default instruction",
tools=[get_math_tool()]
)
defget_agent():
root_agent=get_root_agent()
sub_agent=get_sub_agent()
root_agent.sub_agents= [sub_agent]
sub_agent.parent_agent=root_agentreturnroot_agent
Tool
defget_math_tool():
returnFunctionTool(func=solve_math)
defsolve_math(
tool_context: Optional[ToolContext] =None,
) ->int:
return2+2defget_description():
return"This tool is uesed to get the answer to the math problem"# Set the docstring for the function (required by ADK)solve_math.__doc__=get_description()
NAME=solve_math.__name__
Setup
asyncdefstart_agent_session(
app_name: str, user_id: str, session_id: str,
) ->Tuple[AsyncGenerator, LiveRequestQueue]:
agent=get_agent()
session_service=DatabaseSessionService("sqlite:///./sessions.db")
session=awaitsession_service.get_session(
app_name=app_name, user_id=user_id, session_id=session_id
)
runner=BiDiRunner(
app_name=app_name,
agent=agent,
session_service=session_service,
)
run_config=RunConfig(
streaming_mode=StreamingMode.BIDI,
response_modalities=[types.Modality.AUDIO],
speech_config=types.SpeechConfig(
voice_config=types.VoiceConfig(
prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name=VOICE_NAME)
)
),
output_audio_transcription=types.AudioTranscriptionConfig(),
input_audio_transcription=types.AudioTranscriptionConfig(),
session_resumption=types.SessionResumptionConfig(),
save_live_audio=False,
)
live_request_queue=LiveRequestQueue()
live_events=runner.run_live(
session=session,
live_request_queue=live_request_queue,
run_config=run_config,
)
returnlive_events, live_request_queueCommunication
asyncdefagent_to_client_sse(
live_events: AsyncGenerator, current_session_id: str
) ->AsyncGenerator[str, None]:
try:
asyncforeventinlive_events:
# Check for turn completion or interruptionifevent.turn_completeorevent.interrupted:
ifevent.interrupted:
message= {
"type": "interrupted",
"data": "Response interrupted by user input",
}
yieldf"data: {json.dumps(message)}\n\n"ifevent.turn_complete:
message= {
"type": "turn_complete",
"session_id": current_session_id,
}
yieldf"data: {json.dumps(message)}\n\n"continueif (
hasattr(event, "session_resumption_update")
andevent.session_resumption_update
):
update=event.session_resumption_updateifupdate.resumableandupdate.new_handle:
current_session_id=update.new_handlemessage= {"type": "session_id", "data": current_session_id}
yieldf"data: {json.dumps(message)}\n\n"# Handle contentifevent.contentandevent.content.parts:
forpartinevent.content.parts:
ifhasattr(part, "inline_data") andpart.inline_data:
ifpart.inline_data.mime_type.startswith("audio/pcm"):
audio_data=part.inline_data.dataifaudio_data:
message= {
"type": "audio",
"data": base64.b64encode(audio_data).decode(
"ascii"
),
}
yieldf"data: {json.dumps(message)}\n\n"continueifevent.output_transcription:
output_texts.append(event.output_transcription.text)
message= {
"type": "text",
"role": "model",
"data": event.output_transcription.text,
}
yieldf"data: {json.dumps(message)}\n\n"ifevent.input_transcription:
message= {
"type": "text",
"role": "user",
"data": event.input_transcription.text,
}
yieldf"data: {json.dumps(message)}\n\n"input_texts.append(event.input_transcription.text)
exceptExceptionase:
importtracebacktraceback.print_exc()
error_message= {"type": "error", "data": f"Stream error: {str(e)}"}
yieldf"data: {json.dumps(error_message)}\n\n"asyncdefprocess_client_message(
message_data: Dict[str, Any], live_request_queue: LiveRequestQueue, session_id: str
) ->bool: try:
msg_type=message_data.get("type", "")
ifmsg_type=="audio":
data_b64=message_data.get("data", "")
ifnotdata_b64:
logger.warning("Empty audio payload; dropping")
returnFalsetry:
audio_bytes=base64.b64decode(data_b64, validate=True)
exceptExceptionasde:
logger.warning(f"Invalid base64 audio payload: {de}; dropping")
returnFalseiflen(audio_bytes) ==0or (len(audio_bytes) %2) !=0:
returnFalselive_request_queue.send_realtime(
types.Blob(
data=audio_bytes,
mime_type=f"audio/pcm;rate=16000",
)
)
returnTrueelifmsg_type=="text"ormessage_data.get("mime_type") =="text/plain":
text_content=message_data.get("data", "").strip()
iftext_content:
content=types.Content(
role="user",
parts=[types.Part.from_text(text=text_content)],
)
live_request_queue.send_content(content=content)
else:
returnFalseexceptExceptionase:
importtracebacktraceback.print_exc()
returnFalseRoutes
@app.get(path="/apps/{app_name}/users/{user_id}/sessions/{session_id}/events", )asyncdefsse_endpoint(
app_name: str, user_id: str, session_id: str, is_audio: str="false"
):
try:
live_events, live_request_queue=awaitstart_agent_session(
app_name, user_id, session_id, is_audio=(is_audio.lower() =="true")
)
active_sessions[user_id+session_id] =live_request_queuelogger.info(f"Client #{user_id} connected via SSE, audio mode: {is_audio}")
defcleanup():
active_id=user_id+session_idtry:
live_request_queue.close()
ifactive_idinactive_sessions:
delactive_sessions[active_id]
logger.info(f"Client #{active_id} disconnected from SSE")
exceptExceptionase:
logger.error(f"Error cleaning up session for {active_id}: {e}")
asyncdefevent_generator():
session_info= {
"type": "session_id",
"data": session_id,
}
session_id_message=json.dumps(session_info)
yieldf"data: {session_id_message}\n\n"try:
asyncfordatainagent_to_client_sse(live_events, session_id):
yielddataexceptExceptionase:
logger.error(
f"Error in SSE stream for userId: {user_id}, session_id: {session_id} : {e}"
)
finally:
cleanup()
returnStreamingResponse(
event_generator(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no",
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Headers": "Cache-Control",
},
)
exceptExceptionase:
logger.error(f"Failed to create SSE session for {user_id}: {e}")
raiseHTTPException(status_code=500, detail="Failed to create session")
@app.post(path="/apps/{app_name}/users/{user_id}/sessions/{session_id}/send", )asyncdefsend_message_endpoint(
app_name: str, user_id: str, session_id: str, request: Request
):
try:
live_request_queue=active_sessions.get(user_id+session_id)
ifnotlive_request_queue:
raiseHTTPException(status_code=404, detail="Session not found")
message=awaitrequest.json()
ifmessage.get("type") =="end":
live_request_queue.close()
return {"status": "session ended"}
success=awaitprocess_client_message(
message, live_request_queue, session_id
)
ifnotsuccess:
raiseHTTPException(status_code=400, detail="Failed to process message")
return {"status": "sent"}
exceptHTTPException:
raiseexceptExceptionase:
logger.error(f"Error processing message for {user_id}: {e}")
raiseHTTPException(status_code=500, detail="Internal server error")To Reproduce
Issue 1:
- Transfer between two bidi agents using the gemini models above through Vertex AI API
- Get the agent to call a function
- See multiple texts returned.
Issue 2:
- Transfer between two bidi agents using the gemini models above through Vertex AI API
- Send other messages to the ub agent if you like.
- Wait for session resumption
- Hear agent respond without prompt.
- Print the history of llm_connection.send_history() - (Line 135 of base_llm_flow.py) - during session resumption to see last call is "user" and an agent transfer.
Desktop (please complete the following information):
- OS: Windows (WSL Dev Container)
- Python version(python -V): 3.13.7
- ADK version(pip show google-adk): 1.17.0
Model Information:
- Which model is being used: "gemini-live-2.5-flash" or "gemini-live-2.5-flash-preview-native-audio-09-2025"
Issues
Noticed two issues when running Vertex AI Live agents in with both Audio input and text input (audio out only).
1.) Using an architecture with a parent agent and sub agent where both have a callback context tool. Whenever the user transfers to an agent, if they then call a tool, the agent will respond N + 1 times where N is the number of transfers that have happened in the current conversation (e.g., root_agent -> sub_agent -> root_agent -> tool_call: Will respond 3 times). Depending on latency, the audio will fully play fully or the latter responses will cut off the audio of the earlier ones. The transcription will always output all of the responses in full.
2.) When reconnecting using SessionResumption, if an agent transfer has taken place, only the conversation history up to the agent transfer is sent. Since this is a user turn, the agent will then respond answering the query that caused the agent transfer. This only happens with SessionResumption and does not happen when manually resetting the session. Still happens after transferring back to the root agent.
Not sure if these are both Gemini model issues or ADK issues.
Code
Basic Code For replication is here. Would need to be hooked up to a front end. Running through routes that have been added to ADKs
get_fast_api_app(). All built off the SSE example available on the ADK docs website (or was it looks to have been taken down now).Agents
Tool
Setup
Communication
Routes
To Reproduce
Issue 1:
Issue 2:
Desktop (please complete the following information):
Model Information: