Skip to content

fix: flush buffered OTEL logs - #2772

Open
yashrajshuklaaa wants to merge 2 commits into
kagent-dev:mainfrom
yashrajshuklaaa:fix/flush-otel-logger-provider
Open

fix: flush buffered OTEL logs#2772
yashrajshuklaaa wants to merge 2 commits into
kagent-dev:mainfrom
yashrajshuklaaa:fix/flush-otel-logger-provider

Conversation

@yashrajshuklaaa

Copy link
Copy Markdown
Contributor

What

Fix ForceFlush() so it flushes both the tracer and logger providers.

Why

GenAI audit logs are buffered by the OTEL batch log processor. When an Agent Substrate actor checkpoints right after the A2A response closes, those buffered logs can be left behind and never exported.

Changes

  • Flush the global LoggerProvider in ForceFlush().
  • Keep tracer flushing working as before.
  • Use the same timeout for both providers.
  • Added a regression test to make sure buffered logs are exported, even when the request context is already cancelled.
  • Updated the related comment from spans to telemetry.

Validation

  • go test ./adk/pkg/telemetry ./adk/pkg/a2a/server
  • go build ./adk/pkg/telemetry ./adk/pkg/a2a/server

All passed.

Fixes #2759

Signed-off-by: Yashraj Shukla <shuklayashraj68@gmail.com>
@github-actions github-actions Bot added the bug Something isn't working label Sep 9, 2026
@EItanya

EItanya commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

@krisztianfekete

@krisztianfekete krisztianfekete left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Would you mind also fixing this on the Python side, address the comments and add test coverage for both runtimes and both signals?

Comment thread go/adk/pkg/telemetry/tracing.go Outdated
return
}

flushCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), flushTimeout())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we give each provider its own budget, or flush logs first?

Comment thread go/adk/pkg/telemetry/tracing.go Outdated
// session's last message).
// Uses its own detached timeout because the request context is typically
// already canceled by the time deferred cleanup runs. The timeout defaults to
// 3s and is configurable via KAGENT_TRACE_FLUSH_TIMEOUT_MS.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably want to rename this now that it covers both traces and logs.

Signed-off-by: Yashraj Shukla <shuklayashraj68@gmail.com>
@yashrajshuklaaa
yashrajshuklaaa requested a review from a team as a code owner September 10, 2026 16:04
@yashrajshuklaaa

Copy link
Copy Markdown
Contributor Author

@krisztianfekete Thanks for the review. I’ve addressed the comments and updated the PR. fix now covers both Go and Python, flushing logs first while keeping a shared timeout budget. also added test coverage for both logs and traces in both runtimes and renamed the timeout setting to KAGENT_TELEMETRY_FLUSH_TIMEOUT_MS
Validation is passing locally and the changes are pushed to the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ForceFlush never flushes the LoggerProvider, so GenAI audit records are silently dropped when the Actor checkpoints

3 participants