gen_ai spans (invoke_agent / execute_tool) omit calling user identity and client IP — add enduser.id / client.address

[slreznit]

Summary

The GenAI observability spans emitted for an AgentCore-hosted agent
(invoke_agent, execute_tool, chat, POST /invocations) contain no identity
of the caller who invoked the agent and no client IP address. The invoking IAM
principal / end user and the real client address are absent from every span and from
the correlated GenAI content log records. This makes it impossible to attribute an
agent session (and its tool calls) to who triggered it using CloudWatch
observability alone.

Observed spans — no identity/network attributes

For a full invocation I inspected every span (verified live). None carry any
caller-identity or client-IP attribute:

• invoke_agent Strands Agents — has gen_ai.*, gen_ai.usage.*, harness.id,
  session.id; no user or client address.
• execute_tool <tool> — has gen_ai.tool.*, session.id; no user or client address.
• chat / chat <model> — model attrs only.
• POST /invocations (LOCAL_ROOT) — has only internal transport values
  (net.peer.ip = 127.0.0.1, http.host = 127.0.0.1:8080,
  http.url = http://cell01.us-east-2.prod.arp.kepler-analytics.aws.dev/invocations);
  the external caller's IP/identity is not present.

So neither the identity nor the true client address of the InvokeAgentRuntime
caller is propagated onto the spans.

Relevant OpenTelemetry semantic conventions (the schema gap)

These spans should be able to carry the caller identity and client address using
existing OTel conventions:

End user identity — https://opentelemetry.io/docs/specs/semconv/registry/attributes/enduser/

Attribute	Type	Description
enduser.id	string	Unique identifier of an end user (username/email/…) — PII
enduser.pseudo.id	string	Pseudonymous (non-PII) end-user identifier
user.roles	string[]	(replaces deprecated enduser.role)

Client / network — https://opentelemetry.io/docs/specs/semconv/registry/attributes/client/

Attribute	Stability	Description
client.address	Stable	Client address behind any intermediaries (the real caller IP)
client.port	Stable	Client port
user_agent.original	Stable	Client user agent

None of these are populated on the AgentCore/Strands GenAI spans.

Request

Stamp the invoking caller's identity and client address onto the agent spans
(at minimum the root invoke_agent / POST /invocations span, ideally inherited by
child execute_tool spans), using the OTel conventions above:

• enduser.id (or enduser.pseudo.id for privacy) — the invoking principal / end
  user (e.g. from the InvokeAgentRuntime caller identity, or from the
  X-Amzn-Bedrock-AgentCore-Runtime-User-Id header when
  InvokeAgentRuntimeForUser is used).
• client.address (+ client.port, user_agent.original) — the real client that
  called InvokeAgentRuntime, not the internal 127.0.0.1 loopback currently on
  POST /invocations.

Gating these behind an opt-in (they are sensitive/PII) is fine.

Why this can't be worked around today

The caller identity and real client IP do exist in the InvokeAgentRuntime
CloudTrail data event (userIdentity, sourceIPAddress, userAgent) — but that
event has requestParameters: null and shares no join key with the spans
(session.id is not in CloudTrail; requestID/eventID are not in any span). A full
field-value intersection of the two returns only the region. So identity cannot be
attached to a span/session from telemetry alone — it must be captured client-side. Put
another way: the data exists in CloudTrail but is unjoinable to the spans, which is
exactly why stamping enduser.id / client.address onto the spans is needed.

Environment

• Amazon Bedrock AgentCore managed harness (Strands agent, amazon.nova-lite-v1:0),
  region us-east-2.
• Spans from strands.telemetry.tracer, telemetry.auto.version = 0.18.0-aws,
  delivered to CloudWatch aws/spans (Transaction Search enabled).
• Verified: no enduser.*, client.address, or external sourceIPAddress on any
  span; POST /invocations shows only internal loopback transport values.

Steps to reproduce the behavior

Prereqs: an AgentCore-hosted agent (e.g. managed harness running Strands) with at
least one tool, and CloudWatch Transaction Search enabled (spans in aws/spans).

1. Invoke the agent with a fresh session id, from a client with an identifiable AWS
   principal (assumed-role/SSO) and a routable public IP:

    harnessArn="<HARNESS_ARN>",
    runtimeSessionId=sid,
    messages=[{"role": "user", "content": [{"text":
    "Use the <tool> tool to get info, then summarize it in one sentence."}]}], ) for _ in resp.get("stream", []):
    pass print("session:", sid)

2. Pull the spans for that session from CloudWatch Logs group aws/spans and dump
   every span's attributes:
   limit=5000, filterPattern=f'"{sid}"') for e in r["events"]:
   m = json.loads(e["message"])
   if m.get("name"):
   print(m["name"], "→", sorted(m.get("attributes", {})))

3. Inspect the root span POST /invocations and the invoke_agent / execute_tool 
   spans for any caller-identity or client-address attribute.

Result: no span contains enduser.id , enduser.pseudo.id , client.address ,
 client.port , or user_agent.original . The only network values are internal
( POST /invocations : net.peer.ip = 127.0.0.1 , http.host = 127.0.0.1:8080 ); the
external caller's IP and identity never appear. (The real caller IP + IAM principal are
present only in the InvokeAgentRuntime CloudTrail data event, which shares no join
key with these spans.)

Expected behavior

The invoking caller's identity and real client address should be recorded on the agent
spans using the standard OpenTelemetry conventions, so a session/trace can be attributed
to who triggered it from CloudWatch alone:

• enduser.id (or enduser.pseudo.id for a non-PII identifier) = the invoking
principal / end user (from the InvokeAgentRuntime caller identity, or from
 X-Amzn-Bedrock-AgentCore-Runtime-User-Id when InvokeAgentRuntimeForUser is used).
• client.address (+ client.port , user_agent.original ) = the real client that
called InvokeAgentRuntime — not the internal 127.0.0.1 loopback.

At minimum these should appear on the root invoke_agent / POST /invocations span and
be inheritable by child execute_tool spans. Gating them behind an opt-in (PII) is fine.

Additional context

• This is a service-level gap: the agent runs behind the AgentCore runtime's internal
proxy, so the agent/tracer only sees 127.0.0.1 . Only the managed runtime/service can
inject the true external caller identity/IP — hence filed here rather than against the
Strands SDK.
• Cannot be worked around from telemetry: the caller identity + real IP exist in the
 InvokeAgentRuntime CloudTrail data event ( userIdentity , sourceIPAddress ,
 userAgent ), but that event has requestParameters: null and shares no per-invocation
key with the spans — a full field-value intersection of the two returns only the region.
(Companion request: "Add runtimeSessionId to the InvokeAgentRuntime CloudTrail data
event" — the two together close the attribution gap from either side.)
• OTel references: end-user attrs
https://opentelemetry.io/docs/specs/semconv/registry/attributes/enduser/ ·
client attrs https://opentelemetry.io/docs/specs/semconv/registry/attributes/client/
• Env: managed harness, Strands agent ( amazon.nova-lite-v1:0 ), region us-east-2;
spans from strands.telemetry.tracer , telemetry.auto.version = 0.18.0-aws .