feat(tel): implement otel sink

[Hweinstock]

Problem

The AgentCore CLI is not currently publishing telemetry to our collector.

Solution

• otel sdk requires the resource attributes separated from the metric attributes (even though they are merged on the request) so we explicitly inject those into each sink.
• implement an otel histogram metric sink. We use histogram because our values represent duration, so histogram gives us count, and distributions over the values in a single instrument.

Testing

• run our collector locally, and override local endpoint to point to it. Then verified metrics hit cloudwatch in dev account. (Requires one small backend change that is already in the pipeline).
• added a unit test that spins up a local server, and verifies it only receives request when telemetry is enabled, and requests match otel format.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.48485% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.70%. Comparing base (7e29eda) to head (6806436).
⚠️ Report is 12 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/telemetry/otelSink.tsx	98.07%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1888 +/- ##
=========================================
Coverage 96.69% 96.70% =========================================
Files 291 292 +1 Lines 16012 16068 +56 =========================================
+ Hits 15483 15538 +55 - Misses 529 530 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Hweinstock]

^ the line missing coverage is getName() on the collector sink.

[notgitika]

outside this try block, there is meterProvider.shutdown() which also flushes the metric again.

https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown

can we ensure one export somehow?

[Hweinstock]

This method provides a way for provider to do any cleanup required.
Shutdown MUST be called only once for each MeterProvider instance. After the call to Shutdown, subsequent attempts to get a Meter are not allowed. SDKs SHOULD return a valid no-op Meter for these calls, if possible.
Shutdown SHOULD provide a way to let the caller know whether it succeeded, failed or timed out.
Shutdown SHOULD complete or abort within some timeout. Shutdown MAY be implemented as a blocking API or an asynchronous API which notifies the caller via a callback or an event. [OpenTelemetry SDK](https://opentelemetry.io/docs/specs/otel/overview/#sdk) authors MAY decide if they want to make the shutdown timeout configurable.
Shutdown MUST be implemented at least by invoking Shutdown on all registered [MetricReader](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricreader) and [MetricExporter](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricexporter) instances.

from https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown.

I don't see any explicit lines in the protocol linked for shutdown that it also flushes. Based on some testing, I think it does internally, but I think its safer to make that behavior explicit. If we flush twice, its a no-op anyway.

[notgitika]

it seems like a wrong/malformed endpoint makes getMetricSinks() reject, and shutdown() propagates that rejection, erroring out in the CLI command. Is that understanding correct? can we make it best-effort?

[Hweinstock]

I think we (the dev team) should be the only ones modifying the endpoint for testing purposes. In which case, I think the ideal behavior is that we reject early.

If a user decides to go into the global config and add an invalid override, I think rejecting is reasonable.

[notgitika]

LGMT

[AlexanderRichey]

Do we want to use our AgentCoreError.fromError() here instead?

[Hweinstock]

Yeah I think that makes sense to ensure its consistent.

[Hweinstock]

slack failure appears unrelated.

Configure AWS credentials (central secrets reader)
1m 0s
Run aws-actions/configure-aws-credentials@v6
Retry validateCredentials: attempt 1 of 12 failed: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers. Retrying after 23ms.

[aidandaly24]

I may be missing whether inheriting the caller's OTEL configuration is intentional, but OTLPMetricExporter reads OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_METRICS_HEADERS when headers is omitted. I set dummy authorization and x-api-key values and confirmed both were forwarded to the configured collector. A user running the CLI in an environment configured for another OTLP backend could therefore send those credentials to telemetry.agentcore.aws.dev. Would passing headers: {} here make sense?

[Hweinstock]

nice catch, was not aware of these environment variables. Was going to add this as a follow-up, but let me inject the right header now to avoid this behavior.

[aidandaly24]

I think the resource attributes are still ending up in both places. InMemoryMetricEvent.emit() merges resourceAttributes into the sink attributes, then this line records that merged map after the provider already registered the same fields with resourceFromAttributes. In a local OTLP capture, all eight resource keys appeared under both resource.attributes and histogram.dataPoints[].attributes. Since the goal here is to separate resource and metric attributes, would it make sense for the sink contract to receive only metric attributes, with FileSystemSink composing its JSONL entry from its configured resource attributes? The test could also assert that resource keys are absent from the datapoint. This is really a low finding and could always be changed in the future.

[Hweinstock]

good catch, I think removing is the right call to simplify.

[notgitika]

looks good to me!

[Hweinstock]

Problem

The AgentCore CLI is not currently publishing telemetry to our collector.

Solution

• otel sdk requires the resource attributes separated from the metric attributes (even though they are merged on the request) so we explicitly inject those into each sink.
• implement an otel histogram metric sink. We use histogram because our values represent duration, so histogram gives us count, and distributions over the values in a single instrument.

Testing

• run our collector locally, and override local endpoint to point to it. Then verified metrics hit cloudwatch in dev account. (Requires one small backend change that is already in the pipeline).
• added a unit test that spins up a local server, and verifies it only receives request when telemetry is enabled, and requests match otel format.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.48485% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.70%. Comparing base (7e29eda) to head (6806436).
⚠️ Report is 12 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/telemetry/otelSink.tsx	98.07%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1888 +/- ##
=========================================
Coverage 96.69% 96.70% =========================================
Files 291 292 +1 Lines 16012 16068 +56 =========================================
+ Hits 15483 15538 +55 - Misses 529 530 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Hweinstock]

^ the line missing coverage is getName() on the collector sink.

[notgitika]

outside this try block, there is meterProvider.shutdown() which also flushes the metric again.

https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown

can we ensure one export somehow?

[Hweinstock]

This method provides a way for provider to do any cleanup required.
Shutdown MUST be called only once for each MeterProvider instance. After the call to Shutdown, subsequent attempts to get a Meter are not allowed. SDKs SHOULD return a valid no-op Meter for these calls, if possible.
Shutdown SHOULD provide a way to let the caller know whether it succeeded, failed or timed out.
Shutdown SHOULD complete or abort within some timeout. Shutdown MAY be implemented as a blocking API or an asynchronous API which notifies the caller via a callback or an event. [OpenTelemetry SDK](https://opentelemetry.io/docs/specs/otel/overview/#sdk) authors MAY decide if they want to make the shutdown timeout configurable.
Shutdown MUST be implemented at least by invoking Shutdown on all registered [MetricReader](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricreader) and [MetricExporter](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricexporter) instances.

from https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown.

I don't see any explicit lines in the protocol linked for shutdown that it also flushes. Based on some testing, I think it does internally, but I think its safer to make that behavior explicit. If we flush twice, its a no-op anyway.

[notgitika]

it seems like a wrong/malformed endpoint makes getMetricSinks() reject, and shutdown() propagates that rejection, erroring out in the CLI command. Is that understanding correct? can we make it best-effort?

[Hweinstock]

I think we (the dev team) should be the only ones modifying the endpoint for testing purposes. In which case, I think the ideal behavior is that we reject early.

If a user decides to go into the global config and add an invalid override, I think rejecting is reasonable.

[notgitika]

LGMT

[AlexanderRichey]

Do we want to use our AgentCoreError.fromError() here instead?

[Hweinstock]

Yeah I think that makes sense to ensure its consistent.

[Hweinstock]

slack failure appears unrelated.

Configure AWS credentials (central secrets reader)
1m 0s
Run aws-actions/configure-aws-credentials@v6
Retry validateCredentials: attempt 1 of 12 failed: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers. Retrying after 23ms.

[aidandaly24]

I may be missing whether inheriting the caller's OTEL configuration is intentional, but OTLPMetricExporter reads OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_METRICS_HEADERS when headers is omitted. I set dummy authorization and x-api-key values and confirmed both were forwarded to the configured collector. A user running the CLI in an environment configured for another OTLP backend could therefore send those credentials to telemetry.agentcore.aws.dev. Would passing headers: {} here make sense?

[Hweinstock]

nice catch, was not aware of these environment variables. Was going to add this as a follow-up, but let me inject the right header now to avoid this behavior.

[aidandaly24]

I think the resource attributes are still ending up in both places. InMemoryMetricEvent.emit() merges resourceAttributes into the sink attributes, then this line records that merged map after the provider already registered the same fields with resourceFromAttributes. In a local OTLP capture, all eight resource keys appeared under both resource.attributes and histogram.dataPoints[].attributes. Since the goal here is to separate resource and metric attributes, would it make sense for the sink contract to receive only metric attributes, with FileSystemSink composing its JSONL entry from its configured resource attributes? The test could also assert that resource keys are absent from the datapoint. This is really a low finding and could always be changed in the future.

[Hweinstock]

good catch, I think removing is the right call to simplify.

[notgitika]

looks good to me!

[Hweinstock]

Problem

The AgentCore CLI is not currently publishing telemetry to our collector.

Solution

• otel sdk requires the resource attributes separated from the metric attributes (even though they are merged on the request) so we explicitly inject those into each sink.
• implement an otel histogram metric sink. We use histogram because our values represent duration, so histogram gives us count, and distributions over the values in a single instrument.

Testing

• run our collector locally, and override local endpoint to point to it. Then verified metrics hit cloudwatch in dev account. (Requires one small backend change that is already in the pipeline).
• added a unit test that spins up a local server, and verifies it only receives request when telemetry is enabled, and requests match otel format.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.48485% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.70%. Comparing base (7e29eda) to head (6806436).
⚠️ Report is 12 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/telemetry/otelSink.tsx	98.07%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1888 +/- ##
=========================================
Coverage 96.69% 96.70% =========================================
Files 291 292 +1 Lines 16012 16068 +56 =========================================
+ Hits 15483 15538 +55 - Misses 529 530 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Hweinstock]

^ the line missing coverage is getName() on the collector sink.

[notgitika]

outside this try block, there is meterProvider.shutdown() which also flushes the metric again.

https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown

can we ensure one export somehow?

[Hweinstock]

This method provides a way for provider to do any cleanup required.
Shutdown MUST be called only once for each MeterProvider instance. After the call to Shutdown, subsequent attempts to get a Meter are not allowed. SDKs SHOULD return a valid no-op Meter for these calls, if possible.
Shutdown SHOULD provide a way to let the caller know whether it succeeded, failed or timed out.
Shutdown SHOULD complete or abort within some timeout. Shutdown MAY be implemented as a blocking API or an asynchronous API which notifies the caller via a callback or an event. [OpenTelemetry SDK](https://opentelemetry.io/docs/specs/otel/overview/#sdk) authors MAY decide if they want to make the shutdown timeout configurable.
Shutdown MUST be implemented at least by invoking Shutdown on all registered [MetricReader](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricreader) and [MetricExporter](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricexporter) instances.

from https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown.

I don't see any explicit lines in the protocol linked for shutdown that it also flushes. Based on some testing, I think it does internally, but I think its safer to make that behavior explicit. If we flush twice, its a no-op anyway.

[notgitika]

it seems like a wrong/malformed endpoint makes getMetricSinks() reject, and shutdown() propagates that rejection, erroring out in the CLI command. Is that understanding correct? can we make it best-effort?

[Hweinstock]

I think we (the dev team) should be the only ones modifying the endpoint for testing purposes. In which case, I think the ideal behavior is that we reject early.

If a user decides to go into the global config and add an invalid override, I think rejecting is reasonable.

[notgitika]

LGMT

[AlexanderRichey]

Do we want to use our AgentCoreError.fromError() here instead?

[Hweinstock]

Yeah I think that makes sense to ensure its consistent.

[Hweinstock]

slack failure appears unrelated.

Configure AWS credentials (central secrets reader)
1m 0s
Run aws-actions/configure-aws-credentials@v6
Retry validateCredentials: attempt 1 of 12 failed: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers. Retrying after 23ms.

[aidandaly24]

I may be missing whether inheriting the caller's OTEL configuration is intentional, but OTLPMetricExporter reads OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_METRICS_HEADERS when headers is omitted. I set dummy authorization and x-api-key values and confirmed both were forwarded to the configured collector. A user running the CLI in an environment configured for another OTLP backend could therefore send those credentials to telemetry.agentcore.aws.dev. Would passing headers: {} here make sense?

[Hweinstock]

nice catch, was not aware of these environment variables. Was going to add this as a follow-up, but let me inject the right header now to avoid this behavior.

[aidandaly24]

I think the resource attributes are still ending up in both places. InMemoryMetricEvent.emit() merges resourceAttributes into the sink attributes, then this line records that merged map after the provider already registered the same fields with resourceFromAttributes. In a local OTLP capture, all eight resource keys appeared under both resource.attributes and histogram.dataPoints[].attributes. Since the goal here is to separate resource and metric attributes, would it make sense for the sink contract to receive only metric attributes, with FileSystemSink composing its JSONL entry from its configured resource attributes? The test could also assert that resource keys are absent from the datapoint. This is really a low finding and could always be changed in the future.

[Hweinstock]

good catch, I think removing is the right call to simplify.

[notgitika]

looks good to me!

[Hweinstock]

Problem

The AgentCore CLI is not currently publishing telemetry to our collector.

Solution

• otel sdk requires the resource attributes separated from the metric attributes (even though they are merged on the request) so we explicitly inject those into each sink.
• implement an otel histogram metric sink. We use histogram because our values represent duration, so histogram gives us count, and distributions over the values in a single instrument.

Testing

• run our collector locally, and override local endpoint to point to it. Then verified metrics hit cloudwatch in dev account. (Requires one small backend change that is already in the pipeline).
• added a unit test that spins up a local server, and verifies it only receives request when telemetry is enabled, and requests match otel format.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.48485% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.70%. Comparing base (7e29eda) to head (6806436).
⚠️ Report is 12 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/telemetry/otelSink.tsx	98.07%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1888 +/- ##
=========================================
Coverage 96.69% 96.70% =========================================
Files 291 292 +1 Lines 16012 16068 +56 =========================================
+ Hits 15483 15538 +55 - Misses 529 530 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Hweinstock]

^ the line missing coverage is getName() on the collector sink.

[notgitika]

outside this try block, there is meterProvider.shutdown() which also flushes the metric again.

https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown

can we ensure one export somehow?

[Hweinstock]

This method provides a way for provider to do any cleanup required.
Shutdown MUST be called only once for each MeterProvider instance. After the call to Shutdown, subsequent attempts to get a Meter are not allowed. SDKs SHOULD return a valid no-op Meter for these calls, if possible.
Shutdown SHOULD provide a way to let the caller know whether it succeeded, failed or timed out.
Shutdown SHOULD complete or abort within some timeout. Shutdown MAY be implemented as a blocking API or an asynchronous API which notifies the caller via a callback or an event. [OpenTelemetry SDK](https://opentelemetry.io/docs/specs/otel/overview/#sdk) authors MAY decide if they want to make the shutdown timeout configurable.
Shutdown MUST be implemented at least by invoking Shutdown on all registered [MetricReader](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricreader) and [MetricExporter](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricexporter) instances.

from https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown.

I don't see any explicit lines in the protocol linked for shutdown that it also flushes. Based on some testing, I think it does internally, but I think its safer to make that behavior explicit. If we flush twice, its a no-op anyway.

[notgitika]

it seems like a wrong/malformed endpoint makes getMetricSinks() reject, and shutdown() propagates that rejection, erroring out in the CLI command. Is that understanding correct? can we make it best-effort?

[Hweinstock]

I think we (the dev team) should be the only ones modifying the endpoint for testing purposes. In which case, I think the ideal behavior is that we reject early.

If a user decides to go into the global config and add an invalid override, I think rejecting is reasonable.

[notgitika]

LGMT

[AlexanderRichey]

Do we want to use our AgentCoreError.fromError() here instead?

[Hweinstock]

Yeah I think that makes sense to ensure its consistent.

[Hweinstock]

slack failure appears unrelated.

Configure AWS credentials (central secrets reader)
1m 0s
Run aws-actions/configure-aws-credentials@v6
Retry validateCredentials: attempt 1 of 12 failed: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers. Retrying after 23ms.

[aidandaly24]

I may be missing whether inheriting the caller's OTEL configuration is intentional, but OTLPMetricExporter reads OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_METRICS_HEADERS when headers is omitted. I set dummy authorization and x-api-key values and confirmed both were forwarded to the configured collector. A user running the CLI in an environment configured for another OTLP backend could therefore send those credentials to telemetry.agentcore.aws.dev. Would passing headers: {} here make sense?

[Hweinstock]

nice catch, was not aware of these environment variables. Was going to add this as a follow-up, but let me inject the right header now to avoid this behavior.

[aidandaly24]

I think the resource attributes are still ending up in both places. InMemoryMetricEvent.emit() merges resourceAttributes into the sink attributes, then this line records that merged map after the provider already registered the same fields with resourceFromAttributes. In a local OTLP capture, all eight resource keys appeared under both resource.attributes and histogram.dataPoints[].attributes. Since the goal here is to separate resource and metric attributes, would it make sense for the sink contract to receive only metric attributes, with FileSystemSink composing its JSONL entry from its configured resource attributes? The test could also assert that resource keys are absent from the datapoint. This is really a low finding and could always be changed in the future.

[Hweinstock]

good catch, I think removing is the right call to simplify.

[notgitika]

looks good to me!

[Hweinstock]

Problem

The AgentCore CLI is not currently publishing telemetry to our collector.

Solution

• otel sdk requires the resource attributes separated from the metric attributes (even though they are merged on the request) so we explicitly inject those into each sink.
• implement an otel histogram metric sink. We use histogram because our values represent duration, so histogram gives us count, and distributions over the values in a single instrument.

Testing

• run our collector locally, and override local endpoint to point to it. Then verified metrics hit cloudwatch in dev account. (Requires one small backend change that is already in the pipeline).
• added a unit test that spins up a local server, and verifies it only receives request when telemetry is enabled, and requests match otel format.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.48485% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.70%. Comparing base (7e29eda) to head (6806436).
⚠️ Report is 12 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/telemetry/otelSink.tsx	98.07%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1888 +/- ##
=========================================
Coverage 96.69% 96.70% =========================================
Files 291 292 +1 Lines 16012 16068 +56 =========================================
+ Hits 15483 15538 +55 - Misses 529 530 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Hweinstock]

^ the line missing coverage is getName() on the collector sink.

[notgitika]

outside this try block, there is meterProvider.shutdown() which also flushes the metric again.

https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown

can we ensure one export somehow?

[Hweinstock]

This method provides a way for provider to do any cleanup required.
Shutdown MUST be called only once for each MeterProvider instance. After the call to Shutdown, subsequent attempts to get a Meter are not allowed. SDKs SHOULD return a valid no-op Meter for these calls, if possible.
Shutdown SHOULD provide a way to let the caller know whether it succeeded, failed or timed out.
Shutdown SHOULD complete or abort within some timeout. Shutdown MAY be implemented as a blocking API or an asynchronous API which notifies the caller via a callback or an event. [OpenTelemetry SDK](https://opentelemetry.io/docs/specs/otel/overview/#sdk) authors MAY decide if they want to make the shutdown timeout configurable.
Shutdown MUST be implemented at least by invoking Shutdown on all registered [MetricReader](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricreader) and [MetricExporter](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricexporter) instances.

from https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown.

I don't see any explicit lines in the protocol linked for shutdown that it also flushes. Based on some testing, I think it does internally, but I think its safer to make that behavior explicit. If we flush twice, its a no-op anyway.

[notgitika]

it seems like a wrong/malformed endpoint makes getMetricSinks() reject, and shutdown() propagates that rejection, erroring out in the CLI command. Is that understanding correct? can we make it best-effort?

[Hweinstock]

I think we (the dev team) should be the only ones modifying the endpoint for testing purposes. In which case, I think the ideal behavior is that we reject early.

If a user decides to go into the global config and add an invalid override, I think rejecting is reasonable.

[notgitika]

LGMT

[AlexanderRichey]

Do we want to use our AgentCoreError.fromError() here instead?

[Hweinstock]

Yeah I think that makes sense to ensure its consistent.

[Hweinstock]

slack failure appears unrelated.

Configure AWS credentials (central secrets reader)
1m 0s
Run aws-actions/configure-aws-credentials@v6
Retry validateCredentials: attempt 1 of 12 failed: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers. Retrying after 23ms.

[aidandaly24]

I may be missing whether inheriting the caller's OTEL configuration is intentional, but OTLPMetricExporter reads OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_METRICS_HEADERS when headers is omitted. I set dummy authorization and x-api-key values and confirmed both were forwarded to the configured collector. A user running the CLI in an environment configured for another OTLP backend could therefore send those credentials to telemetry.agentcore.aws.dev. Would passing headers: {} here make sense?

[Hweinstock]

nice catch, was not aware of these environment variables. Was going to add this as a follow-up, but let me inject the right header now to avoid this behavior.

[aidandaly24]

I think the resource attributes are still ending up in both places. InMemoryMetricEvent.emit() merges resourceAttributes into the sink attributes, then this line records that merged map after the provider already registered the same fields with resourceFromAttributes. In a local OTLP capture, all eight resource keys appeared under both resource.attributes and histogram.dataPoints[].attributes. Since the goal here is to separate resource and metric attributes, would it make sense for the sink contract to receive only metric attributes, with FileSystemSink composing its JSONL entry from its configured resource attributes? The test could also assert that resource keys are absent from the datapoint. This is really a low finding and could always be changed in the future.

[Hweinstock]

good catch, I think removing is the right call to simplify.

[notgitika]

looks good to me!

[Hweinstock]

Problem

The AgentCore CLI is not currently publishing telemetry to our collector.

Solution

• otel sdk requires the resource attributes separated from the metric attributes (even though they are merged on the request) so we explicitly inject those into each sink.
• implement an otel histogram metric sink. We use histogram because our values represent duration, so histogram gives us count, and distributions over the values in a single instrument.

Testing

• run our collector locally, and override local endpoint to point to it. Then verified metrics hit cloudwatch in dev account. (Requires one small backend change that is already in the pipeline).
• added a unit test that spins up a local server, and verifies it only receives request when telemetry is enabled, and requests match otel format.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.48485% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.70%. Comparing base (7e29eda) to head (6806436).
⚠️ Report is 12 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/telemetry/otelSink.tsx	98.07%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1888 +/- ##
=========================================
Coverage 96.69% 96.70% =========================================
Files 291 292 +1 Lines 16012 16068 +56 =========================================
+ Hits 15483 15538 +55 - Misses 529 530 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Hweinstock]

^ the line missing coverage is getName() on the collector sink.

[notgitika]

outside this try block, there is meterProvider.shutdown() which also flushes the metric again.

https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown

can we ensure one export somehow?

[Hweinstock]

This method provides a way for provider to do any cleanup required.
Shutdown MUST be called only once for each MeterProvider instance. After the call to Shutdown, subsequent attempts to get a Meter are not allowed. SDKs SHOULD return a valid no-op Meter for these calls, if possible.
Shutdown SHOULD provide a way to let the caller know whether it succeeded, failed or timed out.
Shutdown SHOULD complete or abort within some timeout. Shutdown MAY be implemented as a blocking API or an asynchronous API which notifies the caller via a callback or an event. [OpenTelemetry SDK](https://opentelemetry.io/docs/specs/otel/overview/#sdk) authors MAY decide if they want to make the shutdown timeout configurable.
Shutdown MUST be implemented at least by invoking Shutdown on all registered [MetricReader](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricreader) and [MetricExporter](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricexporter) instances.

from https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown.

I don't see any explicit lines in the protocol linked for shutdown that it also flushes. Based on some testing, I think it does internally, but I think its safer to make that behavior explicit. If we flush twice, its a no-op anyway.

[notgitika]

it seems like a wrong/malformed endpoint makes getMetricSinks() reject, and shutdown() propagates that rejection, erroring out in the CLI command. Is that understanding correct? can we make it best-effort?

[Hweinstock]

I think we (the dev team) should be the only ones modifying the endpoint for testing purposes. In which case, I think the ideal behavior is that we reject early.

If a user decides to go into the global config and add an invalid override, I think rejecting is reasonable.

[notgitika]

LGMT

[AlexanderRichey]

Do we want to use our AgentCoreError.fromError() here instead?

[Hweinstock]

Yeah I think that makes sense to ensure its consistent.

[Hweinstock]

slack failure appears unrelated.

Configure AWS credentials (central secrets reader)
1m 0s
Run aws-actions/configure-aws-credentials@v6
Retry validateCredentials: attempt 1 of 12 failed: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers. Retrying after 23ms.

[aidandaly24]

I may be missing whether inheriting the caller's OTEL configuration is intentional, but OTLPMetricExporter reads OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_METRICS_HEADERS when headers is omitted. I set dummy authorization and x-api-key values and confirmed both were forwarded to the configured collector. A user running the CLI in an environment configured for another OTLP backend could therefore send those credentials to telemetry.agentcore.aws.dev. Would passing headers: {} here make sense?

[Hweinstock]

nice catch, was not aware of these environment variables. Was going to add this as a follow-up, but let me inject the right header now to avoid this behavior.

[aidandaly24]

I think the resource attributes are still ending up in both places. InMemoryMetricEvent.emit() merges resourceAttributes into the sink attributes, then this line records that merged map after the provider already registered the same fields with resourceFromAttributes. In a local OTLP capture, all eight resource keys appeared under both resource.attributes and histogram.dataPoints[].attributes. Since the goal here is to separate resource and metric attributes, would it make sense for the sink contract to receive only metric attributes, with FileSystemSink composing its JSONL entry from its configured resource attributes? The test could also assert that resource keys are absent from the datapoint. This is really a low finding and could always be changed in the future.

[Hweinstock]

good catch, I think removing is the right call to simplify.

[notgitika]

looks good to me!

[Hweinstock]

Problem

The AgentCore CLI is not currently publishing telemetry to our collector.

Solution

• otel sdk requires the resource attributes separated from the metric attributes (even though they are merged on the request) so we explicitly inject those into each sink.
• implement an otel histogram metric sink. We use histogram because our values represent duration, so histogram gives us count, and distributions over the values in a single instrument.

Testing

• run our collector locally, and override local endpoint to point to it. Then verified metrics hit cloudwatch in dev account. (Requires one small backend change that is already in the pipeline).
• added a unit test that spins up a local server, and verifies it only receives request when telemetry is enabled, and requests match otel format.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.48485% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.70%. Comparing base (7e29eda) to head (6806436).
⚠️ Report is 12 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/telemetry/otelSink.tsx	98.07%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1888 +/- ##
=========================================
Coverage 96.69% 96.70% =========================================
Files 291 292 +1 Lines 16012 16068 +56 =========================================
+ Hits 15483 15538 +55 - Misses 529 530 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Hweinstock]

^ the line missing coverage is getName() on the collector sink.

[notgitika]

outside this try block, there is meterProvider.shutdown() which also flushes the metric again.

https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown

can we ensure one export somehow?

[Hweinstock]

This method provides a way for provider to do any cleanup required.
Shutdown MUST be called only once for each MeterProvider instance. After the call to Shutdown, subsequent attempts to get a Meter are not allowed. SDKs SHOULD return a valid no-op Meter for these calls, if possible.
Shutdown SHOULD provide a way to let the caller know whether it succeeded, failed or timed out.
Shutdown SHOULD complete or abort within some timeout. Shutdown MAY be implemented as a blocking API or an asynchronous API which notifies the caller via a callback or an event. [OpenTelemetry SDK](https://opentelemetry.io/docs/specs/otel/overview/#sdk) authors MAY decide if they want to make the shutdown timeout configurable.
Shutdown MUST be implemented at least by invoking Shutdown on all registered [MetricReader](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricreader) and [MetricExporter](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricexporter) instances.

from https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown.

I don't see any explicit lines in the protocol linked for shutdown that it also flushes. Based on some testing, I think it does internally, but I think its safer to make that behavior explicit. If we flush twice, its a no-op anyway.

[notgitika]

it seems like a wrong/malformed endpoint makes getMetricSinks() reject, and shutdown() propagates that rejection, erroring out in the CLI command. Is that understanding correct? can we make it best-effort?

[Hweinstock]

I think we (the dev team) should be the only ones modifying the endpoint for testing purposes. In which case, I think the ideal behavior is that we reject early.

If a user decides to go into the global config and add an invalid override, I think rejecting is reasonable.

[notgitika]

LGMT

[AlexanderRichey]

Do we want to use our AgentCoreError.fromError() here instead?

[Hweinstock]

Yeah I think that makes sense to ensure its consistent.

[Hweinstock]

slack failure appears unrelated.

Configure AWS credentials (central secrets reader)
1m 0s
Run aws-actions/configure-aws-credentials@v6
Retry validateCredentials: attempt 1 of 12 failed: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers. Retrying after 23ms.

[aidandaly24]

I may be missing whether inheriting the caller's OTEL configuration is intentional, but OTLPMetricExporter reads OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_METRICS_HEADERS when headers is omitted. I set dummy authorization and x-api-key values and confirmed both were forwarded to the configured collector. A user running the CLI in an environment configured for another OTLP backend could therefore send those credentials to telemetry.agentcore.aws.dev. Would passing headers: {} here make sense?

[Hweinstock]

nice catch, was not aware of these environment variables. Was going to add this as a follow-up, but let me inject the right header now to avoid this behavior.

[aidandaly24]

I think the resource attributes are still ending up in both places. InMemoryMetricEvent.emit() merges resourceAttributes into the sink attributes, then this line records that merged map after the provider already registered the same fields with resourceFromAttributes. In a local OTLP capture, all eight resource keys appeared under both resource.attributes and histogram.dataPoints[].attributes. Since the goal here is to separate resource and metric attributes, would it make sense for the sink contract to receive only metric attributes, with FileSystemSink composing its JSONL entry from its configured resource attributes? The test could also assert that resource keys are absent from the datapoint. This is really a low finding and could always be changed in the future.

[Hweinstock]

good catch, I think removing is the right call to simplify.

[notgitika]

looks good to me!

[Hweinstock]

Problem

The AgentCore CLI is not currently publishing telemetry to our collector.

Solution

• otel sdk requires the resource attributes separated from the metric attributes (even though they are merged on the request) so we explicitly inject those into each sink.
• implement an otel histogram metric sink. We use histogram because our values represent duration, so histogram gives us count, and distributions over the values in a single instrument.

Testing

• run our collector locally, and override local endpoint to point to it. Then verified metrics hit cloudwatch in dev account. (Requires one small backend change that is already in the pipeline).
• added a unit test that spins up a local server, and verifies it only receives request when telemetry is enabled, and requests match otel format.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.48485% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.70%. Comparing base (7e29eda) to head (6806436).
⚠️ Report is 12 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/telemetry/otelSink.tsx	98.07%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## refactor #1888 +/- ##
=========================================
Coverage 96.69% 96.70% =========================================
Files 291 292 +1 Lines 16012 16068 +56 =========================================
+ Hits 15483 15538 +55 - Misses 529 530 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Hweinstock]

^ the line missing coverage is getName() on the collector sink.

[notgitika]

outside this try block, there is meterProvider.shutdown() which also flushes the metric again.

https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown

can we ensure one export somehow?

[Hweinstock]

This method provides a way for provider to do any cleanup required.
Shutdown MUST be called only once for each MeterProvider instance. After the call to Shutdown, subsequent attempts to get a Meter are not allowed. SDKs SHOULD return a valid no-op Meter for these calls, if possible.
Shutdown SHOULD provide a way to let the caller know whether it succeeded, failed or timed out.
Shutdown SHOULD complete or abort within some timeout. Shutdown MAY be implemented as a blocking API or an asynchronous API which notifies the caller via a callback or an event. [OpenTelemetry SDK](https://opentelemetry.io/docs/specs/otel/overview/#sdk) authors MAY decide if they want to make the shutdown timeout configurable.
Shutdown MUST be implemented at least by invoking Shutdown on all registered [MetricReader](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricreader) and [MetricExporter](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricexporter) instances.

from https://opentelemetry.io/docs/specs/otel/metrics/sdk/#shutdown.

I don't see any explicit lines in the protocol linked for shutdown that it also flushes. Based on some testing, I think it does internally, but I think its safer to make that behavior explicit. If we flush twice, its a no-op anyway.

[notgitika]

it seems like a wrong/malformed endpoint makes getMetricSinks() reject, and shutdown() propagates that rejection, erroring out in the CLI command. Is that understanding correct? can we make it best-effort?

[Hweinstock]

I think we (the dev team) should be the only ones modifying the endpoint for testing purposes. In which case, I think the ideal behavior is that we reject early.

If a user decides to go into the global config and add an invalid override, I think rejecting is reasonable.

[notgitika]

LGMT

[AlexanderRichey]

Do we want to use our AgentCoreError.fromError() here instead?

[Hweinstock]

Yeah I think that makes sense to ensure its consistent.

[Hweinstock]

slack failure appears unrelated.

Configure AWS credentials (central secrets reader)
1m 0s
Run aws-actions/configure-aws-credentials@v6
Retry validateCredentials: attempt 1 of 12 failed: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers. Retrying after 23ms.

[aidandaly24]

I may be missing whether inheriting the caller's OTEL configuration is intentional, but OTLPMetricExporter reads OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_METRICS_HEADERS when headers is omitted. I set dummy authorization and x-api-key values and confirmed both were forwarded to the configured collector. A user running the CLI in an environment configured for another OTLP backend could therefore send those credentials to telemetry.agentcore.aws.dev. Would passing headers: {} here make sense?

[Hweinstock]

nice catch, was not aware of these environment variables. Was going to add this as a follow-up, but let me inject the right header now to avoid this behavior.

[aidandaly24]

I think the resource attributes are still ending up in both places. InMemoryMetricEvent.emit() merges resourceAttributes into the sink attributes, then this line records that merged map after the provider already registered the same fields with resourceFromAttributes. In a local OTLP capture, all eight resource keys appeared under both resource.attributes and histogram.dataPoints[].attributes. Since the goal here is to separate resource and metric attributes, would it make sense for the sink contract to receive only metric attributes, with FileSystemSink composing its JSONL entry from its configured resource attributes? The test could also assert that resource keys are absent from the datapoint. This is really a low finding and could always be changed in the future.

[Hweinstock]

good catch, I think removing is the right call to simplify.

[notgitika]

looks good to me!