Skip to content

feat(dev): collect local OTEL traces in project dev - #1980

Merged
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector
Aug 21, 2026
Merged

feat(dev): collect local OTEL traces in project dev#1980
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector

Conversation

@tejaskash

@tejaskashtejaskash commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Review after #2043

Until #2043 merges, the diff shows both PRs' commits (GitHub's stack tooling wouldn't retarget the base); the work to review here is the single collector commit on top. It collapses to just that once #2043 merges.

What this does

agentcore project dev gives a local agent tracing with no extra config: an in-process OTLP/HTTP receiver starts on a loopback port, the agent's OpenTelemetry SDK is pointed at it via env vars, and traces persist through #2043's TraceStore to agentcore/.cli/traces/. Nothing leaves the machine; --no-traces or instrumentation.enableOtel: false opts out.

  • io/httpServer.ts — loopback-only node:http primitive (body caps, abort-signal close).
  • otel/collector.tsPOST /v1/traces + /v1/logs, protobuf or JSON. Pinned @opentelemetry/otlp-transformer@0.213.0, the last version shipping request decoders — do not bump.
  • Handler wiring — collector lifecycle on the command's abort signal; signal-specific OTEL env vars so stray shell values can't redirect traces; host.docker.internal rewrite for containers; sitecustomize-on-PYTHONPATH so uvicorn --reload workers stay instrumented (a wrapper would only instrument the reloader parent).

Verification

Per-layer tests (receiver over real ephemeral-port HTTP with protobuf + JSON fixtures; handler with fake runners/collector) plus end to end: built bundle → real project createproject dev → real Bedrock invocation → traces on disk. Protobuf decode verified under bun, the Node bundle, and a compiled binary.

@github-actionsgithub-actionsBot added the size/xl PR size: XL label Aug 12, 2026
@github-actionsgithub-actionsBot added agentcore-harness-reviewing AgentCore Harness review in progress size/xl PR size: XL and removed size/xl PR size: XL labels Aug 12, 2026
@github-actionsgithub-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 12, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from b4d0349 to e2e696cCompareAugust 14, 2026 20:38
@github-actionsgithub-actionsBot added size/xl PR size: XL and removed size/xl PR size: XL labels Aug 14, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 56c20f7 to a59ebf1CompareAugust 17, 2026 17:30
@codecov-commenter

codecov-commenter commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.28315% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.17%. Comparing base (a41b4de) to head (10be0a6).

Files with missing linesPatch %Lines
src/io/httpServer.ts97.05%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## refactor #1980 +/- ##
============================================
+ Coverage 97.15% 97.17% +0.02% 
============================================
Files 387 389 +2 Lines 23124 23386 +262 ============================================
+ Hits 22465 22726 +261 - Misses 659 660 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from cad8772 to 7e315b9CompareAugust 17, 2026 17:53
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from be5c3d5 to 4b80241CompareAugust 21, 2026 15:44
Base automatically changed from feat/dev-otel-storage to refactorAugust 21, 2026 17:51
tejaskash added a commit that referenced this pull request Aug 21, 2026
* feat(dev): OTLP trace storage for local dev
Pure OTLP wire handling (per-trace batch partitioning, id normalization,
frontend shaping) and append-only per-trace JSONL storage. A batch routinely
carries spans from several traces, so persistence partitions by trace id —
writing whole batches under the first id corrupts trace identity. Consumed by
the OTLP collector in #1980, which stacks on this.
* fix(dev): match trace filters against every participating service
A distributed trace spans several local agents whose exports append to the
same trace file; filtering by any participant must find it, not only the
first service seen.
* fix(dev): address review on OTLP trace storage
- narrow OtlpAttributes to the key/value wire form (the flat-record variant
had no producer); drop the dead passthrough branches
- flatten array attributes through extractAnyValue so ints stay numeric and
nested kvlists survive (was stringifying and dropping them)
- count rendered (post-filter) spans for the list summary instead of raw
records, so the count matches the waterfall the inspector shows
- surface non-ENOENT fs errors from reads instead of masking them as empty
- add newest-N limit to list() for the inspector's per-invocation poll
* fix(dev): flatten kvlist-valued OTLP attributes instead of dropping them
flattenAttributes hand-rolled a value branch per AnyValue kind and had no
kvlistValue case, so an attribute whose value is a kvlist (or anything the
chain didn't enumerate) silently vanished. Route every attribute value
through extractAnyValue, which already unwraps all variants including
kvlist — smaller and complete.
Addresses Gitika's review on transforms.ts (reuse extractAnyValue; kvlist
must not disappear).
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 4b80241 to 733c274CompareAugust 21, 2026 17:51
@tejaskash

Copy link
Copy Markdown
ContributorAuthor

#2039 was superseded by #2043, which just merged — this is unblocked and rebased on it.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from fce12fd to 27fbae3CompareAugust 21, 2026 17:59
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/io/httpServer.ts
Comment threadsrc/router/flags.tsx Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/codezip.ts
Comment threadsrc/core/dev/codezip.ts Outdated
tejaskash added a commit that referenced this pull request Aug 21, 2026
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
Comment threadsrc/core/dev/container.ts
Comment threadsrc/io/httpServer.ts
Comment threadsrc/core/dev/container.ts
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/handlers/project/dev/index.ts Outdated
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
An in-process OTLP/HTTP receiver (protobuf via the pinned otlp-transformer
decoders, or JSON) persists agent traces through the storage layer. project
dev starts it unless --no-traces or the runtime disables instrumentation,
points every spawned agent at it (signal-specific OTEL env), rewrites the
endpoint to host.docker.internal for containers — with an explicit
host-gateway mapping so Linux Docker Engine resolves it — and keeps uvicorn
--reload workers instrumented via sitecustomize on PYTHONPATH. Oversized
collector requests get a 413 before the connection closes so exporters do
not retry them as transient failures.
Rebuilt from explicit paths: the previous tree-snapshot commit accidentally
reverted unrelated merged work (cdk target guard, config-bundle TUI, error
classification).
A batch that can't be persisted (disk full, permissions) was being
turned into a 500, which the OTEL SDK exporter retries forever while the
user sees nothing. Ack the export (200) so retries stop, and surface the
failure once via an onError sink threaded from the collector to the dev
handler, which owns the IO to warn the user.
Addresses Gitika's review on store.ts:45 (catch in one place; don't let
persistence faults read as a silent, retried loss).
Follow-ups from #2043 review (Gitika, Harrison):
- Add a TraceStore.list spanCount test with transport-noise spans
(1 agent + 4 http-send -> "1"), guarding the post-filter count.
- Rename hexFromB64OrString -> hexFromBase64OrHex; both reviewers misread B64.
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
…cycle
Address reviewer findings on the collector and dev wiring:
- Validate top-level OTLP shape and return 400 instead of mislabeling a bad
payload as a persistence error.
- Guard the shared HTTP server against a client that disconnects mid-response
so it can no longer crash project dev; add an optional bind host.
- Bind the collector to 0.0.0.0 for container runtimes so a container can
reach it over the host bridge on Linux.
- Run the container template under opentelemetry-instrument so it emits traces.
- Keep the collector alive through the child's shutdown grace so final spans
are not lost.
- Force the OTEL settings that would otherwise let shell or .env.local values
disable or break local collection.
- Make the uv sitecustomize discovery abortable and read its path from a
marker rather than the last merged output line.
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from adf371b to 10be0a6CompareAugust 21, 2026 18:55

@aidandaly24aidandaly24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the fixes. This looks good to me!

@tejaskash
tejaskash merged commit f38b18d into refactorAug 21, 2026
11 checks passed
@tejaskash
tejaskash deleted the feat/dev-otel-collector branch August 21, 2026 21:32
tejaskash added a commit that referenced this pull request Aug 21, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
* feat(dev): supervise every runtime in project dev
project dev without --agent now runs all of the project's runtimes at once:
a DevSupervisor owns per-agent lifecycle (sequential port resolution — a
concurrent race would put two agents on one port), merges every runner's
output into one agent-attributed stream ([name] prefixes; an agent field in
NDJSON), and keeps the session alive when one agent crashes. Selecting a
single runtime (--agent, or a one-runtime project) keeps the direct path
where a crash still fails the command.
* fix(dev): bound readiness polling; unchain abort listeners on failed setup
A child that stays alive without ever binding its port previously blocked
every later runtime (starts are sequential) until interrupted; readiness now
gives up after 120s and fails that start. Setup failures before the pump
exists (port resolution, environment) now remove their parent-abort listener
like every other exit path, so Inspector retries of a failing agent cannot
accumulate listeners.
* fix(dev): bind collector to 0.0.0.0 when any runtime is a container
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
* fix(dev): deliver events queued while the supervisor stream drains a batch
events() spliced the queue, yielded each event, then installed its wake
callback and blocked. A push landing during the yields ran while wake was
undefined, so its wake was a no-op, and the newly installed waiter never
noticed the queued event until the next push or shutdown. Re-check the queue
inside the wait so a queued event resolves immediately.
* refactor(dev): route single runtime through the supervisor and address review
Remove the runSingleRuntime special case so every runtime, one or many,
runs through DevSupervisor. Start all agents concurrently and stream their
output live rather than awaiting each readiness first. End the event loop
when no agent is starting or running, and exit non-zero when none started.
Make start() async so an unknown name rejects rather than throwing
synchronously. Time readiness off the agent's last output (idle timeout)
so a slow build that keeps logging is not killed. Move waitForPort into
src/io/port.ts to keep socket IO in the IO layer. Rename the environment
accessor to getDevEnvVarsForRuntime and SupervisedEvent.agent to agentName.
* refactor(dev): always attribute agent output to its runtime
Split renderEvent into renderAgentEvent (agent name required) and
renderStatus for command-level trace-collector lines, so agent output
is always tagged with [name] and the agent-optional branches are gone.
Addresses review feedback on #2041.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xlPR size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@tejaskash@codecov-commenter@Hweinstock@notgitika@aidandaly24
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
feat(dev): collect local OTEL traces in project dev by tejaskash · Pull Request #1980 · aws/agentcore-cli · GitHub
Skip to content

feat(dev): collect local OTEL traces in project dev - #1980

Merged
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector
Aug 21, 2026
Merged

feat(dev): collect local OTEL traces in project dev#1980
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector

Conversation

@tejaskash

@tejaskashtejaskash commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Review after #2043

Until #2043 merges, the diff shows both PRs' commits (GitHub's stack tooling wouldn't retarget the base); the work to review here is the single collector commit on top. It collapses to just that once #2043 merges.

What this does

agentcore project dev gives a local agent tracing with no extra config: an in-process OTLP/HTTP receiver starts on a loopback port, the agent's OpenTelemetry SDK is pointed at it via env vars, and traces persist through #2043's TraceStore to agentcore/.cli/traces/. Nothing leaves the machine; --no-traces or instrumentation.enableOtel: false opts out.

  • io/httpServer.ts — loopback-only node:http primitive (body caps, abort-signal close).
  • otel/collector.tsPOST /v1/traces + /v1/logs, protobuf or JSON. Pinned @opentelemetry/otlp-transformer@0.213.0, the last version shipping request decoders — do not bump.
  • Handler wiring — collector lifecycle on the command's abort signal; signal-specific OTEL env vars so stray shell values can't redirect traces; host.docker.internal rewrite for containers; sitecustomize-on-PYTHONPATH so uvicorn --reload workers stay instrumented (a wrapper would only instrument the reloader parent).

Verification

Per-layer tests (receiver over real ephemeral-port HTTP with protobuf + JSON fixtures; handler with fake runners/collector) plus end to end: built bundle → real project createproject dev → real Bedrock invocation → traces on disk. Protobuf decode verified under bun, the Node bundle, and a compiled binary.

@github-actionsgithub-actionsBot added the size/xl PR size: XL label Aug 12, 2026
@github-actionsgithub-actionsBot added agentcore-harness-reviewing AgentCore Harness review in progress size/xl PR size: XL and removed size/xl PR size: XL labels Aug 12, 2026
@github-actionsgithub-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 12, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from b4d0349 to e2e696cCompareAugust 14, 2026 20:38
@github-actionsgithub-actionsBot added size/xl PR size: XL and removed size/xl PR size: XL labels Aug 14, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 56c20f7 to a59ebf1CompareAugust 17, 2026 17:30
@codecov-commenter

codecov-commenter commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.28315% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.17%. Comparing base (a41b4de) to head (10be0a6).

Files with missing linesPatch %Lines
src/io/httpServer.ts97.05%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## refactor #1980 +/- ##
============================================
+ Coverage 97.15% 97.17% +0.02% 
============================================
Files 387 389 +2 Lines 23124 23386 +262 ============================================
+ Hits 22465 22726 +261 - Misses 659 660 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from cad8772 to 7e315b9CompareAugust 17, 2026 17:53
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from be5c3d5 to 4b80241CompareAugust 21, 2026 15:44
Base automatically changed from feat/dev-otel-storage to refactorAugust 21, 2026 17:51
tejaskash added a commit that referenced this pull request Aug 21, 2026
* feat(dev): OTLP trace storage for local dev
Pure OTLP wire handling (per-trace batch partitioning, id normalization,
frontend shaping) and append-only per-trace JSONL storage. A batch routinely
carries spans from several traces, so persistence partitions by trace id —
writing whole batches under the first id corrupts trace identity. Consumed by
the OTLP collector in #1980, which stacks on this.
* fix(dev): match trace filters against every participating service
A distributed trace spans several local agents whose exports append to the
same trace file; filtering by any participant must find it, not only the
first service seen.
* fix(dev): address review on OTLP trace storage
- narrow OtlpAttributes to the key/value wire form (the flat-record variant
had no producer); drop the dead passthrough branches
- flatten array attributes through extractAnyValue so ints stay numeric and
nested kvlists survive (was stringifying and dropping them)
- count rendered (post-filter) spans for the list summary instead of raw
records, so the count matches the waterfall the inspector shows
- surface non-ENOENT fs errors from reads instead of masking them as empty
- add newest-N limit to list() for the inspector's per-invocation poll
* fix(dev): flatten kvlist-valued OTLP attributes instead of dropping them
flattenAttributes hand-rolled a value branch per AnyValue kind and had no
kvlistValue case, so an attribute whose value is a kvlist (or anything the
chain didn't enumerate) silently vanished. Route every attribute value
through extractAnyValue, which already unwraps all variants including
kvlist — smaller and complete.
Addresses Gitika's review on transforms.ts (reuse extractAnyValue; kvlist
must not disappear).
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 4b80241 to 733c274CompareAugust 21, 2026 17:51
@tejaskash

Copy link
Copy Markdown
ContributorAuthor

#2039 was superseded by #2043, which just merged — this is unblocked and rebased on it.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from fce12fd to 27fbae3CompareAugust 21, 2026 17:59
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/io/httpServer.ts
Comment threadsrc/router/flags.tsx Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/codezip.ts
Comment threadsrc/core/dev/codezip.ts Outdated
tejaskash added a commit that referenced this pull request Aug 21, 2026
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
Comment threadsrc/core/dev/container.ts
Comment threadsrc/io/httpServer.ts
Comment threadsrc/core/dev/container.ts
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/handlers/project/dev/index.ts Outdated
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
An in-process OTLP/HTTP receiver (protobuf via the pinned otlp-transformer
decoders, or JSON) persists agent traces through the storage layer. project
dev starts it unless --no-traces or the runtime disables instrumentation,
points every spawned agent at it (signal-specific OTEL env), rewrites the
endpoint to host.docker.internal for containers — with an explicit
host-gateway mapping so Linux Docker Engine resolves it — and keeps uvicorn
--reload workers instrumented via sitecustomize on PYTHONPATH. Oversized
collector requests get a 413 before the connection closes so exporters do
not retry them as transient failures.
Rebuilt from explicit paths: the previous tree-snapshot commit accidentally
reverted unrelated merged work (cdk target guard, config-bundle TUI, error
classification).
A batch that can't be persisted (disk full, permissions) was being
turned into a 500, which the OTEL SDK exporter retries forever while the
user sees nothing. Ack the export (200) so retries stop, and surface the
failure once via an onError sink threaded from the collector to the dev
handler, which owns the IO to warn the user.
Addresses Gitika's review on store.ts:45 (catch in one place; don't let
persistence faults read as a silent, retried loss).
Follow-ups from #2043 review (Gitika, Harrison):
- Add a TraceStore.list spanCount test with transport-noise spans
(1 agent + 4 http-send -> "1"), guarding the post-filter count.
- Rename hexFromB64OrString -> hexFromBase64OrHex; both reviewers misread B64.
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
…cycle
Address reviewer findings on the collector and dev wiring:
- Validate top-level OTLP shape and return 400 instead of mislabeling a bad
payload as a persistence error.
- Guard the shared HTTP server against a client that disconnects mid-response
so it can no longer crash project dev; add an optional bind host.
- Bind the collector to 0.0.0.0 for container runtimes so a container can
reach it over the host bridge on Linux.
- Run the container template under opentelemetry-instrument so it emits traces.
- Keep the collector alive through the child's shutdown grace so final spans
are not lost.
- Force the OTEL settings that would otherwise let shell or .env.local values
disable or break local collection.
- Make the uv sitecustomize discovery abortable and read its path from a
marker rather than the last merged output line.
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from adf371b to 10be0a6CompareAugust 21, 2026 18:55

@aidandaly24aidandaly24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the fixes. This looks good to me!

@tejaskash
tejaskash merged commit f38b18d into refactorAug 21, 2026
11 checks passed
@tejaskash
tejaskash deleted the feat/dev-otel-collector branch August 21, 2026 21:32
tejaskash added a commit that referenced this pull request Aug 21, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
* feat(dev): supervise every runtime in project dev
project dev without --agent now runs all of the project's runtimes at once:
a DevSupervisor owns per-agent lifecycle (sequential port resolution — a
concurrent race would put two agents on one port), merges every runner's
output into one agent-attributed stream ([name] prefixes; an agent field in
NDJSON), and keeps the session alive when one agent crashes. Selecting a
single runtime (--agent, or a one-runtime project) keeps the direct path
where a crash still fails the command.
* fix(dev): bound readiness polling; unchain abort listeners on failed setup
A child that stays alive without ever binding its port previously blocked
every later runtime (starts are sequential) until interrupted; readiness now
gives up after 120s and fails that start. Setup failures before the pump
exists (port resolution, environment) now remove their parent-abort listener
like every other exit path, so Inspector retries of a failing agent cannot
accumulate listeners.
* fix(dev): bind collector to 0.0.0.0 when any runtime is a container
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
* fix(dev): deliver events queued while the supervisor stream drains a batch
events() spliced the queue, yielded each event, then installed its wake
callback and blocked. A push landing during the yields ran while wake was
undefined, so its wake was a no-op, and the newly installed waiter never
noticed the queued event until the next push or shutdown. Re-check the queue
inside the wait so a queued event resolves immediately.
* refactor(dev): route single runtime through the supervisor and address review
Remove the runSingleRuntime special case so every runtime, one or many,
runs through DevSupervisor. Start all agents concurrently and stream their
output live rather than awaiting each readiness first. End the event loop
when no agent is starting or running, and exit non-zero when none started.
Make start() async so an unknown name rejects rather than throwing
synchronously. Time readiness off the agent's last output (idle timeout)
so a slow build that keeps logging is not killed. Move waitForPort into
src/io/port.ts to keep socket IO in the IO layer. Rename the environment
accessor to getDevEnvVarsForRuntime and SupervisedEvent.agent to agentName.
* refactor(dev): always attribute agent output to its runtime
Split renderEvent into renderAgentEvent (agent name required) and
renderStatus for command-level trace-collector lines, so agent output
is always tagged with [name] and the agent-optional branches are gone.
Addresses review feedback on #2041.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xlPR size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@tejaskash@codecov-commenter@Hweinstock@notgitika@aidandaly24
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' feat(dev): collect local OTEL traces in project dev by tejaskash · Pull Request #1980 · aws/agentcore-cli · GitHub
Skip to content

feat(dev): collect local OTEL traces in project dev - #1980

Merged
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector
Aug 21, 2026
Merged

feat(dev): collect local OTEL traces in project dev#1980
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector

Conversation

@tejaskash

@tejaskashtejaskash commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Review after #2043

Until #2043 merges, the diff shows both PRs' commits (GitHub's stack tooling wouldn't retarget the base); the work to review here is the single collector commit on top. It collapses to just that once #2043 merges.

What this does

agentcore project dev gives a local agent tracing with no extra config: an in-process OTLP/HTTP receiver starts on a loopback port, the agent's OpenTelemetry SDK is pointed at it via env vars, and traces persist through #2043's TraceStore to agentcore/.cli/traces/. Nothing leaves the machine; --no-traces or instrumentation.enableOtel: false opts out.

  • io/httpServer.ts — loopback-only node:http primitive (body caps, abort-signal close).
  • otel/collector.tsPOST /v1/traces + /v1/logs, protobuf or JSON. Pinned @opentelemetry/otlp-transformer@0.213.0, the last version shipping request decoders — do not bump.
  • Handler wiring — collector lifecycle on the command's abort signal; signal-specific OTEL env vars so stray shell values can't redirect traces; host.docker.internal rewrite for containers; sitecustomize-on-PYTHONPATH so uvicorn --reload workers stay instrumented (a wrapper would only instrument the reloader parent).

Verification

Per-layer tests (receiver over real ephemeral-port HTTP with protobuf + JSON fixtures; handler with fake runners/collector) plus end to end: built bundle → real project createproject dev → real Bedrock invocation → traces on disk. Protobuf decode verified under bun, the Node bundle, and a compiled binary.

@github-actionsgithub-actionsBot added the size/xl PR size: XL label Aug 12, 2026
@github-actionsgithub-actionsBot added agentcore-harness-reviewing AgentCore Harness review in progress size/xl PR size: XL and removed size/xl PR size: XL labels Aug 12, 2026
@github-actionsgithub-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 12, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from b4d0349 to e2e696cCompareAugust 14, 2026 20:38
@github-actionsgithub-actionsBot added size/xl PR size: XL and removed size/xl PR size: XL labels Aug 14, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 56c20f7 to a59ebf1CompareAugust 17, 2026 17:30
@codecov-commenter

codecov-commenter commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.28315% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.17%. Comparing base (a41b4de) to head (10be0a6).

Files with missing linesPatch %Lines
src/io/httpServer.ts97.05%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## refactor #1980 +/- ##
============================================
+ Coverage 97.15% 97.17% +0.02% 
============================================
Files 387 389 +2 Lines 23124 23386 +262 ============================================
+ Hits 22465 22726 +261 - Misses 659 660 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from cad8772 to 7e315b9CompareAugust 17, 2026 17:53
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from be5c3d5 to 4b80241CompareAugust 21, 2026 15:44
Base automatically changed from feat/dev-otel-storage to refactorAugust 21, 2026 17:51
tejaskash added a commit that referenced this pull request Aug 21, 2026
* feat(dev): OTLP trace storage for local dev
Pure OTLP wire handling (per-trace batch partitioning, id normalization,
frontend shaping) and append-only per-trace JSONL storage. A batch routinely
carries spans from several traces, so persistence partitions by trace id —
writing whole batches under the first id corrupts trace identity. Consumed by
the OTLP collector in #1980, which stacks on this.
* fix(dev): match trace filters against every participating service
A distributed trace spans several local agents whose exports append to the
same trace file; filtering by any participant must find it, not only the
first service seen.
* fix(dev): address review on OTLP trace storage
- narrow OtlpAttributes to the key/value wire form (the flat-record variant
had no producer); drop the dead passthrough branches
- flatten array attributes through extractAnyValue so ints stay numeric and
nested kvlists survive (was stringifying and dropping them)
- count rendered (post-filter) spans for the list summary instead of raw
records, so the count matches the waterfall the inspector shows
- surface non-ENOENT fs errors from reads instead of masking them as empty
- add newest-N limit to list() for the inspector's per-invocation poll
* fix(dev): flatten kvlist-valued OTLP attributes instead of dropping them
flattenAttributes hand-rolled a value branch per AnyValue kind and had no
kvlistValue case, so an attribute whose value is a kvlist (or anything the
chain didn't enumerate) silently vanished. Route every attribute value
through extractAnyValue, which already unwraps all variants including
kvlist — smaller and complete.
Addresses Gitika's review on transforms.ts (reuse extractAnyValue; kvlist
must not disappear).
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 4b80241 to 733c274CompareAugust 21, 2026 17:51
@tejaskash

Copy link
Copy Markdown
ContributorAuthor

#2039 was superseded by #2043, which just merged — this is unblocked and rebased on it.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from fce12fd to 27fbae3CompareAugust 21, 2026 17:59
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/io/httpServer.ts
Comment threadsrc/router/flags.tsx Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/codezip.ts
Comment threadsrc/core/dev/codezip.ts Outdated
tejaskash added a commit that referenced this pull request Aug 21, 2026
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
Comment threadsrc/core/dev/container.ts
Comment threadsrc/io/httpServer.ts
Comment threadsrc/core/dev/container.ts
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/handlers/project/dev/index.ts Outdated
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
An in-process OTLP/HTTP receiver (protobuf via the pinned otlp-transformer
decoders, or JSON) persists agent traces through the storage layer. project
dev starts it unless --no-traces or the runtime disables instrumentation,
points every spawned agent at it (signal-specific OTEL env), rewrites the
endpoint to host.docker.internal for containers — with an explicit
host-gateway mapping so Linux Docker Engine resolves it — and keeps uvicorn
--reload workers instrumented via sitecustomize on PYTHONPATH. Oversized
collector requests get a 413 before the connection closes so exporters do
not retry them as transient failures.
Rebuilt from explicit paths: the previous tree-snapshot commit accidentally
reverted unrelated merged work (cdk target guard, config-bundle TUI, error
classification).
A batch that can't be persisted (disk full, permissions) was being
turned into a 500, which the OTEL SDK exporter retries forever while the
user sees nothing. Ack the export (200) so retries stop, and surface the
failure once via an onError sink threaded from the collector to the dev
handler, which owns the IO to warn the user.
Addresses Gitika's review on store.ts:45 (catch in one place; don't let
persistence faults read as a silent, retried loss).
Follow-ups from #2043 review (Gitika, Harrison):
- Add a TraceStore.list spanCount test with transport-noise spans
(1 agent + 4 http-send -> "1"), guarding the post-filter count.
- Rename hexFromB64OrString -> hexFromBase64OrHex; both reviewers misread B64.
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
…cycle
Address reviewer findings on the collector and dev wiring:
- Validate top-level OTLP shape and return 400 instead of mislabeling a bad
payload as a persistence error.
- Guard the shared HTTP server against a client that disconnects mid-response
so it can no longer crash project dev; add an optional bind host.
- Bind the collector to 0.0.0.0 for container runtimes so a container can
reach it over the host bridge on Linux.
- Run the container template under opentelemetry-instrument so it emits traces.
- Keep the collector alive through the child's shutdown grace so final spans
are not lost.
- Force the OTEL settings that would otherwise let shell or .env.local values
disable or break local collection.
- Make the uv sitecustomize discovery abortable and read its path from a
marker rather than the last merged output line.
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from adf371b to 10be0a6CompareAugust 21, 2026 18:55

@aidandaly24aidandaly24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the fixes. This looks good to me!

@tejaskash
tejaskash merged commit f38b18d into refactorAug 21, 2026
11 checks passed
@tejaskash
tejaskash deleted the feat/dev-otel-collector branch August 21, 2026 21:32
tejaskash added a commit that referenced this pull request Aug 21, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
* feat(dev): supervise every runtime in project dev
project dev without --agent now runs all of the project's runtimes at once:
a DevSupervisor owns per-agent lifecycle (sequential port resolution — a
concurrent race would put two agents on one port), merges every runner's
output into one agent-attributed stream ([name] prefixes; an agent field in
NDJSON), and keeps the session alive when one agent crashes. Selecting a
single runtime (--agent, or a one-runtime project) keeps the direct path
where a crash still fails the command.
* fix(dev): bound readiness polling; unchain abort listeners on failed setup
A child that stays alive without ever binding its port previously blocked
every later runtime (starts are sequential) until interrupted; readiness now
gives up after 120s and fails that start. Setup failures before the pump
exists (port resolution, environment) now remove their parent-abort listener
like every other exit path, so Inspector retries of a failing agent cannot
accumulate listeners.
* fix(dev): bind collector to 0.0.0.0 when any runtime is a container
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
* fix(dev): deliver events queued while the supervisor stream drains a batch
events() spliced the queue, yielded each event, then installed its wake
callback and blocked. A push landing during the yields ran while wake was
undefined, so its wake was a no-op, and the newly installed waiter never
noticed the queued event until the next push or shutdown. Re-check the queue
inside the wait so a queued event resolves immediately.
* refactor(dev): route single runtime through the supervisor and address review
Remove the runSingleRuntime special case so every runtime, one or many,
runs through DevSupervisor. Start all agents concurrently and stream their
output live rather than awaiting each readiness first. End the event loop
when no agent is starting or running, and exit non-zero when none started.
Make start() async so an unknown name rejects rather than throwing
synchronously. Time readiness off the agent's last output (idle timeout)
so a slow build that keeps logging is not killed. Move waitForPort into
src/io/port.ts to keep socket IO in the IO layer. Rename the environment
accessor to getDevEnvVarsForRuntime and SupervisedEvent.agent to agentName.
* refactor(dev): always attribute agent output to its runtime
Split renderEvent into renderAgentEvent (agent name required) and
renderStatus for command-level trace-collector lines, so agent output
is always tagged with [name] and the agent-optional branches are gone.
Addresses review feedback on #2041.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xlPR size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@tejaskash@codecov-commenter@Hweinstock@notgitika@aidandaly24
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' feat(dev): collect local OTEL traces in project dev by tejaskash · Pull Request #1980 · aws/agentcore-cli · GitHub
Skip to content

feat(dev): collect local OTEL traces in project dev - #1980

Merged
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector
Aug 21, 2026
Merged

feat(dev): collect local OTEL traces in project dev#1980
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector

Conversation

@tejaskash

@tejaskashtejaskash commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Review after #2043

Until #2043 merges, the diff shows both PRs' commits (GitHub's stack tooling wouldn't retarget the base); the work to review here is the single collector commit on top. It collapses to just that once #2043 merges.

What this does

agentcore project dev gives a local agent tracing with no extra config: an in-process OTLP/HTTP receiver starts on a loopback port, the agent's OpenTelemetry SDK is pointed at it via env vars, and traces persist through #2043's TraceStore to agentcore/.cli/traces/. Nothing leaves the machine; --no-traces or instrumentation.enableOtel: false opts out.

  • io/httpServer.ts — loopback-only node:http primitive (body caps, abort-signal close).
  • otel/collector.tsPOST /v1/traces + /v1/logs, protobuf or JSON. Pinned @opentelemetry/otlp-transformer@0.213.0, the last version shipping request decoders — do not bump.
  • Handler wiring — collector lifecycle on the command's abort signal; signal-specific OTEL env vars so stray shell values can't redirect traces; host.docker.internal rewrite for containers; sitecustomize-on-PYTHONPATH so uvicorn --reload workers stay instrumented (a wrapper would only instrument the reloader parent).

Verification

Per-layer tests (receiver over real ephemeral-port HTTP with protobuf + JSON fixtures; handler with fake runners/collector) plus end to end: built bundle → real project createproject dev → real Bedrock invocation → traces on disk. Protobuf decode verified under bun, the Node bundle, and a compiled binary.

@github-actionsgithub-actionsBot added the size/xl PR size: XL label Aug 12, 2026
@github-actionsgithub-actionsBot added agentcore-harness-reviewing AgentCore Harness review in progress size/xl PR size: XL and removed size/xl PR size: XL labels Aug 12, 2026
@github-actionsgithub-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 12, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from b4d0349 to e2e696cCompareAugust 14, 2026 20:38
@github-actionsgithub-actionsBot added size/xl PR size: XL and removed size/xl PR size: XL labels Aug 14, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 56c20f7 to a59ebf1CompareAugust 17, 2026 17:30
@codecov-commenter

codecov-commenter commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.28315% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.17%. Comparing base (a41b4de) to head (10be0a6).

Files with missing linesPatch %Lines
src/io/httpServer.ts97.05%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## refactor #1980 +/- ##
============================================
+ Coverage 97.15% 97.17% +0.02% 
============================================
Files 387 389 +2 Lines 23124 23386 +262 ============================================
+ Hits 22465 22726 +261 - Misses 659 660 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from cad8772 to 7e315b9CompareAugust 17, 2026 17:53
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from be5c3d5 to 4b80241CompareAugust 21, 2026 15:44
Base automatically changed from feat/dev-otel-storage to refactorAugust 21, 2026 17:51
tejaskash added a commit that referenced this pull request Aug 21, 2026
* feat(dev): OTLP trace storage for local dev
Pure OTLP wire handling (per-trace batch partitioning, id normalization,
frontend shaping) and append-only per-trace JSONL storage. A batch routinely
carries spans from several traces, so persistence partitions by trace id —
writing whole batches under the first id corrupts trace identity. Consumed by
the OTLP collector in #1980, which stacks on this.
* fix(dev): match trace filters against every participating service
A distributed trace spans several local agents whose exports append to the
same trace file; filtering by any participant must find it, not only the
first service seen.
* fix(dev): address review on OTLP trace storage
- narrow OtlpAttributes to the key/value wire form (the flat-record variant
had no producer); drop the dead passthrough branches
- flatten array attributes through extractAnyValue so ints stay numeric and
nested kvlists survive (was stringifying and dropping them)
- count rendered (post-filter) spans for the list summary instead of raw
records, so the count matches the waterfall the inspector shows
- surface non-ENOENT fs errors from reads instead of masking them as empty
- add newest-N limit to list() for the inspector's per-invocation poll
* fix(dev): flatten kvlist-valued OTLP attributes instead of dropping them
flattenAttributes hand-rolled a value branch per AnyValue kind and had no
kvlistValue case, so an attribute whose value is a kvlist (or anything the
chain didn't enumerate) silently vanished. Route every attribute value
through extractAnyValue, which already unwraps all variants including
kvlist — smaller and complete.
Addresses Gitika's review on transforms.ts (reuse extractAnyValue; kvlist
must not disappear).
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 4b80241 to 733c274CompareAugust 21, 2026 17:51
@tejaskash

Copy link
Copy Markdown
ContributorAuthor

#2039 was superseded by #2043, which just merged — this is unblocked and rebased on it.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from fce12fd to 27fbae3CompareAugust 21, 2026 17:59
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/io/httpServer.ts
Comment threadsrc/router/flags.tsx Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/codezip.ts
Comment threadsrc/core/dev/codezip.ts Outdated
tejaskash added a commit that referenced this pull request Aug 21, 2026
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
Comment threadsrc/core/dev/container.ts
Comment threadsrc/io/httpServer.ts
Comment threadsrc/core/dev/container.ts
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/handlers/project/dev/index.ts Outdated
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
An in-process OTLP/HTTP receiver (protobuf via the pinned otlp-transformer
decoders, or JSON) persists agent traces through the storage layer. project
dev starts it unless --no-traces or the runtime disables instrumentation,
points every spawned agent at it (signal-specific OTEL env), rewrites the
endpoint to host.docker.internal for containers — with an explicit
host-gateway mapping so Linux Docker Engine resolves it — and keeps uvicorn
--reload workers instrumented via sitecustomize on PYTHONPATH. Oversized
collector requests get a 413 before the connection closes so exporters do
not retry them as transient failures.
Rebuilt from explicit paths: the previous tree-snapshot commit accidentally
reverted unrelated merged work (cdk target guard, config-bundle TUI, error
classification).
A batch that can't be persisted (disk full, permissions) was being
turned into a 500, which the OTEL SDK exporter retries forever while the
user sees nothing. Ack the export (200) so retries stop, and surface the
failure once via an onError sink threaded from the collector to the dev
handler, which owns the IO to warn the user.
Addresses Gitika's review on store.ts:45 (catch in one place; don't let
persistence faults read as a silent, retried loss).
Follow-ups from #2043 review (Gitika, Harrison):
- Add a TraceStore.list spanCount test with transport-noise spans
(1 agent + 4 http-send -> "1"), guarding the post-filter count.
- Rename hexFromB64OrString -> hexFromBase64OrHex; both reviewers misread B64.
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
…cycle
Address reviewer findings on the collector and dev wiring:
- Validate top-level OTLP shape and return 400 instead of mislabeling a bad
payload as a persistence error.
- Guard the shared HTTP server against a client that disconnects mid-response
so it can no longer crash project dev; add an optional bind host.
- Bind the collector to 0.0.0.0 for container runtimes so a container can
reach it over the host bridge on Linux.
- Run the container template under opentelemetry-instrument so it emits traces.
- Keep the collector alive through the child's shutdown grace so final spans
are not lost.
- Force the OTEL settings that would otherwise let shell or .env.local values
disable or break local collection.
- Make the uv sitecustomize discovery abortable and read its path from a
marker rather than the last merged output line.
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from adf371b to 10be0a6CompareAugust 21, 2026 18:55

@aidandaly24aidandaly24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the fixes. This looks good to me!

@tejaskash
tejaskash merged commit f38b18d into refactorAug 21, 2026
11 checks passed
@tejaskash
tejaskash deleted the feat/dev-otel-collector branch August 21, 2026 21:32
tejaskash added a commit that referenced this pull request Aug 21, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
* feat(dev): supervise every runtime in project dev
project dev without --agent now runs all of the project's runtimes at once:
a DevSupervisor owns per-agent lifecycle (sequential port resolution — a
concurrent race would put two agents on one port), merges every runner's
output into one agent-attributed stream ([name] prefixes; an agent field in
NDJSON), and keeps the session alive when one agent crashes. Selecting a
single runtime (--agent, or a one-runtime project) keeps the direct path
where a crash still fails the command.
* fix(dev): bound readiness polling; unchain abort listeners on failed setup
A child that stays alive without ever binding its port previously blocked
every later runtime (starts are sequential) until interrupted; readiness now
gives up after 120s and fails that start. Setup failures before the pump
exists (port resolution, environment) now remove their parent-abort listener
like every other exit path, so Inspector retries of a failing agent cannot
accumulate listeners.
* fix(dev): bind collector to 0.0.0.0 when any runtime is a container
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
* fix(dev): deliver events queued while the supervisor stream drains a batch
events() spliced the queue, yielded each event, then installed its wake
callback and blocked. A push landing during the yields ran while wake was
undefined, so its wake was a no-op, and the newly installed waiter never
noticed the queued event until the next push or shutdown. Re-check the queue
inside the wait so a queued event resolves immediately.
* refactor(dev): route single runtime through the supervisor and address review
Remove the runSingleRuntime special case so every runtime, one or many,
runs through DevSupervisor. Start all agents concurrently and stream their
output live rather than awaiting each readiness first. End the event loop
when no agent is starting or running, and exit non-zero when none started.
Make start() async so an unknown name rejects rather than throwing
synchronously. Time readiness off the agent's last output (idle timeout)
so a slow build that keeps logging is not killed. Move waitForPort into
src/io/port.ts to keep socket IO in the IO layer. Rename the environment
accessor to getDevEnvVarsForRuntime and SupervisedEvent.agent to agentName.
* refactor(dev): always attribute agent output to its runtime
Split renderEvent into renderAgentEvent (agent name required) and
renderStatus for command-level trace-collector lines, so agent output
is always tagged with [name] and the agent-optional branches are gone.
Addresses review feedback on #2041.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xlPR size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@tejaskash@codecov-commenter@Hweinstock@notgitika@aidandaly24
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' feat(dev): collect local OTEL traces in project dev by tejaskash · Pull Request #1980 · aws/agentcore-cli · GitHub
Skip to content

feat(dev): collect local OTEL traces in project dev - #1980

Merged
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector
Aug 21, 2026
Merged

feat(dev): collect local OTEL traces in project dev#1980
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector

Conversation

@tejaskash

@tejaskashtejaskash commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Review after #2043

Until #2043 merges, the diff shows both PRs' commits (GitHub's stack tooling wouldn't retarget the base); the work to review here is the single collector commit on top. It collapses to just that once #2043 merges.

What this does

agentcore project dev gives a local agent tracing with no extra config: an in-process OTLP/HTTP receiver starts on a loopback port, the agent's OpenTelemetry SDK is pointed at it via env vars, and traces persist through #2043's TraceStore to agentcore/.cli/traces/. Nothing leaves the machine; --no-traces or instrumentation.enableOtel: false opts out.

  • io/httpServer.ts — loopback-only node:http primitive (body caps, abort-signal close).
  • otel/collector.tsPOST /v1/traces + /v1/logs, protobuf or JSON. Pinned @opentelemetry/otlp-transformer@0.213.0, the last version shipping request decoders — do not bump.
  • Handler wiring — collector lifecycle on the command's abort signal; signal-specific OTEL env vars so stray shell values can't redirect traces; host.docker.internal rewrite for containers; sitecustomize-on-PYTHONPATH so uvicorn --reload workers stay instrumented (a wrapper would only instrument the reloader parent).

Verification

Per-layer tests (receiver over real ephemeral-port HTTP with protobuf + JSON fixtures; handler with fake runners/collector) plus end to end: built bundle → real project createproject dev → real Bedrock invocation → traces on disk. Protobuf decode verified under bun, the Node bundle, and a compiled binary.

@github-actionsgithub-actionsBot added the size/xl PR size: XL label Aug 12, 2026
@github-actionsgithub-actionsBot added agentcore-harness-reviewing AgentCore Harness review in progress size/xl PR size: XL and removed size/xl PR size: XL labels Aug 12, 2026
@github-actionsgithub-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 12, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from b4d0349 to e2e696cCompareAugust 14, 2026 20:38
@github-actionsgithub-actionsBot added size/xl PR size: XL and removed size/xl PR size: XL labels Aug 14, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 56c20f7 to a59ebf1CompareAugust 17, 2026 17:30
@codecov-commenter

codecov-commenter commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.28315% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.17%. Comparing base (a41b4de) to head (10be0a6).

Files with missing linesPatch %Lines
src/io/httpServer.ts97.05%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## refactor #1980 +/- ##
============================================
+ Coverage 97.15% 97.17% +0.02% 
============================================
Files 387 389 +2 Lines 23124 23386 +262 ============================================
+ Hits 22465 22726 +261 - Misses 659 660 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from cad8772 to 7e315b9CompareAugust 17, 2026 17:53
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from be5c3d5 to 4b80241CompareAugust 21, 2026 15:44
Base automatically changed from feat/dev-otel-storage to refactorAugust 21, 2026 17:51
tejaskash added a commit that referenced this pull request Aug 21, 2026
* feat(dev): OTLP trace storage for local dev
Pure OTLP wire handling (per-trace batch partitioning, id normalization,
frontend shaping) and append-only per-trace JSONL storage. A batch routinely
carries spans from several traces, so persistence partitions by trace id —
writing whole batches under the first id corrupts trace identity. Consumed by
the OTLP collector in #1980, which stacks on this.
* fix(dev): match trace filters against every participating service
A distributed trace spans several local agents whose exports append to the
same trace file; filtering by any participant must find it, not only the
first service seen.
* fix(dev): address review on OTLP trace storage
- narrow OtlpAttributes to the key/value wire form (the flat-record variant
had no producer); drop the dead passthrough branches
- flatten array attributes through extractAnyValue so ints stay numeric and
nested kvlists survive (was stringifying and dropping them)
- count rendered (post-filter) spans for the list summary instead of raw
records, so the count matches the waterfall the inspector shows
- surface non-ENOENT fs errors from reads instead of masking them as empty
- add newest-N limit to list() for the inspector's per-invocation poll
* fix(dev): flatten kvlist-valued OTLP attributes instead of dropping them
flattenAttributes hand-rolled a value branch per AnyValue kind and had no
kvlistValue case, so an attribute whose value is a kvlist (or anything the
chain didn't enumerate) silently vanished. Route every attribute value
through extractAnyValue, which already unwraps all variants including
kvlist — smaller and complete.
Addresses Gitika's review on transforms.ts (reuse extractAnyValue; kvlist
must not disappear).
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 4b80241 to 733c274CompareAugust 21, 2026 17:51
@tejaskash

Copy link
Copy Markdown
ContributorAuthor

#2039 was superseded by #2043, which just merged — this is unblocked and rebased on it.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from fce12fd to 27fbae3CompareAugust 21, 2026 17:59
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/io/httpServer.ts
Comment threadsrc/router/flags.tsx Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/codezip.ts
Comment threadsrc/core/dev/codezip.ts Outdated
tejaskash added a commit that referenced this pull request Aug 21, 2026
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
Comment threadsrc/core/dev/container.ts
Comment threadsrc/io/httpServer.ts
Comment threadsrc/core/dev/container.ts
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/handlers/project/dev/index.ts Outdated
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
An in-process OTLP/HTTP receiver (protobuf via the pinned otlp-transformer
decoders, or JSON) persists agent traces through the storage layer. project
dev starts it unless --no-traces or the runtime disables instrumentation,
points every spawned agent at it (signal-specific OTEL env), rewrites the
endpoint to host.docker.internal for containers — with an explicit
host-gateway mapping so Linux Docker Engine resolves it — and keeps uvicorn
--reload workers instrumented via sitecustomize on PYTHONPATH. Oversized
collector requests get a 413 before the connection closes so exporters do
not retry them as transient failures.
Rebuilt from explicit paths: the previous tree-snapshot commit accidentally
reverted unrelated merged work (cdk target guard, config-bundle TUI, error
classification).
A batch that can't be persisted (disk full, permissions) was being
turned into a 500, which the OTEL SDK exporter retries forever while the
user sees nothing. Ack the export (200) so retries stop, and surface the
failure once via an onError sink threaded from the collector to the dev
handler, which owns the IO to warn the user.
Addresses Gitika's review on store.ts:45 (catch in one place; don't let
persistence faults read as a silent, retried loss).
Follow-ups from #2043 review (Gitika, Harrison):
- Add a TraceStore.list spanCount test with transport-noise spans
(1 agent + 4 http-send -> "1"), guarding the post-filter count.
- Rename hexFromB64OrString -> hexFromBase64OrHex; both reviewers misread B64.
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
…cycle
Address reviewer findings on the collector and dev wiring:
- Validate top-level OTLP shape and return 400 instead of mislabeling a bad
payload as a persistence error.
- Guard the shared HTTP server against a client that disconnects mid-response
so it can no longer crash project dev; add an optional bind host.
- Bind the collector to 0.0.0.0 for container runtimes so a container can
reach it over the host bridge on Linux.
- Run the container template under opentelemetry-instrument so it emits traces.
- Keep the collector alive through the child's shutdown grace so final spans
are not lost.
- Force the OTEL settings that would otherwise let shell or .env.local values
disable or break local collection.
- Make the uv sitecustomize discovery abortable and read its path from a
marker rather than the last merged output line.
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from adf371b to 10be0a6CompareAugust 21, 2026 18:55

@aidandaly24aidandaly24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the fixes. This looks good to me!

@tejaskash
tejaskash merged commit f38b18d into refactorAug 21, 2026
11 checks passed
@tejaskash
tejaskash deleted the feat/dev-otel-collector branch August 21, 2026 21:32
tejaskash added a commit that referenced this pull request Aug 21, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
* feat(dev): supervise every runtime in project dev
project dev without --agent now runs all of the project's runtimes at once:
a DevSupervisor owns per-agent lifecycle (sequential port resolution — a
concurrent race would put two agents on one port), merges every runner's
output into one agent-attributed stream ([name] prefixes; an agent field in
NDJSON), and keeps the session alive when one agent crashes. Selecting a
single runtime (--agent, or a one-runtime project) keeps the direct path
where a crash still fails the command.
* fix(dev): bound readiness polling; unchain abort listeners on failed setup
A child that stays alive without ever binding its port previously blocked
every later runtime (starts are sequential) until interrupted; readiness now
gives up after 120s and fails that start. Setup failures before the pump
exists (port resolution, environment) now remove their parent-abort listener
like every other exit path, so Inspector retries of a failing agent cannot
accumulate listeners.
* fix(dev): bind collector to 0.0.0.0 when any runtime is a container
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
* fix(dev): deliver events queued while the supervisor stream drains a batch
events() spliced the queue, yielded each event, then installed its wake
callback and blocked. A push landing during the yields ran while wake was
undefined, so its wake was a no-op, and the newly installed waiter never
noticed the queued event until the next push or shutdown. Re-check the queue
inside the wait so a queued event resolves immediately.
* refactor(dev): route single runtime through the supervisor and address review
Remove the runSingleRuntime special case so every runtime, one or many,
runs through DevSupervisor. Start all agents concurrently and stream their
output live rather than awaiting each readiness first. End the event loop
when no agent is starting or running, and exit non-zero when none started.
Make start() async so an unknown name rejects rather than throwing
synchronously. Time readiness off the agent's last output (idle timeout)
so a slow build that keeps logging is not killed. Move waitForPort into
src/io/port.ts to keep socket IO in the IO layer. Rename the environment
accessor to getDevEnvVarsForRuntime and SupervisedEvent.agent to agentName.
* refactor(dev): always attribute agent output to its runtime
Split renderEvent into renderAgentEvent (agent name required) and
renderStatus for command-level trace-collector lines, so agent output
is always tagged with [name] and the agent-optional branches are gone.
Addresses review feedback on #2041.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xlPR size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@tejaskash@codecov-commenter@Hweinstock@notgitika@aidandaly24
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' feat(dev): collect local OTEL traces in project dev by tejaskash · Pull Request #1980 · aws/agentcore-cli · GitHub
Skip to content

feat(dev): collect local OTEL traces in project dev - #1980

Merged
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector
Aug 21, 2026
Merged

feat(dev): collect local OTEL traces in project dev#1980
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector

Conversation

@tejaskash

@tejaskashtejaskash commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Review after #2043

Until #2043 merges, the diff shows both PRs' commits (GitHub's stack tooling wouldn't retarget the base); the work to review here is the single collector commit on top. It collapses to just that once #2043 merges.

What this does

agentcore project dev gives a local agent tracing with no extra config: an in-process OTLP/HTTP receiver starts on a loopback port, the agent's OpenTelemetry SDK is pointed at it via env vars, and traces persist through #2043's TraceStore to agentcore/.cli/traces/. Nothing leaves the machine; --no-traces or instrumentation.enableOtel: false opts out.

  • io/httpServer.ts — loopback-only node:http primitive (body caps, abort-signal close).
  • otel/collector.tsPOST /v1/traces + /v1/logs, protobuf or JSON. Pinned @opentelemetry/otlp-transformer@0.213.0, the last version shipping request decoders — do not bump.
  • Handler wiring — collector lifecycle on the command's abort signal; signal-specific OTEL env vars so stray shell values can't redirect traces; host.docker.internal rewrite for containers; sitecustomize-on-PYTHONPATH so uvicorn --reload workers stay instrumented (a wrapper would only instrument the reloader parent).

Verification

Per-layer tests (receiver over real ephemeral-port HTTP with protobuf + JSON fixtures; handler with fake runners/collector) plus end to end: built bundle → real project createproject dev → real Bedrock invocation → traces on disk. Protobuf decode verified under bun, the Node bundle, and a compiled binary.

@github-actionsgithub-actionsBot added the size/xl PR size: XL label Aug 12, 2026
@github-actionsgithub-actionsBot added agentcore-harness-reviewing AgentCore Harness review in progress size/xl PR size: XL and removed size/xl PR size: XL labels Aug 12, 2026
@github-actionsgithub-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 12, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from b4d0349 to e2e696cCompareAugust 14, 2026 20:38
@github-actionsgithub-actionsBot added size/xl PR size: XL and removed size/xl PR size: XL labels Aug 14, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 56c20f7 to a59ebf1CompareAugust 17, 2026 17:30
@codecov-commenter

codecov-commenter commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.28315% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.17%. Comparing base (a41b4de) to head (10be0a6).

Files with missing linesPatch %Lines
src/io/httpServer.ts97.05%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## refactor #1980 +/- ##
============================================
+ Coverage 97.15% 97.17% +0.02% 
============================================
Files 387 389 +2 Lines 23124 23386 +262 ============================================
+ Hits 22465 22726 +261 - Misses 659 660 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from cad8772 to 7e315b9CompareAugust 17, 2026 17:53
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from be5c3d5 to 4b80241CompareAugust 21, 2026 15:44
Base automatically changed from feat/dev-otel-storage to refactorAugust 21, 2026 17:51
tejaskash added a commit that referenced this pull request Aug 21, 2026
* feat(dev): OTLP trace storage for local dev
Pure OTLP wire handling (per-trace batch partitioning, id normalization,
frontend shaping) and append-only per-trace JSONL storage. A batch routinely
carries spans from several traces, so persistence partitions by trace id —
writing whole batches under the first id corrupts trace identity. Consumed by
the OTLP collector in #1980, which stacks on this.
* fix(dev): match trace filters against every participating service
A distributed trace spans several local agents whose exports append to the
same trace file; filtering by any participant must find it, not only the
first service seen.
* fix(dev): address review on OTLP trace storage
- narrow OtlpAttributes to the key/value wire form (the flat-record variant
had no producer); drop the dead passthrough branches
- flatten array attributes through extractAnyValue so ints stay numeric and
nested kvlists survive (was stringifying and dropping them)
- count rendered (post-filter) spans for the list summary instead of raw
records, so the count matches the waterfall the inspector shows
- surface non-ENOENT fs errors from reads instead of masking them as empty
- add newest-N limit to list() for the inspector's per-invocation poll
* fix(dev): flatten kvlist-valued OTLP attributes instead of dropping them
flattenAttributes hand-rolled a value branch per AnyValue kind and had no
kvlistValue case, so an attribute whose value is a kvlist (or anything the
chain didn't enumerate) silently vanished. Route every attribute value
through extractAnyValue, which already unwraps all variants including
kvlist — smaller and complete.
Addresses Gitika's review on transforms.ts (reuse extractAnyValue; kvlist
must not disappear).
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 4b80241 to 733c274CompareAugust 21, 2026 17:51
@tejaskash

Copy link
Copy Markdown
ContributorAuthor

#2039 was superseded by #2043, which just merged — this is unblocked and rebased on it.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from fce12fd to 27fbae3CompareAugust 21, 2026 17:59
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/io/httpServer.ts
Comment threadsrc/router/flags.tsx Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/codezip.ts
Comment threadsrc/core/dev/codezip.ts Outdated
tejaskash added a commit that referenced this pull request Aug 21, 2026
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
Comment threadsrc/core/dev/container.ts
Comment threadsrc/io/httpServer.ts
Comment threadsrc/core/dev/container.ts
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/handlers/project/dev/index.ts Outdated
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
An in-process OTLP/HTTP receiver (protobuf via the pinned otlp-transformer
decoders, or JSON) persists agent traces through the storage layer. project
dev starts it unless --no-traces or the runtime disables instrumentation,
points every spawned agent at it (signal-specific OTEL env), rewrites the
endpoint to host.docker.internal for containers — with an explicit
host-gateway mapping so Linux Docker Engine resolves it — and keeps uvicorn
--reload workers instrumented via sitecustomize on PYTHONPATH. Oversized
collector requests get a 413 before the connection closes so exporters do
not retry them as transient failures.
Rebuilt from explicit paths: the previous tree-snapshot commit accidentally
reverted unrelated merged work (cdk target guard, config-bundle TUI, error
classification).
A batch that can't be persisted (disk full, permissions) was being
turned into a 500, which the OTEL SDK exporter retries forever while the
user sees nothing. Ack the export (200) so retries stop, and surface the
failure once via an onError sink threaded from the collector to the dev
handler, which owns the IO to warn the user.
Addresses Gitika's review on store.ts:45 (catch in one place; don't let
persistence faults read as a silent, retried loss).
Follow-ups from #2043 review (Gitika, Harrison):
- Add a TraceStore.list spanCount test with transport-noise spans
(1 agent + 4 http-send -> "1"), guarding the post-filter count.
- Rename hexFromB64OrString -> hexFromBase64OrHex; both reviewers misread B64.
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
…cycle
Address reviewer findings on the collector and dev wiring:
- Validate top-level OTLP shape and return 400 instead of mislabeling a bad
payload as a persistence error.
- Guard the shared HTTP server against a client that disconnects mid-response
so it can no longer crash project dev; add an optional bind host.
- Bind the collector to 0.0.0.0 for container runtimes so a container can
reach it over the host bridge on Linux.
- Run the container template under opentelemetry-instrument so it emits traces.
- Keep the collector alive through the child's shutdown grace so final spans
are not lost.
- Force the OTEL settings that would otherwise let shell or .env.local values
disable or break local collection.
- Make the uv sitecustomize discovery abortable and read its path from a
marker rather than the last merged output line.
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from adf371b to 10be0a6CompareAugust 21, 2026 18:55

@aidandaly24aidandaly24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the fixes. This looks good to me!

@tejaskash
tejaskash merged commit f38b18d into refactorAug 21, 2026
11 checks passed
@tejaskash
tejaskash deleted the feat/dev-otel-collector branch August 21, 2026 21:32
tejaskash added a commit that referenced this pull request Aug 21, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
* feat(dev): supervise every runtime in project dev
project dev without --agent now runs all of the project's runtimes at once:
a DevSupervisor owns per-agent lifecycle (sequential port resolution — a
concurrent race would put two agents on one port), merges every runner's
output into one agent-attributed stream ([name] prefixes; an agent field in
NDJSON), and keeps the session alive when one agent crashes. Selecting a
single runtime (--agent, or a one-runtime project) keeps the direct path
where a crash still fails the command.
* fix(dev): bound readiness polling; unchain abort listeners on failed setup
A child that stays alive without ever binding its port previously blocked
every later runtime (starts are sequential) until interrupted; readiness now
gives up after 120s and fails that start. Setup failures before the pump
exists (port resolution, environment) now remove their parent-abort listener
like every other exit path, so Inspector retries of a failing agent cannot
accumulate listeners.
* fix(dev): bind collector to 0.0.0.0 when any runtime is a container
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
* fix(dev): deliver events queued while the supervisor stream drains a batch
events() spliced the queue, yielded each event, then installed its wake
callback and blocked. A push landing during the yields ran while wake was
undefined, so its wake was a no-op, and the newly installed waiter never
noticed the queued event until the next push or shutdown. Re-check the queue
inside the wait so a queued event resolves immediately.
* refactor(dev): route single runtime through the supervisor and address review
Remove the runSingleRuntime special case so every runtime, one or many,
runs through DevSupervisor. Start all agents concurrently and stream their
output live rather than awaiting each readiness first. End the event loop
when no agent is starting or running, and exit non-zero when none started.
Make start() async so an unknown name rejects rather than throwing
synchronously. Time readiness off the agent's last output (idle timeout)
so a slow build that keeps logging is not killed. Move waitForPort into
src/io/port.ts to keep socket IO in the IO layer. Rename the environment
accessor to getDevEnvVarsForRuntime and SupervisedEvent.agent to agentName.
* refactor(dev): always attribute agent output to its runtime
Split renderEvent into renderAgentEvent (agent name required) and
renderStatus for command-level trace-collector lines, so agent output
is always tagged with [name] and the agent-optional branches are gone.
Addresses review feedback on #2041.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xlPR size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@tejaskash@codecov-commenter@Hweinstock@notgitika@aidandaly24
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' feat(dev): collect local OTEL traces in project dev by tejaskash · Pull Request #1980 · aws/agentcore-cli · GitHub
Skip to content

feat(dev): collect local OTEL traces in project dev - #1980

Merged
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector
Aug 21, 2026
Merged

feat(dev): collect local OTEL traces in project dev#1980
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector

Conversation

@tejaskash

@tejaskashtejaskash commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Review after #2043

Until #2043 merges, the diff shows both PRs' commits (GitHub's stack tooling wouldn't retarget the base); the work to review here is the single collector commit on top. It collapses to just that once #2043 merges.

What this does

agentcore project dev gives a local agent tracing with no extra config: an in-process OTLP/HTTP receiver starts on a loopback port, the agent's OpenTelemetry SDK is pointed at it via env vars, and traces persist through #2043's TraceStore to agentcore/.cli/traces/. Nothing leaves the machine; --no-traces or instrumentation.enableOtel: false opts out.

  • io/httpServer.ts — loopback-only node:http primitive (body caps, abort-signal close).
  • otel/collector.tsPOST /v1/traces + /v1/logs, protobuf or JSON. Pinned @opentelemetry/otlp-transformer@0.213.0, the last version shipping request decoders — do not bump.
  • Handler wiring — collector lifecycle on the command's abort signal; signal-specific OTEL env vars so stray shell values can't redirect traces; host.docker.internal rewrite for containers; sitecustomize-on-PYTHONPATH so uvicorn --reload workers stay instrumented (a wrapper would only instrument the reloader parent).

Verification

Per-layer tests (receiver over real ephemeral-port HTTP with protobuf + JSON fixtures; handler with fake runners/collector) plus end to end: built bundle → real project createproject dev → real Bedrock invocation → traces on disk. Protobuf decode verified under bun, the Node bundle, and a compiled binary.

@github-actionsgithub-actionsBot added the size/xl PR size: XL label Aug 12, 2026
@github-actionsgithub-actionsBot added agentcore-harness-reviewing AgentCore Harness review in progress size/xl PR size: XL and removed size/xl PR size: XL labels Aug 12, 2026
@github-actionsgithub-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 12, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from b4d0349 to e2e696cCompareAugust 14, 2026 20:38
@github-actionsgithub-actionsBot added size/xl PR size: XL and removed size/xl PR size: XL labels Aug 14, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 56c20f7 to a59ebf1CompareAugust 17, 2026 17:30
@codecov-commenter

codecov-commenter commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.28315% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.17%. Comparing base (a41b4de) to head (10be0a6).

Files with missing linesPatch %Lines
src/io/httpServer.ts97.05%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## refactor #1980 +/- ##
============================================
+ Coverage 97.15% 97.17% +0.02% 
============================================
Files 387 389 +2 Lines 23124 23386 +262 ============================================
+ Hits 22465 22726 +261 - Misses 659 660 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from cad8772 to 7e315b9CompareAugust 17, 2026 17:53
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from be5c3d5 to 4b80241CompareAugust 21, 2026 15:44
Base automatically changed from feat/dev-otel-storage to refactorAugust 21, 2026 17:51
tejaskash added a commit that referenced this pull request Aug 21, 2026
* feat(dev): OTLP trace storage for local dev
Pure OTLP wire handling (per-trace batch partitioning, id normalization,
frontend shaping) and append-only per-trace JSONL storage. A batch routinely
carries spans from several traces, so persistence partitions by trace id —
writing whole batches under the first id corrupts trace identity. Consumed by
the OTLP collector in #1980, which stacks on this.
* fix(dev): match trace filters against every participating service
A distributed trace spans several local agents whose exports append to the
same trace file; filtering by any participant must find it, not only the
first service seen.
* fix(dev): address review on OTLP trace storage
- narrow OtlpAttributes to the key/value wire form (the flat-record variant
had no producer); drop the dead passthrough branches
- flatten array attributes through extractAnyValue so ints stay numeric and
nested kvlists survive (was stringifying and dropping them)
- count rendered (post-filter) spans for the list summary instead of raw
records, so the count matches the waterfall the inspector shows
- surface non-ENOENT fs errors from reads instead of masking them as empty
- add newest-N limit to list() for the inspector's per-invocation poll
* fix(dev): flatten kvlist-valued OTLP attributes instead of dropping them
flattenAttributes hand-rolled a value branch per AnyValue kind and had no
kvlistValue case, so an attribute whose value is a kvlist (or anything the
chain didn't enumerate) silently vanished. Route every attribute value
through extractAnyValue, which already unwraps all variants including
kvlist — smaller and complete.
Addresses Gitika's review on transforms.ts (reuse extractAnyValue; kvlist
must not disappear).
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 4b80241 to 733c274CompareAugust 21, 2026 17:51
@tejaskash

Copy link
Copy Markdown
ContributorAuthor

#2039 was superseded by #2043, which just merged — this is unblocked and rebased on it.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from fce12fd to 27fbae3CompareAugust 21, 2026 17:59
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/io/httpServer.ts
Comment threadsrc/router/flags.tsx Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/codezip.ts
Comment threadsrc/core/dev/codezip.ts Outdated
tejaskash added a commit that referenced this pull request Aug 21, 2026
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
Comment threadsrc/core/dev/container.ts
Comment threadsrc/io/httpServer.ts
Comment threadsrc/core/dev/container.ts
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/handlers/project/dev/index.ts Outdated
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
An in-process OTLP/HTTP receiver (protobuf via the pinned otlp-transformer
decoders, or JSON) persists agent traces through the storage layer. project
dev starts it unless --no-traces or the runtime disables instrumentation,
points every spawned agent at it (signal-specific OTEL env), rewrites the
endpoint to host.docker.internal for containers — with an explicit
host-gateway mapping so Linux Docker Engine resolves it — and keeps uvicorn
--reload workers instrumented via sitecustomize on PYTHONPATH. Oversized
collector requests get a 413 before the connection closes so exporters do
not retry them as transient failures.
Rebuilt from explicit paths: the previous tree-snapshot commit accidentally
reverted unrelated merged work (cdk target guard, config-bundle TUI, error
classification).
A batch that can't be persisted (disk full, permissions) was being
turned into a 500, which the OTEL SDK exporter retries forever while the
user sees nothing. Ack the export (200) so retries stop, and surface the
failure once via an onError sink threaded from the collector to the dev
handler, which owns the IO to warn the user.
Addresses Gitika's review on store.ts:45 (catch in one place; don't let
persistence faults read as a silent, retried loss).
Follow-ups from #2043 review (Gitika, Harrison):
- Add a TraceStore.list spanCount test with transport-noise spans
(1 agent + 4 http-send -> "1"), guarding the post-filter count.
- Rename hexFromB64OrString -> hexFromBase64OrHex; both reviewers misread B64.
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
…cycle
Address reviewer findings on the collector and dev wiring:
- Validate top-level OTLP shape and return 400 instead of mislabeling a bad
payload as a persistence error.
- Guard the shared HTTP server against a client that disconnects mid-response
so it can no longer crash project dev; add an optional bind host.
- Bind the collector to 0.0.0.0 for container runtimes so a container can
reach it over the host bridge on Linux.
- Run the container template under opentelemetry-instrument so it emits traces.
- Keep the collector alive through the child's shutdown grace so final spans
are not lost.
- Force the OTEL settings that would otherwise let shell or .env.local values
disable or break local collection.
- Make the uv sitecustomize discovery abortable and read its path from a
marker rather than the last merged output line.
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from adf371b to 10be0a6CompareAugust 21, 2026 18:55

@aidandaly24aidandaly24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the fixes. This looks good to me!

@tejaskash
tejaskash merged commit f38b18d into refactorAug 21, 2026
11 checks passed
@tejaskash
tejaskash deleted the feat/dev-otel-collector branch August 21, 2026 21:32
tejaskash added a commit that referenced this pull request Aug 21, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
* feat(dev): supervise every runtime in project dev
project dev without --agent now runs all of the project's runtimes at once:
a DevSupervisor owns per-agent lifecycle (sequential port resolution — a
concurrent race would put two agents on one port), merges every runner's
output into one agent-attributed stream ([name] prefixes; an agent field in
NDJSON), and keeps the session alive when one agent crashes. Selecting a
single runtime (--agent, or a one-runtime project) keeps the direct path
where a crash still fails the command.
* fix(dev): bound readiness polling; unchain abort listeners on failed setup
A child that stays alive without ever binding its port previously blocked
every later runtime (starts are sequential) until interrupted; readiness now
gives up after 120s and fails that start. Setup failures before the pump
exists (port resolution, environment) now remove their parent-abort listener
like every other exit path, so Inspector retries of a failing agent cannot
accumulate listeners.
* fix(dev): bind collector to 0.0.0.0 when any runtime is a container
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
* fix(dev): deliver events queued while the supervisor stream drains a batch
events() spliced the queue, yielded each event, then installed its wake
callback and blocked. A push landing during the yields ran while wake was
undefined, so its wake was a no-op, and the newly installed waiter never
noticed the queued event until the next push or shutdown. Re-check the queue
inside the wait so a queued event resolves immediately.
* refactor(dev): route single runtime through the supervisor and address review
Remove the runSingleRuntime special case so every runtime, one or many,
runs through DevSupervisor. Start all agents concurrently and stream their
output live rather than awaiting each readiness first. End the event loop
when no agent is starting or running, and exit non-zero when none started.
Make start() async so an unknown name rejects rather than throwing
synchronously. Time readiness off the agent's last output (idle timeout)
so a slow build that keeps logging is not killed. Move waitForPort into
src/io/port.ts to keep socket IO in the IO layer. Rename the environment
accessor to getDevEnvVarsForRuntime and SupervisedEvent.agent to agentName.
* refactor(dev): always attribute agent output to its runtime
Split renderEvent into renderAgentEvent (agent name required) and
renderStatus for command-level trace-collector lines, so agent output
is always tagged with [name] and the agent-optional branches are gone.
Addresses review feedback on #2041.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xlPR size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@tejaskash@codecov-commenter@Hweinstock@notgitika@aidandaly24
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); feat(dev): collect local OTEL traces in project dev by tejaskash · Pull Request #1980 · aws/agentcore-cli · GitHub
Skip to content

feat(dev): collect local OTEL traces in project dev - #1980

Merged
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector
Aug 21, 2026
Merged

feat(dev): collect local OTEL traces in project dev#1980
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector

Conversation

@tejaskash

@tejaskashtejaskash commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Review after #2043

Until #2043 merges, the diff shows both PRs' commits (GitHub's stack tooling wouldn't retarget the base); the work to review here is the single collector commit on top. It collapses to just that once #2043 merges.

What this does

agentcore project dev gives a local agent tracing with no extra config: an in-process OTLP/HTTP receiver starts on a loopback port, the agent's OpenTelemetry SDK is pointed at it via env vars, and traces persist through #2043's TraceStore to agentcore/.cli/traces/. Nothing leaves the machine; --no-traces or instrumentation.enableOtel: false opts out.

  • io/httpServer.ts — loopback-only node:http primitive (body caps, abort-signal close).
  • otel/collector.tsPOST /v1/traces + /v1/logs, protobuf or JSON. Pinned @opentelemetry/otlp-transformer@0.213.0, the last version shipping request decoders — do not bump.
  • Handler wiring — collector lifecycle on the command's abort signal; signal-specific OTEL env vars so stray shell values can't redirect traces; host.docker.internal rewrite for containers; sitecustomize-on-PYTHONPATH so uvicorn --reload workers stay instrumented (a wrapper would only instrument the reloader parent).

Verification

Per-layer tests (receiver over real ephemeral-port HTTP with protobuf + JSON fixtures; handler with fake runners/collector) plus end to end: built bundle → real project createproject dev → real Bedrock invocation → traces on disk. Protobuf decode verified under bun, the Node bundle, and a compiled binary.

@github-actionsgithub-actionsBot added the size/xl PR size: XL label Aug 12, 2026
@github-actionsgithub-actionsBot added agentcore-harness-reviewing AgentCore Harness review in progress size/xl PR size: XL and removed size/xl PR size: XL labels Aug 12, 2026
@github-actionsgithub-actionsBot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 12, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from b4d0349 to e2e696cCompareAugust 14, 2026 20:38
@github-actionsgithub-actionsBot added size/xl PR size: XL and removed size/xl PR size: XL labels Aug 14, 2026
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 56c20f7 to a59ebf1CompareAugust 17, 2026 17:30
@codecov-commenter

codecov-commenter commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.28315% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.17%. Comparing base (a41b4de) to head (10be0a6).

Files with missing linesPatch %Lines
src/io/httpServer.ts97.05%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## refactor #1980 +/- ##
============================================
+ Coverage 97.15% 97.17% +0.02% 
============================================
Files 387 389 +2 Lines 23124 23386 +262 ============================================
+ Hits 22465 22726 +261 - Misses 659 660 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from cad8772 to 7e315b9CompareAugust 17, 2026 17:53
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from be5c3d5 to 4b80241CompareAugust 21, 2026 15:44
Base automatically changed from feat/dev-otel-storage to refactorAugust 21, 2026 17:51
tejaskash added a commit that referenced this pull request Aug 21, 2026
* feat(dev): OTLP trace storage for local dev
Pure OTLP wire handling (per-trace batch partitioning, id normalization,
frontend shaping) and append-only per-trace JSONL storage. A batch routinely
carries spans from several traces, so persistence partitions by trace id —
writing whole batches under the first id corrupts trace identity. Consumed by
the OTLP collector in #1980, which stacks on this.
* fix(dev): match trace filters against every participating service
A distributed trace spans several local agents whose exports append to the
same trace file; filtering by any participant must find it, not only the
first service seen.
* fix(dev): address review on OTLP trace storage
- narrow OtlpAttributes to the key/value wire form (the flat-record variant
had no producer); drop the dead passthrough branches
- flatten array attributes through extractAnyValue so ints stay numeric and
nested kvlists survive (was stringifying and dropping them)
- count rendered (post-filter) spans for the list summary instead of raw
records, so the count matches the waterfall the inspector shows
- surface non-ENOENT fs errors from reads instead of masking them as empty
- add newest-N limit to list() for the inspector's per-invocation poll
* fix(dev): flatten kvlist-valued OTLP attributes instead of dropping them
flattenAttributes hand-rolled a value branch per AnyValue kind and had no
kvlistValue case, so an attribute whose value is a kvlist (or anything the
chain didn't enumerate) silently vanished. Route every attribute value
through extractAnyValue, which already unwraps all variants including
kvlist — smaller and complete.
Addresses Gitika's review on transforms.ts (reuse extractAnyValue; kvlist
must not disappear).
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from 4b80241 to 733c274CompareAugust 21, 2026 17:51
@tejaskash

Copy link
Copy Markdown
ContributorAuthor

#2039 was superseded by #2043, which just merged — this is unblocked and rebased on it.

@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch 2 times, most recently from fce12fd to 27fbae3CompareAugust 21, 2026 17:59
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/io/httpServer.ts
Comment threadsrc/router/flags.tsx Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/otel/collector.ts
Comment threadsrc/core/dev/codezip.ts
Comment threadsrc/core/dev/codezip.ts Outdated
tejaskash added a commit that referenced this pull request Aug 21, 2026
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
Comment threadsrc/core/dev/container.ts
Comment threadsrc/io/httpServer.ts
Comment threadsrc/core/dev/container.ts
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/handlers/project/dev/index.ts Outdated
Comment threadsrc/handlers/project/dev/index.ts
Comment threadsrc/core/dev/otel/collector.ts Outdated
Comment threadsrc/core/dev/codezip.ts Outdated
An in-process OTLP/HTTP receiver (protobuf via the pinned otlp-transformer
decoders, or JSON) persists agent traces through the storage layer. project
dev starts it unless --no-traces or the runtime disables instrumentation,
points every spawned agent at it (signal-specific OTEL env), rewrites the
endpoint to host.docker.internal for containers — with an explicit
host-gateway mapping so Linux Docker Engine resolves it — and keeps uvicorn
--reload workers instrumented via sitecustomize on PYTHONPATH. Oversized
collector requests get a 413 before the connection closes so exporters do
not retry them as transient failures.
Rebuilt from explicit paths: the previous tree-snapshot commit accidentally
reverted unrelated merged work (cdk target guard, config-bundle TUI, error
classification).
A batch that can't be persisted (disk full, permissions) was being
turned into a 500, which the OTEL SDK exporter retries forever while the
user sees nothing. Ack the export (200) so retries stop, and surface the
failure once via an onError sink threaded from the collector to the dev
handler, which owns the IO to warn the user.
Addresses Gitika's review on store.ts:45 (catch in one place; don't let
persistence faults read as a silent, retried loss).
Follow-ups from #2043 review (Gitika, Harrison):
- Add a TraceStore.list spanCount test with transport-noise spans
(1 agent + 4 http-send -> "1"), guarding the post-filter count.
- Rename hexFromB64OrString -> hexFromBase64OrHex; both reviewers misread B64.
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
— the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
(ack + hand to onError) rather than the caller's report-once behavior; drop
the volatile "matches the reference CLI" aside.
…cycle
Address reviewer findings on the collector and dev wiring:
- Validate top-level OTLP shape and return 400 instead of mislabeling a bad
payload as a persistence error.
- Guard the shared HTTP server against a client that disconnects mid-response
so it can no longer crash project dev; add an optional bind host.
- Bind the collector to 0.0.0.0 for container runtimes so a container can
reach it over the host bridge on Linux.
- Run the container template under opentelemetry-instrument so it emits traces.
- Keep the collector alive through the child's shutdown grace so final spans
are not lost.
- Force the OTEL settings that would otherwise let shell or .env.local values
disable or break local collection.
- Make the uv sitecustomize discovery abortable and read its path from a
marker rather than the last merged output line.
@tejaskash
tejaskashforce-pushed the feat/dev-otel-collector branch from adf371b to 10be0a6CompareAugust 21, 2026 18:55

@aidandaly24aidandaly24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the fixes. This looks good to me!

@tejaskash
tejaskash merged commit f38b18d into refactorAug 21, 2026
11 checks passed
@tejaskash
tejaskash deleted the feat/dev-otel-collector branch August 21, 2026 21:32
tejaskash added a commit that referenced this pull request Aug 21, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 24, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
tejaskash added a commit that referenced this pull request Aug 25, 2026
* feat(dev): supervise every runtime in project dev
project dev without --agent now runs all of the project's runtimes at once:
a DevSupervisor owns per-agent lifecycle (sequential port resolution — a
concurrent race would put two agents on one port), merges every runner's
output into one agent-attributed stream ([name] prefixes; an agent field in
NDJSON), and keeps the session alive when one agent crashes. Selecting a
single runtime (--agent, or a one-runtime project) keeps the direct path
where a crash still fails the command.
* fix(dev): bound readiness polling; unchain abort listeners on failed setup
A child that stays alive without ever binding its port previously blocked
every later runtime (starts are sequential) until interrupted; readiness now
gives up after 120s and fails that start. Setup failures before the pump
exists (port resolution, environment) now remove their parent-abort listener
like every other exit path, so Inspector retries of a failing agent cannot
accumulate listeners.
* fix(dev): bind collector to 0.0.0.0 when any runtime is a container
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
* fix(dev): deliver events queued while the supervisor stream drains a batch
events() spliced the queue, yielded each event, then installed its wake
callback and blocked. A push landing during the yields ran while wake was
undefined, so its wake was a no-op, and the newly installed waiter never
noticed the queued event until the next push or shutdown. Re-check the queue
inside the wait so a queued event resolves immediately.
* refactor(dev): route single runtime through the supervisor and address review
Remove the runSingleRuntime special case so every runtime, one or many,
runs through DevSupervisor. Start all agents concurrently and stream their
output live rather than awaiting each readiness first. End the event loop
when no agent is starting or running, and exit non-zero when none started.
Make start() async so an unknown name rejects rather than throwing
synchronously. Time readiness off the agent's last output (idle timeout)
so a slow build that keeps logging is not killed. Move waitForPort into
src/io/port.ts to keep socket IO in the IO layer. Rename the environment
accessor to getDevEnvVarsForRuntime and SupervisedEvent.agent to agentName.
* refactor(dev): always attribute agent output to its runtime
Split renderEvent into renderAgentEvent (agent name required) and
renderStatus for command-level trace-collector lines, so agent output
is always tagged with [name] and the agent-optional branches are gone.
Addresses review feedback on #2041.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xlPR size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@tejaskash@codecov-commenter@Hweinstock@notgitika@aidandaly24