Improve LangSmith trace coverage by stranske · Pull Request #2127 · stranske/Workflows

stranske · 2026-05-15T14:12:32Z

Summary

add shared LangSmith trace helpers and wire trace metadata through issue formatter, context extraction, task decomposition, validation, topic splitting, and structured-output repair
record format-step trace IDs/URLs in auto-pilot metrics, sanitize env exports, and gate LangSmith tracing when the secret is unavailable
improve LangSmith metrics diagnostics and verifier trace coverage accounting while preserving URL-only trace coverage
sync the consumer template and manifest with the new LangChain tracing files

Validation

python -m py_compile scripts/aggregate_agent_metrics.py templates/consumer-repo/scripts/aggregate_agent_metrics.py scripts/langchain/trace_utils.py scripts/langchain/issue_formatter.py scripts/langchain/context_extractor.py scripts/langchain/task_decomposer.py scripts/langchain/task_validator.py scripts/langchain/structured_output.py scripts/langchain/topic_splitter.py scripts/aggregate_metrics.py
python scripts/validate_workflow_yaml.py .github/workflows/maint-80-langsmith-metrics-dashboard.yml .github/workflows/agents-auto-pilot.yml templates/consumer-repo/.github/workflows/agents-auto-pilot.yml
python scripts/validate_template_completeness.py
python scripts/validate_template_sync.py
uv run pytest tests/scripts/test_trace_utils.py tests/scripts/test_issue_formatter.py tests/scripts/test_context_extractor.py tests/scripts/test_task_validator.py tests/test_structured_output.py tests/scripts/test_aggregate_metrics.py tests/scripts/test_aggregate_agent_metrics.py (132 passed)

Notes

Merged Workflows PRs fix: count LangSmith trace URL coverage #2124 and fix: sanitize auto-pilot LangSmith env exports #2125 first; this branch is rebased on top of them.
Did not merge draft PR fix: gate LangSmith tracing metrics #2126 because its metrics patch would have dropped URL-only trace coverage from fix: count LangSmith trace URL coverage #2124; this branch carries the useful gating behavior with URL-aware metrics.

github-actions · 2026-05-15T14:12:51Z

❌ Sync Manifest Validation Failed

This PR modifies files that should be synced to consumer repos,
but the sync manifest is incomplete.

Required action: Update .github/sync-manifest.yml to include the
new/modified files.

Why this matters

Files not declared in the manifest won't be synced to consumer repos
(Template, Manager-Database, trip-planner, Travel-Plan-Permission),
causing features to silently not work in those repos.

How to fix

Open .github/sync-manifest.yml
Add your new files to the appropriate section (workflows, prompts, scripts, etc.)
Include a description explaining what the file does
If the file should NOT be synced, add it to excluded: with a reason

See the workflow logs for specific files that need to be added.

stranske-keepalive · 2026-05-15T14:12:55Z

Workflow source needed

PR #2127 needs either a linked GitHub issue or one valid non-issue Workflow Source before PR metadata automation can manage it safely.

Please do one of:

Add  or a normal Closes #123 / Related to #123 line.
Check one Workflow Source option in the PR body.
Add a hidden marker such as , , , , or .
Add a workflow source label such as workflow:source-direct-pr, workflow:source-local-request, workflow:source-review-followup, workflow:source-sync, or workflow:no-automation.

Once a valid source is present, this warning will not be reposted.

stranske-keepalive · 2026-05-15T14:15:36Z

Automated Status Summary

Head SHA: 12d8c98
Latest Runs: ⏳ pending — Gate
Required contexts: Gate / gate, Health 45 Agents Guard / guard
Required: core tests (3.12): ⏳ pending, core tests (3.13): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job	Result	Logs
(no jobs reported)	⏳ pending	—

Coverage Overview

Coverage history entries: 1

Coverage Trend

Metric	Value
Current	93.12%
Baseline	85.00%
Delta	+8.12%
Minimum	70.00%
Status	✅ Pass

Top Coverage Hotspots (lowest coverage)

File	Coverage	Missing
`src/cli_parser.py`	81.8%	4
`src/percentile_calculator.py`	95.0%	1
`src/aggregator.py`	95.0%	2
`src/__init__.py`	100.0%	0
`src/ndjson_parser.py`	100.0%	0

Updated automatically; will refresh on subsequent CI/Docker completions.

Keepalive checklist

Scope

No scope information available

Tasks

No tasks defined

Acceptance criteria

No acceptance criteria defined

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d89d3e0e97

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-05-15T14:15:47Z

+    except Exception as first_exc:
+        try:
+            response = runnable.invoke(payload)
+        except Exception:


Retry without config only on signature incompatibility

invoke_with_trace retries every failed runnable.invoke(..., config=...) call without config, regardless of why the first call failed. That means transient/provider errors (timeouts, auth/rate-limit failures, upstream 5xx) trigger a second LLM request unnecessarily, increasing latency/cost and potentially worsening rate limiting for every failed operation. Limit the fallback retry to incompatibility errors (for example a TypeError about an unexpected config argument) so real invocation failures are not duplicated.

Useful? React with 👍 / 👎.

github-actions · 2026-05-15T14:16:00Z

Runner dispatch state for codex on PR #2127. Do not edit.

stranske · 2026-05-15T14:16:06Z

Runner dispatch state for autofix on PR #2127. Do not edit.

Copilot

Pull request overview

This PR threads LangSmith trace metadata through the LangChain script entry points (issue formatter, context extractor, task decomposer, task validator, topic splitter, structured-output repair) by introducing a shared trace_utils helper, records the format-step trace IDs/URLs in auto-pilot metrics with sanitized env exports, gates LANGCHAIN_TRACING_V2 on LANGSMITH_API_KEY availability, deduplicates verifier trace coverage by run key while keeping URL-only traces countable, and adds dashboard diagnostics for the LangSmith metrics workflow. Consumer template files and the sync manifest are updated in lockstep.

Changes:

New scripts/langchain/trace_utils.py (and template copy) plus wiring in formatter/extractor/decomposer/validator/topic-splitter/structured-output to capture and propagate trace IDs/URLs.
Auto-pilot workflow: gate tracing on the secret, sanitize trace env exports, surface format-step trace metadata to the metrics step.
Aggregate metrics: dedupe verifier LangSmith trace coverage by run key while still counting URL-only traces; LangSmith dashboard workflow gains diagnostics output and an artifact.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
scripts/langchain/trace_utils.py	New shared LangSmith trace helper used by all LangChain entry points.
scripts/langchain/issue_formatter.py	Wires `invoke_with_trace`, returns trace metadata in result.
scripts/langchain/context_extractor.py	Same trace wiring for context extraction path.
scripts/langchain/task_decomposer.py	Trace wiring + propagates trace dict in result.
scripts/langchain/task_validator.py	Adds trace fields to `ValidationResult` and refine fates.
scripts/langchain/topic_splitter.py	Logs LangSmith trace URL when available.
scripts/langchain/structured_output.py	Repair callback now uses `invoke_with_trace`; refactors loop and clamps repair attempts.
scripts/aggregate_agent_metrics.py	Dedupes verifier trace refs by run key, keeps URL-only counting.
.github/workflows/agents-auto-pilot.yml	Gates tracing on secret; exports format-step trace IDs/URLs into metrics.
.github/workflows/maint-80-langsmith-metrics-dashboard.yml	Adds diagnostics file/artifact and run/artifact counters.
.github/sync-manifest.yml	Registers new `trace_utils.py` and `task_validator.py` consumer scripts.
templates/consumer-repo/... (multiple)	Mirrors the above changes to the consumer template.
tests/scripts/test_trace_utils.py	New unit tests for `invoke_with_trace`.
tests/scripts/test_issue_formatter.py	Asserts trace ID is propagated through formatter.
tests/scripts/test_task_validator.py	Adds trace-shape assertion for empty refinement.
tests/scripts/test_aggregate_agent_metrics.py	Verifies run-key dedup of verifier trace coverage.

+    try:
+        response = runnable.invoke(payload, config=config)
+    except Exception as first_exc:
+        try:
+            response = runnable.invoke(payload)
+        except Exception:
+            raise first_exc


+    try:
+        response = runnable.invoke(payload, config=config)
+    except Exception as first_exc:
+        try:
+            response = runnable.invoke(payload)
+        except Exception:
+            raise first_exc


Improve LangSmith trace coverage

d89d3e0

Copilot AI review requested due to automatic review settings May 15, 2026 14:12

stranske temporarily deployed to agent-high-privilege May 15, 2026 14:12 — with GitHub Actions Inactive

Copilot started reviewing on behalf of stranske May 15, 2026 14:14 View session

chatgpt-codex-connector Bot reviewed May 15, 2026

View reviewed changes

stranske added the autofix:escalated label May 15, 2026

stranske temporarily deployed to agent-high-privilege May 15, 2026 14:16 — with GitHub Actions Inactive

stranske temporarily deployed to agent-standard May 15, 2026 14:16 — with GitHub Actions Inactive

github-actions Bot added the autofix Opt-in automated formatting & lint remediation label May 15, 2026

stranske temporarily deployed to agent-standard May 15, 2026 14:16 — with GitHub Actions Inactive

Copilot AI reviewed May 15, 2026

View reviewed changes

Fix LangSmith PR validation

9c408d3

stranske temporarily deployed to agent-high-privilege May 15, 2026 14:18 — with GitHub Actions Inactive

stranske merged commit 1ec7ee0 into main May 15, 2026
39 checks passed

stranske deleted the codex/langsmith-observability-fixes branch May 15, 2026 14:26

stranske mentioned this pull request May 15, 2026

fix: gate LangSmith tracing metrics #2126

Closed

Navigation Menu

Improve LangSmith trace coverage - #2127

Improve LangSmith trace coverage#2127
stranske merged 2 commits into
mainfrom
codex/langsmith-observability-fixes

stranske commented May 15, 2026

Uh oh!

github-actions Bot commented May 15, 2026

Uh oh!

stranske-keepalive Bot commented May 15, 2026

Uh oh!

stranske-keepalive Bot commented May 15, 2026 •

edited by agents-workflows-bot Bot

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot May 15, 2026

Uh oh!

github-actions Bot commented May 15, 2026

Uh oh!

stranske commented May 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

stranske commented May 15, 2026

Summary

Validation

Uh oh!

github-actions Bot commented May 15, 2026

❌ Sync Manifest Validation Failed

Why this matters

How to fix

Uh oh!

stranske-keepalive Bot commented May 15, 2026

Workflow source needed

Uh oh!

stranske-keepalive Bot commented May 15, 2026 • edited by agents-workflows-bot Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Automated Status Summary

Coverage Overview

Coverage Trend

Top Coverage Hotspots (lowest coverage)

Keepalive checklist

Scope

Tasks

Acceptance criteria

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 15, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 15, 2026

Uh oh!

stranske commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stranske-keepalive Bot commented May 15, 2026 •

edited by agents-workflows-bot Bot

Loading

stranske commented May 15, 2026 •

edited

Loading