Uh oh!
There was an error while loading. Please reload this page.
feat: add context efficiency metrics (#505) - #515
Conversation
Warning Review limit reached
Next review available in:37 minutes Limit details: You’ve used all 1 included review currently available under your plan. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (21)
📝 WalkthroughWalkthroughThe SDK now computes context-efficiency metrics for turns and sessions. Summary commands and Node bindings expose these metrics. Hotspots support configurable context-to-output ratio findings with minimum context-token thresholds. ChangesContext efficiency
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant LedgerHandle
participant ContextEfficiency
participant Summary
participant CLI_or_Node
LedgerHandle->>ContextEfficiency: compute turn and session metrics
ContextEfficiency->>Summary: populate contextEfficiency
Summary->>CLI_or_Node: serialize or render metrics
LedgerHandle->>ContextEfficiency: evaluate hotspot thresholds
ContextEfficiency-->>LedgerHandle: return ratio findings
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
packages/sdk-node/CHANGELOG.md (1)
5-5: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueSplit the bullet into one entry per API change.
This bullet covers two separate user-visible changes: the
summary()result shape and the newhotspots()options. The content is accurate againstpackages/sdk-node/src/index.d.tsLines 305-320.📝 Proposed changelog edit
-- `summary()` now returns context tokens per normalized generated-output token (including reasoning) plus p50/p95/max context sizes for the ten highest-ratio sessions, where context is input + cache-read + cache-creation tokens; `hotspots()` accepts ratio and minimum-context options for the new cost-independent finding (defaults: 382:1 inclusive and 1M context tokens).+- `summary()` now returns context tokens per normalized generated-output token (including reasoning) plus p50/p95/max context sizes for the ten highest-ratio sessions. Context is input + cache-read + cache-creation tokens.+- `hotspots()` accepts `contextOutputRatioThreshold` and `contextOutputMinTokens` for the new cost-independent finding (defaults: 382:1 inclusive and 1M context tokens).Based on the guideline "Prefer one short bullet per user-visible change: name the command/API/schema touched and the practical effect".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/sdk-node/CHANGELOG.md` at line 5, Split the changelog bullet into two separate entries: one describing the `summary()` result changes, including context-token normalization and context-size statistics, and another describing the `hotspots()` ratio and minimum-context options with their defaults. Keep each entry concise and focused on the affected API and practical user-visible effect.Source: Coding guidelines
crates/relayburn-sdk/src/query_verbs/tests.rs (1)
1130-1137: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winReplace the vacuous negative assertion with a positive detail check.
context_output_ratio_findingincrates/relayburn-sdk/src/analyze/findings.rsbuildsdetailfrom a fixed format string that lists context tokens, output tokens, the threshold, and the minimum context tokens. The substring"169/6,348"belongs to the changelog calibration note, not to that format string. The assertion at Line 1134 therefore always passes and guards nothing. Assert the real contract instead.♻️ Proposed test assertion change
match result { HotspotsResult::Findings { findings, .. } => { assert_eq!(findings.len(), 1); assert_eq!(findings[0].session_id, "incident-382"); - assert!(!findings[0].detail.contains("169/6,348"));+ assert_eq!(findings[0].kind, "context-output-ratio");+ assert!(findings[0].detail.contains("1146000 context tokens"));+ assert!(findings[0].detail.contains("3000 generated output tokens")); } other => panic!("expected findings, got {other:?}"), }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/relayburn-sdk/src/query_verbs/tests.rs` around lines 1130 - 1137, Update the assertion in the HotspotsResult::Findings branch of the test to positively verify the detail format produced by context_output_ratio_finding, including the expected context tokens, output tokens, threshold, and minimum context tokens. Remove the vacuous "169/6,348" negative check and assert the actual contract represented by the fixed detail format string.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CHANGELOG.md`:
- Line 8: Update the changelog bullet for `hotspots --findings` by removing the
private calibration statistics and related corpus wording, including the counts
and percentages. Retain the configurable ratio and context-token floor, the
default ratio, the shipped selection impact if applicable, and the caveat that
this is an inspection signal rather than a length-normalized anomaly score.
In `@packages/sdk-node/src/index.d.ts`:
- Around line 315-318: Update the contextOutputMinTokens declaration in the SDK
options type to accept only bigint, matching the native
context_output_min_tokens: Option<BigInt> interface and ensuring values reach
bigint_to_u64 without number conversion failures.
---
Nitpick comments:
In `@crates/relayburn-sdk/src/query_verbs/tests.rs`:
- Around line 1130-1137: Update the assertion in the HotspotsResult::Findings
branch of the test to positively verify the detail format produced by
context_output_ratio_finding, including the expected context tokens, output
tokens, threshold, and minimum context tokens. Remove the vacuous "169/6,348"
negative check and assert the actual contract represented by the fixed detail
format string.
In `@packages/sdk-node/CHANGELOG.md`:
- Line 5: Split the changelog bullet into two separate entries: one describing
the `summary()` result changes, including context-token normalization and
context-size statistics, and another describing the `hotspots()` ratio and
minimum-context options with their defaults. Keep each entry concise and focused
on the affected API and practical user-visible effect.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 77842559-7eb6-43d4-9941-168fb16c03a4
📒 Files selected for processing (16)
CHANGELOG.mdcrates/relayburn-cli/src/commands/hotspots/mod.rscrates/relayburn-cli/src/commands/summary/human.rscrates/relayburn-cli/src/commands/summary/json.rscrates/relayburn-cli/src/commands/summary/mod.rscrates/relayburn-sdk-node/src/lib.rscrates/relayburn-sdk/src/analyze.rscrates/relayburn-sdk/src/analyze/cost.rscrates/relayburn-sdk/src/analyze/findings.rscrates/relayburn-sdk/src/query_verbs/context_efficiency.rscrates/relayburn-sdk/src/query_verbs/hotspots.rscrates/relayburn-sdk/src/query_verbs/mod.rscrates/relayburn-sdk/src/query_verbs/summary/mod.rscrates/relayburn-sdk/src/query_verbs/tests.rspackages/sdk-node/CHANGELOG.mdpackages/sdk-node/src/index.d.ts
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
willwashburn
commented
Aug 3, 2026
Addressed both review-summary nitpicks in 9bab8ae: the Node changelog now has one bullet per API change, and the incident fixture now positively asserts finding kind plus context/output/threshold/floor detail instead of a vacuous negative calibration-string check. Full workspace tests and strict clippy pass. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit:cca7987a14
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
All reported issues were addressed
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Resolves the merged-main overlap: unioned findings exports and hotspots tests, pricing_status on the ratio finding, context-output options in the MCP hotspots wrapper, and golden snapshots regenerated now that the golden suite is enforced in CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Summary
Validation
Fixes#505
🤖 Generated with Claude Code.