feat(obs): add ttft_ms (time to first token) to UsageEvent - #276

Merged
nic-6443 merged 2 commits into
mainfrom
feat/ttft-usage-event
May 14, 2026
Merged

feat(obs): add ttft_ms (time to first token) to UsageEvent#276
nic-6443 merged 2 commits into
mainfrom
feat/ttft-usage-event

Conversation

@jarvis9443

@jarvis9443jarvis9443 commented May 14, 2026•
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Contributor

Add a ttft_ms field to UsageEvent to track first-token latency on the streaming path.

What changed:

  • UsageEvent gains ttft_ms: u32 — time from request entry to first upstream SSE chunk carrying delta.content or delta.tool_calls (milliseconds). Role-only chunks are skipped. Omitted from the wire when 0 (backwards-compatible with older cp-api).
  • Streaming path in build_sse_stream records elapsed time on the first content/tool-call chunk, stored in StreamCompletion and propagated through UsageExtras → emit_usage_event.
  • Non-streaming, error, and cache-hit paths keep ttft_ms = 0.
  • OTLP trace export includes aisix.ttft_ms span attribute when > 0.

Files:

  • crates/aisix-obs/src/usage.rs — new field + serialization tests
  • crates/aisix-proxy/src/chat.rs — TTFT capture + wiring
  • crates/aisix-obs/src/otlp_http_sink.rs — OTLP attribute + test

Follow-up: cp-api side (telemetryEvent struct, DB migration, dashboard) to persist and surface TTFT.

Summary by CodeRabbit

  • New Features

    • Added time-to-first-token (TTFT) telemetry for chat completions; streaming paths now capture and include TTFT in telemetry while non-streaming emits zero.

  • Tests

    • Extended tests to ensure TTFT is collected for streaming, serialized into telemetry when set, and omitted from payloads when zero.

Review Change Stack

@jarvis9443
feat(obs): add ttft_ms to UsageEvent for first-token latency
c919a4a
Add a ttft_ms field (time to first token, milliseconds) to the
UsageEvent struct. On the streaming path, record the elapsed time
from request entry to the first upstream SSE chunk. Non-streaming,
error, and cache-hit paths emit 0 (omitted from the wire via
skip_serializing_if).
Also wire ttft_ms into the OTLP trace export as aisix.ttft_ms
span attribute when > 0.
CopilotAI review requested due to automatic review settings May 14, 2026 08:28
@coderabbitai

coderabbitaiBot commented May 14, 2026•
edited
Loading

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4e2994f8-c99a-478e-bd75-c4be95e9ac5c

📥 Commits

Reviewing files that changed from the base of the PR and between c919a4a and bfe1ab9.

📒 Files selected for processing (1)
  • crates/aisix-proxy/src/chat.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/aisix-proxy/src/chat.rs
📝 Walkthrough

Walkthrough

This PR adds time-to-first-token (TTFT) telemetry for chat completions. The UsageEvent struct gains a ttft_ms field that streams populate by tracking the first upstream chunk's arrival time; non-streaming paths set it to zero. TTFT is exported as an OTLP span attribute aisix.ttft_ms when present.

Changes

TTFT Telemetry Integration

Layer / File(s)Summary
UsageEvent TTFT field and serialization
crates/aisix-obs/src/usage.rs		UsageEvent gains ttft_ms: u32 field with serde(default, skip_serializing_if = "is_zero_u32") to omit zero values from JSON; tests assert the field is absent when unset and present when assigned.
Chat handler TTFT collection and emission
crates/aisix-proxy/src/chat.rs		UsageExtras and StreamCompletion structures add ttft_ms fields; build_sse_stream accepts started: Instant and streaming loop computes TTFT from first upstream chunk arrival; non-streaming and error paths set ttft_ms to 0; emit_usage_event copies the value into final UsageEvent.
OTLP span attribute encoding
crates/aisix-obs/src/otlp_http_sink.rs		build_otlp_traces_payload conditionally encodes event.ttft_ms as OTLP intValue attribute aisix.ttft_ms only when ttft_ms > 0; tests verify omission when zero and correct JSON encoding when set.

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check nameStatusExplanation
Description Check✅ PassedCheck skipped - CodeRabbit’s high-level summary is enabled.
Title check✅ PassedThe title 'feat(obs): add ttft_ms (time to first token) to UsageEvent' accurately reflects the main change: adding a new ttft_ms field to the UsageEvent struct to track time-to-first-token latency.
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands and usage tips.

CopilotAI reviewed May 14, 2026
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds TTFT telemetry to usage/observability so streaming chat requests can report first-token latency through UsageEvent and OTLP traces.

Changes:

  • Added ttft_ms to UsageEvent, omitted when zero.
  • Wired streaming completion telemetry to carry TTFT from SSE processing.
  • Added OTLP trace attribute emission and serialization tests.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

FileDescription
crates/aisix-proxy/src/chat.rsCaptures TTFT during streaming and forwards it into usage telemetry.
crates/aisix-obs/src/usage.rsAdds the ttft_ms UsageEvent field and serialization coverage.
crates/aisix-obs/src/otlp_http_sink.rsEmits aisix.ttft_ms OTLP span attribute when set.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment threadcrates/aisix-proxy/src/chat.rs
@jarvis9443
fix: gate TTFT on content/tool-call chunks, skip role-only
bfe1ab9
OpenAI streams emit a role-only chunk before actual content.
Record ttft_ms only when the first chunk carrying delta.content
or delta.tool_calls arrives.
@nic-6443
nic-6443 merged commit 068aa1f into mainMay 14, 2026
7 checks passed
@nic-6443
nic-6443 deleted the feat/ttft-usage-event branch May 14, 2026 08:49
@moonmingmoonming mentioned this pull request May 15, 2026
Merged
3 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jarvis9443@nic-6443
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
feat(obs): add ttft_ms (time to first token) to UsageEvent - #276
Merged
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
May 14, 2026
Merged
feat(obs): add ttft_ms (time to first token) to UsageEvent#276
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
Conversation
@jarvis9443
@jarvis9443jarvis9443
commented
May 14, 2026•

edited by coderabbitai
Bot
Loading

Copy link


Copy Markdown

Contributor
Add a ttft_ms field to UsageEvent to track first-token latency on the streaming path.
What changed:
  • UsageEvent gains ttft_ms: u32 — time from request entry to first upstream SSE chunk carrying delta.content or delta.tool_calls (milliseconds). Role-only chunks are skipped. Omitted from the wire when 0 (backwards-compatible with older cp-api).
  • Streaming path in build_sse_stream records elapsed time on the first content/tool-call chunk, stored in StreamCompletion and propagated through UsageExtras → emit_usage_event.
  • Non-streaming, error, and cache-hit paths keep ttft_ms = 0.
  • OTLP trace export includes aisix.ttft_ms span attribute when > 0.
Files:
  • crates/aisix-obs/src/usage.rs — new field + serialization tests
  • crates/aisix-proxy/src/chat.rs — TTFT capture + wiring
  • crates/aisix-obs/src/otlp_http_sink.rs — OTLP attribute + test
Follow-up: cp-api side (telemetryEvent struct, DB migration, dashboard) to persist and surface TTFT.
Summary by CodeRabbit
  • New Features
    • Added time-to-first-token (TTFT) telemetry for chat completions; streaming paths now capture and include TTFT in telemetry while non-streaming emits zero.
  • Tests
    • Extended tests to ensure TTFT is collected for streaming, serialized into telemetry when set, and omitted from payloads when zero.
Review Change Stack
@jarvis9443
feat(obs): add ttft_ms to UsageEvent for first-token latency
c919a4a
Add a ttft_ms field (time to first token, milliseconds) to the
UsageEvent struct. On the streaming path, record the elapsed time
from request entry to the first upstream SSE chunk. Non-streaming,
error, and cache-hit paths emit 0 (omitted from the wire via
skip_serializing_if).
Also wire ttft_ms into the OTLP trace export as aisix.ttft_ms
span attribute when > 0.
CopilotAI
review requested due to automatic review settings
May 14, 2026 08:28
@coderabbitai
coderabbitaiBot
commented
May 14, 2026•

edited

Loading

Copy link


Copy Markdown

No actionable comments were generated in the recent review. 🎉
ℹ️ Recent review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4e2994f8-c99a-478e-bd75-c4be95e9ac5c
📥 Commits
Reviewing files that changed from the base of the PR and between c919a4a and bfe1ab9.
📒 Files selected for processing (1)
  • crates/aisix-proxy/src/chat.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/aisix-proxy/src/chat.rs
📝 Walkthrough
Walkthrough
This PR adds time-to-first-token (TTFT) telemetry for chat completions. The UsageEvent struct gains a ttft_ms field that streams populate by tracking the first upstream chunk's arrival time; non-streaming paths set it to zero. TTFT is exported as an OTLP span attribute aisix.ttft_ms when present.
Changes
TTFT Telemetry Integration
Layer / File(s)Summary
UsageEvent TTFT field and serialization
crates/aisix-obs/src/usage.rs		UsageEvent gains ttft_ms: u32 field with serde(default, skip_serializing_if = "is_zero_u32") to omit zero values from JSON; tests assert the field is absent when unset and present when assigned.
Chat handler TTFT collection and emission
crates/aisix-proxy/src/chat.rs		UsageExtras and StreamCompletion structures add ttft_ms fields; build_sse_stream accepts started: Instant and streaming loop computes TTFT from first upstream chunk arrival; non-streaming and error paths set ttft_ms to 0; emit_usage_event copies the value into final UsageEvent.
OTLP span attribute encoding
crates/aisix-obs/src/otlp_http_sink.rs		build_otlp_traces_payload conditionally encodes event.ttft_ms as OTLP intValue attribute aisix.ttft_ms only when ttft_ms > 0; tests verify omission when zero and correct JSON encoding when set.
🎯 2 (Simple) | ⏱️ ~10 minutes
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check nameStatusExplanation
Description Check✅ PassedCheck skipped - CodeRabbit’s high-level summary is enabled.
Title check✅ PassedThe title 'feat(obs): add ttft_ms (time to first token) to UsageEvent' accurately reflects the main change: adding a new ttft_ms field to the UsageEvent struct to track time-to-first-token latency.
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
✏️ Tip: You can configure your own custom pre-merge checks in the settings.
✨ Finishing Touches
📝 Generate docstrings
  •  Create stacked PR
  •  Commit on current branch
Comment @coderabbitai help to get the list of available commands and usage tips.
CopilotAI
reviewed
May 14, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
Adds TTFT telemetry to usage/observability so streaming chat requests can report first-token latency through UsageEvent and OTLP traces.
Changes:
  • Added ttft_ms to UsageEvent, omitted when zero.
  • Wired streaming completion telemetry to carry TTFT from SSE processing.
  • Added OTLP trace attribute emission and serialization tests.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
FileDescription
crates/aisix-proxy/src/chat.rsCaptures TTFT during streaming and forwards it into usage telemetry.
crates/aisix-obs/src/usage.rsAdds the ttft_ms UsageEvent field and serialization coverage.
crates/aisix-obs/src/otlp_http_sink.rsEmits aisix.ttft_ms OTLP span attribute when set.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment threadcrates/aisix-proxy/src/chat.rs
@jarvis9443
fix: gate TTFT on content/tool-call chunks, skip role-only
bfe1ab9
OpenAI streams emit a role-only chunk before actual content.
Record ttft_ms only when the first chunk carrying delta.content
or delta.tool_calls arrives.
@nic-6443
nic-6443
merged commit 068aa1f
into
mainMay 14, 2026

7 checks passed

@nic-6443
nic-6443
deleted the

feat/ttft-usage-event

branch
May 14, 2026 08:49
@moonmingmoonming
mentioned this pull request
May 15, 2026
 Merged

3 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jarvis9443@nic-6443

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat(obs): add ttft_ms (time to first token) to UsageEvent - #276
Merged
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
May 14, 2026
Merged
feat(obs): add ttft_ms (time to first token) to UsageEvent#276
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
Conversation
@jarvis9443
@jarvis9443jarvis9443
commented
May 14, 2026•

edited by coderabbitai
Bot
Loading

Copy link


Copy Markdown

Contributor
Add a ttft_ms field to UsageEvent to track first-token latency on the streaming path.
What changed:
  • UsageEvent gains ttft_ms: u32 — time from request entry to first upstream SSE chunk carrying delta.content or delta.tool_calls (milliseconds). Role-only chunks are skipped. Omitted from the wire when 0 (backwards-compatible with older cp-api).
  • Streaming path in build_sse_stream records elapsed time on the first content/tool-call chunk, stored in StreamCompletion and propagated through UsageExtras → emit_usage_event.
  • Non-streaming, error, and cache-hit paths keep ttft_ms = 0.
  • OTLP trace export includes aisix.ttft_ms span attribute when > 0.
Files:
  • crates/aisix-obs/src/usage.rs — new field + serialization tests
  • crates/aisix-proxy/src/chat.rs — TTFT capture + wiring
  • crates/aisix-obs/src/otlp_http_sink.rs — OTLP attribute + test
Follow-up: cp-api side (telemetryEvent struct, DB migration, dashboard) to persist and surface TTFT.
Summary by CodeRabbit
  • New Features
    • Added time-to-first-token (TTFT) telemetry for chat completions; streaming paths now capture and include TTFT in telemetry while non-streaming emits zero.
  • Tests
    • Extended tests to ensure TTFT is collected for streaming, serialized into telemetry when set, and omitted from payloads when zero.
Review Change Stack
@jarvis9443
feat(obs): add ttft_ms to UsageEvent for first-token latency
c919a4a
Add a ttft_ms field (time to first token, milliseconds) to the
UsageEvent struct. On the streaming path, record the elapsed time
from request entry to the first upstream SSE chunk. Non-streaming,
error, and cache-hit paths emit 0 (omitted from the wire via
skip_serializing_if).
Also wire ttft_ms into the OTLP trace export as aisix.ttft_ms
span attribute when > 0.
CopilotAI
review requested due to automatic review settings
May 14, 2026 08:28
@coderabbitai
coderabbitaiBot
commented
May 14, 2026•

edited

Loading

Copy link


Copy Markdown

No actionable comments were generated in the recent review. 🎉
ℹ️ Recent review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4e2994f8-c99a-478e-bd75-c4be95e9ac5c
📥 Commits
Reviewing files that changed from the base of the PR and between c919a4a and bfe1ab9.
📒 Files selected for processing (1)
  • crates/aisix-proxy/src/chat.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/aisix-proxy/src/chat.rs
📝 Walkthrough
Walkthrough
This PR adds time-to-first-token (TTFT) telemetry for chat completions. The UsageEvent struct gains a ttft_ms field that streams populate by tracking the first upstream chunk's arrival time; non-streaming paths set it to zero. TTFT is exported as an OTLP span attribute aisix.ttft_ms when present.
Changes
TTFT Telemetry Integration
Layer / File(s)Summary
UsageEvent TTFT field and serialization
crates/aisix-obs/src/usage.rs		UsageEvent gains ttft_ms: u32 field with serde(default, skip_serializing_if = "is_zero_u32") to omit zero values from JSON; tests assert the field is absent when unset and present when assigned.
Chat handler TTFT collection and emission
crates/aisix-proxy/src/chat.rs		UsageExtras and StreamCompletion structures add ttft_ms fields; build_sse_stream accepts started: Instant and streaming loop computes TTFT from first upstream chunk arrival; non-streaming and error paths set ttft_ms to 0; emit_usage_event copies the value into final UsageEvent.
OTLP span attribute encoding
crates/aisix-obs/src/otlp_http_sink.rs		build_otlp_traces_payload conditionally encodes event.ttft_ms as OTLP intValue attribute aisix.ttft_ms only when ttft_ms > 0; tests verify omission when zero and correct JSON encoding when set.
🎯 2 (Simple) | ⏱️ ~10 minutes
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check nameStatusExplanation
Description Check✅ PassedCheck skipped - CodeRabbit’s high-level summary is enabled.
Title check✅ PassedThe title 'feat(obs): add ttft_ms (time to first token) to UsageEvent' accurately reflects the main change: adding a new ttft_ms field to the UsageEvent struct to track time-to-first-token latency.
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
✏️ Tip: You can configure your own custom pre-merge checks in the settings.
✨ Finishing Touches
📝 Generate docstrings
  •  Create stacked PR
  •  Commit on current branch
Comment @coderabbitai help to get the list of available commands and usage tips.
CopilotAI
reviewed
May 14, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
Adds TTFT telemetry to usage/observability so streaming chat requests can report first-token latency through UsageEvent and OTLP traces.
Changes:
  • Added ttft_ms to UsageEvent, omitted when zero.
  • Wired streaming completion telemetry to carry TTFT from SSE processing.
  • Added OTLP trace attribute emission and serialization tests.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
FileDescription
crates/aisix-proxy/src/chat.rsCaptures TTFT during streaming and forwards it into usage telemetry.
crates/aisix-obs/src/usage.rsAdds the ttft_ms UsageEvent field and serialization coverage.
crates/aisix-obs/src/otlp_http_sink.rsEmits aisix.ttft_ms OTLP span attribute when set.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment threadcrates/aisix-proxy/src/chat.rs
@jarvis9443
fix: gate TTFT on content/tool-call chunks, skip role-only
bfe1ab9
OpenAI streams emit a role-only chunk before actual content.
Record ttft_ms only when the first chunk carrying delta.content
or delta.tool_calls arrives.
@nic-6443
nic-6443
merged commit 068aa1f
into
mainMay 14, 2026

7 checks passed

@nic-6443
nic-6443
deleted the

feat/ttft-usage-event

branch
May 14, 2026 08:49
@moonmingmoonming
mentioned this pull request
May 15, 2026
 Merged

3 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jarvis9443@nic-6443

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat(obs): add ttft_ms (time to first token) to UsageEvent - #276
Merged
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
May 14, 2026
Merged
feat(obs): add ttft_ms (time to first token) to UsageEvent#276
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
Conversation
@jarvis9443
@jarvis9443jarvis9443
commented
May 14, 2026•

edited by coderabbitai
Bot
Loading

Copy link


Copy Markdown

Contributor
Add a ttft_ms field to UsageEvent to track first-token latency on the streaming path.
What changed:
  • UsageEvent gains ttft_ms: u32 — time from request entry to first upstream SSE chunk carrying delta.content or delta.tool_calls (milliseconds). Role-only chunks are skipped. Omitted from the wire when 0 (backwards-compatible with older cp-api).
  • Streaming path in build_sse_stream records elapsed time on the first content/tool-call chunk, stored in StreamCompletion and propagated through UsageExtras → emit_usage_event.
  • Non-streaming, error, and cache-hit paths keep ttft_ms = 0.
  • OTLP trace export includes aisix.ttft_ms span attribute when > 0.
Files:
  • crates/aisix-obs/src/usage.rs — new field + serialization tests
  • crates/aisix-proxy/src/chat.rs — TTFT capture + wiring
  • crates/aisix-obs/src/otlp_http_sink.rs — OTLP attribute + test
Follow-up: cp-api side (telemetryEvent struct, DB migration, dashboard) to persist and surface TTFT.
Summary by CodeRabbit
  • New Features
    • Added time-to-first-token (TTFT) telemetry for chat completions; streaming paths now capture and include TTFT in telemetry while non-streaming emits zero.
  • Tests
    • Extended tests to ensure TTFT is collected for streaming, serialized into telemetry when set, and omitted from payloads when zero.
Review Change Stack
@jarvis9443
feat(obs): add ttft_ms to UsageEvent for first-token latency
c919a4a
Add a ttft_ms field (time to first token, milliseconds) to the
UsageEvent struct. On the streaming path, record the elapsed time
from request entry to the first upstream SSE chunk. Non-streaming,
error, and cache-hit paths emit 0 (omitted from the wire via
skip_serializing_if).
Also wire ttft_ms into the OTLP trace export as aisix.ttft_ms
span attribute when > 0.
CopilotAI
review requested due to automatic review settings
May 14, 2026 08:28
@coderabbitai
coderabbitaiBot
commented
May 14, 2026•

edited

Loading

Copy link


Copy Markdown

No actionable comments were generated in the recent review. 🎉
ℹ️ Recent review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4e2994f8-c99a-478e-bd75-c4be95e9ac5c
📥 Commits
Reviewing files that changed from the base of the PR and between c919a4a and bfe1ab9.
📒 Files selected for processing (1)
  • crates/aisix-proxy/src/chat.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/aisix-proxy/src/chat.rs
📝 Walkthrough
Walkthrough
This PR adds time-to-first-token (TTFT) telemetry for chat completions. The UsageEvent struct gains a ttft_ms field that streams populate by tracking the first upstream chunk's arrival time; non-streaming paths set it to zero. TTFT is exported as an OTLP span attribute aisix.ttft_ms when present.
Changes
TTFT Telemetry Integration
Layer / File(s)Summary
UsageEvent TTFT field and serialization
crates/aisix-obs/src/usage.rs		UsageEvent gains ttft_ms: u32 field with serde(default, skip_serializing_if = "is_zero_u32") to omit zero values from JSON; tests assert the field is absent when unset and present when assigned.
Chat handler TTFT collection and emission
crates/aisix-proxy/src/chat.rs		UsageExtras and StreamCompletion structures add ttft_ms fields; build_sse_stream accepts started: Instant and streaming loop computes TTFT from first upstream chunk arrival; non-streaming and error paths set ttft_ms to 0; emit_usage_event copies the value into final UsageEvent.
OTLP span attribute encoding
crates/aisix-obs/src/otlp_http_sink.rs		build_otlp_traces_payload conditionally encodes event.ttft_ms as OTLP intValue attribute aisix.ttft_ms only when ttft_ms > 0; tests verify omission when zero and correct JSON encoding when set.
🎯 2 (Simple) | ⏱️ ~10 minutes
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check nameStatusExplanation
Description Check✅ PassedCheck skipped - CodeRabbit’s high-level summary is enabled.
Title check✅ PassedThe title 'feat(obs): add ttft_ms (time to first token) to UsageEvent' accurately reflects the main change: adding a new ttft_ms field to the UsageEvent struct to track time-to-first-token latency.
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
✏️ Tip: You can configure your own custom pre-merge checks in the settings.
✨ Finishing Touches
📝 Generate docstrings
  •  Create stacked PR
  •  Commit on current branch
Comment @coderabbitai help to get the list of available commands and usage tips.
CopilotAI
reviewed
May 14, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
Adds TTFT telemetry to usage/observability so streaming chat requests can report first-token latency through UsageEvent and OTLP traces.
Changes:
  • Added ttft_ms to UsageEvent, omitted when zero.
  • Wired streaming completion telemetry to carry TTFT from SSE processing.
  • Added OTLP trace attribute emission and serialization tests.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
FileDescription
crates/aisix-proxy/src/chat.rsCaptures TTFT during streaming and forwards it into usage telemetry.
crates/aisix-obs/src/usage.rsAdds the ttft_ms UsageEvent field and serialization coverage.
crates/aisix-obs/src/otlp_http_sink.rsEmits aisix.ttft_ms OTLP span attribute when set.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment threadcrates/aisix-proxy/src/chat.rs
@jarvis9443
fix: gate TTFT on content/tool-call chunks, skip role-only
bfe1ab9
OpenAI streams emit a role-only chunk before actual content.
Record ttft_ms only when the first chunk carrying delta.content
or delta.tool_calls arrives.
@nic-6443
nic-6443
merged commit 068aa1f
into
mainMay 14, 2026

7 checks passed

@nic-6443
nic-6443
deleted the

feat/ttft-usage-event

branch
May 14, 2026 08:49
@moonmingmoonming
mentioned this pull request
May 15, 2026
 Merged

3 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jarvis9443@nic-6443

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
feat(obs): add ttft_ms (time to first token) to UsageEvent - #276
Merged
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
May 14, 2026
Merged
feat(obs): add ttft_ms (time to first token) to UsageEvent#276
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
Conversation
@jarvis9443
@jarvis9443jarvis9443
commented
May 14, 2026•

edited by coderabbitai
Bot
Loading

Copy link


Copy Markdown

Contributor
Add a ttft_ms field to UsageEvent to track first-token latency on the streaming path.
What changed:
  • UsageEvent gains ttft_ms: u32 — time from request entry to first upstream SSE chunk carrying delta.content or delta.tool_calls (milliseconds). Role-only chunks are skipped. Omitted from the wire when 0 (backwards-compatible with older cp-api).
  • Streaming path in build_sse_stream records elapsed time on the first content/tool-call chunk, stored in StreamCompletion and propagated through UsageExtras → emit_usage_event.
  • Non-streaming, error, and cache-hit paths keep ttft_ms = 0.
  • OTLP trace export includes aisix.ttft_ms span attribute when > 0.
Files:
  • crates/aisix-obs/src/usage.rs — new field + serialization tests
  • crates/aisix-proxy/src/chat.rs — TTFT capture + wiring
  • crates/aisix-obs/src/otlp_http_sink.rs — OTLP attribute + test
Follow-up: cp-api side (telemetryEvent struct, DB migration, dashboard) to persist and surface TTFT.
Summary by CodeRabbit
  • New Features
    • Added time-to-first-token (TTFT) telemetry for chat completions; streaming paths now capture and include TTFT in telemetry while non-streaming emits zero.
  • Tests
    • Extended tests to ensure TTFT is collected for streaming, serialized into telemetry when set, and omitted from payloads when zero.
Review Change Stack
@jarvis9443
feat(obs): add ttft_ms to UsageEvent for first-token latency
c919a4a
Add a ttft_ms field (time to first token, milliseconds) to the
UsageEvent struct. On the streaming path, record the elapsed time
from request entry to the first upstream SSE chunk. Non-streaming,
error, and cache-hit paths emit 0 (omitted from the wire via
skip_serializing_if).
Also wire ttft_ms into the OTLP trace export as aisix.ttft_ms
span attribute when > 0.
CopilotAI
review requested due to automatic review settings
May 14, 2026 08:28
@coderabbitai
coderabbitaiBot
commented
May 14, 2026•

edited

Loading

Copy link


Copy Markdown

No actionable comments were generated in the recent review. 🎉
ℹ️ Recent review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4e2994f8-c99a-478e-bd75-c4be95e9ac5c
📥 Commits
Reviewing files that changed from the base of the PR and between c919a4a and bfe1ab9.
📒 Files selected for processing (1)
  • crates/aisix-proxy/src/chat.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/aisix-proxy/src/chat.rs
📝 Walkthrough
Walkthrough
This PR adds time-to-first-token (TTFT) telemetry for chat completions. The UsageEvent struct gains a ttft_ms field that streams populate by tracking the first upstream chunk's arrival time; non-streaming paths set it to zero. TTFT is exported as an OTLP span attribute aisix.ttft_ms when present.
Changes
TTFT Telemetry Integration
Layer / File(s)Summary
UsageEvent TTFT field and serialization
crates/aisix-obs/src/usage.rs		UsageEvent gains ttft_ms: u32 field with serde(default, skip_serializing_if = "is_zero_u32") to omit zero values from JSON; tests assert the field is absent when unset and present when assigned.
Chat handler TTFT collection and emission
crates/aisix-proxy/src/chat.rs		UsageExtras and StreamCompletion structures add ttft_ms fields; build_sse_stream accepts started: Instant and streaming loop computes TTFT from first upstream chunk arrival; non-streaming and error paths set ttft_ms to 0; emit_usage_event copies the value into final UsageEvent.
OTLP span attribute encoding
crates/aisix-obs/src/otlp_http_sink.rs		build_otlp_traces_payload conditionally encodes event.ttft_ms as OTLP intValue attribute aisix.ttft_ms only when ttft_ms > 0; tests verify omission when zero and correct JSON encoding when set.
🎯 2 (Simple) | ⏱️ ~10 minutes
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check nameStatusExplanation
Description Check✅ PassedCheck skipped - CodeRabbit’s high-level summary is enabled.
Title check✅ PassedThe title 'feat(obs): add ttft_ms (time to first token) to UsageEvent' accurately reflects the main change: adding a new ttft_ms field to the UsageEvent struct to track time-to-first-token latency.
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
✏️ Tip: You can configure your own custom pre-merge checks in the settings.
✨ Finishing Touches
📝 Generate docstrings
  •  Create stacked PR
  •  Commit on current branch
Comment @coderabbitai help to get the list of available commands and usage tips.
CopilotAI
reviewed
May 14, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
Adds TTFT telemetry to usage/observability so streaming chat requests can report first-token latency through UsageEvent and OTLP traces.
Changes:
  • Added ttft_ms to UsageEvent, omitted when zero.
  • Wired streaming completion telemetry to carry TTFT from SSE processing.
  • Added OTLP trace attribute emission and serialization tests.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
FileDescription
crates/aisix-proxy/src/chat.rsCaptures TTFT during streaming and forwards it into usage telemetry.
crates/aisix-obs/src/usage.rsAdds the ttft_ms UsageEvent field and serialization coverage.
crates/aisix-obs/src/otlp_http_sink.rsEmits aisix.ttft_ms OTLP span attribute when set.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment threadcrates/aisix-proxy/src/chat.rs
@jarvis9443
fix: gate TTFT on content/tool-call chunks, skip role-only
bfe1ab9
OpenAI streams emit a role-only chunk before actual content.
Record ttft_ms only when the first chunk carrying delta.content
or delta.tool_calls arrives.
@nic-6443
nic-6443
merged commit 068aa1f
into
mainMay 14, 2026

7 checks passed

@nic-6443
nic-6443
deleted the

feat/ttft-usage-event

branch
May 14, 2026 08:49
@moonmingmoonming
mentioned this pull request
May 15, 2026
 Merged

3 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jarvis9443@nic-6443

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat(obs): add ttft_ms (time to first token) to UsageEvent - #276
Merged
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
May 14, 2026
Merged
feat(obs): add ttft_ms (time to first token) to UsageEvent#276
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
Conversation
@jarvis9443
@jarvis9443jarvis9443
commented
May 14, 2026•

edited by coderabbitai
Bot
Loading

Copy link


Copy Markdown

Contributor
Add a ttft_ms field to UsageEvent to track first-token latency on the streaming path.
What changed:
  • UsageEvent gains ttft_ms: u32 — time from request entry to first upstream SSE chunk carrying delta.content or delta.tool_calls (milliseconds). Role-only chunks are skipped. Omitted from the wire when 0 (backwards-compatible with older cp-api).
  • Streaming path in build_sse_stream records elapsed time on the first content/tool-call chunk, stored in StreamCompletion and propagated through UsageExtras → emit_usage_event.
  • Non-streaming, error, and cache-hit paths keep ttft_ms = 0.
  • OTLP trace export includes aisix.ttft_ms span attribute when > 0.
Files:
  • crates/aisix-obs/src/usage.rs — new field + serialization tests
  • crates/aisix-proxy/src/chat.rs — TTFT capture + wiring
  • crates/aisix-obs/src/otlp_http_sink.rs — OTLP attribute + test
Follow-up: cp-api side (telemetryEvent struct, DB migration, dashboard) to persist and surface TTFT.
Summary by CodeRabbit
  • New Features
    • Added time-to-first-token (TTFT) telemetry for chat completions; streaming paths now capture and include TTFT in telemetry while non-streaming emits zero.
  • Tests
    • Extended tests to ensure TTFT is collected for streaming, serialized into telemetry when set, and omitted from payloads when zero.
Review Change Stack
@jarvis9443
feat(obs): add ttft_ms to UsageEvent for first-token latency
c919a4a
Add a ttft_ms field (time to first token, milliseconds) to the
UsageEvent struct. On the streaming path, record the elapsed time
from request entry to the first upstream SSE chunk. Non-streaming,
error, and cache-hit paths emit 0 (omitted from the wire via
skip_serializing_if).
Also wire ttft_ms into the OTLP trace export as aisix.ttft_ms
span attribute when > 0.
CopilotAI
review requested due to automatic review settings
May 14, 2026 08:28
@coderabbitai
coderabbitaiBot
commented
May 14, 2026•

edited

Loading

Copy link


Copy Markdown

No actionable comments were generated in the recent review. 🎉
ℹ️ Recent review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4e2994f8-c99a-478e-bd75-c4be95e9ac5c
📥 Commits
Reviewing files that changed from the base of the PR and between c919a4a and bfe1ab9.
📒 Files selected for processing (1)
  • crates/aisix-proxy/src/chat.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/aisix-proxy/src/chat.rs
📝 Walkthrough
Walkthrough
This PR adds time-to-first-token (TTFT) telemetry for chat completions. The UsageEvent struct gains a ttft_ms field that streams populate by tracking the first upstream chunk's arrival time; non-streaming paths set it to zero. TTFT is exported as an OTLP span attribute aisix.ttft_ms when present.
Changes
TTFT Telemetry Integration
Layer / File(s)Summary
UsageEvent TTFT field and serialization
crates/aisix-obs/src/usage.rs		UsageEvent gains ttft_ms: u32 field with serde(default, skip_serializing_if = "is_zero_u32") to omit zero values from JSON; tests assert the field is absent when unset and present when assigned.
Chat handler TTFT collection and emission
crates/aisix-proxy/src/chat.rs		UsageExtras and StreamCompletion structures add ttft_ms fields; build_sse_stream accepts started: Instant and streaming loop computes TTFT from first upstream chunk arrival; non-streaming and error paths set ttft_ms to 0; emit_usage_event copies the value into final UsageEvent.
OTLP span attribute encoding
crates/aisix-obs/src/otlp_http_sink.rs		build_otlp_traces_payload conditionally encodes event.ttft_ms as OTLP intValue attribute aisix.ttft_ms only when ttft_ms > 0; tests verify omission when zero and correct JSON encoding when set.
🎯 2 (Simple) | ⏱️ ~10 minutes
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check nameStatusExplanation
Description Check✅ PassedCheck skipped - CodeRabbit’s high-level summary is enabled.
Title check✅ PassedThe title 'feat(obs): add ttft_ms (time to first token) to UsageEvent' accurately reflects the main change: adding a new ttft_ms field to the UsageEvent struct to track time-to-first-token latency.
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
✏️ Tip: You can configure your own custom pre-merge checks in the settings.
✨ Finishing Touches
📝 Generate docstrings
  •  Create stacked PR
  •  Commit on current branch
Comment @coderabbitai help to get the list of available commands and usage tips.
CopilotAI
reviewed
May 14, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
Adds TTFT telemetry to usage/observability so streaming chat requests can report first-token latency through UsageEvent and OTLP traces.
Changes:
  • Added ttft_ms to UsageEvent, omitted when zero.
  • Wired streaming completion telemetry to carry TTFT from SSE processing.
  • Added OTLP trace attribute emission and serialization tests.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
FileDescription
crates/aisix-proxy/src/chat.rsCaptures TTFT during streaming and forwards it into usage telemetry.
crates/aisix-obs/src/usage.rsAdds the ttft_ms UsageEvent field and serialization coverage.
crates/aisix-obs/src/otlp_http_sink.rsEmits aisix.ttft_ms OTLP span attribute when set.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment threadcrates/aisix-proxy/src/chat.rs
@jarvis9443
fix: gate TTFT on content/tool-call chunks, skip role-only
bfe1ab9
OpenAI streams emit a role-only chunk before actual content.
Record ttft_ms only when the first chunk carrying delta.content
or delta.tool_calls arrives.
@nic-6443
nic-6443
merged commit 068aa1f
into
mainMay 14, 2026

7 checks passed

@nic-6443
nic-6443
deleted the

feat/ttft-usage-event

branch
May 14, 2026 08:49
@moonmingmoonming
mentioned this pull request
May 15, 2026
 Merged

3 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jarvis9443@nic-6443

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat(obs): add ttft_ms (time to first token) to UsageEvent - #276
Merged
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
May 14, 2026
Merged
feat(obs): add ttft_ms (time to first token) to UsageEvent#276
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
Conversation
@jarvis9443
@jarvis9443jarvis9443
commented
May 14, 2026•

edited by coderabbitai
Bot
Loading

Copy link


Copy Markdown

Contributor
Add a ttft_ms field to UsageEvent to track first-token latency on the streaming path.
What changed:
  • UsageEvent gains ttft_ms: u32 — time from request entry to first upstream SSE chunk carrying delta.content or delta.tool_calls (milliseconds). Role-only chunks are skipped. Omitted from the wire when 0 (backwards-compatible with older cp-api).
  • Streaming path in build_sse_stream records elapsed time on the first content/tool-call chunk, stored in StreamCompletion and propagated through UsageExtras → emit_usage_event.
  • Non-streaming, error, and cache-hit paths keep ttft_ms = 0.
  • OTLP trace export includes aisix.ttft_ms span attribute when > 0.
Files:
  • crates/aisix-obs/src/usage.rs — new field + serialization tests
  • crates/aisix-proxy/src/chat.rs — TTFT capture + wiring
  • crates/aisix-obs/src/otlp_http_sink.rs — OTLP attribute + test
Follow-up: cp-api side (telemetryEvent struct, DB migration, dashboard) to persist and surface TTFT.
Summary by CodeRabbit
  • New Features
    • Added time-to-first-token (TTFT) telemetry for chat completions; streaming paths now capture and include TTFT in telemetry while non-streaming emits zero.
  • Tests
    • Extended tests to ensure TTFT is collected for streaming, serialized into telemetry when set, and omitted from payloads when zero.
Review Change Stack
@jarvis9443
feat(obs): add ttft_ms to UsageEvent for first-token latency
c919a4a
Add a ttft_ms field (time to first token, milliseconds) to the
UsageEvent struct. On the streaming path, record the elapsed time
from request entry to the first upstream SSE chunk. Non-streaming,
error, and cache-hit paths emit 0 (omitted from the wire via
skip_serializing_if).
Also wire ttft_ms into the OTLP trace export as aisix.ttft_ms
span attribute when > 0.
CopilotAI
review requested due to automatic review settings
May 14, 2026 08:28
@coderabbitai
coderabbitaiBot
commented
May 14, 2026•

edited

Loading

Copy link


Copy Markdown

No actionable comments were generated in the recent review. 🎉
ℹ️ Recent review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4e2994f8-c99a-478e-bd75-c4be95e9ac5c
📥 Commits
Reviewing files that changed from the base of the PR and between c919a4a and bfe1ab9.
📒 Files selected for processing (1)
  • crates/aisix-proxy/src/chat.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/aisix-proxy/src/chat.rs
📝 Walkthrough
Walkthrough
This PR adds time-to-first-token (TTFT) telemetry for chat completions. The UsageEvent struct gains a ttft_ms field that streams populate by tracking the first upstream chunk's arrival time; non-streaming paths set it to zero. TTFT is exported as an OTLP span attribute aisix.ttft_ms when present.
Changes
TTFT Telemetry Integration
Layer / File(s)Summary
UsageEvent TTFT field and serialization
crates/aisix-obs/src/usage.rs		UsageEvent gains ttft_ms: u32 field with serde(default, skip_serializing_if = "is_zero_u32") to omit zero values from JSON; tests assert the field is absent when unset and present when assigned.
Chat handler TTFT collection and emission
crates/aisix-proxy/src/chat.rs		UsageExtras and StreamCompletion structures add ttft_ms fields; build_sse_stream accepts started: Instant and streaming loop computes TTFT from first upstream chunk arrival; non-streaming and error paths set ttft_ms to 0; emit_usage_event copies the value into final UsageEvent.
OTLP span attribute encoding
crates/aisix-obs/src/otlp_http_sink.rs		build_otlp_traces_payload conditionally encodes event.ttft_ms as OTLP intValue attribute aisix.ttft_ms only when ttft_ms > 0; tests verify omission when zero and correct JSON encoding when set.
🎯 2 (Simple) | ⏱️ ~10 minutes
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check nameStatusExplanation
Description Check✅ PassedCheck skipped - CodeRabbit’s high-level summary is enabled.
Title check✅ PassedThe title 'feat(obs): add ttft_ms (time to first token) to UsageEvent' accurately reflects the main change: adding a new ttft_ms field to the UsageEvent struct to track time-to-first-token latency.
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
✏️ Tip: You can configure your own custom pre-merge checks in the settings.
✨ Finishing Touches
📝 Generate docstrings
  •  Create stacked PR
  •  Commit on current branch
Comment @coderabbitai help to get the list of available commands and usage tips.
CopilotAI
reviewed
May 14, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
Adds TTFT telemetry to usage/observability so streaming chat requests can report first-token latency through UsageEvent and OTLP traces.
Changes:
  • Added ttft_ms to UsageEvent, omitted when zero.
  • Wired streaming completion telemetry to carry TTFT from SSE processing.
  • Added OTLP trace attribute emission and serialization tests.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
FileDescription
crates/aisix-proxy/src/chat.rsCaptures TTFT during streaming and forwards it into usage telemetry.
crates/aisix-obs/src/usage.rsAdds the ttft_ms UsageEvent field and serialization coverage.
crates/aisix-obs/src/otlp_http_sink.rsEmits aisix.ttft_ms OTLP span attribute when set.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment threadcrates/aisix-proxy/src/chat.rs
@jarvis9443
fix: gate TTFT on content/tool-call chunks, skip role-only
bfe1ab9
OpenAI streams emit a role-only chunk before actual content.
Record ttft_ms only when the first chunk carrying delta.content
or delta.tool_calls arrives.
@nic-6443
nic-6443
merged commit 068aa1f
into
mainMay 14, 2026

7 checks passed

@nic-6443
nic-6443
deleted the

feat/ttft-usage-event

branch
May 14, 2026 08:49
@moonmingmoonming
mentioned this pull request
May 15, 2026
 Merged

3 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jarvis9443@nic-6443

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
feat(obs): add ttft_ms (time to first token) to UsageEvent - #276
Merged
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
May 14, 2026
Merged
feat(obs): add ttft_ms (time to first token) to UsageEvent#276
nic-6443 merged 2 commits into
mainfrom 
feat/ttft-usage-event
Conversation
@jarvis9443
@jarvis9443jarvis9443
commented
May 14, 2026•

edited by coderabbitai
Bot
Loading

Copy link


Copy Markdown

Contributor
Add a ttft_ms field to UsageEvent to track first-token latency on the streaming path.
What changed:
  • UsageEvent gains ttft_ms: u32 — time from request entry to first upstream SSE chunk carrying delta.content or delta.tool_calls (milliseconds). Role-only chunks are skipped. Omitted from the wire when 0 (backwards-compatible with older cp-api).
  • Streaming path in build_sse_stream records elapsed time on the first content/tool-call chunk, stored in StreamCompletion and propagated through UsageExtras → emit_usage_event.
  • Non-streaming, error, and cache-hit paths keep ttft_ms = 0.
  • OTLP trace export includes aisix.ttft_ms span attribute when > 0.
Files:
  • crates/aisix-obs/src/usage.rs — new field + serialization tests
  • crates/aisix-proxy/src/chat.rs — TTFT capture + wiring
  • crates/aisix-obs/src/otlp_http_sink.rs — OTLP attribute + test
Follow-up: cp-api side (telemetryEvent struct, DB migration, dashboard) to persist and surface TTFT.
Summary by CodeRabbit
  • New Features
    • Added time-to-first-token (TTFT) telemetry for chat completions; streaming paths now capture and include TTFT in telemetry while non-streaming emits zero.
  • Tests
    • Extended tests to ensure TTFT is collected for streaming, serialized into telemetry when set, and omitted from payloads when zero.
Review Change Stack
@jarvis9443
feat(obs): add ttft_ms to UsageEvent for first-token latency
c919a4a
Add a ttft_ms field (time to first token, milliseconds) to the
UsageEvent struct. On the streaming path, record the elapsed time
from request entry to the first upstream SSE chunk. Non-streaming,
error, and cache-hit paths emit 0 (omitted from the wire via
skip_serializing_if).
Also wire ttft_ms into the OTLP trace export as aisix.ttft_ms
span attribute when > 0.
CopilotAI
review requested due to automatic review settings
May 14, 2026 08:28
@coderabbitai
coderabbitaiBot
commented
May 14, 2026•

edited

Loading

Copy link


Copy Markdown

No actionable comments were generated in the recent review. 🎉
ℹ️ Recent review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4e2994f8-c99a-478e-bd75-c4be95e9ac5c
📥 Commits
Reviewing files that changed from the base of the PR and between c919a4a and bfe1ab9.
📒 Files selected for processing (1)
  • crates/aisix-proxy/src/chat.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/aisix-proxy/src/chat.rs
📝 Walkthrough
Walkthrough
This PR adds time-to-first-token (TTFT) telemetry for chat completions. The UsageEvent struct gains a ttft_ms field that streams populate by tracking the first upstream chunk's arrival time; non-streaming paths set it to zero. TTFT is exported as an OTLP span attribute aisix.ttft_ms when present.
Changes
TTFT Telemetry Integration
Layer / File(s)Summary
UsageEvent TTFT field and serialization
crates/aisix-obs/src/usage.rs		UsageEvent gains ttft_ms: u32 field with serde(default, skip_serializing_if = "is_zero_u32") to omit zero values from JSON; tests assert the field is absent when unset and present when assigned.
Chat handler TTFT collection and emission
crates/aisix-proxy/src/chat.rs		UsageExtras and StreamCompletion structures add ttft_ms fields; build_sse_stream accepts started: Instant and streaming loop computes TTFT from first upstream chunk arrival; non-streaming and error paths set ttft_ms to 0; emit_usage_event copies the value into final UsageEvent.
OTLP span attribute encoding
crates/aisix-obs/src/otlp_http_sink.rs		build_otlp_traces_payload conditionally encodes event.ttft_ms as OTLP intValue attribute aisix.ttft_ms only when ttft_ms > 0; tests verify omission when zero and correct JSON encoding when set.
🎯 2 (Simple) | ⏱️ ~10 minutes
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check nameStatusExplanation
Description Check✅ PassedCheck skipped - CodeRabbit’s high-level summary is enabled.
Title check✅ PassedThe title 'feat(obs): add ttft_ms (time to first token) to UsageEvent' accurately reflects the main change: adding a new ttft_ms field to the UsageEvent struct to track time-to-first-token latency.
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
✏️ Tip: You can configure your own custom pre-merge checks in the settings.
✨ Finishing Touches
📝 Generate docstrings
  •  Create stacked PR
  •  Commit on current branch
Comment @coderabbitai help to get the list of available commands and usage tips.
CopilotAI
reviewed
May 14, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
Adds TTFT telemetry to usage/observability so streaming chat requests can report first-token latency through UsageEvent and OTLP traces.
Changes:
  • Added ttft_ms to UsageEvent, omitted when zero.
  • Wired streaming completion telemetry to carry TTFT from SSE processing.
  • Added OTLP trace attribute emission and serialization tests.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
FileDescription
crates/aisix-proxy/src/chat.rsCaptures TTFT during streaming and forwards it into usage telemetry.
crates/aisix-obs/src/usage.rsAdds the ttft_ms UsageEvent field and serialization coverage.
crates/aisix-obs/src/otlp_http_sink.rsEmits aisix.ttft_ms OTLP span attribute when set.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment threadcrates/aisix-proxy/src/chat.rs
@jarvis9443
fix: gate TTFT on content/tool-call chunks, skip role-only
bfe1ab9
OpenAI streams emit a role-only chunk before actual content.
Record ttft_ms only when the first chunk carrying delta.content
or delta.tool_calls arrives.
@nic-6443
nic-6443
merged commit 068aa1f
into
mainMay 14, 2026

7 checks passed

@nic-6443
nic-6443
deleted the

feat/ttft-usage-event

branch
May 14, 2026 08:49
@moonmingmoonming
mentioned this pull request
May 15, 2026
 Merged

3 tasks
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jarvis9443@nic-6443