Uh oh!
There was an error while loading. Please reload this page.
add APIs for llm obs sdk - #8135
Merged
Merged
Conversation
gary-huangforce-pushed
the
gary/add-llm-obs-api
branch
from
December 30, 2024 16:41
84eb184 to
85d0946CompareBenchmarksStartupParameters
See matching parameters
SummaryFound 8 performance improvements and 24 performance regressions! Performance is the same for 34 metrics, 5 unstable metrics.
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.49.0-SNAPSHOT~27177c01ea, baseline=1.49.0-SNAPSHOT~e8a21567c5
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.015 s) : 0, 1014514
Total [baseline] (8.671 s) : 0, 8670825
Agent [candidate] (1.056 s) : 0, 1055599
Total [candidate] (8.674 s) : 0, 8673956
section iast
Agent [baseline] (1.138 s) : 0, 1137519
Total [baseline] (9.215 s) : 0, 9215467
Agent [candidate] (1.184 s) : 0, 1184007
Total [candidate] (9.257 s) : 0, 9256591
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.135 s) : 0, 1134881
Total [baseline] (9.156 s) : 0, 9155585
Agent [candidate] (1.185 s) : 0, 1184942
Total [candidate] (9.231 s) : 0, 9231127
section iast_TELEMETRY_OFF
Agent [baseline] (1.134 s) : 0, 1134352
Total [baseline] (9.173 s) : 0, 9173070
Agent [candidate] (1.189 s) : 0, 1188675
Total [candidate] (9.274 s) : 0, 9274014
gantt
title insecure-bank - break down per module: candidate=1.49.0-SNAPSHOT~27177c01ea, baseline=1.49.0-SNAPSHOT~e8a21567c5
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (675.856 ms) : 0, 675856
BytebuddyAgent [candidate] (720.574 ms) : 0, 720574
GlobalTracer [baseline] (242.217 ms) : 0, 242217
GlobalTracer [candidate] (239.391 ms) : 0, 239391
AppSec [baseline] (55.392 ms) : 0, 55392
AppSec [candidate] (55.144 ms) : 0, 55144
Debugger [baseline] (7.631 ms) : 0, 7631
Debugger [candidate] (5.082 ms) : 0, 5082
Remote Config [baseline] (729.7 µs) : 0, 730
Remote Config [candidate] (716.048 µs) : 0, 716
Telemetry [baseline] (9.321 ms) : 0, 9321
Telemetry [candidate] (11.395 ms) : 0, 11395
section iast
BytebuddyAgent [baseline] (790.798 ms) : 0, 790798
BytebuddyAgent [candidate] (838.471 ms) : 0, 838471
GlobalTracer [baseline] (229.73 ms) : 0, 229730
GlobalTracer [candidate] (229.935 ms) : 0, 229935
AppSec [baseline] (56.366 ms) : 0, 56366
AppSec [candidate] (55.173 ms) : 0, 55173
Debugger [baseline] (5.886 ms) : 0, 5886
Debugger [candidate] (4.225 ms) : 0, 4225
Remote Config [baseline] (581.171 µs) : 0, 581
Remote Config [candidate] (603.37 µs) : 0, 603
Telemetry [baseline] (7.883 ms) : 0, 7883
Telemetry [candidate] (8.652 ms) : 0, 8652
IAST [baseline] (22.81 ms) : 0, 22810
IAST [candidate] (23.525 ms) : 0, 23525
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (788.634 ms) : 0, 788634
BytebuddyAgent [candidate] (838.976 ms) : 0, 838976
GlobalTracer [baseline] (229.676 ms) : 0, 229676
GlobalTracer [candidate] (229.782 ms) : 0, 229782
AppSec [baseline] (56.058 ms) : 0, 56058
AppSec [candidate] (53.815 ms) : 0, 53815
Debugger [baseline] (5.812 ms) : 0, 5812
Debugger [candidate] (4.292 ms) : 0, 4292
Remote Config [baseline] (597.411 µs) : 0, 597
Remote Config [candidate] (618.091 µs) : 0, 618
Telemetry [baseline] (7.933 ms) : 0, 7933
Telemetry [candidate] (8.672 ms) : 0, 8672
IAST [baseline] (22.724 ms) : 0, 22724
IAST [candidate] (25.309 ms) : 0, 25309
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (787.202 ms) : 0, 787202
BytebuddyAgent [candidate] (843.053 ms) : 0, 843053
GlobalTracer [baseline] (230.629 ms) : 0, 230629
GlobalTracer [candidate] (230.577 ms) : 0, 230577
AppSec [baseline] (56.442 ms) : 0, 56442
AppSec [candidate] (55.727 ms) : 0, 55727
Debugger [baseline] (5.952 ms) : 0, 5952
Debugger [candidate] (4.143 ms) : 0, 4143
Remote Config [baseline] (603.709 µs) : 0, 604
Remote Config [candidate] (621.166 µs) : 0, 621
Telemetry [baseline] (7.732 ms) : 0, 7732
Telemetry [candidate] (8.665 ms) : 0, 8665
IAST [baseline] (22.433 ms) : 0, 22433
IAST [candidate] (22.366 ms) : 0, 22366
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.49.0-SNAPSHOT~27177c01ea, baseline=1.49.0-SNAPSHOT~e8a21567c5
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.005 s) : 0, 1005084
Total [baseline] (10.475 s) : 0, 10474719
Agent [candidate] (1.057 s) : 0, 1056875
Total [candidate] (10.515 s) : 0, 10514548
section appsec
Agent [baseline] (1.148 s) : 0, 1147726
Total [baseline] (10.665 s) : 0, 10664666
Agent [candidate] (1.203 s) : 0, 1203043
Total [candidate] (10.822 s) : 0, 10821973
section iast
Agent [baseline] (1.139 s) : 0, 1139153
Total [baseline] (10.903 s) : 0, 10903280
Agent [candidate] (1.189 s) : 0, 1189477
Total [candidate] (11.097 s) : 0, 11096664
section profiling
Agent [baseline] (1.254 s) : 0, 1253965
Total [baseline] (10.786 s) : 0, 10785955
Agent [candidate] (1.293 s) : 0, 1293316
Total [candidate] (11.016 s) : 0, 11016391
gantt
title petclinic - break down per module: candidate=1.49.0-SNAPSHOT~27177c01ea, baseline=1.49.0-SNAPSHOT~e8a21567c5
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (670.258 ms) : 0, 670258
BytebuddyAgent [candidate] (721.35 ms) : 0, 721350
GlobalTracer [baseline] (239.978 ms) : 0, 239978
GlobalTracer [candidate] (239.79 ms) : 0, 239790
AppSec [baseline] (54.657 ms) : 0, 54657
AppSec [candidate] (55.218 ms) : 0, 55218
Debugger [baseline] (6.111 ms) : 0, 6111
Debugger [candidate] (4.397 ms) : 0, 4397
Remote Config [baseline] (706.643 µs) : 0, 707
Remote Config [candidate] (700.012 µs) : 0, 700
Telemetry [baseline] (10.019 ms) : 0, 10019
Telemetry [candidate] (11.939 ms) : 0, 11939
section appsec
BytebuddyAgent [baseline] (688.033 ms) : 0, 688033
BytebuddyAgent [candidate] (743.511 ms) : 0, 743511
GlobalTracer [baseline] (236.281 ms) : 0, 236281
GlobalTracer [candidate] (236.811 ms) : 0, 236811
AppSec [baseline] (175.41 ms) : 0, 175410
AppSec [candidate] (175.127 ms) : 0, 175127
Debugger [baseline] (5.804 ms) : 0, 5804
Debugger [candidate] (4.215 ms) : 0, 4215
Remote Config [baseline] (623.952 µs) : 0, 624
Remote Config [candidate] (625.416 µs) : 0, 625
Telemetry [baseline] (7.356 ms) : 0, 7356
Telemetry [candidate] (8.187 ms) : 0, 8187
IAST [baseline] (21.682 ms) : 0, 21682
IAST [candidate] (22.004 ms) : 0, 22004
section iast
BytebuddyAgent [baseline] (792.13 ms) : 0, 792130
BytebuddyAgent [candidate] (841.089 ms) : 0, 841089
GlobalTracer [baseline] (230.116 ms) : 0, 230116
GlobalTracer [candidate] (231.896 ms) : 0, 231896
AppSec [baseline] (56.253 ms) : 0, 56253
AppSec [candidate] (56.295 ms) : 0, 56295
Debugger [baseline] (5.912 ms) : 0, 5912
Debugger [candidate] (4.274 ms) : 0, 4274
Remote Config [baseline] (587.776 µs) : 0, 588
Remote Config [candidate] (615.228 µs) : 0, 615
Telemetry [baseline] (7.902 ms) : 0, 7902
Telemetry [candidate] (8.743 ms) : 0, 8743
IAST [baseline] (22.724 ms) : 0, 22724
IAST [candidate] (23.111 ms) : 0, 23111
section profiling
BytebuddyAgent [baseline] (664.031 ms) : 0, 664031
BytebuddyAgent [candidate] (712.321 ms) : 0, 712321
GlobalTracer [baseline] (374.882 ms) : 0, 374882
GlobalTracer [candidate] (361.595 ms) : 0, 361595
AppSec [baseline] (54.127 ms) : 0, 54127
AppSec [candidate] (53.337 ms) : 0, 53337
Debugger [baseline] (6.075 ms) : 0, 6075
Debugger [candidate] (4.408 ms) : 0, 4408
Remote Config [baseline] (644.93 µs) : 0, 645
Remote Config [candidate] (661.296 µs) : 0, 661
Telemetry [baseline] (8.143 ms) : 0, 8143
Telemetry [candidate] (8.997 ms) : 0, 8997
ProfilingAgent [baseline] (95.737 ms) : 0, 95737
ProfilingAgent [candidate] (101.744 ms) : 0, 101744
Profiling [baseline] (95.761 ms) : 0, 95761
Profiling [candidate] (101.772 ms) : 0, 101772
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 18 unstable metrics. Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.49.0-SNAPSHOT~27177c01ea, baseline=1.49.0-SNAPSHOT~e8a21567c5
dateFormat X
axisFormat %s
section baseline
no_agent (381.538 µs) : 362, 401
. : milestone, 382,
iast (524.519 µs) : 502, 547
. : milestone, 525,
iast_FULL (736.455 µs) : 713, 760
. : milestone, 736,
iast_GLOBAL (571.371 µs) : 548, 594
. : milestone, 571,
iast_HARDCODED_SECRET_DISABLED (512.874 µs) : 490, 536
. : milestone, 513,
iast_INACTIVE (470.395 µs) : 448, 493
. : milestone, 470,
iast_TELEMETRY_OFF (516.565 µs) : 494, 539
. : milestone, 517,
tracing (455.345 µs) : 434, 477
. : milestone, 455,
section candidate
no_agent (376.101 µs) : 356, 396
. : milestone, 376,
iast (519.522 µs) : 498, 542
. : milestone, 520,
iast_FULL (729.133 µs) : 707, 751
. : milestone, 729,
iast_GLOBAL (564.721 µs) : 543, 586
. : milestone, 565,
iast_HARDCODED_SECRET_DISABLED (512.141 µs) : 491, 534
. : milestone, 512,
iast_INACTIVE (469.599 µs) : 448, 491
. : milestone, 470,
iast_TELEMETRY_OFF (502.811 µs) : 481, 525
. : milestone, 503,
tracing (462.07 µs) : 441, 483
. : milestone, 462,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.49.0-SNAPSHOT~27177c01ea, baseline=1.49.0-SNAPSHOT~e8a21567c5
dateFormat X
axisFormat %s
section baseline
no_agent (1.358 ms) : 1339, 1378
. : milestone, 1358,
appsec (1.73 ms) : 1706, 1754
. : milestone, 1730,
appsec_no_iast (1.735 ms) : 1711, 1759
. : milestone, 1735,
code_origins (1.645 ms) : 1618, 1672
. : milestone, 1645,
iast (1.512 ms) : 1487, 1537
. : milestone, 1512,
profiling (1.542 ms) : 1517, 1566
. : milestone, 1542,
tracing (1.497 ms) : 1472, 1522
. : milestone, 1497,
section candidate
no_agent (1.356 ms) : 1336, 1376
. : milestone, 1356,
appsec (1.743 ms) : 1720, 1767
. : milestone, 1743,
appsec_no_iast (1.738 ms) : 1715, 1761
. : milestone, 1738,
code_origins (1.696 ms) : 1668, 1723
. : milestone, 1696,
iast (1.534 ms) : 1510, 1558
. : milestone, 1534,
profiling (1.589 ms) : 1565, 1614
. : milestone, 1589,
tracing (1.522 ms) : 1498, 1547
. : milestone, 1522,
Dacapo |
gary-huangforce-pushed
the
gary/add-llm-obs-api
branch
from
January 2, 2025 19:08
85d0946 to
dd637fdComparemcculls
reviewed
Jan 2, 2025
| private static LLMObsSpanFactory SPAN_FACTORY = NoOpLLMObsSpanFactory.INSTANCE; | ||
| /** | ||
| * This a hook for injecting SpanFactory implementation. It should only be used internally by |
Contributor
There was a problem hiding this comment.
FYI, internal APIs should be added to the internal-api module.
You can keep the same package structure and they will still be visible to the tracer internals.
mccullsforce-pushed
the
gary/add-llmobs-configs
branch
from
January 7, 2025 09:35
14f71b3 to
5d90450Comparegary-huangforce-pushed
the
gary/add-llm-obs-api
branch
from
January 14, 2025 13:24
dd637fd to
85debfcCompareContributor
Hi! 👋 Looks like you updated a Git Submodule.
|
gary-huangforce-pushed
the
gary/add-llm-obs-api
branch
from
January 14, 2025 13:27
85debfc to
98060a9CompareContributor
Hi! 👋 Looks like you updated a Git Submodule.
|
gary-huangforce-pushed
the
gary/add-llm-obs-api
branch
from
January 14, 2025 13:28
98060a9 to
62bc9a0Comparegary-huangforce-pushed
the
gary/add-llm-obs-api
branch
from
January 30, 2025 21:43
2b23b61 to
08b677eComparegary-huangforce-pushed
the
gary/add-llm-obs-api
branch
from
February 12, 2025 06:48
08b677e to
5b12d55Comparegary-huangforce-pushed
the
gary/add-llm-obs-api
branch
from
February 20, 2025 21:15
5b12d55 to
77ddec8Comparegary-huangforce-pushed
the
gary/add-llm-obs-api
branch
6 times, most recently
from
March 7, 2025 13:34
c1f63dc to
6e250a2Comparegary-huangforce-pushed
the
gary/add-llm-obs-api
branch
from
March 26, 2025 01:14
6e250a2 to
d16b87dComparegary-huangforce-pushed
the
gary/add-llm-obs-api
branch
from
April 10, 2025 19:09
d16b87d to
b5c7a0aComparegary-huang
marked this pull request as ready for review
April 23, 2025 12:04
Contributor
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
nayeem-kamal
left a comment
Contributor
There was a problem hiding this comment.
Overall LGTM. I would just address the comment left by Stuart earlier in the PR. Is there a reason this api isn't in internal-api?
PerfectSlayer
left a comment
Collaborator
There was a problem hiding this comment.
Left few comments about API design.
Can you also add your GitHub team as the owner of this new llmobs package from internal API?
| @Nullable String mlApp, | ||
| @Nullable String sessionID) { | ||
| return SPAN_FACTORY.startLLMSpan(spanName, modelName, modelProvider, mlApp, sessionID); |
Collaborator
There was a problem hiding this comment.
Suggested change
| returnSPAN_FACTORY.startLLMSpan(spanName, modelName, modelProvider, mlApp, sessionID); | |
| returnSPAN_FACTORY.startLLMSpan(spanName, modelName, modelProvider, mlApp, sessionId); |
We usually follow the Java naming convention across the code base using xxxId rather than xxxID (like span, trace, etc...).
Comment on lines
+8
to
+9
| public class LLMObs { | ||
| protected LLMObs() {} |
Collaborator
There was a problem hiding this comment.
Should it a public final class with private constructor if it’s only an helper class?
ContributorAuthor
There was a problem hiding this comment.
i extend this class here to allow setting an implemented, non-noop span factory (and in another PR, another factory)
i would say that this is not exactly only a helper class, this is the entry point of the SDK manual API and the calls are registered with the llmobs agent (as seen in #8390)
ContributorAuthor
@nayeem-kamal what Stuart pointed out was a previous function that allows the setting of the span factory - which is meant to be kept internal and out of user's control, my previous implementation had that function in the api package, but that has been moved since https://github.com/DataDog/dd-trace-java/pull/8390/files#diff-ee71e1544f39cc6c551488ad74b877a819b780b9d07c2983be13b80161958aa9R1-R10 these APIs are meant to be for users' to start llm obs spans, so no, they are not meant to be in internal-api |
nayeem-kamal
approved these changes
May 5, 2025
gary-huang added a commit
that referenced
this pull request
May 14, 2025
* add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners
gary-huang added a commit
that referenced
this pull request
Jun 4, 2025
* add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * spotless * add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * rename ID to Id according to java naming conventions * Undo change to integrations-core submodule * fix build gradle * rm empty line * fix test
gary-huang added a commit
that referenced
this pull request
Jul 9, 2025
* add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners
gary-huang added a commit
that referenced
this pull request
Jul 9, 2025
* add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * spotless * add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * rename ID to Id according to java naming conventions * Undo change to integrations-core submodule * fix build gradle * rm empty line * fix test
nayeem-kamal added a commit
that referenced
this pull request
Jul 9, 2025
* add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * implement LLM Obs SDK spans APIs (#8390) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * spotless * add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * rename ID to Id according to java naming conventions * Undo change to integrations-core submodule * fix build gradle * rm empty line * fix test * LLM Obs SDK Mapper (#8372) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <nayeem.kamal@datadoghq.com> * updated to master submodule * LLM Obs SDK use context API for parent children span linkage (#8711) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * impl llmobs agent and llmobs apis * use new ctx api to track parent span * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * ran spotless * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <nayeem.kamal@datadoghq.com> Co-authored-by: Nayeem Kamal <kamal.nayeem12@gmail.com> * LLM Obs SDK evaluation metrics submission (#8688) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * impl llmobs agent and llmobs apis * use new ctx api to track parent span * add api for evals * working impl supporting both agentless and agent * handle null tags and default to default ml app if null or empty string provided in the override * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * ran spotless * ran spotless * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <nayeem.kamal@datadoghq.com> Co-authored-by: Nayeem Kamal <kamal.nayeem12@gmail.com> * fix CODEOWNERS --------- Co-authored-by: Nayeem Kamal <nayeem.kamal@datadoghq.com> Co-authored-by: Nayeem Kamal <kamal.nayeem12@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Add APIs for LLM obs SDK manual instrumentation
Starting spans
Interacting with spans
A sample spring controller using the APIs above
logging for the spans above
Motivation
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: MLOB-1422