LLM Obs SDK Mapper by gary-huang · Pull Request #8372 · DataDog/dd-trace-java

gary-huang · 2025-02-12T05:40:31Z

What Does This Do

Adds the remote mapper and payload definitions for LLMObs spans as well as adding the remote intake api setup for the LLMObs backend endpoint.

Motivation

Support manual sending of LLMObs spans from within the tracer.

Additional Notes

Additional tests will follow in the feature branch.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-02-13T08:17:29Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	gary/add-llm-obs-writer
git_commit_date	1751899764	1751932260
git_commit_sha	`2ee59a9`	`75d85a2`
release_version	1.51.0-SNAPSHOT~2ee59a972d	1.50.0-SNAPSHOT~75d85a24a4

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1751934730	1751934730
ci_job_id	1017407734	1017407734
ci_pipeline_id	69857120	69857120
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-zgj6jfkn 6.8.0-1030-aws #32~22.04.1-Ubuntu SMP Thu Jun 5 08:38:24 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-zgj6jfkn 6.8.0-1030-aws #32~22.04.1-Ubuntu SMP Thu Jun 5 08:38:24 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 1 performance improvements and 7 performance regressions! Performance is the same for 45 metrics, 18 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:insecure-bank:tracing:Agent.start	worse [+21.499ms; +59.462ms] or [+2.148%; +5.942%]	1.041s	1.001s
scenario:startup:insecure-bank:tracing:Remote Config	worse [+53.519µs; +91.515µs] or [+7.982%; +13.649%]	742.984µs	670.467µs
scenario:startup:petclinic:appsec:IAST	better [-1.626ms; -0.883ms] or [-7.017%; -3.811%]	21.919ms	23.174ms
scenario:startup:petclinic:appsec:AppSec	worse [+5.520ms; +8.298ms] or [+3.240%; +4.871%]	177.244ms	170.335ms
scenario:startup:petclinic:appsec:Remote Config	worse [+13.830µs; +59.728µs] or [+2.260%; +9.761%]	648.664µs	611.885µs
scenario:startup:petclinic:profiling:AppSec	worse [+28.448ms; +31.335ms] or [+89.930%; +99.056%]	61.526ms	31.634ms
scenario:startup:petclinic:tracing:Agent.start	worse [+28.949ms; +38.521ms] or [+2.910%; +3.872%]	1028.609ms	994.874ms
scenario:startup:petclinic:tracing:Remote Config	worse [+39.717µs; +100.256µs] or [+5.806%; +14.656%]	754.062µs	684.075µs

Startup time reports for insecure-bank

gantt
title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~75d85a24a4, baseline=1.51.0-SNAPSHOT~2ee59a972d
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.001 s) : 0, 1000684
Total [baseline] (8.581 s) : 0, 8580781
Agent [candidate] (1.041 s) : 0, 1041164
Total [candidate] (8.627 s) : 0, 8627429
section iast
Agent [baseline] (1.132 s) : 0, 1132474
Total [baseline] (9.296 s) : 0, 9295531
Agent [candidate] (1.152 s) : 0, 1152031
Total [candidate] (9.232 s) : 0, 9232244
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.137 s) : 0, 1136589
Total [baseline] (9.243 s) : 0, 9243239
Agent [candidate] (1.152 s) : 0, 1152177
Total [candidate] (9.188 s) : 0, 9188250
section iast_TELEMETRY_OFF
Agent [baseline] (1.132 s) : 0, 1131586
Total [baseline] (9.243 s) : 0, 9243302
Agent [candidate] (1.153 s) : 0, 1153211
Total [candidate] (9.237 s) : 0, 9236572

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.001 s	-
Agent	iast	1.132 s	131.79 ms (13.2%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.137 s	135.906 ms (13.6%)
Agent	iast_TELEMETRY_OFF	1.132 s	130.903 ms (13.1%)
Total	tracing	8.581 s	-
Total	iast	9.296 s	714.751 ms (8.3%)
Total	iast_HARDCODED_SECRET_DISABLED	9.243 s	662.458 ms (7.7%)
Total	iast_TELEMETRY_OFF	9.243 s	662.521 ms (7.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.041 s	-
Agent	iast	1.152 s	110.867 ms (10.6%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.152 s	111.013 ms (10.7%)
Agent	iast_TELEMETRY_OFF	1.153 s	112.047 ms (10.8%)
Total	tracing	8.627 s	-
Total	iast	9.232 s	604.815 ms (7.0%)
Total	iast_HARDCODED_SECRET_DISABLED	9.188 s	560.822 ms (6.5%)
Total	iast_TELEMETRY_OFF	9.237 s	609.144 ms (7.1%)

gantt
title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~75d85a24a4, baseline=1.51.0-SNAPSHOT~2ee59a972d
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (692.578 ms) : 0, 692578
BytebuddyAgent [candidate] (695.179 ms) : 0, 695179
GlobalTracer [baseline] (242.096 ms) : 0, 242096
GlobalTracer [candidate] (245.642 ms) : 0, 245642
AppSec [baseline] (30.206 ms) : 0, 30206
AppSec [candidate] (57.728 ms) : 0, 57728
Debugger [baseline] (6.017 ms) : 0, 6017
Debugger [candidate] (6.248 ms) : 0, 6248
Remote Config [baseline] (670.467 µs) : 0, 670
Remote Config [candidate] (742.984 µs) : 0, 743
Telemetry [baseline] (8.205 ms) : 0, 8205
Telemetry [candidate] (12.015 ms) : 0, 12015
section iast
BytebuddyAgent [baseline] (807.084 ms) : 0, 807084
BytebuddyAgent [candidate] (804.265 ms) : 0, 804265
GlobalTracer [baseline] (232.771 ms) : 0, 232771
GlobalTracer [candidate] (231.301 ms) : 0, 231301
IAST [baseline] (25.264 ms) : 0, 25264
IAST [candidate] (28.479 ms) : 0, 28479
AppSec [baseline] (31.281 ms) : 0, 31281
AppSec [candidate] (50.152 ms) : 0, 50152
Debugger [baseline] (6.711 ms) : 0, 6711
Debugger [candidate] (5.913 ms) : 0, 5913
Remote Config [baseline] (594.193 µs) : 0, 594
Remote Config [candidate] (590.854 µs) : 0, 591
Telemetry [baseline] (7.938 ms) : 0, 7938
Telemetry [candidate] (7.892 ms) : 0, 7892
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (809.88 ms) : 0, 809880
BytebuddyAgent [candidate] (803.06 ms) : 0, 803060
GlobalTracer [baseline] (233.977 ms) : 0, 233977
GlobalTracer [candidate] (232.313 ms) : 0, 232313
IAST [baseline] (27.183 ms) : 0, 27183
IAST [candidate] (26.056 ms) : 0, 26056
AppSec [baseline] (29.44 ms) : 0, 29440
AppSec [candidate] (52.904 ms) : 0, 52904
Debugger [baseline] (6.677 ms) : 0, 6677
Debugger [candidate] (5.904 ms) : 0, 5904
Remote Config [baseline] (575.641 µs) : 0, 576
Remote Config [candidate] (583.864 µs) : 0, 584
Telemetry [baseline] (8.042 ms) : 0, 8042
Telemetry [candidate] (7.869 ms) : 0, 7869
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (806.473 ms) : 0, 806473
BytebuddyAgent [candidate] (805.936 ms) : 0, 805936
GlobalTracer [baseline] (233.495 ms) : 0, 233495
GlobalTracer [candidate] (231.129 ms) : 0, 231129
IAST [baseline] (27.99 ms) : 0, 27990
IAST [candidate] (28.659 ms) : 0, 28659
AppSec [baseline] (28.414 ms) : 0, 28414
AppSec [candidate] (49.485 ms) : 0, 49485
Debugger [baseline] (5.853 ms) : 0, 5853
Debugger [candidate] (6.008 ms) : 0, 6008
Remote Config [baseline] (579.127 µs) : 0, 579
Remote Config [candidate] (607.292 µs) : 0, 607
Telemetry [baseline] (7.904 ms) : 0, 7904
Telemetry [candidate] (7.847 ms) : 0, 7847

Startup time reports for petclinic

gantt
title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~75d85a24a4, baseline=1.51.0-SNAPSHOT~2ee59a972d
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (994.874 ms) : 0, 994874
Total [baseline] (10.627 s) : 0, 10626508
Agent [candidate] (1.029 s) : 0, 1028609
Total [candidate] (10.683 s) : 0, 10682883
section appsec
Agent [baseline] (1.164 s) : 0, 1164353
Total [baseline] (10.751 s) : 0, 10751042
Agent [candidate] (1.173 s) : 0, 1173072
Total [candidate] (10.786 s) : 0, 10785737
section iast
Agent [baseline] (1.135 s) : 0, 1135337
Total [baseline] (10.816 s) : 0, 10815689
Agent [candidate] (1.152 s) : 0, 1151875
Total [candidate] (10.889 s) : 0, 10888694
section profiling
Agent [baseline] (1.245 s) : 0, 1244982
Total [baseline] (10.966 s) : 0, 10966147
Agent [candidate] (1.269 s) : 0, 1269296
Total [candidate] (10.921 s) : 0, 10921137

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	994.874 ms	-
Agent	appsec	1.164 s	169.479 ms (17.0%)
Agent	iast	1.135 s	140.463 ms (14.1%)
Agent	profiling	1.245 s	250.108 ms (25.1%)
Total	tracing	10.627 s	-
Total	appsec	10.751 s	124.534 ms (1.2%)
Total	iast	10.816 s	189.181 ms (1.8%)
Total	profiling	10.966 s	339.638 ms (3.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.029 s	-
Agent	appsec	1.173 s	144.463 ms (14.0%)
Agent	iast	1.152 s	123.266 ms (12.0%)
Agent	profiling	1.269 s	240.687 ms (23.4%)
Total	tracing	10.683 s	-
Total	appsec	10.786 s	102.854 ms (1.0%)
Total	iast	10.889 s	205.811 ms (1.9%)
Total	profiling	10.921 s	238.254 ms (2.2%)

gantt
title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~75d85a24a4, baseline=1.51.0-SNAPSHOT~2ee59a972d
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (686.662 ms) : 0, 686662
BytebuddyAgent [candidate] (684.584 ms) : 0, 684584
GlobalTracer [baseline] (242.062 ms) : 0, 242062
GlobalTracer [candidate] (243.222 ms) : 0, 243222
AppSec [baseline] (30.452 ms) : 0, 30452
AppSec [candidate] (57.236 ms) : 0, 57236
Debugger [baseline] (6.047 ms) : 0, 6047
Debugger [candidate] (6.239 ms) : 0, 6239
Remote Config [baseline] (684.075 µs) : 0, 684
Remote Config [candidate] (754.062 µs) : 0, 754
Telemetry [baseline] (8.18 ms) : 0, 8180
Telemetry [candidate] (12.963 ms) : 0, 12963
section appsec
BytebuddyAgent [baseline] (705.901 ms) : 0, 705901
BytebuddyAgent [candidate] (707.265 ms) : 0, 707265
GlobalTracer [baseline] (239.273 ms) : 0, 239273
GlobalTracer [candidate] (239.962 ms) : 0, 239962
IAST [baseline] (23.174 ms) : 0, 23174
IAST [candidate] (21.919 ms) : 0, 21919
AppSec [baseline] (170.335 ms) : 0, 170335
AppSec [candidate] (177.244 ms) : 0, 177244
Debugger [baseline] (7.305 ms) : 0, 7305
Debugger [candidate] (5.978 ms) : 0, 5978
Remote Config [baseline] (611.885 µs) : 0, 612
Remote Config [candidate] (648.664 µs) : 0, 649
Telemetry [baseline] (8.464 ms) : 0, 8464
Telemetry [candidate] (7.374 ms) : 0, 7374
section iast
BytebuddyAgent [baseline] (809.323 ms) : 0, 809323
BytebuddyAgent [candidate] (803.393 ms) : 0, 803393
GlobalTracer [baseline] (233.45 ms) : 0, 233450
GlobalTracer [candidate] (231.382 ms) : 0, 231382
IAST [baseline] (28.571 ms) : 0, 28571
IAST [candidate] (29.487 ms) : 0, 29487
AppSec [baseline] (28.725 ms) : 0, 28725
AppSec [candidate] (49.659 ms) : 0, 49659
Debugger [baseline] (5.82 ms) : 0, 5820
Debugger [candidate] (5.9 ms) : 0, 5900
Remote Config [baseline] (581.454 µs) : 0, 581
Remote Config [candidate] (596.172 µs) : 0, 596
Telemetry [baseline] (7.993 ms) : 0, 7993
Telemetry [candidate] (7.956 ms) : 0, 7956
section profiling
BytebuddyAgent [baseline] (678.331 ms) : 0, 678331
BytebuddyAgent [candidate] (676.196 ms) : 0, 676196
GlobalTracer [baseline] (362.017 ms) : 0, 362017
GlobalTracer [candidate] (361.013 ms) : 0, 361013
AppSec [baseline] (31.634 ms) : 0, 31634
AppSec [candidate] (61.526 ms) : 0, 61526
Debugger [baseline] (12.049 ms) : 0, 12049
Debugger [candidate] (6.06 ms) : 0, 6060
Remote Config [baseline] (660.085 µs) : 0, 660
Remote Config [candidate] (653.859 µs) : 0, 654
Telemetry [baseline] (8.763 ms) : 0, 8763
Telemetry [candidate] (8.137 ms) : 0, 8137
ProfilingAgent [baseline] (102.937 ms) : 0, 102937
ProfilingAgent [candidate] (104.845 ms) : 0, 104845
Profiling [baseline] (102.96 ms) : 0, 102960
Profiling [candidate] (104.869 ms) : 0, 104869

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-07-08T00:02:29	2025-07-08T00:15:58
git_branch	master	gary/add-llm-obs-writer
git_commit_date	1751899764	1751932260
git_commit_sha	`2ee59a9`	`75d85a2`
release_version	1.51.0-SNAPSHOT~2ee59a972d	1.50.0-SNAPSHOT~75d85a24a4
start_time	2025-07-08T00:02:15	2025-07-08T00:15:43

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1751934098	1751934098
ci_job_id	1017407735	1017407735
ci_pipeline_id	69857120	69857120
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-nnibq57e 6.8.0-1030-aws #32~22.04.1-Ubuntu SMP Thu Jun 5 08:38:24 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-nnibq57e 6.8.0-1030-aws #32~22.04.1-Ubuntu SMP Thu Jun 5 08:38:24 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
thresholds_or_results	results	results
variant	iast	iast

Summary

Found 5 performance improvements and 1 performance regressions! Performance is the same for 6 metrics, 16 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:petclinic:appsec	better [-9.080ms; -8.918ms] or [-87.286%; -85.729%]	unstable [+307.784op/s; +4257.076op/s] or [+64.635%; +893.986%]	1.404ms	2758.621op/s	10.402ms	476.190op/s
scenario:load:petclinic:appsec_no_iast	better [-1.126ms; -0.975ms] or [-62.808%; -54.434%]	unstable [-1447.194op/s; +5630.200op/s] or [-61.506%; +239.284%]	0.742ms	4444.444op/s	1.792ms	2352.941op/s
scenario:load:petclinic:iast	better [-1.100ms; -1.000ms] or [-97.538%; -88.650%]	unstable [+1377.018op/s; +11666.461op/s] or [+39.589%; +335.411%]	0.078ms	10000.000op/s	1.128ms	3478.261op/s
scenario:load:petclinic:no_agent	worse [+2.773ms; +2.916ms] or [+31.811%; +33.451%]	unstable [-165.693op/s; -108.844op/s] or [-29.203%; -19.184%]	11.563ms	430.108op/s	8.718ms	567.376op/s
scenario:load:petclinic:profiling	better [-5.398ms; -5.239ms] or [-73.057%; -70.906%]	unstable [-109.811op/s; +2776.477op/s] or [-16.472%; +416.472%]	2.070ms	2000.000op/s	7.388ms	666.667op/s
scenario:load:petclinic:tracing	better [-5.114ms; -4.915ms] or [-61.746%; -59.349%]	unstable [-312.648op/s; +1780.778op/s] or [-52.369%; +298.280%]	3.267ms	1331.080op/s	8.282ms	597.015op/s

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	gary/add-llm-obs-writer
git_commit_date	1751899764	1751932260
git_commit_sha	`2ee59a9`	`75d85a2`
release_version	1.51.0-SNAPSHOT~2ee59a972d	1.50.0-SNAPSHOT~75d85a24a4

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1751934321	1751934321
ci_job_id	1017407736	1017407736
ci_pipeline_id	69857120	69857120
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-cl7aziws 6.8.0-1030-aws #32~22.04.1-Ubuntu SMP Thu Jun 5 08:38:24 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-cl7aziws 6.8.0-1030-aws #32~22.04.1-Ubuntu SMP Thu Jun 5 08:38:24 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~75d85a24a4, baseline=1.51.0-SNAPSHOT~2ee59a972d
dateFormat X
axisFormat %s
section baseline
no_agent (15.545 s) : 15545000, 15545000
. : milestone, 15545000,
appsec (14.899 s) : 14899000, 14899000
. : milestone, 14899000,
iast (18.704 s) : 18704000, 18704000
. : milestone, 18704000,
iast_GLOBAL (17.93 s) : 17930000, 17930000
. : milestone, 17930000,
profiling (15.782 s) : 15782000, 15782000
. : milestone, 15782000,
tracing (15.077 s) : 15077000, 15077000
. : milestone, 15077000,
section candidate
no_agent (15.26 s) : 15260000, 15260000
. : milestone, 15260000,
appsec (14.69 s) : 14690000, 14690000
. : milestone, 14690000,
iast (18.25 s) : 18250000, 18250000
. : milestone, 18250000,
iast_GLOBAL (17.84 s) : 17840000, 17840000
. : milestone, 17840000,
profiling (15.189 s) : 15189000, 15189000
. : milestone, 15189000,
tracing (14.95 s) : 14950000, 14950000
. : milestone, 14950000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.545 s [15.545 s, 15.545 s]	-
appsec	14.899 s [14.899 s, 14.899 s]	-646.0 ms (-4.2%)
iast	18.704 s [18.704 s, 18.704 s]	3.159 s (20.3%)
iast_GLOBAL	17.93 s [17.93 s, 17.93 s]	2.385 s (15.3%)
profiling	15.782 s [15.782 s, 15.782 s]	237.0 ms (1.5%)
tracing	15.077 s [15.077 s, 15.077 s]	-468.0 ms (-3.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.26 s [15.26 s, 15.26 s]	-
appsec	14.69 s [14.69 s, 14.69 s]	-570.0 ms (-3.7%)
iast	18.25 s [18.25 s, 18.25 s]	2.99 s (19.6%)
iast_GLOBAL	17.84 s [17.84 s, 17.84 s]	2.58 s (16.9%)
profiling	15.189 s [15.189 s, 15.189 s]	-71.0 ms (-0.5%)
tracing	14.95 s [14.95 s, 14.95 s]	-310.0 ms (-2.0%)

Execution time for tomcat

gantt
title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~75d85a24a4, baseline=1.51.0-SNAPSHOT~2ee59a972d
dateFormat X
axisFormat %s
section baseline
no_agent (1.474 ms) : 1462, 1485
. : milestone, 1474,
appsec (2.419 ms) : 2370, 2469
. : milestone, 2419,
iast (2.199 ms) : 2137, 2262
. : milestone, 2199,
iast_GLOBAL (2.233 ms) : 2171, 2295
. : milestone, 2233,
profiling (2.048 ms) : 1998, 2098
. : milestone, 2048,
tracing (2.003 ms) : 1955, 2051
. : milestone, 2003,
section candidate
no_agent (1.477 ms) : 1466, 1489
. : milestone, 1477,
appsec (2.408 ms) : 2359, 2457
. : milestone, 2408,
iast (2.188 ms) : 2127, 2249
. : milestone, 2188,
iast_GLOBAL (2.229 ms) : 2167, 2291
. : milestone, 2229,
profiling (2.039 ms) : 1989, 2089
. : milestone, 2039,
tracing (2.013 ms) : 1965, 2061
. : milestone, 2013,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.474 ms [1.462 ms, 1.485 ms]	-
appsec	2.419 ms [2.37 ms, 2.469 ms]	945.608 µs (64.2%)
iast	2.199 ms [2.137 ms, 2.262 ms]	725.672 µs (49.2%)
iast_GLOBAL	2.233 ms [2.171 ms, 2.295 ms]	759.69 µs (51.5%)
profiling	2.048 ms [1.998 ms, 2.098 ms]	574.363 µs (39.0%)
tracing	2.003 ms [1.955 ms, 2.051 ms]	529.662 µs (35.9%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.477 ms [1.466 ms, 1.489 ms]	-
appsec	2.408 ms [2.359 ms, 2.457 ms]	930.552 µs (63.0%)
iast	2.188 ms [2.127 ms, 2.249 ms]	710.455 µs (48.1%)
iast_GLOBAL	2.229 ms [2.167 ms, 2.291 ms]	751.439 µs (50.9%)
profiling	2.039 ms [1.989 ms, 2.089 ms]	562.057 µs (38.0%)
tracing	2.013 ms [1.965 ms, 2.061 ms]	535.592 µs (36.3%)

* add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <nayeem.kamal@datadoghq.com>

* add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * implement LLM Obs SDK spans APIs (#8390) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * spotless * add APIs for llm obs sdk (#8135) * add APIs for llm obs * add llm message class to support llm spans * follow java convention of naming Id instead of ID * add codeowners * rename ID to Id according to java naming conventions * Undo change to integrations-core submodule * fix build gradle * rm empty line * fix test * LLM Obs SDK Mapper (#8372) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <nayeem.kamal@datadoghq.com> * updated to master submodule * LLM Obs SDK use context API for parent children span linkage (#8711) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * impl llmobs agent and llmobs apis * use new ctx api to track parent span * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * ran spotless * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <nayeem.kamal@datadoghq.com> Co-authored-by: Nayeem Kamal <kamal.nayeem12@gmail.com> * LLM Obs SDK evaluation metrics submission (#8688) * add APIs for llm obs * add llm message class to support llm spans * add llm message class to support llm spans * impl llmobs agent and llmobs apis * support llm messages with tool calls * handle default model name and provider * rm unneeded file * impl llmobs agent and llmobs apis * impl llmobs agent * working writer * add support for llm message and tool calls * impl llmobs agent and llmobs apis * use new ctx api to track parent span * add api for evals * working impl supporting both agentless and agent * handle null tags and default to default ml app if null or empty string provided in the override * cleaned up whitespace * resolve merge conflicts * remaining merge conflicts * fix bad method call * fixed llmobs intake creation if llmobs not enabled * removed print statements * ran spotless * ran spotless * added tests for llmobsspanmapper * fixed coverage for tags --------- Co-authored-by: Nayeem Kamal <nayeem.kamal@datadoghq.com> Co-authored-by: Nayeem Kamal <kamal.nayeem12@gmail.com> * fix CODEOWNERS --------- Co-authored-by: Nayeem Kamal <nayeem.kamal@datadoghq.com> Co-authored-by: Nayeem Kamal <kamal.nayeem12@gmail.com>

datadog-datadog-prod-us1Bot reviewed Feb 12, 2025
View reviewed changes

Comment threaddd-trace-core/src/main/java/datadog/trace/llmobs/writer/ddintake/LLMObsSpanMapper.java Outdated
Comment threaddd-trace-core/src/main/java/datadog/trace/llmobs/writer/ddintake/LLMObsSpanMapper.java Outdated

gary-huang force-pushed the gary/impl-llm-obs-api branch 2 times, most recently from 680ec94 to b1d7aebCompare February 13, 2025 07:39

gary-huang force-pushed the gary/add-llm-obs-writer branch from 113bf8c to 9052dc4Compare February 13, 2025 07:41

gary-huang force-pushed the gary/impl-llm-obs-api branch from b1d7aeb to 3bcbe81Compare February 14, 2025 05:35

gary-huang force-pushed the gary/add-llm-obs-writer branch from 9052dc4 to dd8c1edCompare February 14, 2025 09:13

gary-huang force-pushed the gary/impl-llm-obs-api branch from 3bcbe81 to c437b9cCompare February 19, 2025 14:47

gary-huang force-pushed the gary/add-llm-obs-writer branch 2 times, most recently from 88cbf86 to ce8a6d3Compare February 19, 2025 17:32

gary-huang force-pushed the gary/impl-llm-obs-api branch from c437b9c to 8247b74Compare February 19, 2025 17:50

gary-huang force-pushed the gary/add-llm-obs-writer branch 2 times, most recently from fc30660 to 061b18dCompare February 19, 2025 18:18

gary-huang force-pushed the gary/impl-llm-obs-api branch from 8247b74 to 871d358Compare February 20, 2025 06:56

gary-huang force-pushed the gary/add-llm-obs-writer branch 2 times, most recently from 5c305b9 to 9ee77d9Compare February 20, 2025 15:50

datadog-datadog-prod-us1Bot reviewed Feb 20, 2025
View reviewed changes

Comment threaddd-trace-core/src/main/java/datadog/trace/llmobs/writer/ddintake/LLMObsSpanMapper.java Outdated

gary-huang force-pushed the gary/impl-llm-obs-api branch 6 times, most recently from 94136c5 to 7064f57Compare February 20, 2025 23:13

gary-huang force-pushed the gary/add-llm-obs-writer branch from 9ee77d9 to 0b4f8a8Compare February 24, 2025 19:38

gary-huang force-pushed the gary/impl-llm-obs-api branch from 7064f57 to 9cb3e0aCompare February 28, 2025 05:41

gary-huang force-pushed the gary/add-llm-obs-writer branch from 0b4f8a8 to 738eb77Compare March 4, 2025 16:17

datadog-datadog-prod-us1Bot reviewed Mar 4, 2025
View reviewed changes

Comment threaddd-trace-core/src/main/java/datadog/trace/llmobs/writer/ddintake/LLMObsSpanMapper.java Outdated

gary-huang force-pushed the gary/impl-llm-obs-api branch 3 times, most recently from f9cb23e to 64c94e5Compare March 5, 2025 19:23

rm unneeded file
5e66a2f

gary-huang force-pushed the gary/add-llm-obs-writer branch from f858e30 to fb0af52Compare April 11, 2025 01:54

gary-huang added 4 commits April 11, 2025 21:20

impl llmobs agent and llmobs apis
75290af

impl llmobs agent
7b2db7c

working writer
d76c94d

add support for llm message and tool calls
e0b4510

gary-huang force-pushed the gary/add-llm-obs-writer branch from fb0af52 to e0b4510Compare April 21, 2025 14:04

Base automatically changed from gary/impl-llm-obs-api to gary/llmobs-sdk-mergeJune 4, 2025 19:35

gary-huang changed the title ~~writer impl~~LLM Obs SDK MapperJun 26, 2025

Nayeem Kamal added 4 commits July 2, 2025 11:20

cleaned up whitespace
52d3502

resolve merge conflicts
f54c468

remaining merge conflicts
688d984

merge with feature branch
e62d569

nayeem-kamal approved these changes Jul 2, 2025
View reviewed changes

Nayeem Kamal added 2 commits July 2, 2025 15:12

fix bad method call
157e53e

fixed llmobs intake creation if llmobs not enabled
ade08ca

nayeem-kamal marked this pull request as ready for review July 3, 2025 14:14

nayeem-kamal requested review from a team as code owners July 3, 2025 14:14

nayeem-kamal requested review from daniel-mohedano and mcculls and removed request for daniel-mohedano and mcculls July 3, 2025 14:14

Nayeem Kamal added 3 commits July 3, 2025 10:35

removed print statements
973c09f

added tests for llmobsspanmapper
1eb8db3

fixed coverage for tags
75d85a2

nayeem-kamal merged commit 5ca9c84 into gary/llmobs-sdk-mergeJul 8, 2025
294 of 299 checks passed

nayeem-kamal deleted the gary/add-llm-obs-writer branch July 8, 2025 15:51

Navigation Menu

Uh oh!

LLM Obs SDK Mapper - #8372

LLM Obs SDK Mapper#8372
nayeem-kamal merged 20 commits into
gary/llmobs-sdk-mergefrom
gary/add-llm-obs-writer

gary-huang commented Feb 12, 2025•
edited by nayeem-kamal
Loading

Uh oh!

Uh oh!

Uh oh!

pr-commenterBot commented Feb 13, 2025•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

gary-huang commented Feb 12, 2025• edited by nayeem-kamal Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

Uh oh!

Uh oh!

pr-commenterBot commented Feb 13, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gary-huang commented Feb 12, 2025•
edited by nayeem-kamal
Loading

pr-commenterBot commented Feb 13, 2025•
edited
Loading