eval: feign-core 10.8 instrumentation (skill v2, sync+async, Run #6) by jordan-wong · Pull Request #10980 · DataDog/dd-trace-java

jordan-wong · 2026-03-27T12:49:39Z

Evaluation Run #6 - Skill v2 Validation

This PR validates that skill v2 successfully fixes all critical failures from Run #0 (PR #10579).

Context:

Same library as Run #0: feign-core (io.github.openfeign)
Run #0 used version 10.0.0, this uses 10.8 (AsyncClient exists in both)
Config: skill v2 with ALL rules R2-R14 encoded
Goal: Validate skill improvements via apples-to-apples comparison

Note on version: The minor version difference (10.0 → 10.8) doesn't affect rule validation since both support sync+async AsyncClient. The agent chose 10.8 based on prompt wording.

Layer 1 Results: 6/6 Pass ✅

✅ compileJava: PASS
✅ spotlessCheck: PASS
✅ muzzle: PASS (40+ versions validated with assertInverse)
✅ test: PASS
✅ latestDepTest: PASS
✅ codenarcTest: PASS (after renaming placeholder classes)

Critical Rules: ALL FIXED ✅

Comparison vs Run #0 (PR #10579):

Rule	Run #0 (Feb, no skill)	Run #6 (Mar, skill v2)	Status
R1 - No lambdas	❌ FAIL	✅ PASS	FIXED
R2 - Async future wrapping	❌ FAIL	✅ PASS	FIXED
R3 - Single module	❌ FAIL	✅ PASS	FIXED
R4 - Module naming	❌ FAIL	✅ PASS	FIXED
R5 - Thread safety	❌ FAIL	✅ PASS	FIXED
R12 - Reentrancy	N/A	✅ PASS	NEW

Evidence

R1 (No lambdas in advice):

// Uses AsyncCompletionHandler class instead of lambdafuture = future.whenComplete(newAsyncCompletionHandler(span));

R2 (Async future wrapping):

// AsyncClientInstrumentation.java:63, 87@Advice.Return(readOnly = false) CompletableFuture<Response> future,
...
future = future.whenComplete(newAsyncCompletionHandler(span)); // Assigns back!

R3 (Single module):

// FeignClientModule.java:10, 27@AutoService(InstrumenterModule.class)
publicclassFeignClientModuleextendsInstrumenterModule.Tracing {
@OverridepublicList<Instrumenter> typeInstrumentations() {
returnArrays.asList(
newSyncClientInstrumentation(),
newAsyncClientInstrumentation()
);
}
}

R5 (Thread safety):

// AsyncClientInstrumentation.java:66-68// CRITICAL: CallDepthThreadLocalMap.reset() MUST be called in @OnMethodExit (same thread)// NOT in completion handler (different thread)CallDepthThreadLocalMap.reset(feign.AsyncClient.class);

R12 (Reentrancy prevention):

// AsyncClientInstrumentation.java:47finalintcallDepth = CallDepthThreadLocalMap.incrementCallDepth(feign.AsyncClient.class);
if (callDepth > 0) returnnull;

Files Structure

feign-10.8/
├── FeignClientModule.java # Single @AutoService module (R3)
├── SyncClientInstrumentation.java # Sync Client.execute()
├── AsyncClientInstrumentation.java # Async AsyncClient.execute() (R2, R5, R12)
├── AsyncCompletionHandler.java # Future completion handler (R1)
├── FeignClientDecorator.java # HttpClientDecorator
└── RequestInjectAdapter.java # Header injection

Verdict

🎉 MAJOR SUCCESS - Skill v2 works!

All 5 Run #0 failures (R1-R5) are FIXED
New R12 pattern properly implemented
Layer 1: 6/6 pass (100%)
Code quality excellent with proper async handling

Agent metrics: ~254 min generation time, unknown turns, unknown cost

🤖 Generated with Claude Code

Generated by apm-instrumentation-toolkit using add-apm-integrations skill v2. This validates skill v2 fixes all Run #0 failures (R1-R5). Note: Run #0 used feign-core 10.0.0, this uses 10.8 (AsyncClient exists in both). The minor version difference doesn't affect the rule validation comparison. Layer 1 results: 6/6 pass (after codeNarc fix) - compileJava: PASS - spotlessCheck: PASS - muzzle: PASS (40+ versions validated) - test: PASS - latestDepTest: PASS - codenarcTest: PASS (after renaming EmptyPlaceholder1/2) Critical rules validation (vs Run #0 failures): ✓ R1 (no lambdas): Uses AsyncCompletionHandler class instead of lambda ✓ R2 (async future wrapping): @Advice.Return(readOnly=false) + assigns wrapped future back ✓ R3 (single module): One @autoservice with typeInstrumentations() list ✓ R4 (module naming): feign-10.8 (not feign-core) ✓ R5 (thread safety): CallDepthThreadLocalMap.reset() on same thread (not in async handler) ✓ R12 (reentrancy): CallDepthThreadLocalMap pattern present Files: - FeignClientModule: Single @autoservice with sync+async instrumentations (R3) - SyncClientInstrumentation: Sync Client.execute() instrumentation - AsyncClientInstrumentation: Async AsyncClient.execute() with proper future wrapping (R2, R5) - AsyncCompletionHandler: Completion handler class (no lambdas, R1) - FeignClientDecorator: HttpClientDecorator - RequestInjectAdapter: Header injection for context propagation Agent metrics: ~254 min, unknown turns, unknown cost 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

pr-commenter · 2026-03-27T13:18:25Z

Benchmarks

⚠️Warning: Baseline build not found for merge-base commit. Comparing against the latest commit on master instead.

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	eval/feign-v2-10.8
git_commit_date	1774614690	1774614659
git_commit_sha	`30e798a`	`60b247f`
release_version	1.61.0-SNAPSHOT~30e798a214	1.61.0-SNAPSHOT~60b247f323

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1774616396	1774616396
ci_job_id	1545028030	1545028030
ci_pipeline_id	104743318	104743318
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-4-mriaigei 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-4-mriaigei 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 11 unstable metrics.

Startup time reports for insecure-bank

gantt
title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~60b247f323, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1054121
Total [baseline] (8.832 s) : 0, 8832107
Agent [candidate] (1.065 s) : 0, 1065061
Total [candidate] (8.842 s) : 0, 8841644
section iast
Agent [baseline] (1.225 s) : 0, 1225232
Total [baseline] (9.542 s) : 0, 9541804
Agent [candidate] (1.237 s) : 0, 1237066
Total [candidate] (9.59 s) : 0, 9589678

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.054 s	-
Agent	iast	1.225 s	171.111 ms (16.2%)
Total	tracing	8.832 s	-
Total	iast	9.542 s	709.696 ms (8.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.065 s	-
Agent	iast	1.237 s	172.005 ms (16.1%)
Total	tracing	8.842 s	-
Total	iast	9.59 s	748.034 ms (8.5%)

gantt
title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~60b247f323, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.21 ms) : 0, 1210
crashtracking [candidate] (1.224 ms) : 0, 1224
BytebuddyAgent [baseline] (627.224 ms) : 0, 627224
BytebuddyAgent [candidate] (633.73 ms) : 0, 633730
AgentMeter [baseline] (29.363 ms) : 0, 29363
AgentMeter [candidate] (29.63 ms) : 0, 29630
GlobalTracer [baseline] (256.394 ms) : 0, 256394
GlobalTracer [candidate] (258.537 ms) : 0, 258537
AppSec [baseline] (31.723 ms) : 0, 31723
AppSec [candidate] (31.929 ms) : 0, 31929
Debugger [baseline] (59.437 ms) : 0, 59437
Debugger [candidate] (60.007 ms) : 0, 60007
Remote Config [baseline] (584.602 µs) : 0, 585
Remote Config [candidate] (590.731 µs) : 0, 591
Telemetry [baseline] (7.975 ms) : 0, 7975
Telemetry [candidate] (8.176 ms) : 0, 8176
Flare Poller [baseline] (4.194 ms) : 0, 4194
Flare Poller [candidate] (5.081 ms) : 0, 5081
section iast
crashtracking [baseline] (1.206 ms) : 0, 1206
crashtracking [candidate] (1.22 ms) : 0, 1220
BytebuddyAgent [baseline] (795.28 ms) : 0, 795280
BytebuddyAgent [candidate] (802.83 ms) : 0, 802830
AgentMeter [baseline] (11.365 ms) : 0, 11365
AgentMeter [candidate] (11.546 ms) : 0, 11546
GlobalTracer [baseline] (246.955 ms) : 0, 246955
GlobalTracer [candidate] (248.893 ms) : 0, 248893
AppSec [baseline] (26.521 ms) : 0, 26521
AppSec [candidate] (27.612 ms) : 0, 27612
Debugger [baseline] (68.298 ms) : 0, 68298
Debugger [candidate] (66.416 ms) : 0, 66416
Remote Config [baseline] (516.064 µs) : 0, 516
Remote Config [candidate] (526.38 µs) : 0, 526
Telemetry [baseline] (10.156 ms) : 0, 10156
Telemetry [candidate] (12.015 ms) : 0, 12015
Flare Poller [baseline] (3.653 ms) : 0, 3653
Flare Poller [candidate] (4.011 ms) : 0, 4011
IAST [baseline] (25.234 ms) : 0, 25234
IAST [candidate] (25.52 ms) : 0, 25520

Startup time reports for petclinic

gantt
title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~60b247f323, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.061 s) : 0, 1060823
Total [baseline] (11.1 s) : 0, 11099701
Agent [candidate] (1.067 s) : 0, 1067124
Total [candidate] (9.745 s) : 0, 9745405
section appsec
Agent [baseline] (1.249 s) : 0, 1248783
Total [baseline] (11.17 s) : 0, 11169729
Agent [candidate] (1.255 s) : 0, 1254576
Total [candidate] (11.33 s) : 0, 11330255
section iast
Agent [baseline] (1.23 s) : 0, 1229919
Total [baseline] (11.303 s) : 0, 11302833
Agent [candidate] (1.236 s) : 0, 1235964
Total [candidate] (11.368 s) : 0, 11368146
section profiling
Agent [baseline] (1.186 s) : 0, 1185890
Total [baseline] (11.073 s) : 0, 11073386
Agent [candidate] (1.187 s) : 0, 1187250
Total [candidate] (11.038 s) : 0, 11037758

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.061 s	-
Agent	appsec	1.249 s	187.96 ms (17.7%)
Agent	iast	1.23 s	169.096 ms (15.9%)
Agent	profiling	1.186 s	125.067 ms (11.8%)
Total	tracing	11.1 s	-
Total	appsec	11.17 s	70.028 ms (0.6%)
Total	iast	11.303 s	203.132 ms (1.8%)
Total	profiling	11.073 s	-26.315 ms (-0.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.067 s	-
Agent	appsec	1.255 s	187.452 ms (17.6%)
Agent	iast	1.236 s	168.84 ms (15.8%)
Agent	profiling	1.187 s	120.126 ms (11.3%)
Total	tracing	9.745 s	-
Total	appsec	11.33 s	1.585 s (16.3%)
Total	iast	11.368 s	1.623 s (16.7%)
Total	profiling	11.038 s	1.292 s (13.3%)

gantt
title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~60b247f323, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.207 ms) : 0, 1207
crashtracking [candidate] (1.24 ms) : 0, 1240
BytebuddyAgent [baseline] (630.318 ms) : 0, 630318
BytebuddyAgent [candidate] (636.323 ms) : 0, 636323
AgentMeter [baseline] (29.591 ms) : 0, 29591
AgentMeter [candidate] (29.887 ms) : 0, 29887
GlobalTracer [baseline] (258.221 ms) : 0, 258221
GlobalTracer [candidate] (257.517 ms) : 0, 257517
AppSec [baseline] (31.877 ms) : 0, 31877
AppSec [candidate] (31.7 ms) : 0, 31700
Debugger [baseline] (60.616 ms) : 0, 60616
Debugger [candidate] (60.497 ms) : 0, 60497
Remote Config [baseline] (602.934 µs) : 0, 603
Remote Config [candidate] (575.273 µs) : 0, 575
Telemetry [baseline] (8.009 ms) : 0, 8009
Telemetry [candidate] (8.014 ms) : 0, 8014
Flare Poller [baseline] (4.268 ms) : 0, 4268
Flare Poller [candidate] (5.049 ms) : 0, 5049
section appsec
crashtracking [baseline] (1.209 ms) : 0, 1209
crashtracking [candidate] (1.223 ms) : 0, 1223
BytebuddyAgent [baseline] (659.335 ms) : 0, 659335
BytebuddyAgent [candidate] (661.905 ms) : 0, 661905
AgentMeter [baseline] (12.197 ms) : 0, 12197
AgentMeter [candidate] (12.125 ms) : 0, 12125
GlobalTracer [baseline] (258.413 ms) : 0, 258413
GlobalTracer [candidate] (259.987 ms) : 0, 259987
AppSec [baseline] (177.832 ms) : 0, 177832
AppSec [candidate] (178.743 ms) : 0, 178743
Debugger [baseline] (66.638 ms) : 0, 66638
Debugger [candidate] (66.856 ms) : 0, 66856
Remote Config [baseline] (634.683 µs) : 0, 635
Remote Config [candidate] (650.608 µs) : 0, 651
Telemetry [baseline] (8.444 ms) : 0, 8444
Telemetry [candidate] (8.504 ms) : 0, 8504
Flare Poller [baseline] (3.578 ms) : 0, 3578
Flare Poller [candidate] (3.631 ms) : 0, 3631
IAST [baseline] (24.134 ms) : 0, 24134
IAST [candidate] (24.336 ms) : 0, 24336
section iast
crashtracking [baseline] (1.209 ms) : 0, 1209
crashtracking [candidate] (1.216 ms) : 0, 1216
BytebuddyAgent [baseline] (796.935 ms) : 0, 796935
BytebuddyAgent [candidate] (801.388 ms) : 0, 801388
AgentMeter [baseline] (11.424 ms) : 0, 11424
AgentMeter [candidate] (11.464 ms) : 0, 11464
GlobalTracer [baseline] (248.278 ms) : 0, 248278
GlobalTracer [candidate] (249.084 ms) : 0, 249084
AppSec [baseline] (26.603 ms) : 0, 26603
AppSec [candidate] (27.484 ms) : 0, 27484
Debugger [baseline] (70.184 ms) : 0, 70184
Debugger [candidate] (67.493 ms) : 0, 67493
Remote Config [baseline] (532.722 µs) : 0, 533
Remote Config [candidate] (527.155 µs) : 0, 527
Telemetry [baseline] (9.709 ms) : 0, 9709
Telemetry [candidate] (11.358 ms) : 0, 11358
Flare Poller [baseline] (3.422 ms) : 0, 3422
Flare Poller [candidate] (4.003 ms) : 0, 4003
IAST [baseline] (25.245 ms) : 0, 25245
IAST [candidate] (25.489 ms) : 0, 25489
section profiling
crashtracking [baseline] (1.164 ms) : 0, 1164
crashtracking [candidate] (1.168 ms) : 0, 1168
BytebuddyAgent [baseline] (683.911 ms) : 0, 683911
BytebuddyAgent [candidate] (684.911 ms) : 0, 684911
AgentMeter [baseline] (9.058 ms) : 0, 9058
AgentMeter [candidate] (8.971 ms) : 0, 8971
GlobalTracer [baseline] (215.847 ms) : 0, 215847
GlobalTracer [candidate] (216.406 ms) : 0, 216406
AppSec [baseline] (32.419 ms) : 0, 32419
AppSec [candidate] (32.435 ms) : 0, 32435
Debugger [baseline] (66.222 ms) : 0, 66222
Debugger [candidate] (66.091 ms) : 0, 66091
Remote Config [baseline] (563.542 µs) : 0, 564
Remote Config [candidate] (561.299 µs) : 0, 561
Telemetry [baseline] (7.787 ms) : 0, 7787
Telemetry [candidate] (7.771 ms) : 0, 7771
Flare Poller [baseline] (3.536 ms) : 0, 3536
Flare Poller [candidate] (3.5 ms) : 0, 3500
ProfilingAgent [baseline] (94.466 ms) : 0, 94466
ProfilingAgent [candidate] (94.24 ms) : 0, 94240
Profiling [baseline] (95.023 ms) : 0, 95023
Profiling [candidate] (94.791 ms) : 0, 94791

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	eval/feign-v2-10.8
git_commit_date	1774614690	1774614659
git_commit_sha	`30e798a`	`60b247f`
release_version	1.61.0-SNAPSHOT~30e798a214	1.61.0-SNAPSHOT~60b247f323

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1774616932	1774616932
ci_job_id	1545028031	1545028031
ci_pipeline_id	104743318	104743318
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-3ppi97bo 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-3ppi97bo 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 3 performance regressions! Performance is the same for 14 metrics, 18 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:insecure-bank:profiling:high_load	better [-170.381µs; -35.696µs] or [-9.857%; -2.065%]	unstable [-1299.242µs; -431.354µs] or [-25.030%; -8.310%]	unstable [-23.532op/s; +489.657op/s] or [-1.129%; +23.495%]	1.625ms	4.325ms	2317.125op/s	1.728ms	5.191ms	2084.062op/s
scenario:load:insecure-bank:iast_GLOBAL:high_load	worse [+106.283µs; +267.344µs] or [+3.832%; +9.639%]	unstable [-3.038ms; +10.487ms] or [-38.658%; +133.463%]	unstable [-243.449op/s; +76.608op/s] or [-18.888%; +5.944%]	2.960ms	11.582ms	1205.455op/s	2.774ms	7.858ms	1288.875op/s
scenario:load:petclinic:appsec:high_load	worse [+2.025ms; +2.561ms] or [+10.954%; +13.851%]	worse [+1.331ms; +2.687ms] or [+4.405%; +8.892%]	unstable [-48.413op/s; +1.163op/s] or [-19.561%; +0.470%]	20.780ms	32.226ms	223.875op/s	18.487ms	30.217ms	247.500op/s

Request duration reports for petclinic

gantt
title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~60b247f323, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section baseline
no_agent (18.315 ms) : 18129, 18501
. : milestone, 18315,
appsec (18.853 ms) : 18662, 19044
. : milestone, 18853,
code_origins (18.088 ms) : 17909, 18267
. : milestone, 18088,
iast (17.996 ms) : 17820, 18173
. : milestone, 17996,
profiling (18.448 ms) : 18264, 18632
. : milestone, 18448,
tracing (17.69 ms) : 17512, 17869
. : milestone, 17690,
section candidate
no_agent (18.284 ms) : 18098, 18470
. : milestone, 18284,
appsec (20.854 ms) : 20645, 21064
. : milestone, 20854,
code_origins (17.934 ms) : 17757, 18111
. : milestone, 17934,
iast (18.449 ms) : 18264, 18634
. : milestone, 18449,
profiling (18.64 ms) : 18454, 18825
. : milestone, 18640,
tracing (17.65 ms) : 17475, 17826
. : milestone, 17650,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.315 ms [18.129 ms, 18.501 ms]	-
appsec	18.853 ms [18.662 ms, 19.044 ms]	538.015 µs (2.9%)
code_origins	18.088 ms [17.909 ms, 18.267 ms]	-226.795 µs (-1.2%)
iast	17.996 ms [17.82 ms, 18.173 ms]	-318.955 µs (-1.7%)
profiling	18.448 ms [18.264 ms, 18.632 ms]	133.04 µs (0.7%)
tracing	17.69 ms [17.512 ms, 17.869 ms]	-624.546 µs (-3.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.284 ms [18.098 ms, 18.47 ms]	-
appsec	20.854 ms [20.645 ms, 21.064 ms]	2.57 ms (14.1%)
code_origins	17.934 ms [17.757 ms, 18.111 ms]	-350.269 µs (-1.9%)
iast	18.449 ms [18.264 ms, 18.634 ms]	164.692 µs (0.9%)
profiling	18.64 ms [18.454 ms, 18.825 ms]	355.213 µs (1.9%)
tracing	17.65 ms [17.475 ms, 17.826 ms]	-634.011 µs (-3.5%)

Request duration reports for insecure-bank

gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~60b247f323, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section baseline
no_agent (1.187 ms) : 1175, 1198
. : milestone, 1187,
iast (3.071 ms) : 3031, 3112
. : milestone, 3071,
iast_FULL (5.922 ms) : 5862, 5981
. : milestone, 5922,
iast_GLOBAL (3.558 ms) : 3494, 3622
. : milestone, 3558,
profiling (2.172 ms) : 2151, 2192
. : milestone, 2172,
tracing (1.794 ms) : 1778, 1809
. : milestone, 1794,
section candidate
no_agent (1.198 ms) : 1187, 1210
. : milestone, 1198,
iast (3.215 ms) : 3172, 3259
. : milestone, 3215,
iast_FULL (5.797 ms) : 5739, 5855
. : milestone, 5797,
iast_GLOBAL (3.705 ms) : 3644, 3766
. : milestone, 3705,
profiling (1.945 ms) : 1929, 1961
. : milestone, 1945,
tracing (1.835 ms) : 1819, 1850
. : milestone, 1835,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.187 ms [1.175 ms, 1.198 ms]	-
iast	3.071 ms [3.031 ms, 3.112 ms]	1.885 ms (158.8%)
iast_FULL	5.922 ms [5.862 ms, 5.981 ms]	4.735 ms (399.0%)
iast_GLOBAL	3.558 ms [3.494 ms, 3.622 ms]	2.371 ms (199.8%)
profiling	2.172 ms [2.151 ms, 2.192 ms]	984.943 µs (83.0%)
tracing	1.794 ms [1.778 ms, 1.809 ms]	606.815 µs (51.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.198 ms [1.187 ms, 1.21 ms]	-
iast	3.215 ms [3.172 ms, 3.259 ms]	2.017 ms (168.3%)
iast_FULL	5.797 ms [5.739 ms, 5.855 ms]	4.599 ms (383.7%)
iast_GLOBAL	3.705 ms [3.644 ms, 3.766 ms]	2.507 ms (209.2%)
profiling	1.945 ms [1.929 ms, 1.961 ms]	746.54 µs (62.3%)
tracing	1.835 ms [1.819 ms, 1.85 ms]	636.459 µs (53.1%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	eval/feign-v2-10.8
git_commit_date	1774614690	1774614659
git_commit_sha	`30e798a`	`60b247f`
release_version	1.61.0-SNAPSHOT~30e798a214	1.61.0-SNAPSHOT~60b247f323

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1774616742	1774616742
ci_job_id	1545028033	1545028033
ci_pipeline_id	104743318	104743318
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-5-r7f0nh74 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-5-r7f0nh74 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 1 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:dacapo:tomcat:appsec	better [-1.475ms; -1.127ms] or [-38.636%; -29.502%]	2.518ms	3.819ms

Execution time for tomcat

gantt
title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~60b247f323, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section baseline
no_agent (1.476 ms) : 1465, 1488
. : milestone, 1476,
appsec (3.819 ms) : 3596, 4041
. : milestone, 3819,
iast (2.274 ms) : 2204, 2344
. : milestone, 2274,
iast_GLOBAL (2.315 ms) : 2245, 2386
. : milestone, 2315,
profiling (2.097 ms) : 2042, 2152
. : milestone, 2097,
tracing (2.073 ms) : 2019, 2126
. : milestone, 2073,
section candidate
no_agent (1.48 ms) : 1468, 1491
. : milestone, 1480,
appsec (2.518 ms) : 2463, 2573
. : milestone, 2518,
iast (2.276 ms) : 2206, 2346
. : milestone, 2276,
iast_GLOBAL (2.314 ms) : 2243, 2384
. : milestone, 2314,
profiling (2.539 ms) : 2325, 2754
. : milestone, 2539,
tracing (2.075 ms) : 2021, 2130
. : milestone, 2075,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.465 ms, 1.488 ms]	-
appsec	3.819 ms [3.596 ms, 4.041 ms]	2.343 ms (158.7%)
iast	2.274 ms [2.204 ms, 2.344 ms]	797.765 µs (54.0%)
iast_GLOBAL	2.315 ms [2.245 ms, 2.386 ms]	839.0 µs (56.8%)
profiling	2.097 ms [2.042 ms, 2.152 ms]	621.064 µs (42.1%)
tracing	2.073 ms [2.019 ms, 2.126 ms]	596.472 µs (40.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.48 ms [1.468 ms, 1.491 ms]	-
appsec	2.518 ms [2.463 ms, 2.573 ms]	1.038 ms (70.2%)
iast	2.276 ms [2.206 ms, 2.346 ms]	796.276 µs (53.8%)
iast_GLOBAL	2.314 ms [2.243 ms, 2.384 ms]	833.781 µs (56.3%)
profiling	2.539 ms [2.325 ms, 2.754 ms]	1.06 ms (71.6%)
tracing	2.075 ms [2.021 ms, 2.13 ms]	595.634 µs (40.3%)

Execution time for biojava

gantt
title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~60b247f323, baseline=1.61.0-SNAPSHOT~30e798a214
dateFormat X
axisFormat %s
section baseline
no_agent (15.526 s) : 15526000, 15526000
. : milestone, 15526000,
appsec (15.115 s) : 15115000, 15115000
. : milestone, 15115000,
iast (18.602 s) : 18602000, 18602000
. : milestone, 18602000,
iast_GLOBAL (17.863 s) : 17863000, 17863000
. : milestone, 17863000,
profiling (15.221 s) : 15221000, 15221000
. : milestone, 15221000,
tracing (15.033 s) : 15033000, 15033000
. : milestone, 15033000,
section candidate
no_agent (15.465 s) : 15465000, 15465000
. : milestone, 15465000,
appsec (14.7 s) : 14700000, 14700000
. : milestone, 14700000,
iast (18.417 s) : 18417000, 18417000
. : milestone, 18417000,
iast_GLOBAL (17.961 s) : 17961000, 17961000
. : milestone, 17961000,
profiling (14.74 s) : 14740000, 14740000
. : milestone, 14740000,
tracing (14.924 s) : 14924000, 14924000
. : milestone, 14924000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.526 s [15.526 s, 15.526 s]	-
appsec	15.115 s [15.115 s, 15.115 s]	-411.0 ms (-2.6%)
iast	18.602 s [18.602 s, 18.602 s]	3.076 s (19.8%)
iast_GLOBAL	17.863 s [17.863 s, 17.863 s]	2.337 s (15.1%)
profiling	15.221 s [15.221 s, 15.221 s]	-305.0 ms (-2.0%)
tracing	15.033 s [15.033 s, 15.033 s]	-493.0 ms (-3.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.465 s [15.465 s, 15.465 s]	-
appsec	14.7 s [14.7 s, 14.7 s]	-765.0 ms (-4.9%)
iast	18.417 s [18.417 s, 18.417 s]	2.952 s (19.1%)
iast_GLOBAL	17.961 s [17.961 s, 17.961 s]	2.496 s (16.1%)
profiling	14.74 s [14.74 s, 14.74 s]	-725.0 ms (-4.7%)
tracing	14.924 s [14.924 s, 14.924 s]	-541.0 ms (-3.5%)

jordan-wong assigned jordan-wong and PerfectSlayer and unassigned PerfectSlayer Mar 27, 2026

jordan-wong closed this Mar 27, 2026

jordan-wong deleted the eval/feign-v2-10.8 branch March 27, 2026 13:48

jordan-wong mentioned this pull request Jun 23, 2026
feat(feign-10.8): toolkit-generated feign HTTP client instrumentation #11709
Open

Navigation Menu

Uh oh!

eval: feign-core 10.8 instrumentation (skill v2, sync+async, Run #6) - #10980

eval: feign-core 10.8 instrumentation (skill v2, sync+async, Run #6)#10980
jordan-wong wants to merge 1 commit into
masterfrom
eval/feign-v2-10.8

jordan-wong commented Mar 27, 2026

Uh oh!

pr-commenterBot commented Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jordan-wong commented Mar 27, 2026

Evaluation Run #6 - Skill v2 Validation

Layer 1 Results: 6/6 Pass ✅

Critical Rules: ALL FIXED ✅

Evidence

Files Structure

Verdict

Uh oh!

pr-commenterBot commented Mar 27, 2026

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants