Instrument native threads by zhengyu123 · Pull Request #308 · DataDog/java-profiler

zhengyu123 · 2025-12-09T20:40:29Z

What does this PR do?:
Instrument/profile native threads, the threads that are created/started outside of JVM, on hotspot/non-musl based JVM.

Motivation:
Enhance Java profiler to profiler native threads.

Additional Notes:
The feature is now only enabled for hotspot based JVM running on non-musl Linux platform. The reasons:

A crash seen on aarch64/musl /JDK11. It might not be related to this change, but it is hard to confirm. Disable the feature for musl based Linux for now.
J9 has issues to walk native only thread stack in release build, it shows only
.no_java_frame
while debug build shows correct stack.

How to test the change?:

Regular tests
JDK tier1 tests with profiler agent. Although, there are failures, but match java-profiler main line.
There are failures that are expected:
- HeapMonitor uses agent, which conflicts with profiler agent
- Compiler frame is not compatible with agent.

For Datadog employees:

If this PR touches code that signs or publishes builds or packages, or handles
credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.
This PR doesn't touch any of that.
JIRA: PROF-11577

Unsure? Have a question? Request a review!

jbachorik

Having the libraries patching tied to ctimer (CPU profiling) does not sound right.
We have a bunch of other engines that can be used either for CPU or wallclock profiling and their functionality would be very inconsistent.

I think the library patching should go to a more generic place - and also be called from a more generic place, like eg. profiler.cpp

zhengyu123 · 2025-12-12T13:21:01Z

Having the libraries patching tied to ctimer (CPU profiling) does not sound right. We have a bunch of other engines that can be used either for CPU or wallclock profiling and their functionality would be very inconsistent.
I think the library patching should go to a more generic place - and also be called from a more generic place, like eg. profiler.cpp

Agree. But I want to limit the changes in this PR and address this issue in separate PR.

jbachorik · 2025-12-12T13:38:38Z

Having the libraries patching tied to ctimer (CPU profiling) does not sound right. We have a bunch of other engines that can be used either for CPU or wallclock profiling and their functionality would be very inconsistent.

I think the library patching should go to a more generic place - and also be called from a more generic place, like eg. profiler.cpp

Agree. But I want to limit the changes in this PR and address this issue in separate PR.

Ok. But create a followup ticket for that, plz. And let's move to the proper placement asap - we really don't want to get this partial task accidentally released.

jbachorik · 2025-12-12T13:39:37Z

How will this work with dynamically loaded libraries? Am I reading the code right that we will not patch those?

jbachorik

Approved with the two followup tickets

pr-commenter · 2025-12-12T18:05:56Z

Benchmarks [x86_64 wall]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	off	off
cpu	off	off
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	off	off
modes	wall	wall
wall	on	on

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 15 metrics, 23 unstable metrics.

pr-commenter · 2025-12-12T18:06:19Z

Benchmarks [x86_64 cpu]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	off	off
cpu	on	on
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	off	off
modes	cpu	cpu
wall	off	off

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 14 metrics, 24 unstable metrics.

pr-commenter · 2025-12-12T18:06:25Z

Benchmarks [x86_64 memleak]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	off	off
cpu	off	off
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	on	on
modes	memleak	memleak
wall	off	off

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 16 metrics, 22 unstable metrics.

pr-commenter · 2025-12-12T18:06:34Z

Benchmarks [x86_64 cpu,wall]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	off	off
cpu	on	on
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	off	off
modes	cpu,wall	cpu,wall
wall	on	on

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 15 metrics, 23 unstable metrics.

pr-commenter · 2025-12-12T18:06:48Z

Benchmarks [x86_64 alloc]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	on	on
cpu	off	off
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	off	off
modes	alloc	alloc
wall	off	off

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 15 metrics, 23 unstable metrics.

pr-commenter · 2025-12-12T18:06:51Z

Benchmarks [aarch64 wall]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	off	off
cpu	off	off
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	off	off
modes	wall	wall
wall	on	on

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 18 metrics, 20 unstable metrics.

pr-commenter · 2025-12-12T18:07:20Z

Benchmarks [aarch64 cpu]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	off	off
cpu	on	on
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	off	off
modes	cpu	cpu
wall	off	off

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 19 metrics, 19 unstable metrics.

pr-commenter · 2025-12-12T18:07:37Z

Benchmarks [x86_64 cpu,wall,alloc,memleak]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	on	on
cpu	on	on
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	on	on
modes	cpu,wall,alloc,memleak	cpu,wall,alloc,memleak
wall	on	on

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 14 metrics, 24 unstable metrics.

pr-commenter · 2025-12-12T18:08:22Z

Benchmarks [x86_64 memleak,alloc]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	on	on
cpu	off	off
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	on	on
modes	memleak,alloc	memleak,alloc
wall	off	off

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 14 metrics, 24 unstable metrics.

pr-commenter · 2025-12-12T18:08:58Z

Benchmarks [aarch64 memleak,alloc]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	on	on
cpu	off	off
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	on	on
modes	memleak,alloc	memleak,alloc
wall	off	off

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 18 metrics, 20 unstable metrics.

pr-commenter · 2025-12-12T18:09:11Z

Benchmarks [aarch64 alloc]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	on	on
cpu	off	off
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	off	off
modes	alloc	alloc
wall	off	off

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 16 metrics, 22 unstable metrics.

pr-commenter · 2025-12-12T18:09:33Z

Benchmarks [aarch64 cpu,wall]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	off	off
cpu	on	on
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	off	off
modes	cpu,wall	cpu,wall
wall	on	on

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 18 metrics, 20 unstable metrics.

pr-commenter · 2025-12-12T18:09:58Z

Benchmarks [aarch64 memleak]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	off	off
cpu	off	off
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	on	on
modes	memleak	memleak
wall	off	off

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 17 metrics, 21 unstable metrics.

pr-commenter · 2025-12-12T18:10:03Z

Benchmarks [aarch64 cpu,wall,alloc,memleak]

Parameters

	Baseline	Candidate
config	baseline	candidate
ddprof	1.34.4	1.35.0-zgu_inst_native_thread-SNAPSHOT

See matching parameters

	Baseline	Candidate
alloc	on	on
cpu	on	on
iterations	5	5
java	"11.0.28"	"11.0.28"
memleak	on	on
modes	cpu,wall,alloc,memleak	cpu,wall,alloc,memleak
wall	on	on

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 16 metrics, 22 unstable metrics.

zhengyu123 added 13 commits December 9, 2025 15:06

Instrument native threads
74158e5

Fix
07e480d

J9 stack walk
db6afef

Cleanup
ba40661

Fix bug
3ac6223

Set cstack=dwarf for NativeThreadTest (J9)
0bdded8

Hotspot/Zing only
48354c2

Fix
617ab9a

More fix
b445304

Exclude J9 from NativeThreadTest
77aaedb

Fix
60d8350

Handle musl
8cdabce

Exclude Musl from NativeThreadTest
36fe9b1

zhengyu123 marked this pull request as ready for review December 11, 2025 16:21

zhengyu123 requested review from jbachorik and r1viollet December 11, 2025 16:26

cleanup
5d46345

jbachorik reviewed Dec 11, 2025
View reviewed changes

Comment threadddprof-lib/src/main/cpp/ctimer_linux.cpp

jbachorik requested changes Dec 11, 2025
View reviewed changes

Remove duplicated check
7774a2c

Merge branch 'main' into zgu/inst_native_thread
826dfd5

DataDog deleted a comment from zhengyu123 Dec 12, 2025

jbachorik approved these changes Dec 12, 2025
View reviewed changes

jbachorikand others added 2 commits December 15, 2025 12:21

Merge branch 'main' into zgu/inst_native_thread
566d77c

Restore test parameter
d76aba9

zhengyu123 merged commit a598383 into mainDec 15, 2025
96 checks passed

zhengyu123 deleted the zgu/inst_native_thread branch December 15, 2025 15:31

github-actionsBot added this to the 1.35.0 milestone Dec 15, 2025

Uh oh!

Conversation

zhengyu123 commented Dec 9, 2025• edited by atlassian BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jbachorik left a comment

Choose a reason for hiding this comment

Uh oh!

zhengyu123 commented Dec 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbachorik commented Dec 12, 2025

Uh oh!

jbachorik commented Dec 12, 2025

Uh oh!

jbachorik left a comment

Choose a reason for hiding this comment

Uh oh!

pr-commenterBot commented Dec 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [x86_64 wall]

Parameters

Summary

Uh oh!

pr-commenterBot commented Dec 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [x86_64 cpu]

Parameters

Summary

Uh oh!

pr-commenterBot commented Dec 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [x86_64 memleak]

Parameters

Summary

Uh oh!

pr-commenterBot commented Dec 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [x86_64 cpu,wall]

Parameters

Summary

Uh oh!

pr-commenterBot commented Dec 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [x86_64 alloc]

Parameters

Summary

Uh oh!

pr-commenterBot commented Dec 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [aarch64 wall]

Parameters

Summary

Uh oh!

pr-commenterBot commented Dec 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [aarch64 cpu]

Parameters

Summary

Uh oh!

pr-commenterBot commented Dec 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [x86_64 cpu,wall,alloc,memleak]

Parameters

Summary

Uh oh!

pr-commenterBot commented Dec 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [x86_64 memleak,alloc]

Parameters

Summary

Uh oh!

pr-commenterBot commented Dec 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [aarch64 memleak,alloc]

Parameters

Summary

Uh oh!

pr-commenterBot commented Dec 12, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

zhengyu123 commented Dec 9, 2025•
edited by atlassian Bot
Loading

zhengyu123 commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading

pr-commenterBot commented Dec 12, 2025•
edited
Loading