Skip to content

feat(profiler): native malloc/socket stack capture with profiler-hook frame skipping - #648

Merged
jbachorik merged 14 commits into
mainfrom
jb/malloc_profiles
Jul 23, 2026
Merged

feat(profiler): native malloc/socket stack capture with profiler-hook frame skipping#648
jbachorik merged 14 commits into
mainfrom
jb/malloc_profiles

Conversation

@jbachorik

@jbachorikjbachorik commented Jul 10, 2026•
edited by atlassian Bot
Loading

Copy link
Copy Markdown
Collaborator

What does this PR do?:

Adds native stack capture for malloc (BCI_NATIVE_MALLOC) and native-socket (BCI_NATIVE_SOCKET) samples, stripping profiler-internal hook frames so only real caller frames remain — across all cstack modes (fp/dwarf/vm/vmx). Also renames the JFR event to datadog.NativeMemoryAllocation.

  • Surfaces silent hook-symbol mark-resolution failures via new diagnostic counters and a warning log instead of silently no-opping (markAsyncProfilerHook now returns bool; NativeSocketSampler::start() checks it).
  • Adds a counter for when the MARK_JAVA_PROFILER hook boundary is never found in a callchain, so a fully-discarded/empty native stack is observable rather than silent.
  • Fixes convertNativeTrace's MARK_JAVA_PROFILER reset to gate on the immutable skip_hook_prefix parameter instead of the mutable skipping flag, so a second boundary-marked frame later in the same callchain is still treated as a reset point (matches walkVM's per-frame dispatch).
  • Restores the MARK_COMPILER_ENTRY dispatch branch in walkVM that had been dropped without updating the comp_task flag/docs that still reference it.
  • Drops the now-redundant is_marked field on NativeFrameResolution in favor of a computed mark != 0 accessor, and updates the stale doc comments describing per-mark dispatch semantics.
  • Introduces a single isHookPrefixedSample() helper (BCI and EventType overloads) used at all 4 call sites that previously hardcoded the "is this a malloc/socket sample" predicate independently.
  • Adds stack-trace assertions to NativeSocketStackTraceTest proving a real caller frame (not a profiler-internal hook frame) is present after unwinding.
  • Renames the profiler.Malloc JFR event to datadog.NativeMemoryAllocation and moves it into the Datadog/Profiling JFR category.

Motivation:

Native malloc/socket samples currently include profiler-internal hook frames (and, in some cstack modes, drop the real caller entirely), which pollutes symbolication and makes the resulting stacks hard to read.

Additional Notes:

The restored MARK_COMPILER_ENTRY dispatch is currently unreachable: _features.comp_task is never set to 1 anywhere in the codebase (no CLI flag wires it), so this fix restores doc/dead-code consistency but doesn't change observable behavior. Revisiting that separately.

How to test the change?:

./gradlew assembleDebugJar builds clean; ./gradlew compileTestJava compiles clean. The new/updated assertions in NativeSocketStackTraceTest are parameterized across vm/vmx/dwarf/fp cstack modes and cover the fixed code paths.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-15461

Unsure? Have a question? Request a review!

jbachorikand others added 5 commits July 10, 2026 16:48
@jbachorik
Rename profiler.Malloc JFR event to datadog.NativeMemoryAllocation; f…
60efe7b
…ix missing native frames
The event name/category did not follow the datadog.* convention used by other
events. Separately, Profiler::getNativeTrace unconditionally skipped native
stack unwinding for BCI_NATIVE_MALLOC under the default cstack setting,
leaving only Java frames in malloc call stacks.
@jbachorik
Fix missing native frames for BCI_NATIVE_SOCKET under default cstack
29e859d
Same guard in Profiler::getNativeTrace that was skipping native unwinding
for BCI_NATIVE_MALLOC also applied to BCI_NATIVE_SOCKET; both are non-CPU/WALL
event types and were bypassed when cstack is left at its default.
@jbachorik
Skip profiler-internal hook frames in native malloc/socket call stacks
d2d6893
Marks the socket hook wrapper symbols (send/recv/write/read_hook) with
MARK_ASYNC_PROFILER, mirroring the existing malloc hook marking, and
teaches convertNativeTrace to discard frames captured before reaching
that mark instead of truncating the stack there, so only the real
caller frames remain for BCI_NATIVE_MALLOC/BCI_NATIVE_SOCKET traces.
@jbachorik
Skip profiler-internal hook frames in walkVM (cstack=vm/vmx) too
df9be61
Adds SOCKET_SAMPLE to EventType so BCI_NATIVE_SOCKET gets its own
identity in walkVM, and fixes a bug where the generic is_marked
check in HotspotSupport::walkVM terminated the scan on ANY mark
before the MARK_ASYNC_PROFILER dispatch below it could ever run,
making the malloc hook-skip logic dead code. NativeFrameResolution
now carries the actual mark value so walkVM can dispatch correctly:
MARK_ASYNC_PROFILER resets depth and resumes past the hook boundary
for malloc/socket samples (matching the FP/DWARF path), other marks
still terminate the scan as before.
@jbachorik@claude
Address review findings on malloc/socket hook-skip stackwalking
002040c
Surface silent hook-mark resolution failures via counters/logging, make
convertNativeTrace's MARK_ASYNC_PROFILER reset gate on skip_hook_prefix
instead of the mutable skipping flag, restore the dropped
MARK_COMPILER_ENTRY dispatch in walkVM, drop the now-redundant is_marked
field, de-duplicate the hook-prefixed-sample predicate, and add stack
trace assertions for the socket hook boundary skip.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dd-octo-sts

dd-octo-stsBot commented Jul 10, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

CI Test Results

Run:#30001306855 | Commit:cc8a1af | Duration: 14m 42s (longest job)

✅ All 32 test jobs passed

Status Overview

JDKglibc-aarch64/debugglibc-amd64/debugmusl-aarch64/debugmusl-amd64/debug
8-✅--
8-ibm-✅--
8-j9✅✅--
8-librca--✅✅
8-orcl-✅--
11-✅--
11-j9✅✅--
11-librca--✅✅
17✅✅--
17-graal✅✅--
17-j9✅✅--
17-librca--✅✅
21✅✅--
21-graal✅✅--
21-librca--✅✅
25✅✅--
25-graal✅✅--
25-librca--✅✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0

Updated: 2026-07-23 11:15:14 UTC

@jbachorikjbachorik changed the title Address review findings on malloc/socket hook-skip stackwalkingfix(profiler): complete malloc/native-socket hook-prefix-skip stackwalkingJul 10, 2026
@jbachorik

jbachorik commented Jul 10, 2026

Copy link
Copy Markdown
CollaboratorAuthor

@codex review

@chatgpt-codex-connector

chatgpt-codex-connectorBot commented Jul 10, 2026

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit:002040ca88

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-datadog-prod-us1

This comment has been minimized.

Sign in to view
@dd-octo-sts

dd-octo-stsBot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 15f3132)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124275785 Commit: 15f31325ea71609000dcd76a830ddf0329e2fd58

⚠️ Significant outliers

  • 🟢 future-genetic (JDK 21): runtime -3% (2113→2049 ms)
  • 🔴 future-genetic (JDK 25): runtime +2.5% (2070→2122 ms)
Runtime details (per benchmark × JDK)
BenchmarkJDKLatestDevΔ (dev vs latest)Issues L/D
akka-uct21✅ 10394 ms (21 iters)✅ 10298 ms (21 iters)≈ -0.9% (±11.4%)— / —
akka-uct25✅ 8826 ms (24 iters)✅ 8991 ms (24 iters)≈ +1.9% (±10.3%)— / —
finagle-chirper21✅ 6006 ms (33 iters)✅ 5950 ms (33 iters)≈ -0.9% (±24.8%)⚠️ W:3 / ⚠️ W:3
finagle-chirper25✅ 5496 ms (36 iters)✅ 5422 ms (36 iters)≈ -1.3% (±23.9%)⚠️ W:3 / ⚠️ W:3
fj-kmeans21✅ 2714 ms (69 iters)✅ 2695 ms (69 iters)≈ -0.7% (±2.8%)— / —
fj-kmeans25✅ 2810 ms (67 iters)✅ 2810 ms (66 iters)≈ 0% (±2.6%)— / —
future-genetic21✅ 2113 ms (88 iters)✅ 2049 ms (91 iters)🟢 -3%— / —
future-genetic25✅ 2070 ms (90 iters)✅ 2122 ms (87 iters)🔴 +2.5%— / —
naive-bayes21✅ 1253 ms (136 iters)✅ 1275 ms (134 iters)≈ +1.8% (±32.9%)— / —
naive-bayes25✅ 982 ms (174 iters)✅ 1016 ms (168 iters)≈ +3.5% (±32.7%)— / —
reactors21✅ 16060 ms (15 iters)✅ 16114 ms (15 iters)≈ +0.3% (±8.8%)— / —
reactors25✅ 18976 ms (15 iters)✅ 18213 ms (15 iters)≈ -4% (±4%)— / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

BenchmarkJDKDropped recDropped jvmtiDropped traceSkipped WCAGCT failUnwind fail
akka-uct21✅ / ✅✅ / ✅1 / 32044 / 2050✅ / ✅✅ / ✅
akka-uct25✅ / ✅✅ / ✅1 / 62480 / 2333✅ / ✅✅ / ✅
finagle-chirper21✅ / ✅✅ / ✅4 / 28280 / 8924✅ / ✅✅ / ✅
finagle-chirper25✅ / ✅✅ / ✅1 / ✅8266 / 8237✅ / ✅✅ / ✅
fj-kmeans21✅ / ✅✅ / ✅✅ / 21276 / 1261✅ / ✅✅ / ✅
fj-kmeans25✅ / ✅✅ / ✅✅ / 21288 / 1276✅ / ✅✅ / ✅
future-genetic21✅ / ✅✅ / ✅3 / 23037 / 3032✅ / ✅✅ / ✅
future-genetic25✅ / ✅✅ / ✅✅ / ✅2854 / 2928✅ / ✅✅ / ✅
naive-bayes21✅ / ✅✅ / ✅3 / 43486 / 3500✅ / ✅✅ / ✅
naive-bayes25✅ / ✅✅ / ✅6 / 43445 / 3496✅ / ✅✅ / ✅
reactors21✅ / ✅✅ / ✅1 / 11539 / 1646✅ / ✅✅ / ✅
reactors25✅ / ✅✅ / ✅1 / 11889 / 1884✅ / ✅✅ / ✅

@dd-octo-sts

dd-octo-stsBot commented Jul 13, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Reliability & Chaos Results

✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/124386433

@dd-octo-sts

dd-octo-stsBot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit d27a19e)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124386442 Commit: d27a19ec12a09006024dbeecb5582c766645c31a

⚠️ Significant outliers

  • 🔴 fj-kmeans (JDK 21): runtime +5.3% (2644→2783 ms)
  • 🔴 future-genetic (JDK 21): runtime +4.2% (2063→2149 ms)
  • 🔴 future-genetic (JDK 25): runtime +3.4% (2024→2093 ms)
  • 🟢 reactors (JDK 21): runtime -8.3% (17474→16021 ms)
Runtime details (per benchmark × JDK)
BenchmarkJDKLatestDevΔ (dev vs latest)Issues L/D
akka-uct21✅ 10361 ms (21 iters)✅ 10194 ms (21 iters)≈ -1.6% (±11.5%)— / —
akka-uct25✅ 8822 ms (24 iters)✅ 8881 ms (24 iters)≈ +0.7% (±10%)— / —
finagle-chirper21✅ 5917 ms (33 iters)✅ 6021 ms (33 iters)≈ +1.8% (±25.6%)⚠️ W:3 / ⚠️ W:3
finagle-chirper25✅ 5469 ms (36 iters)✅ 5514 ms (36 iters)≈ +0.8% (±24.8%)⚠️ W:4 / ⚠️ W:3
fj-kmeans21✅ 2644 ms (72 iters)✅ 2783 ms (67 iters)🔴 +5.3%— / —
fj-kmeans25✅ 2843 ms (66 iters)✅ 2848 ms (66 iters)≈ +0.2% (±2.6%)— / —
future-genetic21✅ 2063 ms (90 iters)✅ 2149 ms (86 iters)🔴 +4.2%— / —
future-genetic25✅ 2024 ms (92 iters)✅ 2093 ms (88 iters)🔴 +3.4%— / —
naive-bayes21✅ 1269 ms (135 iters)✅ 1297 ms (132 iters)≈ +2.2% (±32.6%)— / —
naive-bayes25✅ 1014 ms (169 iters)✅ 1009 ms (169 iters)≈ -0.5% (±31.9%)— / —
reactors21✅ 17474 ms (15 iters)✅ 16021 ms (15 iters)🟢 -8.3%— / —
reactors25✅ 18483 ms (15 iters)✅ 18350 ms (15 iters)≈ -0.7% (±5.2%)— / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

BenchmarkJDKDropped recDropped jvmtiDropped traceSkipped WCAGCT failUnwind fail
akka-uct21✅ / ✅✅ / ✅2 / ✅2004 / 2048✅ / ✅✅ / ✅
akka-uct25✅ / ✅✅ / ✅4 / 22226 / 2232✅ / ✅✅ / ✅
finagle-chirper21✅ / ✅✅ / ✅3 / 17951 / 8195✅ / ✅✅ / ✅
finagle-chirper25✅ / ✅✅ / ✅1 / 18519 / 8278✅ / ✅✅ / ✅
fj-kmeans21✅ / ✅✅ / ✅3 / ✅1280 / 1269✅ / ✅✅ / ✅
fj-kmeans25✅ / ✅✅ / ✅✅ / 41289 / 1281✅ / ✅✅ / ✅
future-genetic21✅ / ✅✅ / ✅4 / ✅2931 / 2913✅ / ✅✅ / ✅
future-genetic25✅ / ✅✅ / ✅1 / 32772 / 2826✅ / ✅✅ / ✅
naive-bayes21✅ / ✅✅ / ✅4 / 23493 / 3499✅ / ✅✅ / ✅
naive-bayes25✅ / ✅✅ / ✅5 / 13516 / 3479✅ / ✅✅ / ✅
reactors21✅ / ✅✅ / ✅1 / 21786 / 1703✅ / ✅✅ / ✅
reactors25✅ / ✅✅ / ✅2 / ✅1880 / 1796✅ / ✅✅ / ✅

@dd-octo-sts

dd-octo-stsBot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit bc3ec7d)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124681069 Commit: bc3ec7ded2a9859b04949499d441938a666f0f69

⚠️ Significant outliers

  • 🔴 future-genetic (JDK 25): runtime +3.7% (2003→2077 ms)
Runtime details (per benchmark × JDK)
BenchmarkJDKLatestDevΔ (dev vs latest)Issues L/D
akka-uct21✅ 10229 ms (21 iters)✅ 10257 ms (21 iters)≈ +0.3% (±11.7%)— / —
akka-uct25✅ 8894 ms (24 iters)✅ 8893 ms (24 iters)≈ -0% (±10.2%)— / —
finagle-chirper21✅ 5935 ms (33 iters)✅ 5986 ms (33 iters)≈ +0.9% (±25.8%)⚠️ W:3 / ⚠️ W:4
finagle-chirper25✅ 5454 ms (36 iters)✅ 5481 ms (36 iters)≈ +0.5% (±24.4%)⚠️ W:3 / ⚠️ W:3
fj-kmeans21✅ 2749 ms (68 iters)✅ 2766 ms (67 iters)≈ +0.6% (±2.7%)— / —
fj-kmeans25✅ 2803 ms (66 iters)✅ 2832 ms (66 iters)≈ +1% (±2.6%)— / —
future-genetic21✅ 2048 ms (90 iters)✅ 2070 ms (89 iters)≈ +1.1% (±2.6%)— / —
future-genetic25✅ 2003 ms (92 iters)✅ 2077 ms (89 iters)🔴 +3.7%— / —
naive-bayes21✅ 1249 ms (137 iters)✅ 1286 ms (133 iters)≈ +3% (±33.7%)— / —
naive-bayes25✅ 1002 ms (171 iters)✅ 1016 ms (168 iters)≈ +1.4% (±31.4%)— / —
reactors21✅ 16793 ms (15 iters)✅ 16897 ms (15 iters)≈ +0.6% (±6.4%)— / —
reactors25✅ 18363 ms (15 iters)✅ 18376 ms (15 iters)≈ +0.1% (±4.4%)— / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

BenchmarkJDKDropped recDropped jvmtiDropped traceSkipped WCAGCT failUnwind fail
akka-uct21✅ / ✅✅ / ✅1 / 31991 / 2023✅ / ✅✅ / ✅
akka-uct25✅ / ✅✅ / ✅1 / 22163 / 2366✅ / ✅✅ / ✅
finagle-chirper21✅ / ✅✅ / ✅4 / 48457 / 8255✅ / ✅✅ / ✅
finagle-chirper25✅ / ✅✅ / ✅2 / 18831 / 8311✅ / ✅✅ / ✅
fj-kmeans21✅ / ✅✅ / ✅4 / 11296 / 1255✅ / ✅✅ / ✅
fj-kmeans25✅ / ✅✅ / ✅1 / 41260 / 1271✅ / ✅✅ / ✅
future-genetic21✅ / ✅✅ / ✅1 / 22950 / 2924✅ / ✅✅ / ✅
future-genetic25✅ / ✅✅ / ✅3 / ✅2839 / 2915✅ / ✅✅ / ✅
naive-bayes21✅ / ✅✅ / ✅2 / 103524 / 3505✅ / ✅✅ / ✅
naive-bayes25✅ / ✅✅ / ✅5 / 13471 / 3494✅ / ✅✅ / ✅
reactors21✅ / ✅✅ / ✅1 / 11701 / 1691✅ / ✅✅ / ✅
reactors25✅ / ✅✅ / ✅✅ / 31839 / 1836✅ / ✅✅ / ✅

CopilotAI review requested due to automatic review settings July 21, 2026 08:31
CopilotAI reviewed Jul 21, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@dd-octo-sts

dd-octo-stsBot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit d061e9a)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125889277 Commit: d061e9a685c18fe6b73f71f1d08aa19ca3a3e232

⚠️ Significant outliers

  • 🟢 future-genetic (JDK 21): runtime -4.1% (2140→2052 ms)
  • 🟢 future-genetic (JDK 25): runtime -3.6% (2049→1975 ms)
Runtime details (per benchmark × JDK)
BenchmarkJDKLatestDevΔ (dev vs latest)Issues L/D
akka-uct21✅ 10242 ms (21 iters)✅ 10393 ms (21 iters)≈ +1.5% (±11.7%)— / —
akka-uct25✅ 8872 ms (24 iters)✅ 8832 ms (24 iters)≈ -0.5% (±10.5%)— / —
finagle-chirper25✅ 5503 ms (36 iters)✅ 5447 ms (36 iters)≈ -1% (±24%)⚠️ W:4 / ⚠️ W:3
fj-kmeans21✅ 2758 ms (67 iters)✅ 2745 ms (68 iters)≈ -0.5% (±2.7%)— / —
fj-kmeans25✅ 2829 ms (66 iters)✅ 2827 ms (66 iters)≈ -0.1% (±2.6%)— / —
future-genetic21✅ 2140 ms (87 iters)✅ 2052 ms (91 iters)🟢 -4.1%— / —
future-genetic25✅ 2049 ms (92 iters)✅ 1975 ms (94 iters)🟢 -3.6%— / —
naive-bayes25✅ 1034 ms (165 iters)✅ 1024 ms (167 iters)≈ -1% (±31.6%)— / —
reactors21✅ 16809 ms (15 iters)✅ 15992 ms (15 iters)≈ -4.9% (±6.9%)— / —
reactors25✅ 18306 ms (15 iters)✅ 18481 ms (15 iters)≈ +1% (±4.9%)— / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

BenchmarkJDKDropped recDropped jvmtiDropped traceSkipped WCAGCT failUnwind fail
akka-uct21✅ / ✅✅ / ✅2 / 11967 / 2019✅ / ✅✅ / ✅
akka-uct25✅ / ✅✅ / ✅6 / 32358 / 2103✅ / ✅✅ / ✅
finagle-chirper21✅ / ✅✅ / ✅2 / 38340 / 8558✅ / ✅✅ / ✅
finagle-chirper25✅ / ✅✅ / ✅✅ / 28331 / 8123✅ / ✅✅ / ✅
fj-kmeans21✅ / ✅✅ / ✅✅ / 51256 / 1255✅ / ✅✅ / ✅
fj-kmeans25✅ / ✅✅ / ✅2 / 51279 / 1251✅ / ✅✅ / ✅
future-genetic21✅ / ✅✅ / ✅✅ / 42965 / 3033✅ / ✅✅ / ✅
future-genetic25✅ / ✅✅ / ✅2 / 12898 / 2829✅ / ✅✅ / ✅
naive-bayes21✅ / ✅✅ / ✅3 / 23491 / 3504✅ / ✅✅ / ✅
naive-bayes25✅ / ✅✅ / ✅4 / 53500 / 3493✅ / ✅✅ / ✅
reactors21✅ / ✅✅ / ✅2 / 11755 / 1655✅ / ✅✅ / ✅
reactors25✅ / ✅✅ / ✅✅ / 11897 / 1883✅ / ✅✅ / ✅

CopilotAI review requested due to automatic review settings July 21, 2026 09:35
CopilotAI reviewed Jul 21, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@dd-octo-sts

dd-octo-stsBot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit d669181)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125904596 Commit: d669181182c19fdab334ff0e4bf35be3b6956285

⚠️ Significant outliers

  • 🔴 future-genetic (JDK 25): runtime +3.9% (2041→2120 ms)
Runtime details (per benchmark × JDK)
BenchmarkJDKLatestDevΔ (dev vs latest)Issues L/D
akka-uct21✅ 10272 ms (21 iters)✅ 10242 ms (21 iters)≈ -0.3% (±11.1%)— / —
akka-uct25✅ 8962 ms (24 iters)✅ 8803 ms (24 iters)≈ -1.8% (±10.7%)— / —
finagle-chirper21✅ 6007 ms (33 iters)✅ 5965 ms (33 iters)≈ -0.7% (±25%)⚠️ W:3 / ⚠️ W:3
finagle-chirper25✅ 5466 ms (36 iters)✅ 5520 ms (36 iters)≈ +1% (±24.4%)⚠️ W:3 / ⚠️ W:3
fj-kmeans21✅ 2710 ms (69 iters)✅ 2648 ms (71 iters)≈ -2.3% (±2.6%)— / —
fj-kmeans25✅ 2830 ms (66 iters)✅ 2821 ms (66 iters)≈ -0.3% (±2.6%)— / —
future-genetic21✅ 2117 ms (88 iters)✅ 2159 ms (86 iters)≈ +2% (±2.7%)— / —
future-genetic25✅ 2041 ms (91 iters)✅ 2120 ms (88 iters)🔴 +3.9%— / —
naive-bayes21✅ 1268 ms (135 iters)✅ 1234 ms (137 iters)≈ -2.7% (±32%)— / —
naive-bayes25✅ 1022 ms (167 iters)✅ 1008 ms (170 iters)≈ -1.4% (±30.9%)— / —
reactors21✅ 15764 ms (15 iters)✅ 16111 ms (16 iters)≈ +2.2% (±7.7%)— / —
reactors25✅ 18555 ms (15 iters)✅ 18451 ms (15 iters)≈ -0.6% (±4.3%)— / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

BenchmarkJDKDropped recDropped jvmtiDropped traceSkipped WCAGCT failUnwind fail
akka-uct21✅ / ✅✅ / ✅4 / 21947 / 1993✅ / ✅✅ / ✅
akka-uct25✅ / ✅✅ / ✅3 / 22303 / 2295✅ / ✅✅ / ✅
finagle-chirper21✅ / ✅✅ / ✅4 / 48615 / 8656✅ / ✅✅ / ✅
finagle-chirper25✅ / ✅✅ / ✅1 / ✅8520 / 8206✅ / ✅✅ / ✅
fj-kmeans21✅ / ✅✅ / ✅✅ / 31277 / 1247✅ / ✅✅ / ✅
fj-kmeans25✅ / ✅✅ / ✅✅ / ✅1271 / 1266✅ / ✅✅ / ✅
future-genetic21✅ / ✅✅ / ✅4 / ✅2980 / 2989✅ / ✅✅ / ✅
future-genetic25✅ / ✅✅ / ✅1 / 32909 / 2987✅ / ✅✅ / ✅
naive-bayes21✅ / ✅✅ / ✅8 / 43535 / 3496✅ / ✅✅ / ✅
naive-bayes25✅ / ✅✅ / ✅4 / 23506 / 3503✅ / ✅✅ / ✅
reactors21✅ / ✅✅ / ✅✅ / ✅1593 / 1651✅ / ✅✅ / ✅
reactors25✅ / ✅✅ / ✅✅ / ✅1885 / 1816✅ / ✅✅ / ✅

@jbachorik
jbachorik marked this pull request as ready for review July 21, 2026 10:42
@jbachorik
jbachorik requested a review from a team as a code ownerJuly 21, 2026 10:42
CopilotAI review requested due to automatic review settings July 21, 2026 10:42
chatgpt-codex-connector[bot]
chatgpt-codex-connectorBot reviewed Jul 21, 2026
View reviewed changes

@chatgpt-codex-connectorchatgpt-codex-connectorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:fb32d57629

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment threadddprof-lib/src/main/cpp/jfrMetadata.cpp
@dd-octo-sts

dd-octo-stsBot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit fb32d57)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125918579 Commit: fb32d576296e73162858a157c7a1eb934f06b5db

⚠️ Significant outliers

  • 🟢 fj-kmeans (JDK 21): runtime -2.8% (2766→2688 ms)
  • 🔴 future-genetic (JDK 21): runtime +2.7% (2067→2123 ms)
  • 🟢 future-genetic (JDK 25): runtime -3.4% (2074→2004 ms)
Runtime details (per benchmark × JDK)
BenchmarkJDKLatestDevΔ (dev vs latest)Issues L/D
akka-uct21✅ 10282 ms (21 iters)✅ 10328 ms (21 iters)≈ +0.4% (±11.6%)— / —
akka-uct25✅ 8786 ms (24 iters)✅ 8868 ms (24 iters)≈ +0.9% (±10.4%)— / —
finagle-chirper21✅ 5939 ms (33 iters)✅ 5975 ms (33 iters)≈ +0.6% (±25.3%)⚠️ W:4 / ⚠️ W:3
finagle-chirper25✅ 5421 ms (36 iters)✅ 5493 ms (36 iters)≈ +1.3% (±24.8%)⚠️ W:3 / ⚠️ W:3
fj-kmeans21✅ 2766 ms (68 iters)✅ 2688 ms (70 iters)🟢 -2.8%— / —
fj-kmeans25✅ 2834 ms (66 iters)✅ 2832 ms (66 iters)≈ -0.1% (±2.6%)— / —
future-genetic21✅ 2067 ms (89 iters)✅ 2123 ms (88 iters)🔴 +2.7%— / —
future-genetic25✅ 2074 ms (89 iters)✅ 2004 ms (93 iters)🟢 -3.4%— / —
naive-bayes21✅ 1248 ms (136 iters)✅ 1273 ms (134 iters)≈ +2% (±33%)— / —
naive-bayes25✅ 1028 ms (167 iters)✅ 952 ms (179 iters)≈ -7.4% (±30.3%)— / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

BenchmarkJDKDropped recDropped jvmtiDropped traceSkipped WCAGCT failUnwind fail
akka-uct21✅ / ✅✅ / ✅5 / 31946 / 2044✅ / ✅✅ / ✅
akka-uct25✅ / ✅✅ / ✅2 / ✅2395 / 2209✅ / ✅✅ / ✅
finagle-chirper21✅ / ✅✅ / ✅5 / 38382 / 8478✅ / ✅✅ / ✅
finagle-chirper25✅ / ✅✅ / ✅✅ / 18226 / 8413✅ / ✅✅ / ✅
fj-kmeans21✅ / ✅✅ / ✅✅ / 21277 / 1284✅ / ✅✅ / ✅
fj-kmeans25✅ / ✅✅ / ✅1 / 41299 / 1263✅ / ✅✅ / ✅
future-genetic21✅ / ✅✅ / ✅1 / 42868 / 2926✅ / ✅✅ / ✅
future-genetic25✅ / ✅✅ / ✅6 / ✅2850 / 2904✅ / ✅✅ / ✅
naive-bayes21✅ / ✅✅ / ✅8 / 33509 / 3466✅ / ✅✅ / ✅
naive-bayes25✅ / ✅✅ / ✅3 / 63497 / 3466✅ / ✅✅ / ✅
reactors25✅ / ✅✅ / ✅1 / 21903 / 1862✅ / ✅✅ / ✅

@rkennke

rkennke commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Before we start reviewing, I think the PR title/description need to be brought in line with what actually lands on main when this merges. Three asks:

  1. Re-scope the description to the whole effort, not just the last commit. The current text reads as a follow-up that "closes the gaps before the branch merges," but nothing here is on main yet — the entire feature lands at once: the datadog.NativeMemoryAllocation JFR event rename, native frame capture for BCI_NATIVE_MALLOC/BCI_NATIVE_SOCKET under default cstack, the hook-prefix-skip logic (FP/DWARF + walkVM), and the review fixups. The description should let a reviewer understand all of that, not just the fixup commit.

  2. Reconsider the fix(...) prefix. Net effect on main is introducing native malloc/socket stack capture, so feat(profiler): is more accurate for the changelog. The "fix" framing is only true relative to intermediate commits on the branch.

  3. Update stale terminology. The tip commit renames MARK_ASYNC_PROFILER → MARK_JAVA_PROFILER across the code, but the description still refers to MARK_ASYNC_PROFILER throughout — it's now stale against the branch's own final commit.

The bullet-by-bullet breakdown itself is great; this is just a re-scope + rename, not a rewrite.

@jbachorik
jbachorik marked this pull request as draft July 21, 2026 11:18
@jbachorik

jbachorik commented Jul 21, 2026

Copy link
Copy Markdown
CollaboratorAuthor

Sorry, wanted to click on 'Update branch' and did 'Ready for review' before it was actually ready :/

@jbachorikjbachorik changed the title fix(profiler): complete malloc/native-socket hook-prefix-skip stackwalkingNative malloc/socket stackwalking fixes, fault injection, and ProfiledThread TLS refactorJul 21, 2026
@jbachorikjbachorik changed the title Native malloc/socket stackwalking fixes, fault injection, and ProfiledThread TLS refactorfix(profiler): complete malloc/native-socket hook-prefix-skip stackwalkingJul 21, 2026
@jbachorik
jbachorik marked this pull request as ready for review July 21, 2026 11:44
@jbachorik

jbachorik commented Jul 21, 2026

Copy link
Copy Markdown
CollaboratorAuthor

@rkennke The PR should be ready now

@rkennke

rkennke commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Thanks for the JIRA link and the event-rename bullet. Two of the three asks are still open though, and both are quick — here's concrete text so it's copy-paste:

Ask #2 — title prefix. Change:

fix(profiler): complete malloc/native-socket hook-prefix-skip stackwalking

to something like:

feat(profiler): native malloc/socket stack capture with profiler-hook frame skipping

(net effect on main is introducing the feature, not fixing a shipped regression).

Ask #3 — stale marker name. The tip commit renames MARK_ASYNC_PROFILER → MARK_JAVA_PROFILER in the code, but the description still says MARK_ASYNC_PROFILER in three bullets (the counter bullet, the convertNativeTrace reset bullet). A find-replace of MARK_ASYNC_PROFILER → MARK_JAVA_PROFILER across the description fixes it.

Ask #1 — framing (partially done). The new event-rename bullet helps, but the opening still reads as a follow-up. Suggest reframing the intro from:

"Fixes up gaps in the malloc/native-socket hook-prefix-skip stackwalking logic introduced earlier on this branch ... which was incomplete"

to something that leads with what the PR delivers, e.g.:

"Adds native stack capture for malloc (BCI_NATIVE_MALLOC) and native-socket (BCI_NATIVE_SOCKET) samples, stripping profiler-internal hook frames so only real caller frames remain — across all cstack modes (fp/dwarf/vm/vmx). Also renames the JFR event to datadog.NativeMemoryAllocation."

...then keep the existing bullet list as the detailed breakdown. Same content, just scoped to what lands on main rather than to the intermediate commits.

Happy to start the review as soon as #2 and #3 are in.

@jbachorikjbachorik changed the title fix(profiler): complete malloc/native-socket hook-prefix-skip stackwalkingfeat(profiler): native malloc/socket stack capture with profiler-hook frame skippingJul 21, 2026
@jbachorik

jbachorik commented Jul 21, 2026

Copy link
Copy Markdown
CollaboratorAuthor

@rkennke 🤞 all PR desc changes are in

rkennke
rkennke reviewed Jul 21, 2026
View reviewed changes

@rkennkerkennke left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the net diff against main. The logic is sound — I traced both unwinding paths and the central fix is real: on main, walkVM's if (resolution.is_marked) break; ran before the mark-dispatch chain, making the whole MARK_* dispatch dead code (every marked frame terminated the scan). The new per-mark dispatch with else { break; } fixes that while keeping VM_RUNTIME/INTERPRETER as terminators. convertNativeTrace gating the reset on the immutable skip_hook_prefix is correct and now matches walkVM. The socket test is a genuine positive+negative assertion (real caller present, no *_hook leak), parameterized across cstack modes.

One intentional behavior change worth flagging in the description: the walkVM MARK_JAVA_PROFILER path now does depth = 0; goto dwarf_unwind;, which skips the fillFrame that previously emitted the hook frame — so the hook frame is now dropped for malloc under vm/vmx too (not just socket). Deliberate and consistent, just wire-visible for existing malloc consumers.

Inline notes below — all minor; #1 and #3 are the ones I'd actually action.

Comment threadddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp
Comment threadddprof-lib/src/main/cpp/profiler.cpp
Comment threadddprof-lib/src/main/cpp/nativeSocketSampler.cpp Outdated
Comment threadddprof-lib/src/main/cpp/codeCache.h Outdated
Comment threadddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp
Comment threadddprof-lib/src/main/cpp/profiler.h Outdated
Comment threadddprof-lib/src/main/cpp/nativeSocketSampler.cpp
hooks_marked &= markAsyncProfilerHook((void*)&NativeSocketSampler::recv_hook);
hooks_marked &= markAsyncProfilerHook((void*)&NativeSocketSampler::write_hook);
hooks_marked &= markAsyncProfilerHook((void*)&NativeSocketSampler::read_hook);
if (!hooks_marked) {

@rkennkerkennkeJul 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coverage gap (acceptable): nothing exercises this failure branch or the new NATIVE_HOOK_MARK_RESOLVE_FAILED / NATIVE_TRACE_HOOK_PREFIX_NOT_FOUND counters. Genuinely hard to unit-test, so just noting it rather than requesting a test.

@jbachorik@claude
Address PR review feedback: mirror hook-boundary counter in walkVM, r…
ce3a047
…ename markAsyncProfilerHook, fix stale comment
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CopilotAI review requested due to automatic review settings July 22, 2026 10:12
CopilotAI reviewed Jul 22, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@jbachorik@claude
Document mark field value domain in NativeFrameResolution
8e3bb25
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CopilotAI review requested due to automatic review settings July 22, 2026 10:20
CopilotAI reviewed Jul 22, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@datadog-datadog-prod-us1

datadog-datadog-prod-us1Bot commented Jul 22, 2026•
edited by datadog-datadog-us1-prod Bot
Loading

Copy link
Copy Markdown
Contributor

View session in Datadog

Bits Code status: ✅ Done

Comment @DataDog to request changes

CopilotAI review requested due to automatic review settings July 23, 2026 10:57
CopilotAI reviewed Jul 23, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@dd-octo-sts

dd-octo-stsBot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 6aff489)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126483699 Commit: 6aff489f66b90a05ec29fef2663640305b6e109a

⚠️ Significant outliers

  • 🟢 future-genetic (JDK 25): runtime -2.7% (2057→2001 ms)
Runtime details (per benchmark × JDK)
BenchmarkJDKLatestDevΔ (dev vs latest)Issues L/D
akka-uct21✅ 10345 ms (21 iters)✅ 10172 ms (21 iters)≈ -1.7% (±10.9%)— / —
akka-uct25✅ 8909 ms (24 iters)✅ 8975 ms (24 iters)≈ +0.7% (±10.8%)— / —
finagle-chirper21✅ 5938 ms (33 iters)✅ 5947 ms (33 iters)≈ +0.2% (±25.3%)⚠️ W:3 / ⚠️ W:3
finagle-chirper25💥 0 ms (0 iters)✅ 5551 ms (35 iters)—— / ⚠️ W:4
fj-kmeans21✅ 2722 ms (68 iters)✅ 2775 ms (67 iters)≈ +1.9% (±2.7%)— / —
fj-kmeans25✅ 2837 ms (66 iters)✅ 2845 ms (66 iters)≈ +0.3% (±2.6%)— / —
future-genetic21✅ 2092 ms (88 iters)✅ 2120 ms (87 iters)≈ +1.3% (±2.7%)— / —
future-genetic25✅ 2057 ms (90 iters)✅ 2001 ms (93 iters)🟢 -2.7%— / —
naive-bayes21✅ 1302 ms (132 iters)✅ 1315 ms (130 iters)≈ +1% (±32.6%)— / —
naive-bayes25✅ 1030 ms (166 iters)✅ 1047 ms (164 iters)≈ +1.7% (±31.6%)— / —
reactors21✅ 16328 ms (15 iters)✅ 16273 ms (15 iters)≈ -0.3% (±7.5%)— / —
reactors25✅ 18661 ms (15 iters)✅ 19023 ms (15 iters)≈ +1.9% (±5.6%)— / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

BenchmarkJDKDropped recDropped jvmtiDropped traceSkipped WCAGCT failUnwind fail
akka-uct21✅ / ✅✅ / ✅✅ / 42020 / 1966✅ / ✅✅ / ✅
akka-uct25✅ / ✅✅ / ✅3 / 22268 / 2076✅ / ✅✅ / ✅
finagle-chirper21✅ / ✅✅ / ✅✅ / ✅✅ / ✅✅ / ✅✅ / ✅
finagle-chirper25· / ✅· / ✅· / ✅· / 8906· / ✅· / ✅
fj-kmeans21✅ / ✅✅ / ✅2 / 31253 / 1260✅ / ✅✅ / ✅
fj-kmeans25✅ / ✅✅ / ✅2 / 21316 / 1284✅ / ✅✅ / ✅
future-genetic21✅ / ✅✅ / ✅2 / 13018 / 3033✅ / ✅✅ / ✅
future-genetic25✅ / ✅✅ / ✅1 / 12861 / 2837✅ / ✅✅ / ✅
naive-bayes21✅ / ✅✅ / ✅5 / 63545 / 3501✅ / ✅✅ / ✅
naive-bayes25✅ / ✅✅ / ✅5 / 33482 / 3556✅ / ✅✅ / ✅
reactors21✅ / ✅✅ / ✅1 / 11594 / 1750✅ / ✅✅ / ✅
reactors25✅ / ✅✅ / ✅1 / 21997 / 1951✅ / ✅✅ / ✅

rkennke
rkennke approved these changes Jul 23, 2026
View reviewed changes

@rkennkerkennke left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, looks good to me. Thanks!

@jbachorik
jbachorik merged commit 4e9bc17 into mainJul 23, 2026
105 checks passed
@jbachorik
jbachorik deleted the jb/malloc_profiles branch July 23, 2026 15:24
@github-actionsgithub-actionsBot added this to the 1.48.0 milestone Jul 23, 2026
kaahos pushed a commit that referenced this pull request Jul 24, 2026
@jbachorik@kaahos
feat(profiler): native malloc/socket stack capture with profiler-hook…
86e6bef
… frame skipping (#648)
@kaahoskaahos mentioned this pull request Jul 28, 2026
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@chatgpt-codex-connectorchatgpt-codex-connector[bot]chatgpt-codex-connector[bot] left review comments
@rkennkerkennkerkennke approved these changes

Assignees

No one assigned

Labels

AI

Projects

None yet

Milestone

1.48.0

Development

Successfully merging this pull request may close these issues.

3 participants

@jbachorik@rkennke