Skip to content

Implement 64-bit trace ID system with double-buffered storage and liveness tracking - #262

Merged
jbachorik merged 8 commits into
mainfrom
jb/liveness_1
Sep 8, 2025
Merged

Implement 64-bit trace ID system with double-buffered storage and liveness tracking#262
jbachorik merged 8 commits into
mainfrom
jb/liveness_1

Conversation

@jbachorik

@jbachorikjbachorik commented Aug 18, 2025

Copy link
Copy Markdown
Collaborator

What does this PR do?:

This PR implements a liveness-aware double-buffered call trace storage system with several key improvements:

  1. Liveness-aware trace management with selective preservation across JFR dumps
  2. Contention handling with dropped trace visibility in JFR output
  3. Double-buffered storage with active/standby hash table instances
  4. 64-bit trace ID system with instance-based collision avoidance
  5. Modular hash table architecture with dedicated CallTraceHashTable class

Motivation:

The changes address critical issues in call trace management:

  • Random CI test failures due to lock contention causing dropped samples without visibility
  • Liveness tracking requirements for preserving traces of live objects across garbage collection
  • Trace ID stability needed for consistent liveness tracking across storage swaps
  • Performance and modularity improvements through specialized hash table implementation

Additional Notes:

Key Features:

Liveness-Aware Storage:

  • Callback-based liveness checker registration system
  • Selective trace preservation during storage transitions
  • Coordinated trace collection through processTraces() method
  • Support for multiple concurrent liveness checkers

Contention Handling:

  • Special dropped trace with reserved ID (1ULL) for contention visibility
  • <dropped due to contention> shown in JFR stack traces instead of null entries
  • Platform-specific ASGCT_CallFrame alignment using LP64_ONLY macro
  • BCI_ERROR routing for proper native method resolution

Double-Buffered Architecture:

  • Active/standby hash table pattern for lock-free JFR operations
  • Instance-based trace IDs: (instance_id << 32) | slot preventing collisions
  • Atomic storage swapping with minimal profiling overhead
  • Thread-safe instance ID generation across storage transitions

Hash Table Improvements:

  • Extracted dedicated CallTraceHashTable class (441 lines)
  • Concurrent table expansion with proper synchronization
  • Overflow trace handling for hash table limits
  • Lock-free put operations with retry-based contention handling

Implementation Details:

Core Storage Refactoring:

  • CallTraceStorage reduced from 265→142 lines through hash table extraction
  • Dual active/standby storage instances with atomic swapping
  • Liveness preservation system integrated with JFR dump cycles

64-bit Trace ID Migration:

  • Updated all profiling interfaces: recordJVMTISample(), recordSample(), recordDeferredSample()
  • Modified LivenessTracker for 64-bit trace ID handling
  • JFR integration updated for 64-bit trace ID constant pool support
  • Instance-based ID generation preventing cross-storage collisions

Platform Compatibility:

  • COMMA macro factored to arch_dd.h for consistent designated initializer syntax
  • LP64_ONLY macro usage for proper 64-bit platform struct alignment
  • Cross-platform ASGCT_CallFrame structure handling

New Files:

  • callTraceHashTable.{h,cpp} - Dedicated hash table implementation (441 lines)
  • test_callTraceStorage.cpp - Comprehensive unit tests with liveness scenarios (387 lines)
  • LivenessTrackingTest.java - Java integration test for end-to-end validation (246 lines)
  • ContendedCallTraceStorageTest.java - Contention measurement and validation test (249 lines)

Modified Files:

  • Core profiler: profiler.{h,cpp}, objectSampler.cpp, wallClock.{h,cpp} - 64-bit trace ID adoption
  • Storage: callTraceStorage.{h,cpp} - major refactoring with liveness integration
  • JFR: flightRecorder.{h,cpp} - 64-bit trace ID support and dropped trace handling
  • Liveness: livenessTracker.{h,cpp} - 64-bit trace ID migration
  • Architecture: arch_dd.h - COMMA macro consolidation

How to test the change?:

# Run comprehensive test suite
./gradlew testDebug
# C++ unit tests for storage and liveness
./gradlew gtestDebug # Build verification across configurations
./gradlew buildDebug buildRelease
# Code formatting
./gradlew spotlessApply

The implementation includes extensive test coverage:

  • 9 C++ unit tests for CallTraceStorage liveness scenarios
  • Java integration tests for end-to-end liveness tracking
  • Contention measurement and validation tests
  • Platform-specific compatibility tests

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.
  • This PR doesn't touch any of that.
  • JIRA: PROF-12316

Summary: +1689 lines, -447 lines (net +1239 lines)

This implementation provides a robust foundation for liveness-aware profiling with clear visibility into contention issues while maintaining high performance through lock-free operations and efficient storage management.

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (2)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (2)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (2)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (2)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (2)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (2)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (2)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (2)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (2)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (2)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (2)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (2)

Warnings (8)

Style Violations (299)

1 similar comment
@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (2)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (1)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (3)

Warnings (8)

Style Violations (299)

1 similar comment
@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (3)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (3)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (3)

Warnings (8)

Style Violations (299)

@jbachorik
jbachorik marked this pull request as ready for review August 21, 2025 17:24
@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (3)

Warnings (8)

Style Violations (299)

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements a comprehensive 64-bit trace ID system with double-buffered storage and liveness tracking for call traces. The changes enhance call trace management by adding liveness-aware preservation across JFR dumps, contention handling with visibility into dropped traces, and a modular hash table architecture with instance-based collision avoidance.

  • Adds liveness-aware double-buffered storage with selective trace preservation
  • Implements 64-bit trace ID system with instance-based collision avoidance
  • Introduces contention handling with dropped trace visibility in JFR output
  • Extracts dedicated CallTraceHashTable class for improved modularity

Reviewed Changes

Copilot reviewed 21 out of 22 changed files in this pull request and generated 6 comments.

Show a summary per file
FileDescription
LivenessTrackingTest.javaComprehensive Java integration test for end-to-end liveness tracking validation
TagContextTest.javaEnhanced test with dropped sample tracking and counter validation improvements
ContendedCallTraceStorageTest.javaNew test for measuring and validating contention in CallTraceStorage operations
test_callTraceStorage.cppExtensive C++ unit tests covering liveness scenarios and concurrent operations
wallClock.h/cppUpdated copyright headers and 64-bit trace ID migration
thread.hUpdated trace ID fields and methods to use 64-bit values
profiler.h/cppMajor refactoring for 64-bit trace IDs and liveness checker integration
objectSampler.cppUpdated copyright and 64-bit trace ID adoption
livenessTracker.h/cppEnhanced with 64-bit trace IDs and self-registration with profiler
flightRecorder.h/cppUpdated for 64-bit trace ID support and improved trace processing
counters.hAdded new counter for tracking dropped traces
callTraceStorage.h/cppMajor refactoring with double-buffering and liveness integration
callTraceHashTable.h/cppNew dedicated hash table implementation extracted from storage
arch_dd.hAdded COMMA macro consolidation
CLAUDE.mdNew documentation file with project guidance and architecture details
Comments suppressed due to low confidence (1)

ddprof-lib/src/main/cpp/callTraceStorage.cpp:125

  • Copying the entire unordered_set could be expensive for large sets. Consider using std::move or passing the set by reference to avoid the copy overhead, especially since this is in the critical processTraces path.
 preserve_set = _preserve_set; // Copy the set for lock-free processing

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment threadddprof-lib/src/test/cpp/test_callTraceStorage.cpp
Comment threadddprof-lib/src/main/cpp/callTraceHashTable.cpp
Comment threadddprof-lib/src/main/cpp/callTraceStorage.cpp
Comment threadddprof-lib/src/main/cpp/callTraceHashTable.cpp
@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (3)

Warnings (8)

Style Violations (299)

2 similar comments
@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (3)

Warnings (8)

Style Violations (299)

@dd-octo-sts

Copy link
Copy Markdown
Contributor

CppCheck Report

Errors (3)

Warnings (8)

Style Violations (299)

@zhengyu123zhengyu123 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the second parameters for __atomic_compare_exchange_n() were used correctly. They should be values, not addresses to the value. Please check

Comment threadddprof-lib/src/main/cpp/callTraceHashTable.cpp Outdated
Comment threadddprof-lib/src/main/cpp/callTraceHashTable.h
Comment threadddprof-lib/src/main/cpp/callTraceStorage.cpp
jbachorikand others added 7 commits September 5, 2025 11:16
Major architectural changes:
- Replace monolithic CallTraceStorage with double-buffered hash table design
- Add CallTraceHashTable with lock-free concurrent access and instance-based trace IDs
- Implement liveness tracking system to preserve active traces across JFR dumps
- Add dropped trace handling for lock contention with proper JFR integration
Key features:
- 64-bit trace IDs combining instance ID and slot for collision avoidance
- Split-lock strategy minimizing exclusive lock time during trace collection
- Platform-specific ASGCT_CallFrame alignment using LP64_ONLY macro
- Comprehensive test coverage including contention and liveness scenarios
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
…essTrackingTest.java
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…TraceStorageTest.java
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@jbachorik
jbachorik merged commit fab8fdf into mainSep 8, 2025
182 of 183 checks passed
@jbachorik
jbachorik deleted the jb/liveness_1 branch September 8, 2025 07:15
@github-actionsgithub-actionsBot added this to the 1.32.0 milestone Sep 8, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@jbachorik@zhengyu123