Remap thread id to avoid bitmap contention

[zhengyu123]

What does this PR do?:

An application starts a few threads in short period of window, the newly created threads are likely to have consecutive or close thread ids. When ThreadFilter maps them to the bits on bitmap, there are high chances that they may be mapped to a single cache line, that results cache line contention when updating the bitmap concurrently.

The PR purposes to remap thread id by reversing lower 2 bytes of thread id, so that consecutive/close thread ids map to different cache lines.

The reason for only reversing lower 2 bytes, is that, thread id is remapped within the same bitmap.

Motivation:

Reduce cache line contention, improve performance.

Additional Notes:

How to test the change?:

For Datadog employees:

• If this PR touches code that signs or publishes builds or packages, or handles
  credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.
• This PR doesn't touch any of that.
• JIRA: PROF-12002

Unsure? Have a question? Request a review!

[github-actions]

🔧 Report generated by pr-comment-cppcheck

CppCheck Report

Errors (2)

• There is an unknown macro here somewhere. Configuration is required. If LP64_ONLY is a macro then please configure it.
• syntax error

Warnings (4)

• The class 'RecordingBuffer' defines member function with name 'flushIfNeeded' also defined in its parent class 'Buffer'.
• Member variable 'JitWriteProtection::_restore' is not initialized in the constructor.
• Member variable 'JitWriteProtection::_restore' is not initialized in the constructor.
• Member variable 'ElfParser::_vaddr_diff' is not initialized in the constructor.

Style Violations (295)

• Class 'Arguments' has a constructor with 1 argument that is not explicit.
• struct member 'Multiplier::symbol' is never used.
• struct member 'Multiplier::multiplier' is never used.
• class member 'Error::OK' is never used.
• Class 'AsyncSampleMutex' has a constructor with 1 argument that is not explicit.
• The class 'NativeFunc' does not declare a constructor although it has private member variables which likely require initialization.
• Local variable 'count' shadows outer function
• Local variable 'blob' shadows outer function
• Local variable 'i' shadows outer variable
• Parameter 'prefixes' can be declared as reference to const
• Variable 'cb1' can be declared as pointer to const
• Variable 'cb2' can be declared as pointer to const
• Variable 'fd1' can be declared as pointer to const
• Variable 'fd2' can be declared as pointer to const
• Parameter 'name' can be declared as pointer to const
• Consider using std::transform algorithm instead of a raw loop.
• struct member 'ContextPage::capacity' is never used.
• struct member 'ContextPage::storage' is never used.
• class member 'Contexts::_max_pages' is never used.
• class member 'Contexts::_pages' is never used.
• The function 'check' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'registerThread' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'unregisterThread' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'name' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'interval' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'check' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'stop' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'enableEvents' overrides a function in a base class but is not marked with a 'override' specifier.
• class member 'CTimer::_cstack' is never used.
• class member 'CTimer::_signal' is never used.
• class member 'CTimer::_max_timers' is never used.
• class member 'CTimer::_timers' is never used.
• class member 'Shims::_tid_setter_ref' is never used.
• Class 'Dictionary' has a constructor with 1 argument that is not explicit.
• struct member 'DictRow::keys' is never used.
• struct member 'DictRow::next' is never used.
• Operator '|' with one operand equal to zero is redundant.
• Operator '|' with one operand equal to zero is redundant.
• Operator '|' with one operand equal to zero is redundant.
• Local variable 'table' shadows outer function
• Local variable 'pc_off' shadows outer variable
• Variable 'table' can be declared as pointer to const
• struct member 'FrameDesc::fp_off' is never used.
• struct member 'FrameDesc::pc_off' is never used.
• class member 'DwarfParser::_name' is never used.
• class member 'DwarfParser::_image_base' is never used.
• class member 'DwarfParser::_capacity' is never used.
• class member 'DwarfParser::_prev' is never used.
• class member 'DwarfParser::_data_align' is never used.
• Class 'WallClockEpochEvent' has a constructor with 1 argument that is not explicit.
• class member 'LockEvent::_address' is never used.
• class member 'LockEvent::_timeout' is never used.
• The function 'name' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'interval' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'check' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'stop' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'enableEvents' overrides a function in a base class but is not marked with a 'override' specifier.
• class member 'ITimer::_cstack' is never used.
• The class 'J9WallClock' does not declare a constructor although it has private member variables which likely require initialization.
• The function 'name' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'interval' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'stop' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'enableEvents' overrides a function in a base class but is not marked with a 'override' specifier.
• class member 'J9WallClock::_max_stack_depth' is never used.
• class member 'J9WallClock::_running' is never used.
• Class 'Element' has a constructor with 1 argument that is not explicit.
• Class 'NoField' has a constructor with 1 argument that is not explicit.
• The function 'is_nofield' overrides a function in a base class but is not marked with a 'override' specifier.
• class member 'JfrMetadata::_initialized' is never used.
• class member 'JVM::_is_readable_pointer_resolved' is never used.
• class member 'JVM::_is_readable_pointer' is never used.
• The class 'VirtualSpaceSummary' does not declare a constructor although it has private member variables which likely require initialization.
• The class 'GCHeapSummary' does not declare a constructor although it has private member variables which likely require initialization.
• The class 'CollectedHeapWrapper' does not declare a constructor although it has private member variables which likely require initialization.
• class member 'VirtualSpaceSummary::_committed_end' is never used.
• class member 'GCHeapSummary::vptr' is never used.
• class member 'CollectedHeapWrapper::vptr' is never used.
• class member 'CollectedHeapWrapper::_gc_heap_log' is never used.
• class member 'CollectedHeapWrapper::_capacity_at_last_gc' is never used.
• class member 'CollectedHeapWrapper::_used_at_last_gc' is never used.
• struct member 'Chunk::prev' is never used.
• struct member 'Chunk::offs' is never used.
• struct member 'Chunk::_padding' is never used.
• class member 'LinearAllocator::_chunk_size' is never used.
• class member 'LinearAllocator::_tail' is never used.
• class member 'LinearAllocator::_reserve' is never used.
• Class 'SpinLock' has a constructor with 1 argument that is not explicit.
• struct member 'TrackingEntry::skipped' is never used.
• class member 'LivenessTracker::MAX_TRACKING_TABLE_SIZE' is never used.
• class member 'LivenessTracker::MIN_SAMPLING_INTERVAL' is never used.
• class member 'LivenessTracker::_table_lock' is never used.
• class member 'Log::_file' is never used.
• class member 'Log::LEVEL_NAME' is never used.
• The function 'check' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'stop' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'interval' overrides a function in a base class but is not marked with a 'override' specifier.
• class member 'ObjectSampler::CONFIG_UPDATE_CHECK_PERIOD_SECS' is never used.
• class member 'JitWriteProtection::_restore' is never used.
• class member 'OS::page_size' is never used.
• class member 'OS::page_mask' is never used.
• The function 'rewind' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'next' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'size' overrides a function in a base class but is not marked with a 'override' specifier.
• Variable 'entry' can be declared as pointer to const
• The function 'rewind' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'next' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'size' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'check' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'enableEvents' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'registerThread' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'unregisterThread' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'interval' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'enableEvents' overrides a function in a base class but is identical to the overridden function
• The function 'registerThread' overrides a function in a base class but is identical to the overridden function
• The function 'unregisterThread' overrides a function in a base class but is identical to the overridden function
• The function 'check' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'stop' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'registerThread' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'unregisterThread' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'interval' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'name' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'enableEvents' overrides a function in a base class but is not marked with a 'override' specifier.
• class member 'PerfEvents::_max_events' is never used.
• class member 'PerfEvents::_events' is never used.
• class member 'PerfEvents::_event_type' is never used.
• class member 'PerfEvents::_ring' is never used.
• class member 'PerfEvents::_cstack' is never used.
• class member 'PerfEvents::_use_mmap_page' is never used.
• Class 'ReservoirSampler' has a constructor with 1 argument that is not explicit.
• Parameter 'ptr' can be declared as pointer to const
• Parameter 'ptr' can be declared as pointer to const. However it seems that 'load32' is a callback function, if 'ptr' is declared with const you might also need to cast function pointer(s).
• Class 'SymbolDesc' has a constructor with 1 argument that is not explicit.
• Class 'MemoryMapDesc' has a constructor with 1 argument that is not explicit.
• Local variable 'section' shadows outer function
• Local variable 'section' shadows outer function
• Local variable 'section' shadows outer function
• Variable 'ident' can be declared as pointer to const
• Parameter 'section' can be declared as pointer to const
• Variable 'checked' can be declared as pointer to const
• Variable 'pheader' can be declared as pointer to const
• Variable 'sym' can be declared as pointer to const
• Variable 'sym' can be declared as pointer to const
• Variable 'chain' can be declared as pointer to const
• Variable 'plt' can be declared as pointer to const
• Variable 'note' can be declared as pointer to const
• Variable 'sym' can be declared as pointer to const
• struct member 'RootSymbolEntry::name' is never used.
• struct member 'RootSymbolEntry::kind' is never used.
• class member 'ElfParser::_root_symbols' is never used.
• Class 'ExclusiveLock' has a constructor with 1 argument that is not explicit.
• Class 'SharedLock' has a constructor with 1 argument that is not explicit.
• Same expression on both sides of '-'.
• Local variable 'count' shadows outer function
• Parameter 'failures' can be declared as reference to const
• class member 'ProfiledThread::_buffer_size' is never used.
• class member 'ProfiledThread::_buffer' is never used.
• Local variable 'word' shadows outer function
• class member 'ThreadInfo::_thread_names' is never used.
• class member 'ThreadInfo::_thread_ids' is never used.
• Condition 'enabled()' is always false
• The function 'loadAcquire' is never used.
• The function 'message' is never used.
• The function 'hasOption' is never used.
• The function 'cpuSamplerInterval' is never used.
• The function 'acquired' is never used.
• The function 'offset' is never used.
• The function 'skip' is never used.
• The function 'put16' is never used.
• The function 'put64' is never used.
• The function 'putFloat' is never used.
• The function 'putVar64' is never used.
• The function 'findBlobByAddress' is never used.
• The function 'binarySearch' is never used.
• The function 'dump' is never used.
• The function 'findSymbolsByPrefix' is never used.
• The function 'findImport' is never used.
• The function 'patchImport' is never used.
• The function 'findFrameDesc' is never used.
• The function 'libIndex' is never used.
• The function 'isMarked' is never used.
• The function 'minAddress' is never used.
• The function 'maxAddress' is never used.
• The function 'contains' is never used.
• The function 'get' is never used.
• The function 'getPage' is never used.
• The function 'get_tag' is never used.
• The function 'getCounters' is never used.
• The function 'getCounter' is never used.
• The function 'describeCounters' is never used.
• The function 'units' is never used.
• The function 'check' is never used.
• The function 'start' is never used.
• The function 'supported' is never used.
• The function 'setSighandlerTid' is never used.
• The function 'bounded_lookup' is never used.
• The function 'stop' is never used.
• The function 'interval' is never used.
• The function 'enableEvents' is never used.
• The function 'updateCounter' is never used.
• The function 'registerThread' is never used.
• The function 'unregisterThread' is never used.
• The function 'hasChanged' is never used.
• The function 'updateNumSamplableThreads' is never used.
• The function 'updateNumSuccessfulSamples' is never used.
• The function 'updateNumFailedSamples' is never used.
• The function 'updateNumExitedThreads' is never used.
• The function 'updateNumPermissionDenied' is never used.
• The function 'endEpoch' is never used.
• The function 'clean' is never used.
• The function 'newEpoch' is never used.
• The function 'encode' is never used.
• The function 'decode' is never used.
• The function 'threadEntry' is never used.
• The function 'sampleIdleThreads' is never used.
• The function 'field' is never used.
• The function 'category' is never used.
• The function 'root' is never used.
• The function 'strings' is never used.
• The function 'jniExceptionCheck' is never used.
• The function 'is_readable_pointer' is never used.
• The function 'used' is never used.
• The function 'alloc' is never used.
• The function 'level' is never used.
• The function 'next' is never used.
• The function 'cputime' is never used.
• The function 'micros' is never used.
• The function 'processStartTime' is never used.
• The function 'sleep' is never used.
• The function 'ntoh64' is never used.
• The function 'threadId' is never used.
• The function 'schedPolicy' is never used.
• The function 'threadName' is never used.
• The function 'listThreads' is never used.
• The function 'isLinux' is never used.
• The function 'installSignalHandler' is never used.
• The function 'replaceSigsegvHandler' is never used.
• The function 'replaceSigbusHandler' is never used.
• The function 'sendSignalToThread' is never used.
• The function 'getCpuDescription' is never used.
• The function 'getProcessCpuTime' is never used.
• The function 'getTotalCpuTime' is never used.
• The function 'copyFile' is never used.
• The function 'truncateFile' is never used.
• The function 'fileSize' is never used.
• The function 'freePageCache' is never used.
• The function 'mallocArenaMax' is never used.
• The function 'walkKernel' is never used.
• The function 'resetBuffer' is never used.
• The function 'getEventName' is never used.
• The function 'compute' is never used.
• The function 'sample' is never used.
• The function 'is_probably_rust_legacy' is never used.
• The function 'demangle' is never used.
• The function 'skipLoad' is never used.
• The function 'skipLoadArg' is never used.
• The function 'tryLockShared' is never used.
• The function 'withinCurrentStack' is never used.
• The function 'restore' is never used.
• The function 'stackAt' is never used.
• The function 'clearParsingCaches' is never used.
• The function 'parseLibraries' is never used.
• The function 'isRootSymbol' is never used.
• The function 'forTid' is never used.
• The function 'inBuffer' is never used.
• The function 'tid' is never used.
• The function 'noteCPUSample' is never used.
• The function 'lookupWallclockCallTraceId' is never used.
• The function 'recordCallTraceId' is never used.
• The function 'enterCrashHandler' is never used.
• The function 'exitCrashHandler' is never used.
• The function 'resetCrashHandler' is never used.
• The function 'isDeepCrashHandler' is never used.
• The function 'unwindFailures' is never used.
• The function 'accept' is never used.
• The function 'remove' is never used.
• The function 'getThreadId' is never used.
• The function 'clearAll' is never used.
• The function 'updateThreadName' is never used.
• The function 'reportCounters' is never used.
• The function 'convertJvmExecutionState' is never used.
• The function 'initialized' is never used.
• The function 'ticks' is never used.
• The function 'frequency' is never used.
• The function 'ticks_to_millis' is never used.
• The function 'record' is never used.
• The function 'forEach' is never used.
• The function 'countFailures' is never used.
• The function 'collectAndReset' is never used.

[github-actions]

🔧 Report generated by pr-comment-scanbuild

[r1viollet]

Would the contention not remain if you only swap lower bits ?
I also thought of swapping all bits, however that would lead to other issues (fragmentation in the radix tree). Happy to talk this through. I'll bench it to see what the results have to say.

[r1viollet]

I think the proposal here is still performing better:

Operations/second/thread: 769,128

Versus this one:

Operations/second/thread: 576,013

As a first step can we deliver the bench in a separate PR to get it running in CI ? This way, we could get more stable results. My laptop has a lot of noise.

[r1viollet]

Also a different bench would be interesting.

[zhengyu123]

I think the proposal here is still performing better:

Operations/second/thread: 769,128

Versus this one:

Operations/second/thread: 576,013

As a first step can we deliver the bench in a separate PR to get it running in CI ? This way, we could get more stable results. My laptop has a lot of noise.

As discussed, I updated PR to reverse lower 32 bits that will further more distant 2 consecutive number to ensure they are not mapped to the same cache line.

[jbachorik]

How do the benchmarks look with the latest changes?
Otherwise, LGTM!

[zhengyu123]

JFR recordings from Erwan's benchmark:

Basline:

After patch:

[r1viollet]

This is faster than main, so LGTM
I would still need to check the #224 PR to understand the impact of JNI

[jbachorik]

Is this superseded by #229 ?
If yes, we should close this one.