Skip to content

fix: detached debug symbols from guix - #6971

Merged
PastaPastaPasta merged 6 commits into
dashpay:developfrom
PastaPastaPasta:fix/detached-debug-symbols
Nov 19, 2025
Merged

fix: detached debug symbols from guix#6971
PastaPastaPasta merged 6 commits into
dashpay:developfrom
PastaPastaPasta:fix/detached-debug-symbols

Conversation

@PastaPastaPasta

@PastaPastaPastaPastaPastaPasta commented Nov 13, 2025

Copy link
Copy Markdown
Member

Issue being fixed or feature implemented

The current debug symbols shipped are not sufficient for using tools like perf

What was done?

See changes in commits; debug artifacts are now ~1.2GB per, much bigger sadly.

How Has This Been Tested?

Following process worked:

✅ Perf + Docker + External Debug Tarball: Full Symbolization Checklist

  1. Extract the debug tarball

tar -xvf dashcore-*.debug.tar.gz

You should now have:
• dashd.dbg binaries
• /usr/lib/debug/.build-id/.../*.debug files

  1. Find the build-id of the running dashd

sudo readelf -n /proc/$(pidof dashd)/root/usr/local/bin/dashd | grep 'Build ID'

Example output:

Build ID: 4bec1dd76cd952657cbb3e8d3b9086bdfb651904

  1. Install the matching .debug file on the host

The .debug file must live at:

/usr/lib/debug/.build-id/4b/ec1dd76c...debug

So copy them:

sudo rsync -a dashcore-*/usr/lib/debug/.build-id/ /usr/lib/debug/.build-id/

  1. Ensure the .debug file has the correct perf symlink

Perf expects:

/usr/lib/debug/.build-id/4b/
/usr/lib/debug/.build-id/4b/.debug

Create the symlink if needed:

sudo ln -sf
/usr/lib/debug/.build-id/4b/.debug
/usr/lib/debug/.build-id/4b/

You already did this step.

  1. Copy the running binary itself to the host

Perf must see the exact same ELF that produced the build-id.

sudo mkdir -p /usr/local/bin
sudo cp /proc/$(pidof dashd)/root/usr/local/bin/dashd /usr/local/bin/dashd

  1. Register the binary into perf’s build-id cache

sudo perf buildid-cache -a /usr/local/bin/dashd

  1. Record + Report (dwarf mode recommended)

Record:

sudo perf record -F 99 -g --call-graph dwarf -p "$(pidof dashd)" -- sleep 30

Report with source lines:

sudo perf report --stdio -g srcline

You should now see:
• full symbol names
• file paths
• line numbers
• normal call chains

🎯 Summary

Whenever doing perf profiling inside Docker using external debug symbols:
1. Extract debug tarball
2. Find running binary’s build-id
3. Place matching .debug file under /usr/lib/debug/.build-id/
4. Create build-id symlink
5. Copy the running ELF binary to the host at the same path
6. Add binary to perf’s build-id cache
7. Run perf again — you now get full symbols

Breaking Changes

Checklist:

Go over all the following points, and put an x in all the boxes that apply.

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation
  • I have assigned this pull request to a milestone (for repository code-owners and collaborators only)

@github-actions

github-actionsBot commented Nov 13, 2025

Copy link
Copy Markdown

✅ No Merge Conflicts Detected

This PR currently has no conflicts with other open PRs.

@DashCoreAutoGuix

Copy link
Copy Markdown

Guix Automation has failed due to the HEAD commit not being signed by an authorized core-team member. Please rebase and sign or push a new empty signed commit to allow Guix build to happen.

@PastaPastaPasta
PastaPastaPastaforce-pushed the fix/detached-debug-symbols branch from e83b604 to ef44418CompareNovember 13, 2025 19:05
@DashCoreAutoGuix

Copy link
Copy Markdown

Guix Automation has began to build this PR tagged as v23.0.0-devpr6971.ef444188. A new comment will be made when the image is pushed.

- Added compression for DWARF sections in debug files.
- Implemented creation of .build-id tree for automatic discovery by perf.
- Added verification checks for build-ids and debug links in ELF files.
@DashCoreAutoGuix

Copy link
Copy Markdown

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.0.0-devpr6971.ef444188. The image should be on dockerhub soon.

@DashCoreAutoGuix

Copy link
Copy Markdown

Guix Automation has began to build this PR tagged as v23.0.0-devpr6971.38934351. A new comment will be made when the image is pushed.

@PastaPastaPasta

Copy link
Copy Markdown
MemberAuthor

for the first commit; --enable-debug guix builds compared to latest nightly have only 80bytes bigger shipped binary, and 64k bigger dash-qt; Negligible.

@PastaPastaPastaPastaPastaPasta changed the title fix: detached debug symbols from guisfix: detached debug symbols from guixNov 13, 2025
@DashCoreAutoGuix

Copy link
Copy Markdown

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.0.0-devpr6971.38934351. The image should be on dockerhub soon.

@DashCoreAutoGuix

Copy link
Copy Markdown

Guix Automation has began to build this PR tagged as v23.0.0-devpr6971.badabf97. A new comment will be made when the image is pushed.

@PastaPastaPasta

PastaPastaPasta commented Nov 14, 2025

Copy link
Copy Markdown
MemberAuthor

Second commit also appears to cause no increase in shipped binaries (or minimal like 80 bytes etc)

@DashCoreAutoGuix

Copy link
Copy Markdown

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.0.0-devpr6971.badabf97. The image should be on dockerhub soon.

@DashCoreAutoGuix

Copy link
Copy Markdown

Guix Automation has began to build this PR tagged as v23.0.0-devpr6971.48446f9d. A new comment will be made when the image is pushed.

@DashCoreAutoGuix

Copy link
Copy Markdown

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.0.0-devpr6971.48446f9d. The image should be on dockerhub soon.

@DashCoreAutoGuix

Copy link
Copy Markdown

Guix Automation has began to build this PR tagged as v23.0.0-devpr6971.857f860a. A new comment will be made when the image is pushed.

@DashCoreAutoGuix

Copy link
Copy Markdown

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.0.0-devpr6971.857f860a. The image should be on dockerhub soon.

@PastaPastaPastaPastaPastaPasta added this to the 23.1 milestone Nov 14, 2025
@PastaPastaPasta
PastaPastaPasta marked this pull request as ready for review November 15, 2025 16:22
@coderabbitai

coderabbitaiBot commented Nov 15, 2025

Copy link
Copy Markdown

Walkthrough

This pull request adds Linux-specific build-id linking and refines debug symbol handling. configure.ac appends -Wl,--build-id=sha1 for Linux targets (guarded by the existing LDFLAG_WERROR logic). contrib/devtools/split-debug.sh.in changes strip flags from -s to --strip-debug --strip-unneeded. contrib/guix/libexec/build.sh (Linux path) now compresses DWARF in .dbg files via objcopy, verifies ELF files have a Build ID and a .gnu_debuglink, builds a .build-id tree for perf discovery, and splits packaging into main and debug tarballs (excluding/including .dbg and the build-id tree accordingly).

Sequence Diagram(s)

sequenceDiagram
participant Builder as Builder (guix build)
participant Configure as configure.ac
participant Linker as Linker
participant Objcopy as objcopy
participant FS as Filesystem
participant Tar as Tarball packaging
Note over Configure,Linker: Linux link-time flag selection
Configure->>Linker: Add -Wl,--build-id=sha1 (when TARGET_OS=linux)
Linker-->>Builder: ELF binaries with Build ID
Builder->>Objcopy: Extract debug info -> produce .dbg
Objcopy->>Objcopy: --strip-debug --strip-unneeded (split-debug)
Objcopy->>Objcopy: Compress DWARF sections in .dbg
Objcopy-->>FS: .dbg files + stripped binaries
Builder->>Builder: Verify each ELF has Build ID and .gnu_debuglink
alt All verified
Builder->>FS: Create .build-id tree from .dbg files (perf discovery)
Builder->>Tar: Package main tarball (exclude .dbg + build-id tree)
Builder->>Tar: Package debug tarball (include .dbg + build-id tree)
else Any missing
Builder-->>Builder: Fail build with error
end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

  • Files needing closer attention:
    • configure.ac — ensure correct TARGET_OS detection and interaction with LDFLAG_WERROR
    • contrib/devtools/split-debug.sh.in — verify strip flag semantics and edge cases
    • contrib/guix/libexec/build.sh — validate ELF detection, Build ID and .gnu_debuglink checks, objcopy compression options, file permissions, .build-id tree creation, and tarball include/exclude patterns (Linux-only conditionals)

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)
Check nameStatusExplanation
Title check✅ PassedThe title accurately reflects the main change: detached debug symbols are being implemented in the guix build process.
Description check✅ PassedThe description comprehensively documents the issue, implementation approach, testing methodology, and provides detailed steps for using the new debug symbols with perf.
Docstring Coverage✅ PassedNo functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Tip

📝 Customizable high-level summaries are now available in beta!

You can now customize how CodeRabbit generates the high-level summary in your pull requests — including its content, structure, tone, and formatting.

  • Provide your own instructions using the high_level_summary_instructions setting.
  • Format the summary however you like (bullet lists, tables, contributor stats, etc.).
  • Use high_level_summary_in_walkthrough to move the summary from the description to the walkthrough section.

Example:

"Create a concise high-level summary as a bullet-point list. Then include a Markdown table showing lines added and removed by each contributing author."

Note: This feature is currently in beta for Pro-tier users, and pricing will be announced later.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Comment threadcontrib/guix/libexec/build.sh Outdated
Comment threadcontrib/guix/libexec/build.sh Outdated
Fix verification block to properly capture exit status and prevent
build from continuing on failure. Change objcopy command to use &&
instead of semicolon to ensure chmod only runs if mv succeeds.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>

@coderabbitaicoderabbitaiBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
contrib/guix/libexec/build.sh (1)

345-366: Minor: Silent failures in .build-id tree creation aren't caught by verification.

Lines 350–366 create the .build-id tree for perf auto-discovery. If mkdir -p or cp fail on lines 360–361, the loop continues silently. The verification phase (lines 367–396) only checks that binaries have a .gnu_debuglink section; it doesn't verify that the corresponding .debug file was successfully placed in the .build-id tree.

If the tree creation fails, perf still won't discover symbols, but the build succeeds. This is likely acceptable since the debug tarball can be extracted manually, but consider adding explicit error handling (e.g., cp ... || { echo "Error copying ..."; exit 1; }) if robustness is critical.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 857f860 and 81cd240.

📒 Files selected for processing (1)
  • contrib/guix/libexec/build.sh (2 hunks)
🧰 Additional context used
🧠 Learnings (1)
📓 Common learnings
Learnt from: kwvg
Repo: dashpay/dash PR: 6754
File: contrib/containers/guix/docker-compose.yml:18-19
Timestamp: 2025-07-14T10:11:05.011Z
Learning: In the Guix build process for Dash Core, the `guix.sigs` directory requires write access as signatures are written to it during the build process, and `dash-detached-sigs` may be updated with `git pull` operations, so both directories need rw permissions in the Docker volume mounts.
🔇 Additional comments (3)
contrib/guix/libexec/build.sh (3)

368-396: Verification block now correctly captures exit status.

The previous review flagged that the verification block ran in a subshell (due to the pipe) and the exit status wasn't properly propagated. This has been fixed: the exit status is now correctly captured via verification_status=$? after the command substitution, and the script exits with exit 1 if verification fails. Good work addressing the prior concern.


348-348: LGTM: Objcopy error handling is correct.

The && chaining between objcopy, mv, and chmod ensures that if any step fails, subsequent steps don't run. The previous review concern about error handling has been addressed.


435-446: Tarball separation logic is sound.

The main tarball (lines 436–440) correctly excludes *.dbg files and the /usr/lib/debug tree. The debug tarball (lines 442–446) correctly includes both via the find predicate \( -name "*.dbg" -o -path "${DISTNAME}/usr/lib/debug/*" \). The split is clean and deterministic.

@DashCoreAutoGuix

Copy link
Copy Markdown

Guix Automation has began to build this PR tagged as v23.0.0-devpr6971.81cd2400. A new comment will be made when the image is pushed.

@DashCoreAutoGuix

Copy link
Copy Markdown

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.0.0-devpr6971.81cd2400. The image should be on dockerhub soon.

@UdjinM6UdjinM6 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK 81cd240

@knstknst left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 81cd240

@PastaPastaPasta
PastaPastaPasta merged commit 86e84d7 into dashpay:developNov 19, 2025
51 of 61 checks passed
@PastaPastaPasta
PastaPastaPasta deleted the fix/detached-debug-symbols branch November 19, 2025 16:06
@kwvgkwvg mentioned this pull request Nov 20, 2025
5 tasks
PastaPastaPasta added a commit that referenced this pull request Nov 21, 2025
c359fb2 ci: bump to Clang 19 (Kittywhiskers Van Gogh)
9c9e876 partial bitcoin#33185: update time-machine to 5cb84f2013c5b1e48a7d0e617032266f1e6059e2 (Kittywhiskers Van Gogh)
7fef63d merge bitcoin#31529: latest 2.31 glibc (Kittywhiskers Van Gogh)
de1b467 merge bitcoin#30730: Bump time machine to 53396a22afc04536ddf75d8f82ad2eafa5082725 (Kittywhiskers Van Gogh)
Pull request description:
## Additional Information
Spun-off from [dash#6998](#6988) to remedy CI failures. Potential alternative to [dash#6991](#6991) as Clang 19 does not seem to exhibit failures observed with Clang 18 ([comment](#6991 (comment))) caused by change in compile flags in [dash#6966](#6966).
**Note:** ~~The base of this pull request does not use latest `develop` (86e84d7) due to build issues potentially related to [dash#6971](#6971) (see below) and uses a slightly out-of-date base to sidestep the issue. The build issue was found to be present independent of this PR using the Guix container.~~ Fixed with [dash#6997](#6997).
<details>
<summary>Build error:</summary>
```
make[2]: Nothing to be done for 'install-exec-am'.
/home/ubuntu/.guix-profile/bin/mkdir -p '/distsrc-base/distsrc-23.0.0-rc.3-453-g86e84d701641-x86_64-linux-gnu/installed/dashcore-23.0.0-rc.3-453-g86e84d701641//lib/pkgconfig'
/home/ubuntu/.guix-profile/bin/install -c -m 644 libdashconsensus.pc '/distsrc-base/distsrc-23.0.0-rc.3-453-g86e84d701641-x86_64-linux-gnu/installed/dashcore-23.0.0-rc.3-453-g86e84d701641//lib/pkgconfig'
make[2]: Leaving directory '/distsrc-base/distsrc-23.0.0-rc.3-453-g86e84d701641-x86_64-linux-gnu'
make[1]: Leaving directory '/distsrc-base/distsrc-23.0.0-rc.3-453-g86e84d701641-x86_64-linux-gnu'
objcopy: Unable to recognise the format of the input file `dashcore-23.0.0-rc.3-453-g86e84d701641/bin/dash-tx.dbg'
objcopy: Unable to recognise the format of the input file `dashcore-23.0.0-rc.3-453-g86e84d701641/lib/libdashconsensus.so.0.0.0.dbg'
objcopy: Unable to recognise the format of the input file `dashcore-23.0.0-rc.3-453-g86e84d701641/bin/dash-cli.dbg'
objcopy: Unable to recognise the format of the input file `dashcore-23.0.0-rc.3-453-g86e84d701641/bin/dash-wallet.dbg'
objcopy: Unable to recognise the format of the input file `dashcore-23.0.0-rc.3-453-g86e84d701641/bin/dash-qt.dbg'
objcopy: Unable to recognise the format of the input file `dashcore-23.0.0-rc.3-453-g86e84d701641/bin/test_dash.dbg'
objcopy: Unable to recognise the format of the input file `dashcore-23.0.0-rc.3-453-g86e84d701641/bin/dashd.dbg'
```
</details>
## Breaking Changes
None expected.
## Checklist
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
- [x] I have added or updated relevant unit/integration/functional/e2e tests
- [x] I have made corresponding changes to the documentation **(note: N/A)**
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
UdjinM6:
utACK c359fb2
Tree-SHA512: 69576f6e6c0c101be434b273e9a84d1ab9f9341734241e794585615f194b453d2983dce077f9a2efa04b93bfba2374274aad0f90a1a07aed4ef87ff3f22ddd59
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@PastaPastaPasta@DashCoreAutoGuix@knst@UdjinM6