Enable ASan, TSan, UBSan for aarch64-apple-darwin. by frewsxcv · Pull Request #79883 · rust-lang/rust

frewsxcv · 2020-12-10T04:56:46Z

I confirmed ASan, TSan, UBSan all work for me locally with clang on my new Macbook Air.

~~This requires rust-lang/llvm-project#86~~

rust-highfive · 2020-12-10T04:56:50Z

r? @lcnr

(rust-highfive has picked a reviewer for you, use r? to override)

frewsxcv · 2020-12-10T04:57:30Z

I'm compiling Rust locally with these changes to confirm.

frewsxcv · 2020-12-10T05:24:24Z

/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/bin/rustc -Zsanitizer=address /Users/coreyf/dev/rust-lang/rust/src/test/ui/sanitize/new-llvm-pass-manager-thin-lto.rs
error: linking with `cc` failed: exit code: 1
|
= note: "cc" "-arch" "arm64" "-Wl,-rpath" "-Xlinker" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib" "-lrustc-dev_rt.asan" "-L" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.0.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.1.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.2.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.3.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.4.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.5.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.6.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.7.rcgu.o" "-o" "new-llvm-pass-manager-thin-lto" "new-llvm-pass-manager-thin-lto.3s7blind72jn2fhm.rcgu.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libstd-65ab0e7d9cf0fb87.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libpanic_unwind-4c9cd63d6513f642.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libobject-a259f338f0b7b73d.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libaddr2line-df5c21a60c2d9580.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libgimli-912b769937cf57e6.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc_demangle-50dd149dc108cbb5.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libhashbrown-02799d7e76ed6237.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_alloc-4896ae0049a4a17c.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libunwind-5cf7fb4e1b850582.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libcfg_if-44d6c655565382dd.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/liblibc-83d4eff421d392fe.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/liballoc-e50135f33201856a.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_core-405f59a762fe09ad.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libcore-cd4ecd8429490fa9.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-8f64d161accbb588.rlib" "-lSystem" "-lresolv" "-lc" "-lm"
= note: ld: library not found for -lrustc-dev_rt.asan
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: aborting due to previous error

I will investigate tomorrow

lcnr · 2020-12-10T15:47:05Z

can't really review this, as you've already requested a review by @shepmaster

r? @shepmaster

frewsxcv · 2020-12-10T17:40:34Z

Regarding my last comment, I forgot to enable sanitizers in my config.toml. I then had an issue where running a binary generated with -Zsanitizer=address would result in a sigkill. I then found this in the compiler build logs:

/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.asan.dylib (for architecture x86_64)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.asan.dylib (for architecture x86_64h)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.asan.dylib (for architecture arm64)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.lsan.dylib (for architecture x86_64)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.lsan.dylib (for architecture x86_64h)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.lsan.dylib (for architecture arm64)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.tsan.dylib (for architecture x86_64)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.tsan.dylib (for architecture x86_64h)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.tsan.dylib (for architecture arm64)

These warning occur when we run install_name_tool (e.g. install_name_tool -id @rpath/librustc-dev_rt.asan.dylib /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.asan.dylib).

So it seems that by changing the dynamic libraries' names, we invalidate the signature. If I manually re-sign via:

codesign -f -s - ./build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.asan.dylib

Then ASAN seems to work:

% cat badfree.rs
// needs-sanitizer-support
// needs-sanitizer-address
//
// compile-flags: -Z sanitizer=address -O
//
// run-fail
// error-pattern: AddressSanitizer: SEGV
use std::ffi::c_void;
extern "C" {
fn free(ptr: *mut c_void);
}
fn main() {
unsafe {
free(1 as *mut c_void);
}
}

% /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/bin/rustc -Zsanitizer=address badfree.rs

% ./badfree
==21477==ERROR: AddressSanitizer failed to allocate 0x200004000 (8589950976) bytes at address fffffc000 (errno: 12)
==21477==ReserveShadowMemoryRange failed while trying to map 0x200004000 bytes. Perhaps you're using ulimit -v
zsh: abort ./badfree

alex · 2020-12-10T17:43:04Z

That's not really the ASAN error I'd expect from that code, FWIW -- what I get on x86 looks way more sensible.

frewsxcv · 2020-12-10T17:44:51Z

That's not really the ASAN error I'd expect from that code, FWIW -- what I get on x86 looks way more sensible.

I was just wondering about this. It seems like ASan is failing to allocate memory for itself and this is just the error log? Hm

frewsxcv · 2020-12-10T17:45:48Z

Here's the LLDB output, which maybe confirms that:

(lldb) target create "./badfree"
Current executable set to '/Users/coreyf/dev/rust-lang/rust/badfree' (arm64).

(lldb) run
Process 21550 launched: '/Users/coreyf/dev/rust-lang/rust/badfree' (arm64)
==21550==ERROR: AddressSanitizer failed to allocate 0x200004000 (8589950976) bytes at address fffffc000 (errno: 12)
==21550==ReserveShadowMemoryRange failed while trying to map 0x200004000 bytes. Perhaps you're using ulimit -v
Process 21550 stopped
* thread #1, stop reason = signal SIGABRT
frame #0: 0x000000018be3fcec libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`__pthread_kill:
-> 0x18be3fcec <+8>: b.lo 0x18be3fd0c ; <+40>
0x18be3fcf0 <+12>: pacibsp 0x18be3fcf4 <+16>: stp x29, x30, [sp, #-0x10]!
0x18be3fcf8 <+20>: mov x29, sp
Target 0: (badfree) stopped.

(lldb) thread backtrace
* thread #1, stop reason = signal SIGABRT
* frame #0: 0x000000018be3fcec libsystem_kernel.dylib`__pthread_kill + 8
frame #1: 0x000000018be70c24 libsystem_pthread.dylib`pthread_kill + 292
frame #2: 0x000000018bdb8864 libsystem_c.dylib`abort + 104
frame #3: 0x00000001001bf054 librustc-dev_rt.asan.dylib`::Abort() at sanitizer_posix_libcdep.cpp:155:3 [opt]
frame #4: 0x00000001001ab200 librustc-dev_rt.asan.dylib`::ReserveShadowMemoryRange() at asan_shadow_setup.cpp:38:5 [opt]
frame #5: 0x00000001001ab27c librustc-dev_rt.asan.dylib`::InitializeShadowMemory() at asan_shadow_setup.cpp:129:7 [opt]
frame #6: 0x00000001001aa6f0 librustc-dev_rt.asan.dylib`::AsanInitInternal() at asan_rtl.cpp:463:3 [opt]
frame #7: 0x00000001001aa5a0 librustc-dev_rt.asan.dylib`::AsanInitFromRtl() at asan_rtl.cpp:537:3 [opt] [artificial]
frame #8: 0x00000001001a1fa0 librustc-dev_rt.asan.dylib`::wrap_malloc_default_zone() at sanitizer_malloc_mac.inc:86:3 [opt]
frame #9: 0x000000018bca48c0 libsystem_malloc.dylib`__malloc_init + 896
frame #10: 0x00000001954c27c4 libSystem.B.dylib`libSystem_initializer + 184
frame #11: 0x000000010007f90c dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 868
frame #12: 0x000000010007fb94 dyld`ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 56
frame #13: 0x000000010007984c dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 620
frame #14: 0x0000000100079794 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 436
frame #15: 0x0000000100079794 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 436
frame #16: 0x0000000100079794 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 436
frame #17: 0x0000000100077300 dyld`ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 192
frame #18: 0x00000001000773cc dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 96
frame #19: 0x000000010006284c dyld`dyld::initializeMainExecutable() + 220
frame #20: 0x0000000100068b98 dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 7388
frame #21: 0x0000000100061258 dyld`dyldbootstrap::start(dyld3::MachOLoaded const*, int, char const**, dyld3::MachOLoaded const*, unsigned long*) + 476
frame #22: 0x0000000100061038 dyld`_dyld_start + 56

alex · 2020-12-10T17:49:17Z

I agree that it looks like ASAN is failing to set itself up.

frewsxcv · 2020-12-10T23:30:25Z

@saleemrashid pointed me to llvm/llvm-project@176a6e7 which I cherry-picked on top of Rust's current LLVM commit and it fixes the previous issue! d517333

frewsxcv · 2020-12-11T00:19:03Z

This is ready for review! Note that this pull request is blocked on rust-lang/llvm-project#86

frewsxcv · 2020-12-18T22:14:45Z

rust-lang/llvm-project#86 has merged so it's now just waiting for a review

bors · 2021-01-01T04:03:56Z

☔ The latest upstream changes (presumably #80566) made this pull request unmergeable. Please resolve the merge conflicts.

frewsxcv · 2021-01-01T04:28:03Z

Merge conflicts resolved

alex

One question, otherwise looks good to me (with the caveat that I'm far from a rustc expert!)

alex · 2021-01-01T16:26:37Z

+ "x86_64-unknown-linux-gnu",
+ ];
 const MSAN_SUPPORTED_TARGETS: &[&str] =
 &["aarch64-unknown-linux-gnu", "x86_64-unknown-freebsd", "x86_64-unknown-linux-gnu"];


Why not MSAN :-)

I returned my new Macbook Air (need more memory and waiting for the new one to ship), so I can't easily confirm, but I recall there being an error message saying macOS is not supported by MSan, which is maybe why x86_64-apple-darwin is also not listed here. These docs also back up this claim
https://clang.llvm.org/docs/MemorySanitizer.html#supported-platforms
https://firefox-source-docs.mozilla.org/tools/sanitizer/memory_sanitizer.html#build-prerequisites

I clearly have the reading comprehension of a goldfish, I totally missed that no macOS platforms were there.

https://github.com/llvm/llvm-project/blob/62ec4ac90738a5f2d209ed28c822223e58aaaeb7/compiler-rt/cmake/config-ix.cmake#L696-L701
if (COMPILER_RT_HAS_SANITIZER_COMMON AND MSAN_SUPPORTED_ARCH AND OS_NAME MATCHES"Linux|FreeBSD|NetBSD") set(COMPILER_RT_HAS_MSAN TRUE) else() set(COMPILER_RT_HAS_MSAN FALSE) endif()

shepmaster · 2021-01-01T19:24:33Z

+ // Upon renaming the install name, the code signature of the file will invalidate,
+ // so we will sign it again.


I don't follow this. Renaming other signed files doesn't cause problems; why does this?

Renaming the file with mv doesn't invalidate the signature, but renaming the install name (within the contents of the file) with install_name_tool does invalidate.
Searching for "install_name_tool invalidate signature" on Google returns other projects that have hit this. For example: https://community.intel.com/t5/Intel-oneAPI-Threading-Building/Mac-install-location-conflicts-with-code-signing/m-p/1008483/highlight/true#M12505

shepmaster · 2021-01-01T20:08:46Z

@bors r+ rollup

My understanding is that these tools aren't critical so if we realize that this breaks them somehow, it won't be a big thing until we can fix them.

bors · 2021-01-01T20:08:47Z

📌 Commit d482de3 has been approved by shepmaster

frewsxcv · 2021-01-01T20:14:50Z

@bors r+ rollup
My understanding is that these tools aren't critical so if we realize that this breaks them somehow, it won't be a big thing until we can fix them.

Thanks for the review! What you said sounds right. The only functionality I'm changing for existing platforms is the addition of that extra code signing step for macOS machines. So if we saw an regressions on other platforms, I'd expect it to be just limited to macOS, and that code addition in particular.

bors · 2021-01-02T06:59:05Z

⌛ Testing commit d482de3 with merge 5986dd8...

bors · 2021-01-02T09:52:20Z

☀️ Test successful - checks-actions
Approved by: shepmaster
Pushing 5986dd8 to master...

Enable ASan, TSan, UBSan for aarch64-apple-darwin.
5940c19

rust-highfive assigned lcnr Dec 10, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 10, 2020

frewsxcv requested a review from shepmaster December 10, 2020 04:56

frewsxcv added 2 commits December 10, 2020 00:18

rustfmt
7420776

If "aarch64-apple-darwin", copy sanitizers.
d1911dd

Fix sanitizer test output matching
01029e2

rust-highfive assigned shepmaster and unassigned lcnr Dec 10, 2020

Use fork of llvm-project that includes Apple Silicon fix
d517333

frewsxcv mentioned this pull request Dec 10, 2020
Cherry-pick commit that fixes a sanitizer bug on Apple Silicon rust-lang/llvm-project#86
Merged

Manually code-sign after running install_name_tool
25bfa15

frewsxcv added the A-sanitizers Area: Sanitizers for correctness and code quality label Dec 11, 2020

frewsxcv mentioned this pull request Dec 11, 2020
Tracking issue for supporting macOS on Apple Silicon (a.k.a arm64, M1, M2, aarch64) #73908
Closed
31 tasks

frewsxcv added 2 commits December 10, 2020 19:06

Fix typo in code comment
5ab1602

Update sanitizer supported docs to reflect reality
7ee606f

Switch back to rust-lang/llvm-project now that the patch has merged
26cc060

frewsxcv added the O-macos Operating system: macOS label Dec 18, 2020

tmiasko approved these changes Dec 23, 2020
View reviewed changes

Merge remote-tracking branch 'origin/master' into frewsxcv-san
d482de3

alex reviewed Jan 1, 2021
View reviewed changes

alex approved these changes Jan 1, 2021
View reviewed changes

shepmaster reviewed Jan 1, 2021
View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 1, 2021

bors added the merged-by-bors This PR was explicitly merged by bors. label Jan 2, 2021

bors merged commit 5986dd8 into rust-lang:masterJan 2, 2021

rustbot added this to the 1.51.0 milestone Jan 2, 2021

frewsxcv deleted the frewsxcv-san branch January 2, 2021 14:51

petrochenkov mentioned this pull request Sep 22, 2021
Disable the leak sanitizer on Macos aarch64 for now #89170
Merged

		// Upon renaming the install name, the code signature of the file will invalidate,
		// so we will sign it again.

Uh oh!

Uh oh!

Conversation

frewsxcv commented Dec 10, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rust-highfive commented Dec 10, 2020

Uh oh!

frewsxcv commented Dec 10, 2020

Uh oh!

frewsxcv commented Dec 10, 2020

Uh oh!

lcnr commented Dec 10, 2020

Uh oh!

frewsxcv commented Dec 10, 2020

Uh oh!

alex commented Dec 10, 2020

Uh oh!

frewsxcv commented Dec 10, 2020

Uh oh!

frewsxcv commented Dec 10, 2020

Uh oh!

alex commented Dec 10, 2020

Uh oh!

frewsxcv commented Dec 10, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

frewsxcv commented Dec 11, 2020

Uh oh!

frewsxcv commented Dec 18, 2020

Uh oh!

bors commented Jan 1, 2021

Uh oh!

frewsxcv commented Jan 1, 2021

Uh oh!

alex left a comment

Choose a reason for hiding this comment

Uh oh!

alexJan 1, 2021

Choose a reason for hiding this comment

Uh oh!

frewsxcvJan 1, 2021

Choose a reason for hiding this comment

Uh oh!

alexJan 1, 2021

Choose a reason for hiding this comment

Uh oh!

frewsxcvJan 1, 2021

Choose a reason for hiding this comment

Uh oh!

shepmasterJan 1, 2021

Choose a reason for hiding this comment

Uh oh!

frewsxcvJan 1, 2021

Choose a reason for hiding this comment

Uh oh!

shepmaster commented Jan 1, 2021

Uh oh!

bors commented Jan 1, 2021

Uh oh!

frewsxcv commented Jan 1, 2021

Uh oh!

bors commented Jan 2, 2021

Uh oh!

bors commented Jan 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

frewsxcv commented Dec 10, 2020•
edited
Loading

frewsxcv commented Dec 10, 2020•
edited
Loading