stage2: support dynamically linking musl libc

[ifreund]

Closes#5364

This is a draft as I still need to (partially) automate generation of the libc.s file for easier maintenance.

[LemonBoy]

If you want to do some more filtering check out glibc's abilist.awk (see here. Musl's libc.so is quite tame if compared to glibc's one so most of that is not needed (yet?)/

One thing that's wrong even in the glibc parsing scripts is the use of .type N, @function;, that's ok for functions but not really ok for other symbol kinds (eg. environ is marked as DO instead of DF in objdump's output).

Since the symbol types "percolate" to the final object this may be a problem...

Here's the objdump's output of a simple program referencing environ, the first part is when musl's libc is used, the second when the generated libc.so is used instead:

 5: 0000000000004028 8 OBJECT WEAK DEFAULT 20 _environ
6: 0000000000004028 8 OBJECT WEAK DEFAULT 20 environ
7: 0000000000004028 8 OBJECT GLOBAL DEFAULT 20 __environ
9: 0000000000004028 8 OBJECT WEAK DEFAULT 20 ___environ
48: 0000000000004028 8 OBJECT GLOBAL DEFAULT 20 __environ
50: 0000000000004028 8 OBJECT WEAK DEFAULT 20 environ

 523: 0000000000001016 0 FUNC GLOBAL DEFAULT 6 _environ
552: 0000000000001016 0 FUNC GLOBAL DEFAULT 6 __environ
703: 0000000000001016 0 FUNC GLOBAL DEFAULT 6 environ
820: 0000000000001016 0 FUNC GLOBAL DEFAULT 6 ___environ

Will this be a problem? In theory yes, every tool that checks the symbol type will be extremely confused by this.

[ifreund]

Got rid of the code generating libc_nonshared.a for musl as it is not necessary. Also stopped passing -e _dlstart on linking libc.so as leaving it out doesn't seem to cause any issues at link or runtime. I'll be looking at improving the quality of the generated libc.s stub file next.

[LemonBoy]

I'll drop the big question, how do we integrate this new target in the testing setup?

[ifreund]

The stubs are now generated with proper types and the output is sorted alphabetically to make diffs nicer when updating in the future. I haven't implemented any filtering beyond skipping non-dynamic symbols as I'm not sure if it would be worth it.

[ifreund]

I'll drop the big question, how do we integrate this new target in the testing setup?

Actually if I'm understanding correctly the drone CI is already testing this target by default as it runs on alpine which uses musl as the system libc.

[ifreund]

Now only difference (aside from addresses) in the output of objdump --dynamic-syms when run on the stub libc.so vs the real one produced by musl's Makefile is the section names. The stub libc.so has everything in .text. If this is an issue I can make gen_stubs.zig aware of them as well though I don't think they matter for our purposes.

[ifreund]

I've run into an issue. If I delete the stub libc.so in my cache, binaries linked with it fail to run:

./hello
Error loading shared library /home/ifreund/.cache/zig/o/3381f1c8880ba3f32ae5ccde1f762eae/libc.so: No such file or directory (needed by ./hello)
Error relocating ./hello: __libc_start_main: symbol not found
Error relocating ./hello: __stack_chk_fail: symbol not found
Error relocating ./hello: printf: symbol not found
Error relocating ./hello: abort: symbol not found
Error relocating ./hello: sigaction: symbol not found
Error relocating ./hello: memset: symbol not found
Error relocating ./hello: write: symbol not found
Error relocating ./hello: memcpy: symbol not found
Error relocating ./hello: __errno_location: symbol not found
Error relocating ./hello: dl_iterate_phdr: symbol not found
Error relocating ./hello: mmap: symbol not found
Error relocating ./hello: close: symbol not found
Error relocating ./hello: sched_yield: symbol not found
Error relocating ./hello: munmap: symbol not found
Error relocating ./hello: getenv: symbol not found
Error relocating ./hello: openat: symbol not found
Error relocating ./hello: flock: symbol not found
Error relocating ./hello: fstat: symbol not found
Error relocating ./hello: read: symbol not found
Error relocating ./hello: isatty: symbol not found
Error relocating ./hello: __errno_location: symbol not found
Error relocating ./hello: environ: symbol not found

I'm investigating a fix.

[andrewrk]

Hmm this should be solved by passing the dynamic linker path in the link/Elf.zig code

[ifreund]

Hmm this should be solved by passing the dynamic linker path in the link/Elf.zig code

I believe we already are with -dynamic-linker /lib/ld-musl-x86_64.so.1. Could it be that we do infact need a libc_nonshared_a for the (afaik statically linked) crt1 files?

[LemonBoy]

What's the output of ldd ./hello ?

[ifreund]

What's the output of ldd ./hello ?

Before removing the global cache dir:

$ ldd ./hello /lib/ld-musl-x86_64.so.1 (0x7fb2075ce000)
libc.so => /lib/ld-musl-x86_64.so.1 (0x7fb2075ce000)

After:

$ ldd ./hello /lib/ld-musl-x86_64.so.1 (0x7f922062d000)
Error loading shared library /home/ifreund/.cache/zig/o/3381f1c8880ba3f32ae5ccde1f762eae/libc.so: No such file or directory (needed by ./hello)
Error relocating ./hello: __libc_start_main: symbol not found
Error relocating ./hello: __stack_chk_fail: symbol not found
Error relocating ./hello: printf: symbol not found
Error relocating ./hello: abort: symbol not found
Error relocating ./hello: sigaction: symbol not found
Error relocating ./hello: memset: symbol not found
Error relocating ./hello: write: symbol not found
Error relocating ./hello: memcpy: symbol not found
Error relocating ./hello: __errno_location: symbol not found
Error relocating ./hello: dl_iterate_phdr: symbol not found
Error relocating ./hello: mmap: symbol not found
Error relocating ./hello: close: symbol not found
Error relocating ./hello: sched_yield: symbol not found
Error relocating ./hello: munmap: symbol not found
Error relocating ./hello: getenv: symbol not found
Error relocating ./hello: openat: symbol not found
Error relocating ./hello: flock: symbol not found
Error relocating ./hello: fstat: symbol not found
Error relocating ./hello: read: symbol not found
Error relocating ./hello: isatty: symbol not found
Error relocating ./hello: __errno_location: symbol not found
Error relocating ./hello: environ: symbol not found

[LemonBoy]

The problem is the full path, you want only libc.so there so that the dynamic linker can find musl's libc.
What's the lld invocation like?

[ifreund]

The problem is the full path, you want only libc.so there so that the dynamic linker can find musl's libc.
What's the lld invocation like?

ld.lld -error-limit=0 -z stack-size=16777216 --gc-sections -m elf_x86_64 -o hello /home/ifreund/.cache/zig/o/0c1194ab18f26e5c323c3f1905afade8/Scrt1.o
/home/ifreund/.cache/zig/o/1ff9911c2476e7f81b8c0e7972443185/crti.o -dynamic-linker /lib/ld-musl-x86_64.so.1 zig-cache/o/27256662811ff9fc56f4ecd9595467ee/hello.o /home/ifreund/.cache/zig/o/7006e974fe31fc900fc06808f3e38c81/libcompiler_rt.a /home/ifreund/.cache/zig/o/8d18b2b411365c815ed51d78d2554b0c/libunwind.a /home/ifreund/.cache/zig/o/3381f1c8880ba3f32ae5ccde1f762eae/libc.so /home/ifreund/.cache/zig/o/7e28e375e056f71e600bcca3caeafb6a/crtn.o

[ifreund]

Setting the soname to libc.so when building the libc stub seems to have fixed the issue. Thanks @LemonBoy for the tip!

[ifreund]

The objdump --dynamic-syms output for the stub libc.so and the real version from musl is now identical except for the offsets/addresses. Symbols are in the proper section and the .protected attribute is handled.

[andrewrk]

Really nice work here! This is already in a mergeable state, but I do have a couple requests for ya

[ifreund]

Fixed both comments. For the record, to diff the output of objdump --dynamic-syms on real and stub so files the following commands can be used (with the proper paths of course):

$ objdump --dynamic-syms /home/ifreund/.cache/zig/o/94cd8ea1da001f912f2f7d259446424d/libc.so | tail -n +5 | sed -E -e 's/[0-9a-f]{16}//g' | sort > stubs.txt
$ objdump --dynamic-syms ../musl/lib/libc.so | tail -n +5 | sed -E -e 's/[0-9a-f]{16}//g' | sort > musl.txt $ diff musl.txt stubs.txt

If all is well the diff command will give no output