Alpine Linux ARM64 bring up

[qmfrederik]

Opening an issue to track the efforts to bring up coreclr/corefx on Alpine Linux ARM.

We've re-scoped this issue to track getting "Hello World!" to run on 64-bit Linux. As a next step, we'll work to get exception handling to work on Linux - see dotnet/coreclr#9370

Related threads:

• Support Alpine Linux (inadvertently to conform with libc for embedded systems) #4210
• coreclr on Android Android Build Issue Discussion cydhaselton/coreclr#1

Current approach is to use a docker container on an ARM device for building coreclr; that one is stored here https://github.com/qmfrederik/dotnet-alpine-arm

Building the native components for Alpine arm64:

• Compile coreclr: ./build.sh arm64 skipgenerateversion skipmscorlib
• Run PAL tests src/pal/tests/palsuite/runpaltests.sh /home/coreclr/bin/obj/Linux.arm64.Debug/
• Compile corefx: CPLUS_INCLUDE_PATH=/usr/include/c++/6.3.0/aarch64-alpine-linux-musl/:/usr/include/c++/6.3.0/ src/Native/build-native.sh arm64
• Cross compile mscorlib on Linux for arm: ./build.sh skipnative arm64 debug verbose -rebuild

Putting together a sample app:

• Compile and publish a demo stand-alone .NET core app, copy that one over to Alpine ARM
• Copy /home/corefx/bin/Linux.arm64.Debug/Native/* and /home/coreapp # cp ../coreclr/bin/Product/Linux.arm64.Debug/* to the app dir, as well as bin/Product/Linux.arm64.Debug/mscorlib.dll and bin/Product/Linux.arm64.Debug/System.Private.CoreLib.dll from the cross-build on Linux x64.

Backlog:

• Fix libunwind getcontext call (currently stubbed out)
• Hardware exception handling on aarm64
• Unit tests for the various functions that had to be re-written in assembly?

Open items:
Compile clang 3.9.1 for Alpine; this version may include some welcome bug fixes

Sources are here: http://releases.llvm.org/download.html#3.9.1 , build process here: http://llvm.org/docs/CMake.html
Build process:

mkdir mybuilddir
cd mybuilddir
cmake path/to/llvm/source/root
cmake --build . -- -j 92
cmake --build . --target install

or

cmake -DCMAKE_INSTALL_PREFIX=/usr -P cmake_install.cmake

and at least patches 2, 3 from http://git.alpinelinux.org/cgit/aports/tree/main/llvm?h=3.5-stable

• the same for clang, lld?

[qmfrederik]

Copying relevant comments:

@janvorli

@qmfrederik we already have Alpine on x64 running and we have some ARM32 Linuxes working, like Raspbian. So I think it should not be difficult to bring up Alpine ARM. I'll be happy to help you with that. The bringup of Alpine x64 was my work. To begin, it would be the best to start with cross compilation of the managed assemblies instead of trying to do it the mono way that @cydhaselton was forced to use. Once we have stuff working with the cross compiled managed stuff and locally compiled native code, we would create the RID and dotnet packages.

@janvorli

@qmfrederik it is a little different. The first step is to build the native components of coreclr and corefx on your target Alpine ARM system. So you would just clone the repos and for coreclr run ./build.sh arm skipgenerateversion skipmscorlib, for corefx run src\Native\build-native.sh arm
This would get you the native components for coreclr and corefx.
Once you have this building correctly, we will need to build libuv and stuff from core-setup repo.
Finally, I would recommend getting existing dotnet sdk tarball for ARM32 Linux and just replace the native components in it with the newly built stuff.

@janvorli

@qmfrederik I've forgotten to say that in the last phase, you'll need to make sure that the package you use was built from the same version of sources as the binaries you are sticking in. But let's discuss it once we are there.

@janvorli

We would cross compile in the CI. But we would need to enable our cross compiling to build rootfs for Alpine first. So for the bring up, it is the easiest to start with building the native components on the real hardware.

@qmfrederik

@janvorli So, I found some time to try & build coreclr for Alpine on ARM.

I'm building om aarch64 (so 64-bit version of ARM), and Alpine seems to have all dependencies except for libunwind. I built libunwind from source.

Building coreclr via ./build.sh arm skipgenerateversion skipmscorlib manages to compile a fair part of coreclr, but it fails when linking with libunwind.so because it cannot resolve getcontext.

Here's (part of) the build output:

/usr/bin/ld: ../libcoreclrpal.a(seh.cpp.o)(.debug_info+0x774): R_AARCH64_ABS64 used with TLS symbol _ZZ29PAL_ThrowExceptionFromContextE27threadLocalExceptionStorage
/usr/bin/ld: ../libcoreclrpal.a(seh.cpp.o)(.debug_info+0x82c): R_AARCH64_ABS64 used with TLS symbol _ZL27t_nativeExceptionHolderHead
/usr/bin/../lib/gcc/aarch64-alpine-linux-musl/6.3.0/../../../libunwind.so: undefined reference to `getcontext'
clang-3.8: error: linker command failed with exit code 1 (use -v to see invocation)
I used a Docker container for building, so the entire VM setup & build process is documented here:

https://github.com/qmfrederik/dotnet-alpine-arm/blob/master/Dockerfile

I'm assuming something went wrong when building libunwind. I'll dig around but if you have any pointers, they would be greatly appreciated.

@am11

@qmfrederik, aports has libunwind package in their current release, and they didn't enabled AARCH64 yet:
https://pkgs.alpinelinux.org/packages?name=libunwind
https://github.com/alpinelinux/aports/tree/master/main/libunwind (check the APKBUILD file (build script) and the patches which you may also need to apply for aarch64)

@janvorli

@qmfrederik can you please create a separate issue for the Linux aarch64? This one is already extremely long.
One thing I can see in your message - you need to use ./build.sh arm64 skipgenerateversion skipmscorlib, not arm, arm is arm32.
Also, you will need a bunch of changes that @cydhaselton has made first - adding and fixing various asm helpers, for example. They will need to be pulled out selectively, since he was also making changes that were specific for the Android termux environment, like tmp path location etc.
As for the getcontext, I guess the issue is in the missing

if(PAL_CMAKE_PLATFORM_ARCH_ARM64)
find_library(UNWIND_ARCH NAMES unwind-aarch64)
endif()
in the src/pal/src/CMakeLists.txt around line 290 (I am not sure if the unwind-aarch64 is the right name, just extrapolating from the names for other architectures)

[qmfrederik]

@am11 re: libunwind on Alpine:

I replicated the build script as good as I could, including the parameters passed to ./configure and the patches. I applied the gist of 10-disable-tests.patch through a sed statement and force-enable-man.patch does not seem to be relevant.
You can find the build script for libunwind here: https://github.com/qmfrederik/dotnet-alpine-arm/blob/master/Dockerfile#L19

Do you have any idea as to why Alpine didn't enable builds of libunwind for aarch64?

[am11]

@qmfrederik, perhaps this is because nobody has asked for AARCH64 package. I have only tested building libsass and some other native libraries on Alpine (x86_64) in past. One approach is to send a message on their mailing list or IRC channel of aports to add the package or (like in this case) to support particular architecture. We can also send a PR to aports repo, and our changes will be verified with their dedicated build systems (which runs few times a day to rebuild all packages; and is different/detailed than CI checks per PR).

You can find the build script for libunwind here: https://github.com/qmfrederik/dotnet-alpine-arm/blob/master/Dockerfile#L19

This looks good. 👍
Is it functioning properly and ldd path/to/libunwind.so etc. works?

[qmfrederik]

@am11 Nope, it looks like it's libunwind.so which references the getcontext symbol:

/home/coreclr # ldd /usr/lib/libunwind.so
 ldd (0xaaaad8854000)
 libc.musl-aarch64.so.1 => ldd (0xaaaad8854000)
Error relocating /usr/lib/libunwind.so: getcontext: symbol not found

[janvorli]

I believe the symbol is defined in the libunwind-aarch64.so. On Amd64, we link to libunwind.so, libunwind-x86_64.so and libunwind-generic.so. On aarch 64, the CMakeLists was missing the architecture specific file.

[am11]

@qmfrederik, can you try making this kind of patch for libunwind: https://github.com/alpinelinux/aports/blob/f82ed56f8dd3e5a8ab11a39e75d7c37922cd3691/community/chromium/no-getcontext.patch Basically ifdef it out for GLIBC. Also you may want to use specific version: http://download.savannah.gnu.org/releases/libunwind/libunwind-1.2-rc1.tar.gz, same one that is used by APKBUILD script.

[qmfrederik]

@janvorli You are correct:

/home/libunwind-manual $ ldd /usr/lib/libunwind-aarch64.so
 ldd (0xaaaac1fba000)
 libc.musl-aarch64.so.1 => ldd (0xaaaac1fba000)

It looks like somehow the CoreCLR build managed to reference libunwind.so instead of libunwind-aarch64.so. I'll double-check CMakeLists and let you know.

[janvorli]

@qmfrederik it needs to reference all of the three. The libunwind-aarch64.so is missing, as I've said before.

[qmfrederik]

@janvorli I added the reference to libunwind-aarch64, see this commit

Just to be sure - since you say "it needs to reference all of the three", does this mean that coreclr will also link against libunwind.so and if ldd libunwind.so yields errors, the build will fail?

[janvorli]

It won't fail, the libunwind-aarch64.so will satisfy the symbol need for the libunwind.so. Actually, the libunwind-generic.so seems to not to be needed, but it doesn't hurt.
Look at what we have on amd64:

ldd bin/Product/Linux.x64.Debug/libcoreclr.so
 linux-vdso.so.1 => (0x00007fffb84dd000)
 libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9e34ff7000)
 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9e34dd9000)
 librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f9e34bd0000)
 libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9e349cc000)
 libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f9e347c7000)
 libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 (0x00007f9e345ab000)
 libunwind-x86_64.so.8 => /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8 (0x00007f9e3438c000)
 libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f9e34088000)
 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9e33d81000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9e339bc000)
 /lib64/ld-linux-x86-64.so.2 (0x00007f9e3632e000)
 liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f9e3379a000)

[qmfrederik]

@janvorli I'm relatively sure the issue is in libunwind itself.

libunwind defines unw_tdep_getcontext. On arm a stub is defined in libunwind-arm.h#L262; for aarch64 it is simply defined as getcontext in libunwind-aarch64.h#L17

I'm assuming getcontext doesn't exist on Alpine so that's why we're getting the linker errors.

Question: I can try to copy the definition of getcontext from arm to aarch64. Do you think that will work?

[janvorli]

@qmfrederik ok, thanks for the explanation. As for copying the code it won't be as simple as that. The aarch64 uses different registers and instructions. You may get some idea on how to do that in the LLVM libunwind below:
https://github.com/llvm-mirror/libunwind/blob/master/src/UnwindRegistersSave.S#L252

[qmfrederik]

@janvorli Thanks for the pointer. Is llvm libunwind supposed to be API-compatible with libunwind? I had a cursory look at the header, and the declarations seem to be identical.

Yes, it seems the file you referenced basically implements unw_tdep_getcontext for aarch64, so it should work.

I have to admit I've taken the easy path for now, and just stubbed the function as { return -1; }. That should indicate a failure to the calling code.

Good news! coreclr now compiles successfully on Alpine aarch64.

Are there any coreclr tests I can run to get a grasp of the stability of this version of coreclr?

[janvorli]

You can run the PAL tests. After building coreclr (without skiptests argument), run
src/pal/tests/palsuite/runpaltests.sh bin/obj/Linux.arm64.Debug/

[qmfrederik]

Thanks, I"ll run the PAL tests and post back the results.

Meanwhile, I also attempted to build the native part of corefx, but some standard C++ includes like limits are not found:

/home/corefx/src/Native/Unix/Common/pal_utilities.h:12:10: fatal error: 'limits' file not found
#include <limits>
 ^
1 error generated.

I can export CPLUS_INCLUDE_PATH=/usr/include/c+\+/6.3.0/ only to encounter the next error:

/usr/include/c++/6.3.0/limits:42:10: fatal error: 'bits/c++config.h' file not found
#include <bits/c++config.h>
 ^
1 error generated.

Which C++ package did you use for Alpine x64?