Add an easy way to build from source without LLVM

[andrewrk]

In celebration of #17871, this adds a new file, bootstrap.c, for building from source without LLVM. The new README instructions are reproduced here:

If you don't need your Zig compiler to have LLVM extensions enabled, you can
follow these instructions instead.

In this case, the only system dependency is a C compiler.

cc -o bootstrap bootstrap.c
./bootstrap build

You can pass any options to this that you would pass to zig build.

When a zig compiler without LLVM extensions is satisfactory, this greatly simplified build-from-source process can be used.

This could be useful for users who only want to contribute to the standard library, for example. For me this takes 11 minutes, or 3 minutes if I change -O2 to -O0 when compiling zig2.c. I'm not sure which one it should default to. On one hand building faster is nice; on the other hand, since we don't have optimizations without LLVM yet, it's nice to have a stage2 build handy that is optimized. The unoptimized one takes about 5m to build the compiler, while the optimized one completes in 25 seconds.

If we look at total time to stage3:

• compile zig2.c with -O2: 11m
• compile zig2.c with -O0: 8m

I think I like the -O2 version because even though we wait 3 more minutes, we can rebuild faster, and we have a handy optimized zig2 sitting around.

Either way, this is compared to building LLVM from source, which is a show-stopper for many people, or ~30 minutes if ya nasty.

Related:

• make the main zig executable no longer depend on LLVM, LLD, and Clang libraries #16270
• unable to build zig with x86 backend using ReleaseFast optimization mode #17885

[kubkon]

Sick!

[andrewrk]

Since rebasing on master, this now fails with:

zig build-exe zig Debug native: error: the following command terminated unexpectedly:
/home/andy/Downloads/zig/zig2 build-exe --stack 33554432 /home/andy/Downloads/zig/src/main.zig --cache-dir /home/andy/Downloads/zig/zig-cache --global-cache-dir /home/andy/.cache/zig --name zig --mod Builtins/Builtin.def::/home/andy/Downloads/zig/zig-cache/o/906eedbdff28488a384cbc5c459094f7/Builtin.zig --mod build_options::/home/andy/Downloads/zig/zig-cache/c/a6728411f520fff30d715d0fc2039540/options.zig --mod Attribute/names.def::/home/andy/Downloads/zig/zig-cache/o/1549e4ba81bd7c0fe1b9db446e20a68c/names.zig --mod aro:Builtins/Builtin.def,Attribute/names.def:/home/andy/Downloads/zig/deps/aro/lib.zig --deps Builtins/Builtin.def,Attribute/names.def,aro,build_options --listen=- Build Summary: 3/6 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
└─ install zig transitive failure
└─ zig build-exe zig Debug native failure
error: the following build command failed with exit code 1:
/home/andy/Downloads/zig/zig-cache/o/ae1cf8addedba3a0e1fa61a8c26eed33/build /home/andy/Downloads/zig/zig2 /home/andy/Downloads/zig /home/andy/Downloads/zig/zig-cache /home/andy/.cache/zig --seed 0xa6705ffd -Dno-lib

Thread 1 "zig2" received signal SIGSEGV, Segmentation fault.
0x000000000066cdcc in Air_typeOfIndex__57870 ()
(gdb) bt
#0 0x000000000066cdcc in Air_typeOfIndex__57870 ()
#1 0x0000000000776b41 in Sema_coerceExtra__99923 ()
#2 0x00000000006f6f56 in Sema_zirStructInitEmptyResult__99763 ()
#3 0x00000000006869fb in Sema_analyzeBodyInner__99437 ()
#4 0x0000000000683317 in Sema_analyzeBodyBreak__99435 ()
#5 0x000000000068074c in Sema_resolveStructFieldInits__100012 ()
#6 0x00000000007bd394 in Sema_finishStructInit__99769 ()
#7 0x000000000078c8a6 in Sema_structInitEmpty__99764 ()
#8 0x0000000000697ef5 in Sema_analyzeBodyInner__99437 ()
#9 0x0000000000683317 in Sema_analyzeBodyBreak__99435 ()
#10 0x000000000068074c in Sema_resolveStructFieldInits__100012 ()
#11 0x00000000007bd394 in Sema_finishStructInit__99769 ()
#12 0x000000000078c8a6 in Sema_structInitEmpty__99764 ()
#13 0x0000000000697ef5 in Sema_analyzeBodyInner__99437 ()
#14 0x0000000000683317 in Sema_analyzeBodyBreak__99435 ()
#15 0x0000000000cafcfa in Module_semaDecl__5758 ()
#16 0x000000000062d332 in Module_ensureDeclAnalyzed__5753 ()
#17 0x00000000006692f8 in Sema_ensureDeclAnalyzed__99963 ()
#18 0x0000000000668f37 in Sema_analyzeDeclRefInner__99967 ()
#19 0x00000000006906da in Sema_analyzeBodyInner__99437 ()
#20 0x000000000083e6ee in Sema_resolveBlockBody__99554 ()
#21 0x00000000006933ae in Sema_analyzeBodyInner__99437 ()
#22 0x00000000006bf0af in Sema_analyzeBodyInner__99437 ()
#23 0x000000000083e6ee in Sema_resolveBlockBody__99554 ()
#24 0x00000000006933ae in Sema_analyzeBodyInner__99437 ()
#25 0x000000000083e6ee in Sema_resolveBlockBody__99554 ()
#26 0x00000000006933ae in Sema_analyzeBodyInner__99437 ()
#27 0x000000000083e6ee in Sema_resolveBlockBody__99554 ()
#28 0x00000000006933ae in Sema_analyzeBodyInner__99437 ()
#29 0x000000000062afc7 in Module_ensureFuncBodyAnalyzed__5754 ()
#30 0x0000000000572971 in Compilation_processOneJob__5530 ()
#31 0x0000000000565131 in Compilation_update__5510 ()
#32 0x00000000004f7615 in main_updateModule__216 ()
#33 0x00000000004b5b7d in main_buildOutputType__207 ()
#34 0x0000000000403198 in main ()

[andrewrk]

I bisected it to b3462b7.

[kcbanner]

I tried reproducing this but I get:

~/dev/zig$ clang -o zig2-clang-debug zig2.c compiler_rt.c -g -O0 -std=c99 -fno-stack-protector -Wl,-z,stack-size=0x10000000 -Istage1
~/dev/zig$ ./zig2-clang-debug build
run docgen (langref.html): error: lib/std/Thread.zig:342:9: error: Cannot spawn thread when building in single-threaded mode
@compileError("Cannot spawn thread when building in single-threaded mode");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
referenced by:
test.thread local storage: docgen_tmp/test_thread_local_variables.zig:8:41
remaining reference traces hidden; use '-freference-trace' to see all reference traces
The following command exited with code 1:
/home/kcbanner/dev/zig/zig2-clang-debug test docgen_tmp/test_thread_local_variables.zig
/home/kcbanner/dev/zig/doc/langref.html.in:1475:54: error: test failed
{#code_begin|test|test_thread_local_variables#}
error: ParseError
run docgen (langref.html): error: the following command exited with error code 1:
/home/kcbanner/dev/zig/zig-cache/o/c5662d6cd7e2dcd5608a60a6b5101f0c/docgen --zig /home/kcbanner/dev/zig/zig2-clang-debug /home/kcbanner/dev/zig/doc/langref.html.in /home/kcbanner/dev/zig/zig-cache/o/e7b26f6846737cda08cba96428072c07/langref.html
install generated/: error: unable to open source directory '/home/kcbanner/dev/zig//home/kcbanner/dev/zig/zig-cache/o/dab2e9187bc70268df3330e8e43618c7/docs': FileNotFound
Build Summary: 6/10 steps succeeded; 2 failed (disable with --summary none)
install transitive failure
├─ install generated to doc/langref.html transitive failure
│ └─ run docgen (langref.html) failure
└─ install generated/ failure
error: the following build command failed with exit code 1:
/home/kcbanner/dev/zig/zig-cache/o/4870aaa689bf3884f97704887e789250/build /home/kcbanner/dev/zig/zig2-clang-debug /home/kcbanner/dev/zig /home/kcbanner/dev/zig/zig-cache /home/kcbanner/.cache/zig --seed 0x127f5569

[andrewrk]

Pass -Dno-lib

[kcbanner]

Hm, I can't seem to reproduce the issue - the build completes successfully for me:

$ clang --version
Ubuntu clang version 14.0.0-1ubuntu1.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
$ clang -o zig2-clang-release zig2.c compiler_rt.c -O2 -std=c99 -fno-stack-protector -Wl,-z,stack-size=0x10000000 -Istage1
$ ./zig2-clang-release build -Dno-lib
$ ./zig-out/bin/zig version
0.12.0-dev.1503+f2a1f2b04

[jacobly0]

Note that at least some, if not all, of these CI failures are due to a gcc miscompilation. (hint: that's way too many fld instructions)

[andrewrk]

Nice reduction.

Unfortunately Zig's C backend has to care about generating code that is compatible with GCC versions that are widely in use, so even if they fixed this immediately, we would still need a workaround.

This looks like a pain in the ass to work around though. If I'm reading this right, we can't rely on the assignment operation when the type is an aggregate that contains a f128, meaning that all f128 types would need to be lowered as char[16] instead of __float128.

I requested an account on gcc bugzilla so I can make the report once I get an email response.

[jacobly0]

This looks like a pain in the ass to work around though. If I'm reading this right, we can't rely on the assignment operation when the type is an aggregate that contains a f128, meaning that all f128 types would need to be lowered as char[128] instead of __float128.

As I understand it, gcc believes that the _Float128 and __float128 types fit in an st (80-bit) register. The correct workaround should be to disable the _Float128, and __float128 branches and let it hit the fallback instead on affected gcc versions.

See also this diff on godbolt:

--- a/example.c+++ b/example.c@@ -15,7 +15,7 @@ typedef _Float16 f16;
typedef float f32;
typedef double f64;
typedef long double f80;
-typedef __float128 f128;+typedef u128 f128;
extern void *memcpy(void *restrict, void const *restrict, usize);

[andrewrk]

Ah that makes sense, thanks!

[andrewrk]

Another local miscompilation with gcc 12.2.0:

gcc -o zig2 zig2.c compiler_rt.c -std=c99 -O2 -fno-stack-protector -Wl,-z,stack-size=0x10000000 -Istage1 -pthread
zig2.c: In function ‘unicode_utf8ToUtf16Le__105860’:
zig2.c:1121800: warning: assignment to ‘const uint16_t (*)[16]’ {aka ‘const short unsigned int (*)[16]’} from incompatible pointer type ‘const uint16_t *’ {aka ‘const short unsigned int *’} [-Wincompatible-pointer-types]
1121800 | t26 = t25.ptr;
| In function ‘mem_readPackedIntLittle__anon_256327__256327’,
inlined from ‘mem_readPackedInt__anon_220037__220037’ at zig2.c:2443512:0,
inlined from ‘value_Value_readFromPackedMemory__22823’ at zig2.c:2021538:0:
zig2.c:2862799: warning: ‘memcpy’ reading 16 bytes from a region of size 10 [-Wstringop-overread]
2862799 | memcpy(&t11, &t10, sizeof(zig_u128));
| zig2.c: In function ‘value_Value_readFromPackedMemory__22823’:
zig2.c:2862776: note: source object ‘t10’ of size 10
2862776 | uint8_t t10[10];
| In function ‘mem_readPackedIntBig__anon_256328__256328’,
inlined from ‘mem_readPackedInt__anon_220037__220037’ at zig2.c:2443516:0,
inlined from ‘value_Value_readFromPackedMemory__22823’ at zig2.c:2021538:0:
zig2.c:2862873: warning: ‘memcpy’ reading 16 bytes from a region of size 10 [-Wstringop-overread]
2862873 | memcpy(&t13, &t12, sizeof(zig_u128));
| zig2.c: In function ‘value_Value_readFromPackedMemory__22823’:
zig2.c:2862835: note: source object ‘t12’ of size 10
2862835 | uint8_t t12[10];
| zig2.c: In function ‘value_Value_readFromMemory__22822’:
zig2.c:1604604: warning: ‘memcpy’ reading 16 bytes from a region of size 10 [-Wstringop-overread]
1604604 | memcpy(&t47, &t46, sizeof(zig_u128));
| zig2.c:1604463: note: source object ‘t46’ of size 10
1604463 | uint8_t t46[10];
| error: sub-compilation of zig_libc failed
lib/std/target.zig:713:60: note: expected type 'type', found '[1]usize'
pub const empty = Set{ .ints = [1]usize{0} ** usize_count };
~~~~~~~~~~~~^~~~~~~~~~~~~~
referenced by:
expected type 'type', found '[1]usize': pub const empty = Set{ .ints = [1]usize{0} ** usize_count };:1:2
8 reference(s) hidden; use '-freference-trace=10' to see all references
sub-compilation of compiler_rt failed: pub const empty = Set{ .ints = [1]usize{0} ** usize_count };:1:2
error: sub-compilation of compiler_rt failed
lib/std/target.zig:713:60: note: expected type 'type', found '[1]usize'
pub const empty = Set{ .ints = [1]usize{0} ** usize_count };
~~~~~~~~~~~~^~~~~~~~~~~~~~
referenced by:
expected type 'type', found '[1]usize': pub const empty = Set{ .ints = [1]usize{0} ** usize_count };:1:2
25 reference(s) hidden; use '-freference-trace=27' to see all references
lib/std/target.zig: �:3323:3342
lib/std/target.zig:713:60: error: expected type 'type', found '[1]usize'
pub const empty = Set{ .ints = [1]usize{0} ** usize_count };
~~~~~~~~~~~~^~~~~~~~~~~~~~
referenced by:
featureSet: lib/std/target.zig:812:36
featureSet: lib/std/target/x86.zig:184:59
remaining reference traces hidden; use '-freference-trace' to see all reference traces

[andrewrk]

@jacobly0 I didn't want your nice test case to go to waste so I filed a gcc bug here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112521