Generate synthetic object file to ensure all exported and used symbols participate in the linking

[nbdd0121]

Fix#50007 and #47384

This is the synthetic object file approach that I described in #95363 (comment), allowing all exported and used symbols to be linked while still allowing them to be GCed.

Related #93791, #95363

r? @petrochenkov
cc @carbotaniuman

[petrochenkov]

One reason why I like this approach more than #95363 is that I have no idea how widely linker script files like forcelink.ld from #95363 are supported.
LLD, for example, had issues with supporting version script files used for exporting symbols on PE/COFF targets.

A "root" object file, on another hand, is as portable as it gets, with the same logic being reusable for very different targets.

[carbotaniuman]

Yeah this seems like a cleaner approach. This should fix the linkme style (test untested on this branch) of test cases as well as the linker-script style ones.

This doesn't fix #47384 fully however as it still requires #[used(linker)], but the combination of both works for all available examples.

[nbdd0121]

Yeah this seems like a cleaner approach. This should fix the linkme style (test untested on this branch) of test cases as well as the linker-script style ones.

This doesn't fix #47384 fully however as it still requires #[used(linker)], but the combination of both works for all available examples.

What do you mean? This certainly fixes the issue described by the OP in #47384.

[carbotaniuman]

@nbdd0121

What do you mean? This certainly fixes the issue described by the OP in #47384.

It doesn't fix the issue in rust-audit. #47384 has like 4 different issues bundled into it, which makes testing quote difficult.

[nbdd0121]

Ah, the rust-audit case does require a custom linker script or #[used(linker)], but I think it's rightfully so.

[petrochenkov]

#47384 discusses so many different things that I'd like to set some agreements on what specifically this PR is supposed to assume and fix.

1. Do not change meaning of #[used].
   Whether #[used] defaults to #[used(compiler)] or #[used(linker)] is a matter for a separate PR, like Only compile #[used] as llvm.compiler.used for ELF targets #93718 or other.
   Right now #[used] defaults to #[used(compiler)], aka CodegenFnAttrFlags::USED, aka the symbols must exist in object files.

   This PR changes the meaning of #[used(compiler)] and makes it closer to #[used(linker)] by marking #[used(compiler)] symbols as exported and keeping them alive for longer, until the linker GC.

2. Visibility of #[used(...)] symbols.
   RFC: #[used] static variables rfcs#2386 specifies that used is orthogonal to symbols visibility (exported vs everything else non-exported).

   This PR marks both #[used(compiler)] and #[used(linker)] symbols as exported, even if they wouldn't be exported without that, which contradicts the RFC.
   I'm not actually sure whether you can guaranteedly keep a non-exported symbol alive until linker GC, but for #[used(compiler)] symbols that's not necessary and they certainly can stay non-exported as stated in the RFC.

3. Let's assume -C export-executable-symbols already exists.
   RFC: -C export-executable-symbols rfcs#2841 is an accepted feature, and the implementation seems to be in progress, so we just need to not forget to include it into the picture. Executables with -C export-executable-symbols may end up being identical to cdylibs in these regards though.

[petrochenkov]

@nbdd0121
Could you actually split this PR into two?

• The first PR is just keeping what is already exported, using symbols.o.
• The second PR is for used-related changes.

I'll be able to approve the first PR almost immediately, and it will then unblock #95818.
Then we'll be able to discuss used in more detail.

[nbdd0121]

#47384 discusses so many different things that I'd like to set some agreements on what specifically this PR is supposed to assume and fix.

1. Do not change meaning of #[used].
   Whether #[used] defaults to #[used(compiler)] or #[used(linker)] is a matter for a separate PR, like Only compile #[used] as llvm.compiler.used for ELF targets #93718 or other.
   Right now #[used] defaults to #[used(compiler)], aka CodegenFnAttrFlags::USED, aka the symbols must exist in object files.
   This PR changes the meaning of #[used(compiler)] and makes it closer to #[used(linker)] by marking #[used(compiler)] symbols as exported and keeping them alive for longer, until the linker GC.

No. This PR does not change the meaning of #[used(compiler)]. Instead, it fixes the existing bug to ensure that it conforms to the desired behaviour. The PR ensures that #[used(compiler)] reaches the linker, but it doesn't modify the linker behaviour.

2. Visibility of #[used(...)] symbols.
   RFC: #[used] static variables rfcs#2386 specifies that used is orthogonal to symbols visibility (exported vs everything else non-exported).
   This PR marks both #[used(compiler)] and #[used(linker)] symbols as exported, even if they wouldn't be exported without that, which contradicts the RFC.

No. Neither of these are exported. The export symbol list is not changed at all.

3. Let's assume -C export-executable-symbols already exists.
   RFC: -C export-executable-symbols rfcs#2841 is an accepted feature, and the implementation seems to be in progress, so we just need to not forget to include it into the picture. Executables with -C export-executable-symbols may end up being identical to cdylibs in these regards though.

See above.

[petrochenkov]

Neither of these are exported. The export symbol list is not changed at all.

As I understand the used symbols get into the reachable set (the reachable.rs change), and from there to the exported set, because the exported set is seeded with reachable set (reachable_set -> reachable_non_generics -> exported_symbols ).

[nbdd0121]

The #[used] symbols would have export level "Rust" instead of "C" so they won't be exported for cdylib or binary.

[nbdd0121]

There is an "exported_symbols" query and a "exported_symbols" function in linker.rs that produces the final exported symbols. They are actually different things; the first contains all reachables, latter does a filter to the former to get an actual list for export.

[petrochenkov]

The #[used] symbols would have export level "Rust".

Yes, but they shouldn't have any exported level at all (unless they are exported for other reasons unrelated to used).

the first contains all reachables, latter does a filter to the former to get an actual list for export.

The list of symbols to filter in exported_symbols2 is seeded with the result of exported_symbols1, and the filtering is threshhold-based, so the SymbolExportLevel::Rust symbols will still stay exported in dylibs.

[nbdd0121]

The #[used] symbols would have export level "Rust".

Yes, but they shouldn't have any exported level at all (unless they are exported for other reasons unrelated to used).

"Rust" level export will be exported in dylib anyway and we already export an awful lot of symbols in such case.

I don't see why A in

#[used]staticA:u32 = 1;

shouldn't be exported while A in

staticA:u32 = 1;#[inline]pubfnfoo() -> &u32{&A}

should.

[nbdd0121]

In a sense I don't treat "Rust" export level as truly being exports; it's basically just an export level for "everything that might get used" and I view it as an implementation detail to make dylibs work.

[petrochenkov]

Ok, point understood.

I agree that in practice it's not too important to prune such used symbols from dylib exports, and we can skip it if it helps to simplify the implementation or fix other more important issues sooner.

I'd still consider it as a rough edge of the implementation, because when spec / mental model say one thing and the implementation do another it often leads to misunderstanding and mess.
Could you add a FIXME comment to reachable.rs telling that after being marked as reachable used symbols will have SymbolExportLevel::Rust and may end up being exported from dylibs, but that's mostly ok?

[petrochenkov]

The issue 1 (#[used(compiler)] symbols being made closer to #[used(linker)] due to being over-exposed from dylibs) is a consequence of the issue 2, so the comment from #95604 (comment) will address it as well.

[petrochenkov]

Otherwise LGTM, no need to split into two PRs (#95604 (comment)) anymore.

[nbdd0121]

I left it empty because it isn't currently needed.