rustc_codegen: fix musttail returns for cast/indirect ABIs

[InvalidPathException]

Fixes#148239#144986

Explicit tail calls trigger bug! for any callee whose ABI returns via PassMode::Cast, and we forgot to to forward the hidden sret out-pointer when the ABI requested an indirect return. The former causes ICE, the latter produced malformed IR (wrong codegen) if the return value is large enough to need sret.

Updated the musttail helper to accept cast-mode returns, made it so that we pass the return pointer through the tail-call path.

Added two UI tests to demonstrate each case.

This is my first time contributing, please do check if I did it right.

r? theemathas

[rustbot]

Some changes occurred in compiler/rustc_codegen_ssa

cc @WaffleLapkin

[WaffleLapkin]

The best practice I'm usually trying to follow is: add tests in a separate commit (and make them "pass" there, possibly with known bug annotations, etc). That way the second commit with the impl clearly shows how the behavior changed, in the tests' diff.

If you have time I'd recommend you do the same here (tip: jj makes splitting/modifying commits easy).

And congrats on your first contribution! I'll try to review your PR shortly :)

[InvalidPathException]

Made changes to address existing comments, the generated IR for tests/ui/explicit-tail-calls/become-indirect-return.rs is provided below:
shouldbegood.txt

Lines 110-125

; become_indirect_return::dispatch
; Function Attrs: noinline uwtable
define internal void @_ZN22become_indirect_return8dispatch17hd1719c44a86d65c3E(ptr sret([24 x i8]) align 1 %_0, ptr align 8 %param) unnamed_addr #1 {
start:
; call become_indirect_return::op_dummy
musttail call void @_ZN22become_indirect_return8op_dummy17h31e000dd5109bff4E(ptr sret([24 x i8]) align 1 %_0, ptr align 8 %param)
ret void
}
; become_indirect_return::op_dummy
; Function Attrs: noinline uwtable
define internal void @_ZN22become_indirect_return8op_dummy17h31e000dd5109bff4E(ptr sret([24 x i8]) align 1 %_0, ptr align 8 %_param) unnamed_addr #1 {
start:
call void @llvm.memset.p0.i64(ptr align 1 %_0, i8 1, i64 24, i1 false)
ret void

[InvalidPathException]

The best practice I'm usually trying to follow is: add tests in a separate commit (and make them "pass" there, possibly with known bug annotations, etc). That way the second commit with the impl clearly shows how the behavior changed, in the tests' diff.

If you have time I'd recommend you do the same here (tip: jj makes splitting/modifying commits easy).

And congrats on your first contribution! I'll try to review your PR shortly :)

How can we properly test this if run-pass causes gcc tests to fail? I cannot find an existing example on how to handle this (other than making the tests type check only).

[bjorn3]

You can add an //@ ignore-backends: gcc annotation to disable the test with GCC.

[InvalidPathException]

Tested with a real use case here and it compiles and runs fine. Did not notice any issue + verified it is not lowered to unoptimized recursive calls in IR.

[WaffleLapkin]

@bors r+

[bors]

📌 Commit 52959a8 has been approved by WaffleLapkin

It is now in the queue for this repository.

[matthiaskrgr]

@bors r- #148382 (comment)

[InvalidPathException]

@rustbot ready

[WaffleLapkin]

@InvalidPathException I think you forgot to push the changes addressing CI failure.
@rustbot author

[rustbot]

Reminder, once the PR becomes ready for a review, use @rustbot ready.

[InvalidPathException]

@InvalidPathException I think you forgot to push the changes addressing CI failure. @rustbot author

72c9762f500ec24b017506d01427594f34bce250 This should fix it, no?

[InvalidPathException]

Might be because the comment was timestamped weirdly so the timeline became unclear. Not sure why it's dated "in the future".
@rustbot ready

[WaffleLapkin]

my bad, didn't notice the commit in all the backlinks 🤦🏻
@bors r+

[bors]

📌 Commit 72c9762 has been approved by WaffleLapkin

It is now in the queue for this repository.