JIT: fix gc hole in peephole optimizations

[AndyAyersMS]

We cannot safely peephole instructions that straddle a gc enable boundary. Detecting when this might happen is a bit subtle; currently we rely on emitForceNewIG to be set.

Add a new utility emitCannotPeepholeLastIns to centralize the logic that decides whether basing a peephole opt on emitLastIns is safe.

Closes#77661.

[AndyAyersMS]

@BruceForstall PTAL
cc @dotnet/jit-contrib

One diff on arm64, still running x64 locally but don't expect much there.

[kunalspathak]

LGTM

[BruceForstall]

One suggestion, one question. Let me know if you want to leave it as-is.

[BruceForstall]

Looks great.

I do think we should back-port.

[AndyAyersMS]

One other thought -- if we had emitted the GC liveness update even though we didn't emit the instruction then the peephole in #77661 would have been ok.

Kind of makes me think that instead of not emitting instructions we should just shrink them to zero size or something similar, But that would have some annoying aspects too (perhaps we could discard them after the gc updates are done).

[AndyAyersMS]

Per SPMI, just one arm64 method with diffs -- see #77661 (comment).

[AndyAyersMS]

/backport to release/7.0

[github-actions]

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3429844601

[BruceForstall]

One other thought -- if we had emitted the GC liveness update even though we didn't emit the instruction then the peephole in #77661 would have been ok.

Are you suggesting that an instruction that gets removed by a peephole optimization would still have its GC effect applied? That seems dangerous. It seems like the peephole optimization needs to always do the right thing w.r.t. GC no matter what that might be.

[AndyAyersMS]

One other thought -- if we had emitted the GC liveness update even though we didn't emit the instruction then the peephole in #77661 would have been ok.

Are you suggesting that an instruction that gets removed by a peephole optimization would still have its GC effect applied? That seems dangerous. It seems like the peephole optimization needs to always do the right thing w.r.t. GC no matter what that might be.

The net effect on GC needs to be the same whether we do the peephole or not. Maybe we can at least sanity check this is true somehow? Say save gc state before, then compare after states with/without peephole, they should match. Probably tricky to pull off.

[AndyAyersMS]

FYI I think #77153 is the same bug -- so this may be more widespread than I first thought.

I tracked this down by doing an SPMI collect of the (intermittently failing) libraries run and then running SPMI diffs with and without the fix here.