Converting a `float/double` to `byte/sbyte` or `short/ushort` does not correctly account for overflow

[tannergooding]

Issue

Take the following program:

using System;

class Program
{
 static void Main()
 {
 double aslocal;

 aslocal = 65535.17567;
 Console.WriteLine(unchecked((short)aslocal));

 aslocal = 65536.17567;
 Console.WriteLine(unchecked((short)aslocal));
 }
}

Both Intel and AMD produce the same result (whether using the x87 FPU or SSE+):

-1
0

Both inputs generate the same assembly.

SSE+:

00007FFDC205093F vmovsd xmm0,qword ptr [address]
00007FFDC2050948 vmovsd qword ptr [stack_addr],xmm0 
00007FFDC205094E vcvttsd2si ecx,mmword ptr [stack_addr] 
00007FFDC2050954 movsx rcx,cx 
00007FFDC2050958 call 00007FFE17952980 
00007FFDC205095D nop 

x87 FPU:

00DF08C7 DD 05 10 09 DF 00 fld qword ptr ds:[addr] 
00DF08CD DD 5D C0 fstp qword ptr [stack_addr] 
00DF08D0 F2 0F 10 45 C0 movsd xmm0,mmword ptr [stack_addr] 
00DF08D5 F2 0F 2C C8 cvttsd2si ecx,xmm0 
00DF08D9 0F BF C9 movsx ecx,cx 
00DF08DC E8 37 11 1C 73 call 73FB1A18 
00DF08E1 90 nop 

The SSE+ instructions that support converting (cvtsd2si, cvtss2si, cvttsd2si, and cvttss2si) all deal with only 32-bit or 64-bit results.

The x87 FPU instructions that support converting (fist and fistp) supports 16-bit, 32-bit, or 64-bit results. However, it looks like even under the legacy JIT, it still emits cvtts*2si rather than fist/fistp

Given that the instruction emitted only supports 32-bit/64-bit results, the results are 65535 (0xFFFF) and 65536 (0x10000). These, when cast to short and then sign-extended back to int are -1 and 0, respectively.

Proposed Fix

The .NET Runtime should correctly account for overflow and return a "correct" value in this scenario. On x86, the "indefinite integer value" is defined to be:

(2^(w-1), where w represents the number of bits in the destination format)

[tannergooding]

See dotnet/roslyn#7333 (comment) and surrounding discussion for more details.

[tannergooding]

CC. @gafter

[tannergooding]

It looks like ARM might have similar problems given the conversion instructions I see are available.

[tannergooding]

More noticeable errors can be seen with values such as 65538.0 which returns 2, etc.

C# has started using 0 as the result for cases of constant folding to ensure cross-platform/architecture determinism.

[jkotas]

This behavior is unspecified per the ECMA-335 spec.

If overflow occurs converting a floating-point type to an integer, or if the floating-point value
being converted to an integer is a NaN, the value returned is unspecified.

[tannergooding]

Hmmm, that's unfortunate.

Perhaps this is something we could normalize if we ever provided a "strict"/"determinstic" vs "fast" mode for floating-point operations.