Make Enum.GetValues AOT-safe

[MichalStrehovsky]

...at the cost of a small compat break. We return int[] instead of SomeInt32Enum[].

Fixes#72140.

We can also delete intrinsic handling of Enum.GetValues in dataflow analysis but I don't want to conflict with Vitek's #71485 right now.

Cc @dotnet/ilc-contrib

[ghost]

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, to please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

[agocke]

Who owns the System.Enum surface area? I presume this needs to go through a back-compat review

[MichalStrehovsky]

Who owns the System.Enum surface area? I presume this needs to go through a back-compat review

Only changing it for NativeAOT.

[agocke]

Woops, missed that this was the nativeaot runtime

[vitek-karas]

What guarantees that int[] will exist though? Or for that matter other enum backing types, like byte[], short[] and so on.

The comment here assumes that they are available, without explanation. I think it would be good to add a short explanation.

[MichalStrehovsky]

What guarantees that int[] will exist though?

Good old new keyword:

runtime/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/EnumInfo.cs

Lines 53 to 64 in 6f13196

	ValuesAsUnderlyingType=Type.GetTypeCode(UnderlyingType)switch
	{
	TypeCode.Byte=>newbyte[numValues],
	TypeCode.SByte=>newsbyte[numValues],
	TypeCode.UInt16=>newushort[numValues],
	TypeCode.Int16=>newshort[numValues],
	TypeCode.UInt32=>newuint[numValues],
	TypeCode.Int32=>newint[numValues],
	TypeCode.UInt64=>newulong[numValues],
	TypeCode.Int64=>newlong[numValues],
	_ =>thrownewNotSupportedException(),
	};

I'll add a comment.

[vitek-karas]

Sorry - I was too lazy to lookup what ValuesAsUnderlyingType does :-(

[MichalStrehovsky]

/azp run runtime-extra-platforms

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[LakshanF]

LGTM but there seems to be some related test failures

• System.Reflection.Tests
• System.Runtime.Tests
• System.IO.FileSystem.DisabledFileLocking.Tests, not clear if the failure is related to the enum change
• System.Collections.NonGeneric.Tests
• System.Numerics.Vectors.Tests

[MichalStrehovsky]

LGTM but there seems to be some related test failures

The System.Numerics.Vectors.Tests is #72149. I think that failure is non-deterministic. I'm not sure if it really should be closed unfortunately.

I'm looking into the appcompat implications of the other failures (whether we just fixup/disable/add new tests and be done with it, or whether it's blockers).

[MichalStrehovsky]

I looked at the failing tests. The main implication is that if one does Array Enum.GetValues(Type) and then uses the non-generic IEnumerable to iterate over the elements of the arrays, it will return boxed underlying type instead of the boxed enum.

So the compat concern is basically that. Would someone call GetValues and then do a foreach, calling ToString on each element? We would see different values. Or would they call object.Equals on these? They would not be equal to boxed enums.

It's basically what the tests do:

Assert.Equal() Failure
↓ (pos 0)
Expected: [Enum1, Enum2, Enum10, Enum18, Enum45]
Actual: [1, 2, 10, 18, 45]
↑ (pos 0)

Couple other alternative:

• Introduce a new annotations (e.g. DynamicallyCreatedArrayAttribute?). This would be similar to DynamicallyAccessedMembers but would imply the presence of an array of that type. We can make Enum.GetValues, and Array.CreateInstance AOT safe with this. Might be getting late for that.
• Shared code over enums. It's getting late for that too.
• Keep it as AOT-unsafe

[LakshanF]

This is unfortunate. I would expect in most code, the expectations on post operations after the call to this API, would be to see the Enum type and not the underlying type. I think we need to give an indicator to the developer of the behavioral change in native AOT but can see as you indicated. the options you listed above might be getting too late now.

I think keeping it unsafe with the recommended message to use the generic API might be the best option :-(

[stephentoub]

Would someone call GetValues and then do a foreach, calling ToString on each element?

I looked around usage in grep.app. While by far the most common usage is just foreach'ing, there are a non-trivial number of uses that do databinding, e.g.
https://github.com/HearthSim/Hearthstone-Deck-Tracker/blob/4272f593cdb2282e65dcd73312cb4500ab1cb5da/Hearthstone%20Deck%20Tracker/Controls/Stats/ConstructedFilters.xaml.cs#L36
or that use ToString, either directly, e.g.:
https://github.com/Jimmey-Jiang/Common.Utility/blob/103fb680d2818144fe635f957cb262f7f9b810b6/src/Common.Utility_EN/RenumDropList/RenumDropList.cs#L17-L34
or via interpolation, e.g.:
https://github.com/AlanMorel/MapleServer2/blob/38e9e05f63eaf1f7c78de2c141d659537a4ab9a5/MapleServer2/Commands/Game/GMCommands.cs#L179-L182
or that do type tests, e.g.:
https://github.com/ShareX/ShareX/blob/6dec8a9a10a379fe3ae557813592bbfc252d97b8/ShareX.HelpersLib/Helpers/Helpers.cs#L187-L190
or that use the non-generic Array.IndexOf, e.g.:
https://github.com/ShareX/ShareX/blob/6dec8a9a10a379fe3ae557813592bbfc252d97b8/ShareX.HelpersLib/Extensions/EnumExtensions.cs#L86-L90
Seems this change has the potential to break all such usage (with AOT).

[MichalPetryka]

The biggest issue is that the generic overload wasn't there in .Net Standard 2.0, so a lot of libraries targetting that use the type one. As such, having it work properly would be really useful. Personally I've always copied the data to properly typed arrays, but I've seen samples that just cast the arrays to the enum array type.

[jkotas]

If we cannot fix the existing API, can we introduce a new one? I am thinking GetEnumValuesAsUnderlyingType.

Libraries that target NS2.0 would be able to use this API using reflection light-up and still be AOT compatible. It is not the case with the generic overload. Calling the generic overload via reflection requires MakeGenericMethod that comes with the same AOT problem.

[jkotas]

GetEnumValuesAsUnderlyingType would be also good for non-runtime reflection. System.Reflection.MetadataLoadContext does not support GetEnumValues today since it cannot create the enum array.

[jkotas]

Another option that does not require new APIs and that is compatible with NS2.0: Enumerate the enum members as field info. It comes with performance penalty.

using System.Reflection;
foreach (var f in typeof(DayOfWeek).GetFields(BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static))
{
Console.WriteLine(f.Name);
Console.WriteLine(f.GetRawConstantValue()); // This returns the field value as boxed underlying type
}

[teo-tsirpanis]

Can we make the NativeAOT compiler recognize Enum.GetValues(typeof(T)) and transform it into Enum.GetValues<T>()? It would still raise a warning to suggest users to migrate to the generic overload, but will do what is expected to do at runtime. All but the last examples @stephentoub provided would work, and the last can be easily changed.

[jkotas]

Can we make the NativeAOT compiler recognize Enum.GetValues(typeof(T)) and transform it into Enum.GetValues()?

The AOT compiler does that today:

runtime/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/ReflectionMethodBodyScanner.cs

Lines 549 to 571 in 5fc5c68

	caseIntrinsicId.Enum_GetValues:
	{
	// Enum.GetValues returns System.Array, but it's the array of the enum type under the hood
	// and people depend on this undocumented detail (could have returned enum of the underlying
	// type instead).
	//
	// At least until we have shared enum code, this needs extra handling to get it right.
	foreach(varvalueinmethodParams[0])
	{
	if(valueisSystemTypeValuesystemTypeValue
	&&!systemTypeValue.RepresentedType.Type.IsGenericDefinition
	&&!systemTypeValue.RepresentedType.Type.ContainsSignatureVariables(treatGenericParameterLikeSignatureVariable:true))
	{
	if(systemTypeValue.RepresentedType.Type.IsEnum)
	{
	_reflectionMarker.Dependencies.Add(_factory.ConstructedTypeSymbol(systemTypeValue.RepresentedType.Type.MakeArrayType()),"Enum.GetValues");
	}
	}
	else
	CheckAndReportRequires(calledMethod,newMessageOrigin(callingMethodBody,offset),RequiresDynamicCodeAttribute);
	}
	}
	break;

The problem is what to do with Enum.GetValues where the type is not statically known. For example,

runtime/src/libraries/System.ComponentModel.TypeConverter/src/System/ComponentModel/EnumConverter.cs

Line 174 in 215b39a

ArrayobjValues=Enum.GetValues(EnumType);

[MichalStrehovsky]

Thanks for the data Stephen! The data binding scenario is very likely the reason why .NET Native backtracked on this too (nobody could remember why we backtracked there, only that we backtracked).

I've submitted an API proposal for the API Jan proposed here: #72498. I marked it blocking. If we can get it approved soon, I think it would be reasonable to land before RC 1 snaps off.

[jkotas]

The next step is to get #72498 API approved. This PR can be closed.