[generator] Support Kotlin's unsigned types (#539)

Fixes: #525
Context: dotnet/android#4054
Context: https://github.com/Kotlin/KEEP/blob/13b67668ccc5b4741ecc37d0dd050fd77227c035/proposals/unsigned-types.md
Context: https://kotlinlang.org/docs/reference/basic-types.html#unsigned-integers
Another place where Kotlin makes use of "name mangling" -- see also
commit f3553f4 -- is in the use of unsigned types such as `UInt`.
At the JVM ABI level, Kotlin treats unsigned types as their signed
counterparts, e.g. `kotlin.UInt` is an `int` and `kotlin.UIntArray`
is an `int[]`:
// Kotlin
public class Example {
public fun value(value: UInt) : UInt {
return value
}
public fun array(value: UIntArray) : UIntArray {
return value
}
}
// `javap` output:
public final class Example {
public final int value-WZ4Q5Ns(int);
public final int[] array--ajY-9A(int[]);
} Kotlin uses Java Annotations to determine whether a parameter or
return type is actually an unsigned type instead of a signed type.
Update `Xamarin.Android.Tools.Bytecode` and `generator` to bind e.g.:
* `kotlin.UInt` as `System.UInt32`
* `kotlin.UIntArray` as a `System.UInt32[]`
and likewise for the other unsigned types `ushort`, `ulong`, `ubyte`.
In order to do this, we pretend that they are native Java types and
just translate a few places where we need to tell Java the real type.
~~ Xamarin.Android.Tools.Bytecode / class-parse ~~
When we read the Kotlin metadata in the Java bytecode, if we come
across one of these types we store it within an additional
`FieldInfo.KotlinType` property that we can access later. When we
are generating the XML we check this additional flag and if it's one
of our types we emit it instead of the native Java type.
For example:
<method
abstract="false"
deprecated="not deprecated"
final="false"
name="unsignedAbstractMethod-WZ4Q5Ns"
native="false"
return="uint"
jni-return="I"
static="false"
synchronized="false"
visibility="public"
bridge="false"
synthetic="false"
jni-signature="(I)I">
<parameter
name="value"
type="uint"
jni-type="I" />
</method>
Here we see that even though `@jni-return` is `I` -- meaning `int` --
the `@return` property is `uint`. Likewise `parameter/@jni-type` and
`parameter/@type`. The JNI ABI is `int`, but we bind in C# as `uint`.
~~ ApiXmlAdjuster ~~
Update `JavaTypeReference` to contain unsigned types:
UInt = new JavaTypeReference ("uint");
UShort = new JavaTypeReference ("ushort");
ULong = new JavaTypeReference ("ulong");
UByte = new JavaTypeReference ("ubyte");
~~ generator ~~
`generator` has the 4 new types added to the `SymbolTable` as
`SimpleSymbols`:
AddType (new SimpleSymbol ("0", "uint", "uint", "I", returnCast: "(uint)"));
AddType (new SimpleSymbol ("0", "ushort", "ushort", "S", returnCast: "(ushort)"));
AddType (new SimpleSymbol ("0", "ulong", "ulong", "J", returnCast: "(ulong)"));
AddType (new SimpleSymbol ("0", "ubyte", "byte", "B", returnCast: "(byte)"));
There are 2 fixups we have to make because we use `GetIntValue(...)`,
etc. instead of having unsigned versions:
* Override name of which method to call, e.g.: `GetIntValue()`
instead of `GetUintValue()`.
* Cast the `int` value returned to `uint`. This is accomplished
via the new `ISymbol.ReturnCast` property.
~~ A Note On API Compatibility ~~
Bindings which use Kotlin Unsigned Types will *only* work on
Xamarin.Android 10.2.0 or later ("Visual Studio 16.5").
The problem is that while we *can* emit C# source code which will
*compile* against older versions of Xamarin.Android, if they use
arrays they will not *run* under older versions of Xamarin.Android.
For example, imagine this binding code for the above Kotlin
`Example.array()` method:
// C# Binding of Example.array()
partial class Example {
public unsafe uint[] Array(uint[] value)
{
const string __id = "array--ajY-9A.([I)[I";
IntPtr native_value = JNIEnv.NewArray ((int[]) (object) value); // Works!...ish?
JniArgumentValue* __args = stackalloc JniArgumentValue [1];
__args [0] = new JniArgumentValue (native_value);
JniObjectReference r = _members.InstanceMethods.InvokeVirtualIntMethod (__id, this, __args);
return (uint[]) JNIEnv.GetArray (r.Handle, JniHandleOwnership.DoNotTransfer, typeof (uint));
}
}
That could conceivably *compile* against older Xamarin.Android
versions. However, that cannot *run* against older Xamarin.Android
versions, as *eventually* `JNIEnv.GetArray()` will hit some
dictionaries to determine how to marshal `IntPtr` to a `uint[]`, at
which point things will fail because there is no such mapping *until*
Xamarin.Android 10.2.0.
We feel that a "hard" ABI requirement will have more "graceful"
failure conditions than a solution which doesn't add ABI requirements.
In this case, if you create a Kotlin library binding which exposes
unsigned types, attempting to build an app in Release configuration
against older Xamarin.Android versions will result in a linker error,
as the required `JNIEnv` methods will not be resolvable.

Author: jpobst

src/Java.Interop.Tools.TypeNameMappings/Java.Interop.Tools.TypeNameMappings/JavaNativeTypeManager.cs

@@ -232,10 +232,16 @@ static string GetPrimitiveClass (Type type)
 return"F";
 if(type==typeof(int))
 return"I";
+if(type==typeof(uint))
+return"I";
 if(type==typeof(long))
 return"J";
+if(type==typeof(ulong))
+return"J";
 if(type==typeof(short))
 return"S";
+if(type==typeof(ushort))
+return"S";
 if(type==typeof(bool))
 return"Z";
 returnnull;

src/Java.Interop/Java.Interop/JniArgumentValue.cs

@@ -34,6 +34,8 @@ public JniArgumentValue (sbyte value)
 b=value;
 }
 
+publicJniArgumentValue(bytevalue):this((sbyte)value){}
+
 publicJniArgumentValue(charvalue)
 {
 this=newJniArgumentValue();
@@ -46,18 +48,24 @@ public JniArgumentValue (short value)
 s=value;
 }
 
+publicJniArgumentValue(ushortvalue):this((short)value){}
+
 publicJniArgumentValue(intvalue)
 {
 this=newJniArgumentValue();
 i=value;
 }
 
+publicJniArgumentValue(uintvalue):this((int)value){}
+
 publicJniArgumentValue(longvalue)
 {
 this=newJniArgumentValue();
 j=value;
 }
 
+publicJniArgumentValue(ulongvalue):this((long)value){}
+
 publicJniArgumentValue(floatvalue)
 {
 this=newJniArgumentValue();

src/Xamarin.Android.Tools.ApiXmlAdjuster/JavaTypeReference.cs

@@ -16,7 +16,11 @@ public class JavaTypeReference
 publicstaticreadonlyJavaTypeReferenceFloat;
 publicstaticreadonlyJavaTypeReferenceDouble;
 publicstaticreadonlyJavaTypeReferenceGenericWildcard;
-
+publicstaticreadonlyJavaTypeReferenceUInt;
+publicstaticreadonlyJavaTypeReferenceUShort;
+publicstaticreadonlyJavaTypeReferenceULong;
+publicstaticreadonlyJavaTypeReferenceUByte;
+
 internalstaticJavaTypeReferenceGetSpecialType(stringname)
 {
 switch(name){
@@ -29,6 +33,10 @@ internal static JavaTypeReference GetSpecialType (string name)
 case"long":returnLong;
 case"float":returnFloat;
 case"double":returnDouble;
+case"uint":returnUInt;
+case"ushort":returnUShort;
+case"ulong":returnULong;
+case"ubyte":returnUByte;
 case"?":returnGenericWildcard;
 }
 returnnull;
@@ -46,8 +54,12 @@ static JavaTypeReference ()
 Float=newJavaTypeReference("float");
 Double=newJavaTypeReference("double");
 GenericWildcard=newJavaTypeReference("?");
+UInt=newJavaTypeReference("uint");
+UShort=newJavaTypeReference("ushort");
+ULong=newJavaTypeReference("ulong");
+UByte=newJavaTypeReference("ubyte");
 }
-
+
 JavaTypeReference(stringspecialName)
 {
 SpecialName=specialName;

src/Xamarin.Android.Tools.Bytecode/Fields.cs

@@ -33,6 +33,7 @@ public sealed class FieldInfo {
 publicConstantPoolConstantPool{get;privateset;}
 publicFieldAccessFlagsAccessFlags{get;privateset;}
 publicAttributeCollectionAttributes{get;privateset;}
+publicstringKotlinType{get;set;}
 
 publicFieldInfo(ConstantPoolconstantPool,Streamstream)
 {

src/Xamarin.Android.Tools.Bytecode/Kotlin/KotlinClassMetadata.cs

@@ -1,12 +1,15 @@
 usingSystem;
 usingSystem.Collections.Generic;
+usingSystem.Diagnostics;
 usingSystem.Linq;
 usingSystem.Text;
 usingorg.jetbrains.kotlin.metadata.jvm;
+usingProtoBuf;
 usingType=org.jetbrains.kotlin.metadata.jvm.Type;
 
 namespaceXamarin.Android.Tools.Bytecode
 {
+// https://github.com/JetBrains/kotlin/blob/master/core/metadata.jvm/src/jvm_metadata.proto
 publicclassKotlinFile
 {
 publicList<KotlinFunction>Functions{get;set;}
@@ -220,6 +223,8 @@ internal static KotlinExpression FromProtobuf (Expression exp, JvmNameResolver r
 publicclassKotlinFunction:KotlinMethodBase
 {
 publicstringName{get;set;}
+publicstringJvmName{get;set;}
+publicstringJvmSignature{get;set;}
 publicKotlinFunctionFlagsFlags{get;set;}
 publicKotlinTypeReturnType{get;set;}
 publicintReturnTypeId{get;set;}
@@ -235,9 +240,13 @@ internal static KotlinFunction FromProtobuf (Function f, JvmNameResolver resolve
 if(fisnull)
 returnnull;
 
+varsig=Extensible.GetValue<JvmMethodSignature>(f,100);
+
 returnnewKotlinFunction{
 Flags=(KotlinFunctionFlags)f.Flags,
 Name=resolver.GetString(f.Name),
+JvmName=resolver.GetString((sig?.Name??0)>0?sig.Name:f.Name),
+JvmSignature=sigisnull?null:resolver.GetString(sig.Desc),
 ReturnType=KotlinType.FromProtobuf(f.ReturnType,resolver),
 ReturnTypeId=f.ReturnTypeId,
 ReceiverType=KotlinType.FromProtobuf(f.ReceiverType,resolver),

src/Xamarin.Android.Tools.Bytecode/Kotlin/KotlinFixups.cs

@@ -45,13 +45,15 @@ public static void Fixup (IList<ClassFile> classes)
 FixupJavaMethods(c.Methods);
 
 foreach(varmetinmetadata.Functions)
-FixupFunction(FindJavaMethod(class_metadata,met,c),met,class_metadata);
+FixupFunction(FindJavaMethod(metadata,met,c),met,class_metadata);
 
 foreach(varpropinmetadata.Properties){
-vargetter=FindJavaPropertyGetter(class_metadata,prop,c);
-varsetter=FindJavaPropertySetter(class_metadata,prop,c);
+vargetter=FindJavaPropertyGetter(metadata,prop,c);
+varsetter=FindJavaPropertySetter(metadata,prop,c);
 
 FixupProperty(getter,setter,prop);
+
+FixupField(FindJavaFieldProperty(metadata,prop,c),prop);
 }
 
 }catch(Exceptionex){
@@ -96,7 +98,6 @@ static void FixupConstructor (MethodInfo method, KotlinConstructor metadata)
 Log.Debug($"Kotlin: Hiding internal constructor {method.DeclaringType?.ThisClass.Name.Value} - {metadata.GetSignature()}");
 method.AccessFlags=MethodAccessFlags.Private;
 }
-
 }
 
 staticvoidFixupFunction(MethodInfomethod,KotlinFunctionmetadata,KotlinClasskotlinClass)
@@ -111,18 +112,24 @@ static void FixupFunction (MethodInfo method, KotlinFunction metadata, KotlinCla
 return;
 }
 
-// Kotlin provides actual parameter names
 varjava_parameters=method.GetFilteredParameters();
 
 for(vari=0;i<java_parameters.Length;i++){
 varjava_p=java_parameters[i];
 varkotlin_p=metadata.ValueParameters[i];
 
+// Kotlin provides actual parameter names
 if(TypesMatch(java_p.Type,kotlin_p.Type,kotlinClass)&&java_p.IsUnnamedParameter()&&!kotlin_p.IsUnnamedParameter()){
 Log.Debug($"Kotlin: Renaming parameter {method.DeclaringType?.ThisClass.Name.Value} - {method.Name} - {java_p.Name} -> {kotlin_p.Name}");
 java_p.Name=kotlin_p.Name;
 }
+
+// Handle erasure of Kotlin unsigned types
+java_p.KotlinType=GetKotlinType(java_p.Type.TypeSignature,kotlin_p.Type.ClassName);
 }
+
+// Handle erasure of Kotlin unsigned types
+method.KotlinReturnType=GetKotlinType(method.ReturnType.TypeSignature,metadata.ReturnType.ClassName);
 }
 
 staticvoidFixupExtensionMethod(MethodInfomethod)
@@ -158,16 +165,32 @@ static void FixupProperty (MethodInfo getter, MethodInfo setter, KotlinProperty
 return;
 }
 
+// Handle erasure of Kotlin unsigned types
+if(getter!=null)
+getter.KotlinReturnType=GetKotlinType(getter.ReturnType.TypeSignature,metadata.ReturnType.ClassName);
+
 if(setter!=null){
 varsetter_parameter=setter.GetParameters().First();
 
-if(setter_parameter.IsUnnamedParameter()){
+if(setter_parameter.IsUnnamedParameter()||setter_parameter.Name=="<set-?>"){
 Log.Debug($"Kotlin: Renaming setter parameter {setter.DeclaringType?.ThisClass.Name.Value} - {setter.Name} - {setter_parameter.Name} -> value");
 setter_parameter.Name="value";
 }
+
+// Handle erasure of Kotlin unsigned types
+setter_parameter.KotlinType=GetKotlinType(setter_parameter.Type.TypeSignature,metadata.ReturnType.ClassName);
 }
 }
 
+staticvoidFixupField(FieldInfofield,KotlinPropertymetadata)
+{
+if(fieldisnull)
+return;
+
+// Handle erasure of Kotlin unsigned types
+field.KotlinType=GetKotlinType(field.Descriptor,metadata.ReturnType.ClassName);
+}
+
 staticMethodInfoFindJavaConstructor(KotlinClasskotlinClass,KotlinConstructorconstructor,ClassFileklass)
 {
 varall_constructors=klass.Methods.Where(method =>method.Name=="<init>"||method.Name=="<clinit>");
@@ -181,16 +204,16 @@ static MethodInfo FindJavaConstructor (KotlinClass kotlinClass, KotlinConstructo
 returnnull;
 }
 
-staticMethodInfoFindJavaMethod(KotlinClasskotlinClass,KotlinFunctionfunction,ClassFileklass)
+staticMethodInfoFindJavaMethod(KotlinFilekotlinFile,KotlinFunctionfunction,ClassFileklass)
 {
-varpossible_methods=klass.Methods.Where(method =>method.GetMethodNameWithoutSuffix()==function.Name&&
+varpossible_methods=klass.Methods.Where(method =>method.Name==function.JvmName&&
 method.GetFilteredParameters().Length==function.ValueParameters.Count);
 
 foreach(varmethodinpossible_methods){
-if(!TypesMatch(method.ReturnType,function.ReturnType,kotlinClass))
+if(!TypesMatch(method.ReturnType,function.ReturnType,kotlinFile))
 continue;
 
-if(!ParametersMatch(kotlinClass,method,function.ValueParameters))
+if(!ParametersMatch(kotlinFile,method,function.ValueParameters))
 continue;
 
 returnmethod;
@@ -199,7 +222,15 @@ static MethodInfo FindJavaMethod (KotlinClass kotlinClass, KotlinFunction functi
 returnnull;
 }
 
-staticMethodInfoFindJavaPropertyGetter(KotlinClasskotlinClass,KotlinPropertyproperty,ClassFileklass)
+staticFieldInfoFindJavaFieldProperty(KotlinFilekotlinClass,KotlinPropertyproperty,ClassFileklass)
+{
+varpossible_methods=klass.Fields.Where(field =>field.Name==property.Name&&
+TypesMatch(newTypeInfo(field.Descriptor,field.Descriptor),property.ReturnType,kotlinClass));
+
+returnpossible_methods.FirstOrDefault();
+}
+
+staticMethodInfoFindJavaPropertyGetter(KotlinFilekotlinClass,KotlinPropertyproperty,ClassFileklass)
 {
 varpossible_methods=klass.Methods.Where(method =>(string.Compare(method.GetMethodNameWithoutSuffix(),$"get{property.Name}",true)==0||
 string.Compare(method.GetMethodNameWithoutSuffix(),property.Name,true)==0)&&
@@ -209,7 +240,7 @@ static MethodInfo FindJavaPropertyGetter (KotlinClass kotlinClass, KotlinPropert
 returnpossible_methods.FirstOrDefault();
 }
 
-staticMethodInfoFindJavaPropertySetter(KotlinClasskotlinClass,KotlinPropertyproperty,ClassFileklass)
+staticMethodInfoFindJavaPropertySetter(KotlinFilekotlinClass,KotlinPropertyproperty,ClassFileklass)
 {
 varpossible_methods=klass.Methods.Where(method =>string.Compare(method.GetMethodNameWithoutSuffix(),$"set{property.Name}",true)==0&&
 property.ReturnType!=null&&
@@ -219,7 +250,7 @@ static MethodInfo FindJavaPropertySetter (KotlinClass kotlinClass, KotlinPropert
 returnpossible_methods.FirstOrDefault();
 }
 
-staticboolParametersMatch(KotlinClasskotlinClass,MethodInfomethod,List<KotlinValueParameter>kotlinParameters)
+staticboolParametersMatch(KotlinFilekotlinClass,MethodInfomethod,List<KotlinValueParameter>kotlinParameters)
 {
 varjava_parameters=method.GetFilteredParameters();
 
@@ -237,13 +268,13 @@ static bool ParametersMatch (KotlinClass kotlinClass, MethodInfo method, List<Ko
 returntrue;
 }
 
-staticboolTypesMatch(TypeInfojavaType,KotlinTypekotlinType,KotlinClasskotlinClass)
+staticboolTypesMatch(TypeInfojavaType,KotlinTypekotlinType,KotlinFilekotlinFile)
 {
 // Generic type
 if(!string.IsNullOrWhiteSpace(kotlinType.TypeParameterName)&&$"T{kotlinType.TypeParameterName};"==javaType.TypeSignature)
 returntrue;
 
-if(javaType.BinaryName==KotlinUtilities.ConvertKotlinTypeSignature(kotlinType,kotlinClass))
+if(javaType.BinaryName==KotlinUtilities.ConvertKotlinTypeSignature(kotlinType,kotlinFile))
 returntrue;
 
 // Could be a generic type erasure
@@ -253,17 +284,40 @@ static bool TypesMatch (TypeInfo javaType, KotlinType kotlinType, KotlinClass ko
 // Sometimes Kotlin keeps its native types rather than converting them to Java native types
 // ie: "Lkotlin/UShort;" instead of "S"
 if(javaType.BinaryName.StartsWith("L",StringComparison.Ordinal)&&javaType.BinaryName.EndsWith(";",StringComparison.Ordinal)){
-if(KotlinUtilities.ConvertKotlinClassToJava(javaType.BinaryName.Substring(1,javaType.BinaryName.Length-2))==KotlinUtilities.ConvertKotlinTypeSignature(kotlinType,kotlinClass))
+if(KotlinUtilities.ConvertKotlinClassToJava(javaType.BinaryName.Substring(1,javaType.BinaryName.Length-2))==KotlinUtilities.ConvertKotlinTypeSignature(kotlinType,kotlinFile))
 returntrue;
 }
 
 // Same for some arrays
 if(javaType.BinaryName.StartsWith("[L",StringComparison.Ordinal)&&javaType.BinaryName.EndsWith(";",StringComparison.Ordinal)){
-if("["+KotlinUtilities.ConvertKotlinClassToJava(javaType.BinaryName.Substring(2,javaType.BinaryName.Length-3))==KotlinUtilities.ConvertKotlinTypeSignature(kotlinType,kotlinClass))
+if("["+KotlinUtilities.ConvertKotlinClassToJava(javaType.BinaryName.Substring(2,javaType.BinaryName.Length-3))==KotlinUtilities.ConvertKotlinTypeSignature(kotlinType,kotlinFile))
 returntrue;
 }
 
 returnfalse;
 }
+
+staticstringGetKotlinType(stringjvmType,stringkotlinClass)
+{
+// Handle erasure of Kotlin unsigned types
+if(jvmType=="I"&&kotlinClass=="kotlin/UInt;")
+return"uint";
+if(jvmType=="[I"&&kotlinClass=="kotlin/UIntArray;")
+return"uint[]";
+if(jvmType=="S"&&kotlinClass=="kotlin/UShort;")
+return"ushort";
+if(jvmType=="[S"&&kotlinClass=="kotlin/UShortArray;")
+return"ushort[]";
+if(jvmType=="J"&&kotlinClass=="kotlin/ULong;")
+return"ulong";
+if(jvmType=="[J"&&kotlinClass=="kotlin/ULongArray;")
+return"ulong[]";
+if(jvmType=="B"&&kotlinClass=="kotlin/UByte;")
+return"ubyte";
+if(jvmType=="[B"&&kotlinClass=="kotlin/UByteArray;")
+return"ubyte[]";
+
+returnnull;
+}
 }
 }

src/Xamarin.Android.Tools.Bytecode/Kotlin/KotlinUtilities.cs

@@ -8,15 +8,15 @@ namespace Xamarin.Android.Tools.Bytecode
 {
 publicstaticclassKotlinUtilities
 {
-publicstaticstringConvertKotlinTypeSignature(KotlinTypetype,KotlinClassklass=null)
+publicstaticstringConvertKotlinTypeSignature(KotlinTypetype,KotlinFilemetadata=null)
 {
 if(typeisnull)
 returnstring.Empty;
 
 varclass_name=type.ClassName;
 
 if(string.IsNullOrWhiteSpace(class_name)){
-if(klassisobject){
+if(metadataisKotlinClassklass){
 
 vartp=klass.TypeParameters.FirstOrDefault(t =>t.Id==type.TypeParameter);
 

src/Xamarin.Android.Tools.Bytecode/Methods.cs

@@ -31,6 +31,7 @@ public sealed class MethodInfo {
 publicClassFileDeclaringType{get;privateset;}
 publicMethodAccessFlagsAccessFlags{get;set;}
 publicAttributeCollectionAttributes{get;privateset;}
+publicstringKotlinReturnType{get;set;}
 
 publicMethodInfo(ConstantPoolconstantPool,ClassFiledeclaringType,Streamstream)
 {
@@ -290,6 +291,7 @@ public sealed class ParameterInfo : IEquatable<ParameterInfo> {
 publicstringName;
 publicintPosition;
 publicTypeInfoType=newTypeInfo();
+publicstringKotlinType;
 
 publicMethodParameterAccessFlagsAccessFlags;
 

src/Xamarin.Android.Tools.Bytecode/Xamarin.Android.Tools.Bytecode.csproj

@@ -13,7 +13,7 @@
  </PropertyGroup>
 
  <ItemGroup>
- <PackageReferenceInclude="protobuf-net"Version="2.4.1" />
+ <PackageReferenceInclude="protobuf-net"Version="2.4.4" />
  </ItemGroup>
 
 </Project>