Unity 2019 UWP Hololens compatibility

[sBailey5449]

The README instructions mention using the 2017 version of MRTK, which is not compatible with Unity 2019.x out of the box. MRTKv2 works fine but does not include the Newtonsoft.json.dlls needed by this repository (at least for commits before the 2019 update on master). I noticed that by default the Newtonsoft.json.dll included in this repo is now set to "any platforms" which leads me to think the dependency on MRTK 2017 has been removed? When I try to build for the Hololens using Unity 2019.1.7f1, the latest version of MRTKv2 (RC2.1), and the latest commit on master for this repository (4ccf45f) I see the following error when running in debug mode on the Hololens. Everything works fine in the Editor:

NotSupportedException: System.Reflection.Emit.DynamicMethod::.ctor
  at System.Reflection.Emit.DynamicMethod..ctor (System.String name, System.Type returnType, System.Type[] parameterTypes, System.Type owner, System.Boolean skipVisibility) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Utilities.DynamicReflectionDelegateFactory.CreateDynamicMethod (System.String name, System.Type returnType, System.Type[] parameterTypes, System.Type owner) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Utilities.DynamicReflectionDelegateFactory.CreateGet[T] (System.Reflection.FieldInfo fieldInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Utilities.ReflectionDelegateFactory.CreateGet[T] (System.Reflection.MemberInfo memberInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.DynamicValueProvider.GetValue (System.Object target) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues (Newtonsoft.Json.JsonWriter writer, System.Object value, Newtonsoft.Json.Serialization.JsonContainerContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonProperty property, Newtonsoft.Json.Serialization.JsonContract& memberContract, System.Object& memberValue) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject (Newtonsoft.Json.JsonWriter writer, System.Object value, Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract collectionContract, Newtonsoft.Json.Serialization.JsonProperty containerProperty) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue (Newtonsoft.Json.JsonWriter writer, System.Object value, Newtonsoft.Json.Serialization.JsonContract valueContract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerProperty) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize (Newtonsoft.Json.JsonWriter jsonWriter, System.Object value, System.Type objectType) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonSerializer.SerializeInternal (Newtonsoft.Json.JsonWriter jsonWriter, System.Object value, System.Type objectType) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonSerializer.Serialize (Newtonsoft.Json.JsonWriter jsonWriter, System.Object value, System.Type objectType) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonConvert.SerializeObjectInternal (System.Object value, System.Type type, Newtonsoft.Json.JsonSerializer jsonSerializer) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonConvert.SerializeObject (System.Object value, System.Type type, Newtonsoft.Json.JsonSerializerSettings settings) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonConvert.SerializeObject (System.Object value) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Serialize[T] (T obj) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Send[T] (T communication) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Subscribe[T] (System.String topic, RosSharp.RosBridgeClient.SubscriptionHandler`1[T] subscriptionHandler, System.Int32 throttle_rate, System.Int32 queue_length, System.Int32 fragment_size, System.String compression) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.Subscriber`1[T].Start () [0x00000] in <00000000000000000000000000000000>:0 
...
Rethrow as JsonSerializationException: Error getting value from 'topic' on 'RosSharp.RosBridgeClient.Subscription'.
  at Newtonsoft.Json.Serialization.DynamicValueProvider.GetValue (System.Object target) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues (Newtonsoft.Json.JsonWriter writer, System.Object value, Newtonsoft.Json.Serialization.JsonContainerContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonProperty property, Newtonsoft.Json.Serialization.JsonContract& memberContract, System.Object& memberValue) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject (Newtonsoft.Json.JsonWriter writer, System.Object value, Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract collectionContract, Newtonsoft.Json.Serialization.JsonProperty containerProperty) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue (Newtonsoft.Json.JsonWriter writer, System.Object value, Newtonsoft.Json.Serialization.JsonContract valueContract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerProperty) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize (Newtonsoft.Json.JsonWriter jsonWriter, System.Object value, System.Type objectType) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonSerializer.SerializeInternal (Newtonsoft.Json.JsonWriter jsonWriter, System.Object value, System.Type objectType) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonSerializer.Serialize (Newtonsoft.Json.JsonWriter jsonWriter, System.Object value, System.Type objectType) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonConvert.SerializeObjectInternal (System.Object value, System.Type type, Newtonsoft.Json.JsonSerializer jsonSerializer) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonConvert.SerializeObject (System.Object value, System.Type type, Newtonsoft.Json.JsonSerializerSettings settings) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonConvert.SerializeObject (System.Object value) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Serialize[T] (T obj) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Send[T] (T communication) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Subscribe[T] (System.String topic, RosSharp.RosBridgeClient.SubscriptionHandler`1[T] subscriptionHandler, System.Int32 throttle_rate, System.Int32 queue_length, System.Int32 fragment_size, System.String compression) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.Subscriber`1[T].Start () [0x00000] in <00000000000000000000000000000000>:0 
...

After doing some poking around it seems like this might be an issue related to the IL2CPP scripting backend ( Unity 2019.1 deprecates the .NET scripting backend). Not sure if there's a few steps missing from the README, or if the Newtonsoft.json.dll needs to be updated. Any help is greatly appreciated!

[sBailey5449]

Tried using MRTKv2 with current tip of master, with MRTK 2017's bundled versions of Newtonsoft.json.dlls but still had IL2CPP related problems. Current workaround is to use latest commit on UWP branch with unity 2018.3 and MRTK 2017, but not sure if this the only/best option.

[MischaRo]

Hi @sBailey5449
I also tried to build the project for Hololens MRTKv2 using Unity 2019.1.0f2 and got the same issues. Have you made any progress with the MRTKv2 and Newtonsoft.json.dlls / IL2CPP related problems?

[thommi1609]

Same here, can't get it to work. Any news on this issue?

[curthenrichs]

I have had success with using the AOT version of Newtonsoft as recommended in issue #2269 in the mixed-reality-toolkit: microsoft/MixedRealityToolkit-Unity#2269

https://assetstore.unity.com/packages/tools/input-management/json-net-for-unity-11347

Notes:

• I needed to disable the Netwonsoft.json in RosSharp/Plugins for all platforms.
• When building the application it threw 21 errors but finished the build.

I have only tested this approach on a Hololens v1 with a simple application subscribing to joint state topic to update a URDF robot.

[tgroechel]

Can also confirm @curthenrichs fix works for Hololens 2 + BSON, have tried with laser scan and image messages for subscribing and publishing twist

[sBailey5449]

I haven't had any success using @curthenrichs approach. It will build despite throwing a few dozen errors, but when running on HoloLens1 I just see
Exception thrown at<memory location> in <application name>: Microsoft C++ exception: Il2CppExceptionWrapper at memory location <memory location>. over and over. Trying to deserialize a PointCloud2 message, using Unity 2019.3.6f1. Everything works perfectly in the editor.

[tgroechel]

@sBailey5449 Try building for debug and running which should give you better info in visual studio.

My guess is you are getting NaN or Inf values. I had this issue with laser messages and isolated by using other more stable messages to confirm they work in the Hololens (e.g. Odom). My fix was to use BSON serialization instead of JSON as well as check for NaN and Inf inside of C# scripts.

[sBailey5449]

@tgroechel I should have given more information in my last post, that error is from running in debug mode and occurs each time I try to deserialize a PointCloud2 message. Stepping through the generated c++ code I can see that the error occurs when trying to deserialize a message, so it's coming from the Newtonsoft.Json code and not ROS#. Using Unity 2018.4 I can make a .NET backend build that works fine on the HoloLens1, but as soon as I switch to IL2CPP I see the problem I described above (even when using the AOT Newtonsoft.json dll mentioned above). I suspect that other types of messages will probably work but I haven't tried yet. I also tried using BSON, but I still have the same errors.

The real problem seems to be that we need a modern, UWP-compatible version of the newtonsoft.json dependency. The JSON.Net for Unity asset was last updated in 2017 and uses an older version of the newtonsoft.json code (looks like version 8.0.0.0), ROS# currently depends on a more recent version (looks like version 12.0.0.0).

[aioniosfoititis]

I have been using the AOT Json files with 2019.2.11f unity and works on hololens 1.I have also noticed that for 2019.3 unity this setup is not working.

Maybe stick with 2019.2?

[dnlwbr]

For me @curthenrichs approach (AOT Json files) worked with Unity 2018.4.22f and IL2CPP as well. At least in principle. I successfully subscibed to CompressedImage messages and published Twist messages. However when I try to subscribe to Odometry I get the following error:

NullReferenceException: Object reference not set to an instance of an object.
  at RosSharp.RosBridgeClient.Communicator.GetRosName[T] () [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.Subscriber`1[T]..ctor (System.String id, System.String topic, RosSharp.RosBridgeClient.SubscriptionHandler`1[T] subscriptionHandler, RosSharp.RosBridgeClient.Subscription& subscription, System.Int32 throttle_rate, System.Int32 queue_length, System.Int32 fragment_size, System.String compression) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Subscribe[T] (System.String topic, RosSharp.RosBridgeClient.SubscriptionHandler`1[T] subscriptionHandler, System.Int32 throttle_rate, System.Int32 queue_length, System.Int32 fragment_size, System.String compression) [0x00000] in <00000000000000000000000000000000>:0 
  at Calibration.Start () [0x00000] in <00000000000000000000000000000000>:0

What could be the reason for that? In the Unity Simulation everything works fine.

[EricVoll]

This person created a Unity compatible version of Newtonsoft.Json
I just tested it with Unity 2019.3.7f1 and it did build and work without a problem for Windows Standalone and UWP. Thus it should also be HoloLens compatible. I wonder why I did not find it earlier...

https://github.com/jilleJr/Newtonsoft.Json-for-Unity

As @curthenrichs said, you have to disable the "original" Newtonsoft.json.dll for all platforms.

[dnlwbr]

Yeah, you're right. Thank you very much. It compiles without errors and works for UWP.

But still I have the same problem with the odometry topic as mentioned before. Could this be a bug in Ros#, since it works in Unity and subscribing to CompressedImage works on the device?

[EricVoll]

Yeah about the Communicator class. They use Reflection in the Communicator.GetRosName method. They had to do that, because the Message class does not have the RosMessageName field on it, but every class inheriting from Message implements it on its own.

Generally, Reflection does not work on HoloLens / UWP Builds, because of the AOT compilation. I did read, that Reflection itself shouldn't be a problem and that only the Emit namespace does not work, but I experienced, that everything that has to do with reflection fails.

I fixed this issue in my fork here.:
EricVoll@a7d1401
I received the exact same Exception as you are. In short: Add a RosMessageName : string field to the Message class and then you can easily remove the reflection code.
I'd even suggest, that the main repository would also do that, because Reflection is very inefficient, compared to something like this. If the maintainers want, I can create a PR with that change.

[dnlwbr]

Since I am not an expert, I am not sure wether I am able to do that myself without further issues. I would really welcome a PR. Therefore I would be happy if the maintainers agree. Is that possible @dwhit ?

[dnlwbr]

@luchspeter Is there anything important missing in your fork compared to this one? Otherwise, if you just added some features and fixed the issue mentioned above, I will simply use your fork.