Documentation on unpacking of dynamic types could use some examples

[chrish42]

Hello,

I am transmitting a dict over MessagePack from Python to C#, which either has a key with name "results", pointing to a list of doubles, or a key "error", pointing to a string. I thought the "dynamic" API of the C# MessagePack library would allow me to programmatically unpack this, but after a fair bit of time trying different things and reading some of the source code of the library, I gave up.

I got things working by creating a helper class with the right member variables and passing that to Unpack(). But I think the documentation in https://github.com/msgpack/msgpack-cli/wiki/Messagepackobject could be improved, as I really wasn't able to figure out the "dynamic" way of doing what I wanted from said documentation. For one, adding a few examples of unpacking dynamic, multi-level data using the MessagePackObject, etc. API would be really helpful. Thanks!

[yfakariya]

Hi

You can get MessagePackObject manually through Unpacker.Data property, or you can use MessagePackObject type property in your deserialization target object. Then, you can use MessagePackObject.UnderlyingType property or IsXxx properties to get the convertible type.
As you said, they are not obvious so documents should be improved, and I will provide documentation for 'using MPO to interpret dynamic type field'.

BTW, the entire code (with descriptive assertions) for dynamic typing looks like following:

// Gets dict which stores dynamic key/value pairs as dictionary object.
Dictionary<MessagePackObject, MessagePackObject> mpoDict = MessagePackSerializer.Create<Dictionary<MessagePackObject, MessagePackObject>>().UnpackFrom(...);

Debug.Assert( dict.Keys.All( v => v.IsTypeOf<String>() );
// Converts dict keys. Note that this line assumes there are no key duplications.
Dictionary<String, MessagePackObject> dict= mpoDict.ToDictionary( kv => (String)kv.Key, kv => kv.Value );

MessaePackObject theValue;
if ( dict.TryGetValue( "error", out theValue ) )
{
     // The data must be error.
     Debug.Assert( theValue.IsTypeOf<String>() );
    // Convert utf-8 binaries to a String object.
     var error = (String) theValue;
     ...
}
else if ( dict.TryGetValue( "results", out theValue ) )
{
    Debug.Assert( theData.IsList );
    // First get values as list.
    IList<MessagePackObject> values = (IList<MessagePackObject>)theValue;
    Debug.Assert( !values.Any() || values.All( v => v.IsTypeOf<Double>()) );
    // Convert values.
    var results = values.Select( v => (double )v );
    ...
}

Thanks!

Side note: I guess the source code of MPO looked like complex for numerics compatibility. By this feature, you can use floats or ints in 'results' field because MPO supports 'natural' numerics conversion. MPO just stores binary primitives and its type code for scalars, or object reference for non-scalar values, and makes efforts to handle numerics conversion properly.

[chrish42]

Thanks, this is useful. Having that in the documentation would have helped.

[yfakariya]

I updated wiki to describe issue14 related topic, and polymorphic scenario.

https://github.com/msgpack/msgpack-cli/wiki/Dynamic-type-handling

[chrish42]

I tried the sample code, and it does not work for me. I created a sample MsgPack message using the following Python code:

import msgpack
s = msgpack.dumps({'results': [1.0, 2.0]})
print s.encode('base64')

Feeding the output of the code above into this:

var s = "gadyZXN1bHRzkss/8AAAAAAAAMtAAAAAAAAAAA==";

byte[] rawReply = System.Convert.FromBase64String(s);

// Gets an object which stores dynamic key/value pairs as dictionary object.
Dictionary<MessagePackObject, MessagePackObject> mpoDict = 
  MessagePackSerializer.Create<Dictionary<MessagePackObject, MessagePackObject>>().Unpack(new MemoryStream(rawReply));

// Converts dict keys. Note that this line assumes there are no key duplications.
Dictionary<String, MessagePackObject> dict = mpoDict.ToDictionary( kv => (String)kv.Key, kv => kv.Value );

MessagePackObject theValue;
if (dict.TryGetValue( "error", out theValue ) )
{
     // The data must be error.
     Debug.Assert(theValue.IsTypeOf<String>() ?? false);
    // Convert utf-8 binaries to a String object.
     var error = (String)theValue;
}
else if ( dict.TryGetValue( "results", out theValue ) )
{
    Debug.Assert(theValue.IsList);
    // First get values as list.
    IList<MessagePackObject> values = theValue.AsList();
    // Convert values.
    var results = values.Select( v => (double?)v ?? -1000.0 );
}

... I get the following exception:

InvalidOperationException: Do not convert System.Byte (binary:0x2) MessagePackObject to System.Collections.Generic.IList`1[MsgPack.MessagePackObject].

on the line that calls theValue.AsList(). Inspecting the dict variable, it indeed seems to be a dict with a key of "results" and a corresponding value of a MessagePackObject for the number 2.

Let me know if you can reproduce this or now. Thanks!

[chrish42]

However, the following program works fine:

public class ReplyMessage
{
    // ReSharper disable InconsistentNaming
    public double[] results;
    public string error;
    // ReSharper restore InconsistentNaming
}


void Main()
{
    var s = "gadyZXN1bHRzkss/8AAAAAAAAMtAAAAAAAAAAA==";
    byte[] rawReply = System.Convert.FromBase64String(s);

    var serializer = MessagePackSerializer.Create<ReplyMessage>();
    var stream = new MemoryStream(rawReply);
    var replyMsg = serializer.Unpack(stream);

    if (replyMsg.error != null)
    {
        Console.WriteLine("Exception!");
    }
    else
    {
        replyMsg.results.Dump();
    }
}

(The Dump() call is a Linqpad thing... Feel free to replace it by Console.WriteLine(), etc.)

To my (potentially uneducated) eye, this looks like a bug...

[yfakariya]

Sorry for slow response.

It seems to be serializer bug of MPO as your smart eyes found. (You does not have to be so modest :) )

To workaround this, you can rewrite your code (from line 4 to line 9) as following:

// Because you know it should be treated as Dictionary, so just unpack it.
var dict = Unpacking.UnpackDictionary( rawReply ).Value;

Unpacking is utility providing convenient static APIs around Unpacker, and it does not use serialization stack (this means it is faster than MessagePackSerializer for this case).

I will fix this bug on next release. Thank you for your careful reporting!

[yfakariya]

I fixed this issue and released as 0.4.1.

[yfakariya]

There is a sample code now.
https://github.com/msgpack/msgpack-cli/blob/master/samples/Samples/Sample02_HandlingDynamicObject.cs