Support `boost::variant` of known (custom) classes in array format

[starthal]

In my application, I use MSGPACK_DEFINE to pack custom classes into the array format. The format is known on both ends and they are both C++ (for now).

I'd like to introduce a limited form of "dynamic typing" to the message. I want it include a heterogeneous array, where each element may either be Foo or Bar.

Internally I have this implemented using std::vector<boost::variant<Foo, Bar> >, and I am hacking into the message a format basically equivalent to msgpack-cli's known-subtype polymorphism. I use typeid(T).name() as the type indicator, which is of course unportable. And the custom (un)pack is verbose and specific to variant<Foo, Bar>.

I wonder if there is a more robust way to do this with templates, and if so, is it worth implementing into the library?

Of course the map format doesn't have this issue because everything is dynamically typed. And I know it's already possible to convert objects to generic recursive variants. But the compact and static array format suits my application better.

[redboltz]

Internally I have this implemented using std::vector<boost::variant<Foo, Bar> >, and I am hacking into the message a format basically equivalent to msgpack-cli's known-subtype polymorphism. I use typeid(T).name() as the type indicator, which is of course unportable. And the custom (un)pack is verbose and specific to variant<Foo, Bar>.

I'm not sure but Boost.TypeIndex could be more portable.
http://www.boost.org/doc/libs/1_60_0/doc/html/boost_typeindex.html
What do you think?

I wonder if there is a more robust way to do this with templates, and if so, is it worth implementing into the library?

That's interesting. I implemented known-subtype polymorphism. It is just a PoC.

http://wandbox.org/permlink/TLPCRAvfZzZuPpKa

namespace msgpack {
MSGPACK_API_VERSION_NAMESPACE(MSGPACK_DEFAULT_API_NS) {
namespace adaptor {

template <typename... Args>
struct pack<boost::variant<Args...>> {
    template <typename Stream>
    msgpack::packer<Stream>& operator()(msgpack::packer<Stream>& s, boost::variant<Args...> const& v) const {
        boost::apply_visitor([&s](auto const& e){
            s.pack_array(2);
            s << boost::typeindex::type_id<decltype(e)>().pretty_name();
            s << e;
        }, v);
        return s;
    }
};

template<typename... Args>
struct convert<boost::variant<Args...>> {
    msgpack::object const& operator()(msgpack::object const& o, boost::variant<Args...>& v) const {
        std::map<std::string, std::function<void(boost::variant<Args...>&)>> table;
        boost::mpl::for_each<typename boost::variant<Args...>::types>(
            [&table, &v](auto e){
                table.emplace(
                    boost::typeindex::type_id<decltype(e)>().pretty_name(), 
                    [&e](boost::variant<Args...>& v) {
                        v = decltype(e)();
                    }
                );
            }
        );
        table[std::string(o.via.array.ptr[0].via.str.ptr, o.via.array.ptr[0].via.str.size)](v);

        boost::apply_visitor([&o](auto& e){
            o.via.array.ptr[1].convert(e);
        }, v);
        return o;
    }
};

}}}

I'm not sure it is suitable for a boost::variant's default adaptor. Perhaps it should provide as a utility. After I implemented the PoC code, I come up with some question.

• In the convert class template, the variable table creates every time. Is there any good way to avoid it?
• Is that possible to implement as class template that doesn't require default-constructible to variant arguments types? I guess it's not possible at least in generic implementation.
• I think that I can use a type index instead of a type name. It's smaller size and faster. And also solves type name potability problem. However, both packer and converter need to know the order of variant arguments.

I started thinking that it is difficult to decide by library developer. I think that adding documents that contain the implementation above to https://github.com/msgpack/msgpack-c/wiki is good solution.

Any ideas?

[starthal]

I love that implementation!

Unfortunately my application is using msgpack-rpc and it doesn't seem to support the C++11 headers yet, so I have to build with -std=c++03. Does that prevent a msgpack-rpc user from using variadic templates in this way?

I think that I can use a type index instead of a type name. It's smaller size and faster. And also solves type name potability problem. However, both packer and converter need to know the order of variant arguments.

This is true, but I think the sender and receiver can be expected to know the order just as easily as they know the name. However, it looks like there is also a type_id<T>.hash_code() which could work as long as it's portable.

I started thinking that it is difficult to decide by library developer. I think that adding documents that contain the implementation above to https://github.com/msgpack/msgpack-c/wiki is good solution.

I think I agree. Including it in the library implies that the [type_spec, object] is some standard way of describing the types, but there won't actually be a standard way unless the spec evolves to include this kind of capability.

---

BTW, I thought of another nice use case that others may be interested in. When I implemented a custom messaging channel using msgpack for a different application, I specified the message format using a single struct. But when I needed to extend the system to support different types of messages on each channel, I had to design my own method of including an index and discriminating between types. This variant method would have eliminated those issues.

[davidchappelle]

Pretty sure that msgpack-rpc-boost is c++11 compliant. If not I believe I
have a branch that is.

On Tue, May 3, 2016 at 4:18 PM, Stephen Albert notifications@github.com
wrote:

I love that implementation!

Unfortunately my application is using msgpack-rpc and it doesn't seem to
support the C++11 headers yet, so I have to build with -std=c++03. Does
that prevent a msgpack-rpc user from using variadic templates in this way?

I think that I can use a type index instead of a type name. It's smaller
size and faster. And also solves type name potability problem. However,
both packer and converter need to know the order of variant arguments.

This is true, but I think the sender and receiver can be expected to know
the order just as easily as they know the name. However, it looks like
there is also a type_id.hash_code() which could work as long as it's
portable.

I started thinking that it is difficult to decide by library developer. I
think that adding documents that contain the implementation above to
https://github.com/msgpack/msgpack-c/wiki is good solution.

I think I agree. Including it in the library implies that the [type_spec,
object] is some standard way of describing the types, but there won't
actually be a standard way unless the spec evolves to include this kind of

capability.

BTW, I thought of another nice use case that others may be interested in.
When I implemented a custom messaging channel using msgpack for a different
application, I specified the message format using a single struct. But when
I needed to extend the system to support different types of messages on
each channel, I had to design my own method of including an index and
discriminating between types. This variant method would have eliminated
those issues.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#462 (comment)

[starthal]

@davidchappelle Thanks. I wasn't aware of this fork -- I've been using the Jubatus one. Any quirks I should be aware of if I migrate?

[davidchappelle]

Shouldn't be any other than the fact that you will be using boost::asio
instead of whatever event loop that msgpack-rpc is using. The fork we were
working on is:

https://github.com/davidchappelle/msgpack-rpc-boost

Pretty sure that the upstream fork is way out of date at this point. I
spent a lot of work cleaning things up and making things c++11 compliant.

On Tue, May 3, 2016 at 5:59 PM, Stephen Albert notifications@github.com
wrote:

@davidchappelle https://github.com/davidchappelle Thanks. I wasn't
aware of this fork -- I've been using the Jubatus
https://github.com/jubatus/jubatus-msgpack-rpc one. Any quirks I should
be aware of if I migrate?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#462 (comment)

[starthal]

@davidchappelle It's working great for me (after a brief battle with the linker regarding asio). I appreciate your work on this. You should consider asking msgpack-rpc-cpp to link to this fork as well as Jubatus'.

[redboltz]

Unfortunately my application is using msgpack-rpc and it doesn't seem to support the C++11 headers yet, so I have to build with -std=c++03. Does that prevent a msgpack-rpc user from using variadic templates in this way?

I think that there are two ways to do that. One choice is a code generation. msgpack-c uses a code generation mechanism by ruby.

Source:
https://github.com/msgpack/msgpack-c/blob/master/erb/v1/cpp03_msgpack_tuple.hpp.erb

Generator:
https://github.com/msgpack/msgpack-c/blob/master/preprocess

Generated code:
https://raw.githubusercontent.com/msgpack/msgpack-c/master/include/msgpack/v1/adaptor/detail/cpp03_msgpack_tuple.hpp

The other choice is using Boost.Preprocessor. http://www.boost.org/doc/libs/1_60_0/libs/preprocessor/doc/index.html

[starthal]

I think a solution is implemented with #1075.