[QUESTION]: Example for wrapping std::variant

[gabrieldevillers]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

Problem description

Hello,

I am evaluating pybind11 and I was able to wrap many things without trouble, until trying to wrap a function taking a std::variant as argument. In the documentation I understand that I do not need to use type_caster because it it seems to me that it is already defined in pybind11/stl.h for std::variant. However I think I am missing something, because I would expect to have to call some py::class_ function specifically for the variant (?).

I found no example for wrapping a std::variant in the documentation or elsewhere on the internet or in pybind's test suite. Sorry if this is actually trivial.

Here is an example code and associated compilation error I have with MSVC:

pybind11\include\pybind11/pybind11.h(167): error C2280: 'pybind11::detail::argument_loader<pybind11::detail::value_and_holder &,MyVariant>::argument_loader(void)': attempting to reference a deleted function
pybind11\include\pybind11\cast.h(1445): note: compiler has generated 'pybind11::detail::argument_loader<pybind11::detail::value_and_holder &,MyVariant>::argument_loader' here
pybind11\include\pybind11\cast.h(1445): note: 'pybind11::detail::argument_loader<pybind11::detail::value_and_holder &,MyVariant>::argument_loader(void)': function was implicitly deleted because a data member 'pybind11::detail::argument_loader<pybind11::detail::value_and_holder &,MyVariant>::argcasters' has either no appropriate default constructor or overload resolution was ambiguous
pybind11\include\pybind11\cast.h(1444): note: see declaration of 'pybind11::detail::argument_loader<pybind11::detail::value_and_holder &,MyVariant>::argcasters'
ybind11\include\pybind11/pybind11.h(99): note: see reference to function template instantiation 'void pybind11::cpp_function::initialize<Ty,R,pybind11::detail::value_and_holder&,MyVariant,pybind11::name,pybind11::is_method,pybind11::sibling,pybind11::detail::is_new_style_constructor>(Func &&,Return (cdecl *)(pybind11::detail::value_and_holder &,MyVariant),const pybind11::name &,const pybind11::is_method &,const pybind11::sibling &,const pybind11::detail::is_new_style_constructor &)' being compiled
with
[
Ty=pybind11::detail::initimpl::constructor::execute::<lambda_4bce0e84265d89e846bdadb006aa47c7>,
R=void,
Func=pybind11::detail::initimpl::constructor::execute::<lambda_4bce0e84265d89e846bdadb006aa47c7>,
Return=void
]
pybind11\include\pybind11/pybind11.h(1557): note: see reference to function template instantiation 'pybind11::cpp_function::cpp_function<Ty,pybind11::name,pybind11::is_method,pybind11::sibling,pybind11::detail::is_new_style_constructor,void>(Func &&,const pybind11::name &,const pybind11::is_method &,const pybind11::sibling &,const pybind11::detail::is_new_style_constructor &)' being compiled
with
[
Ty=pybind11::detail::initimpl::constructor::execute::<lambda_4bce0e84265d89e846bdadb006aa47c7>,
Func=pybind11::detail::initimpl::constructor::execute::<lambda_4bce0e84265d89e846bdadb006aa47c7>
]
pybind11\include\pybind11\detail/init.h(199): note: see reference to function template instantiation 'pybind11::class &pybind11::class::def<pybind11::detail::initimpl::constructor::execute::<lambda_4bce0e84265d89e846bdadb006aa47c7>,pybind11::detail::is_new_style_constructor>(const char *,Func &&,const pybind11::detail::is_new_style_constructor &)' being compiled
with
[
Func=pybind11::detail::initimpl::constructor::execute::<lambda_4bce0e84265d89e846bdadb006aa47c7>
]
pybind11\include\pybind11\detail/init.h(200): note: see reference to function template instantiation 'pybind11::class &pybind11::class::def<pybind11::detail::initimpl::constructor::execute::<lambda_4bce0e84265d89e846bdadb006aa47c7>,pybind11::detail::is_new_style_constructor>(const char *,Func &&,const pybind11::detail::is_new_style_constructor &)' being compiled
with
[
Func=pybind11::detail::initimpl::constructor::execute::<lambda_4bce0e84265d89e846bdadb006aa47c7>
]
pybind11\include\pybind11/pybind11.h(1594): note: see reference to function template instantiation 'void pybind11::detail::initimpl::constructor::execute<pybind11::class,,0>(Class &)' being compiled
with
[
Class=pybind11::class
]
pybind11\include\pybind11/pybind11.h(1596): note: see reference to function template instantiation 'void pybind11::detail::initimpl::constructor::execute<pybind11::class_,,0>(Class &)' being compiled
with
[
Class=pybind11::class_
]
pybind_example.cpp(27): note: see reference to function template instantiation 'pybind11::class_ &pybind11::class_::def<MyVariant,>(const pybind11::detail::initimpl::constructor &)' being compiled
pybind_example.cpp(28): note: see reference to function template instantiation 'pybind11::class_ &pybind11::class_::def<MyVariant,>(const pybind11::detail::initimpl::constructor &)' being compiled

Reproducible example code

#include <variant>
#include <vector>

class MyClassA
{
public:
    MyClassA(double x) {}
};

class MyClassB
{
public:
    MyClassB(const std::vector<double>& x) {}
};

using MyVariant = std::variant<MyClassA, MyClassB>;

class MyTestClass
{
public:
    MyTestClass(const MyVariant& variant) {}
};

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
namespace py = pybind11;

PYBIND11_MODULE(MyModule, m) {

    py::class_<MyClassA>(m, "MyClassA")
        .def(py::init<double>());

    py::class_<MyClassB>(m, "MyClassB")
        .def(py::init<std::vector<double>>());

    py::class_<MyTestClass>(m, "MyTestClass")
        .def(py::init<MyVariant>()); // compilation problem here

}

Example code which does not reproduce

#include <variant>

class MyClassA
{
public:
    MyClassA() {}
};

class MyClassB
{
public:
    MyClassB(double x) {}
};

using MyVariant = std::variant<MyClassA, MyClassB>;

class MyTestClass
{
public:
    MyTestClass(const MyVariant& variant) {}
};

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
namespace py = pybind11;

PYBIND11_MODULE(MyModule, m) {

    py::class_<MyClassA>(m, "MyClassA")
        .def(py::init<>());

    py::class_<MyClassB>(m, "MyClassB")
        .def(py::init<double>());

    py::class_<MyTestClass>(m, "MyTestClass")
        .def(py::init<MyVariant>()); // compilation problem here

}

[gabrieldevillers]

Sorry, my code does not reproduce my error, I will try again to extract something which does.

[gabrieldevillers]

Code updated, I think that it reproduces. The difference between the code that does not shows the error and the one that does, is that one of the two classes constructor takes a const std::vector& as argument.

[gabrieldevillers]

I think that the problem is literally that 'pybind11::detail::argument_loader<pybind11::detail::value_and_holder &,MyVariant>::argcasters' has either no appropriate default constructor because MyVariant has a default constructor only in the example that does not reproduce.

I think this pull request integrated in 2.10.0 could help. I may just have to figure out the syntax.

[gabrieldevillers]

So changing

using MyVariant = std::variant<MyClassA, MyClassB>;

to

using MyVariant = std::variant<std::monostate, MyClassA, MyClassB>;

solves the compilation problem. But this is not ideal for me because that requires changing the C++ code and forces to handle a new case that might make no sense in the codebase. If there was a way to disable some constructors on the C++ side that would be better but I found no way to do that.

[gabrieldevillers]

Proposition of alternative solution, that allows not polluting the C++ code with unwanted std::monostate in variants:

py::class_<MyTestClass>(m, "MyTestClass")
    .def(py::init<>([](const monostated<MyVariant>::type& v)
    {
        return MyTestClass(variant_cast_no_monostate(v));
    }));

The Python constructor MyTestClass will throw std::runtime_error("variant_cast_no_monostate received a value of type std::monostate, cannot cast it"); if it receives None.

Definition of template variant_cast_no_monostate:

// helper type for std::visit
template<class... Ts> struct overloaded : Ts... { using Ts::operator()...; };
// explicit deduction guide (not needed as of C++20)
template<class... Ts> overloaded(Ts...) -> overloaded<Ts...>;


// Implementation of variant_cast_no_monostate()
// inspired by https://stackoverflow.com/a/47204507
template <class... Args>
struct variant_cast_no_monostate_impl
{
    std::variant<Args...> v;

    template <class... ToArgs>
    operator std::variant<ToArgs...>() const
    {
        return std::visit(overloaded {
                          [](const std::monostate&) -> std::variant<ToArgs...>
                          {
                              throw std::runtime_error("variant_cast_no_monostate received a value of type std::monostate, cannot cast it");
                          },
                          [](auto&& arg) -> std::variant<ToArgs...>
                          {
                              return arg ;
                          },
                          }, v);
    }
};

// Will convert one variant value into a value of another variant type that
// needs not to have a std::monostate in his alternative types. So the value
// converted must not be of type std::monostate else an exception is thrown.
template <class... Args>
auto variant_cast_no_monostate(const std::variant<Args...>& v) -> variant_cast_no_monostate_impl<Args...>
{
    return {v};
}

Definition of template monostated:

// inspired by https://stackoverflow.com/a/52393977
// this struct is not meant to be instantiated, but to statically provide ::type
template <typename T, typename... Args> struct variant_prepender;

template <typename... Args0, typename... Args1>
struct variant_prepender<std::variant<Args0...>, Args1...> {
    using type = std::variant<Args1..., Args0...>;
};

// Create a variant type from another variant type by prepending its list of
// alternative types with std::monostate
// this struct is not meant to be instantiated, but to statically provide ::type
template <typename Variant> struct monostated : public variant_prepender<Variant, std::monostate>
{};

Does anyone think there is a simpler solution than this ? Are there drawbacks to this solution ?
I am a beginner with pybind11 so looking for opinion of experts. @rwgk @Skylion007

[rwgk]

Hm ... it almost looks like we have a general problem for std::variant<NoDefaultCtor>, does that sound right?

E.g. this does not compile:

#include <variant>

struct NoDefaultCtor {
    NoDefaultCtor(int) {}
};

int main() {
    std::variant<NoDefaultCtor> value;
    return 0;
}

$ clang++ -std=c++17 variant_not_default_ctor.cpp
variant_not_default_ctor.cpp:8:33: error: call to implicitly-deleted default constructor of 'std::variant<NoDefaultCtor>'
    std::variant<NoDefaultCtor> value;
                                ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1411:7: note: explicitly defaulted function was implicitly deleted here
      variant() = default;
      ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/variant:1345:7: note: default constructor of 'variant<NoDefaultCtor>' is implicitly deleted because base class '_Enable_default_constructor<__detail::__variant::_Traits<NoDefaultCtor>::_S_default_ctor, variant<NoDefaultCtor>>' has a deleted default constructor
      private _Enable_default_constructor<
      ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/enable_special_members.h:113:15: note: '_Enable_default_constructor' has been explicitly marked deleted here
    constexpr _Enable_default_constructor() noexcept = delete;
              ^
1 error generated.

The existing variant_caster uses PYBIND11_TYPE_CASTER, which requires that the value it holds is default-constructible.

The existing variant_caster could be changed to accommodate that situation, by not using PYBIND11_TYPE_CASTER, which means there will be more code.

If someone wants to try:

• Start a PR with a minimal unit test that fails ("test-driven development"). I think the above std::variant<NoDefaultCtor> could be such a minimal test.
• Work on the existing variant_caster implementation until the new test passes along will all existing tests.

I think the main trick will be to not hold the std::variant by value in the caster, but by e.g. std::unique_ptr, then to construct the held std::variant from the passed Python object(s).

It's almost a nicely confined exercise, but it doesn't look trivial.

[gnaggnoyil]

I'm too suffering from the fact that pybind11 requires std::variant to be default constructible when using type_caster that @rwgk mentioned. If this is intended behavior, please update the doc to clarify that.