Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce recursive_container_traits #4623

Merged
merged 16 commits into from
May 5, 2023

Conversation

franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented Apr 14, 2023

Description

This fix is necessary when trying to bind recursive map types such as:

struct RecursiveMap : std::map<int, RecursiveMap> {}

The old check (negation<std::is_same<Container, typename Container::value_type>>) is insufficient here, as Container::value_type is std::pair<int, RecursiveMap>. The check needs to additionally consider Container::mapped_type.

This PR introduces a new trait recursive_container_traits that defines a dependent type recursive_container_traits<Container>::type_to_check_recursively that guides the recursion scheme in recursively defined traits such as is_move_assignable and similar.

More complicated self-referential patterns (such as mutually referential lists and maps) are considered by this PR, but require manual intervention by the downstream user. The type_to_check_recursively needs to be manually specified. An example is given in tests/test_stl_binders.cpp.

Suggested changelog entry:

Introduce recursive_container_traits

@Skylion007
Copy link
Collaborator

@franzpoeschel Can you add test to our test suite that compiles (and maybe uses) one of these recursive types?

@rwgk
Copy link
Collaborator

rwgk commented Apr 14, 2023

@franzpoeschel Can you add test to our test suite that compiles (and maybe uses) one of these recursive types?

I took a quick look and have questions about details, but without any tests I want to wait.

I'll look again when there are new test cases that fail without changes. (In the long run the tests are more important than the fix.)

@franzpoeschel
Copy link
Contributor Author

Is there a test where adding this would logically make sense or should I just create a new one?

@rwgk
Copy link
Collaborator

rwgk commented Apr 14, 2023

Is there a test where adding this would logically make sense or should I just create a new one?

Adding is a little easier to get started.

First quick guess: test_stl.cpp,py

We can easily move the new test code elsewhere at the end, when we see the full picture.

FWIW

If I anticipate I may need to experiment a lot, I usually start a new test, to not have to deal with the overhead & distractions of existing tests in the same files.

@franzpoeschel
Copy link
Contributor Author

Thanks for the guidance!
I don't expect much experimentation as this fix is for a very clearly-defined use case, so the STL test sounds like a good fit.
I will continue with this next week.

struct is_comparable<T,
enable_if_t<container_traits<T>::is_vector &&
// Avoid this instantiation if the type is recursive
negation<std::is_same<T, typename T::value_type>>::value>> {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment above the specialization mentions map types, but the enable_if condition requires vector types.
Hence, I only added the simple check here.

@franzpoeschel
Copy link
Contributor Author

I have added a test, and additionally an optional PR that shows the boundaries of the recursion check and an alternative way to bind more complex recursive map/vector types.
I think, someone needs to start the CI since I'm not approved to (last week, it started automatically after the precommit bot's push)

@rwgk
Copy link
Collaborator

rwgk commented Apr 18, 2023

someone needs to start the CI
Done

(will look at the code later today)

py::bind_vector<RecursiveVector>(m, "RecursiveVector");
py::bind_map<RecursiveMap>(m, "RecursiveMap");
py::bind_map<MutuallyRecursiveMap>(m, "MutuallyRecursiveMap");
py::bind_vector<MutuallyRecursiveVector>(m, "MutuallyRecursiveVector");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like these need to go before this block:

    // The rest depends on numpy:
    try {
        py::module_::import("numpy");
    } catch (...) {
        return;
    }

… to avoid the CI failures:


    def test_recursive_containers():
>       recursive_vector = m.RecursiveVector()
E       AttributeError: module 'pybind11_tests.stl_binders' has no attribute 'RecursiveVector'

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like these need to go before this block:

At first glance: that doesn't sound good. "Something else is probably wrong, somewhere."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that there is a large problem. I just previously put the bindings in a place that was only active if numpy is available, so any test without numpy failed as a consequence. Should be fixed now.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, no problem at all: looking now at your latest commit, I misunderstood the before/after in your previous comment.

@rwgk
Copy link
Collaborator

rwgk commented Apr 19, 2023

(Waiting for CI to finish. The 1 failure is almost certainly a flake. I'll rerun.)

@@ -827,6 +827,18 @@ using movable_cast_op_type
template <typename T, typename SFINAE = void>
struct is_copy_constructible : std::is_copy_constructible<T> {};

// True if Container has a dependent type mapped_type that is equivalent
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we generalize this and at the same time make this more uniform?

But a silly thing first: the naming here completely throws me off. It kind-of makes sense with my mind in the context (rare occasion), but imaging myself skimming through this while debugging or working on something completely different (the usual situation), my snap-judgement of what this is doing would go in a completely wrong direction.

I'll use is_container_with_self_referential_mapped_type below.

Currently we have two special cases:

std::is_same<Container, Container::value_type>
is_container_with_self_referential_mapped_type<Container>

Could we turn this into

is_self_referential_container_type<Container>

that

  1. invokes the other two?
  2. can be specialized by users?
struct is_self_referential_container_type<MySelfReferentialContainerType> : std::true_type {};

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have done this and I like the way that it turned out, especially since adding custom user specializations now involves overriding a single struct, which just tells Pybind that this type is recursive:

namespace pybind11 {
namespace detail {
template <>
struct is_container_with_self_referential_mapped_type<MutuallyRecursiveContainerPairB>
    : std::true_type {};
template <>
struct is_container_with_self_referential_mapped_type<MutuallyRecursiveContainerPairA>
    : std::true_type {};
} // namespace detail
} // namespace pybind11

// to Container itself
template <typename Container, typename MappedType = Container>
struct map_self_referential {
constexpr static bool value = false;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we inherit from std::false_type here (and std::true_type below)? I think it would be better/more readable to do that consistently, at least in all new or changed code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I only found out that you typically use these after writing the above code. Changed it.

/*
* Pybind11 does not catch more complicated recursion schemes, such as mutual
* recursion.
* In that case, an alternative is to add a custom overload.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you mean "custom template specialization"? (The term "overload" is usually only used for function overloads.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's what I meant, it's updated now.

namespace pybind11 {
namespace detail {
template <>
struct is_comparable<MutuallyRecursiveVector, void> : std::true_type {};
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unusual to specify the conventional SFINAE void in specializations, could that be omitted here (and everywhere below)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've separated the SFINAE type now from the publically specializable type:

template <typename Container, typename MappedType = Container>
struct sfinae_is_container_with_self_referential_mapped_type : std::false_type {};

// ... specializations for mapped_type and value_type ...

// wrapper type without SFINAE parameter
template <typename Container>
struct is_container_with_self_referential_mapped_type
    : sfinae_is_container_with_self_referential_mapped_type<Container> {};

This gives the struct a cleaner public API for specialization purposes, and the SFINAE parameter can be publically ignored.

* In that case, an alternative is to add a custom overload.
*/
struct MutuallyRecursiveMap;
struct MutuallyRecursiveVector;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's only test code, but anyway, careful naming in tests is important, too.

What do you think about

struct MutuallyRecursiveContainerPairA;
struct MutuallyRecursiveContainerPairB;

?

Two motivations:

  • Bake "pair" into the name. (Only a pair can have something mutual.)

  • Bake in a hint that Map and Vector are more-or-less arbitrary choices for the purpose of testing something more general.

These are long names, but they are also very clear at first sight, which I think is best for something that only ever appears in a few places.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've used these suggestions now. (Personally, I would prefer keeping the reference to "vector" and "map", since that information is very relevant in using the types later on. But since you're the maintainer, this is your decision more than it is mine.)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about making this

struct MutuallyRecursiveContainerPairMV;
struct MutuallyRecursiveContainerPairVM;

?
I.e. a compromise. The Pair is there, and M referencing V, V referencing M (I think M and V are clear enough, we don't need to spell out MutuallyRecursiveContainerPairMapVector, MutuallyRecursiveContainerPairVectorMap).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally, I find this far less confusing, I'm using that now.

tests/test_stl_binders.py Show resolved Hide resolved

recursive_map = m.RecursiveMap()
recursive_map[1] = m.RecursiveMap()
recursive_map[1][1] = m.RecursiveMap()
Copy link
Collaborator

@rwgk rwgk Apr 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test failures later (new compiler with bugs, refactoring with bugs) will be easier to debug if you only go as deep as you need to. I think that's just one level here. The extra level is is also useful as an additional stress test, but I'd definitely not skip the more minimal easier to debug test. I think it's only a couple lines extra (assert list(recursive_map[1].keys()) == [1, 2]).

In a similar vein, although slightly on the paranoid side, using values less special than 0, 1, 2 can help a lot in troubleshooting later, e.g. 101, 202, anything that's more unique, easier to pin-point in a pile of debug output, and less likely to appear by accident.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reduced the recursion depth for each test and used more Ctrl+F-friendly numbers for the RecursiveMap-test now


// Use a helper struct in order to give this a nicer public API without helper template parameter.
// This makes this struct nicer to specialize by users.
template <typename Container>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

template <typename Container, typename SFINAE = void>

We don't want SFINAE to stop here, users may still need enable_if for their own types.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I've now done it in such a way that the SFINAE parameter of the publically specializable struct defaults to void as is usual for SFINAE.

// Use a helper struct in order to give this a nicer public API without helper template parameter.
// This makes this struct nicer to specialize by users.
template <typename Container>
struct is_container_with_self_referential_mapped_type
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure, but tossing this around in my mind, I feel

is_recursive_container

would be great. It takes a small mental leap back to "self-referential", vs a small leap forward from "self-referential" to "recursive". "recursive" is shorter, a little more intuitive at least in my mind, and we don't have the trouble with mapped_type vs value_type, which is is more-or-less a case-specific detail.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

// The specializations are only valid if both conditions are fulfilled:
// 1) The mapped_type (respectively value_type) exists
// 2) And it is equivalent to Container
// So, in each case, only one specialization will activate.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a really neat trick.

What do you think about is_recursive_container_common as a name (see other comment for the is_recursive_container part).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you noticed below, there is a catch to the trick, so I've split this in two now for structs like

struct Self
{
    using mapped_type = Self;
    using value_type = Self;
};

I like is_recursive_container, using that now


template <typename Container>
struct sfinae_is_container_with_self_referential_mapped_type<Container,
typename Container::value_type>
Copy link
Collaborator

@rwgk rwgk Apr 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mainly just disclosing that I'm not sure about something: could there ever be a non-non-sensical container that has both mapped_type and value_type? And what would happen then?

Maybe that's a non-sense thought? And in the worst case we could fix when we actually get into trouble.

But also thinking: do we really need this helper type? Totally untested:

struct is_recursive_container<Container, enable_if_t<std::is_same<Container, Container::value_type>::value>> : std::true_type;

struct is_recursive_container<Container, enable_if_t<std::is_same<Container, Container::mapped_type>::value>> : std::true_type;

or

struct is_recursive_container<Container, enable_if_t<!std::is_same<Container, Container::value_type>::value && std::is_same<Container, Container::mapped_type>::value>> : std::true_type;

in case that could ever happen?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are right. And it's not so nonsensical either, I can imagine that some vector-like structs exist somewhere, that additionally define mapped_type for some reason.

The following struct fails selecting a correct instantiation:

struct Self
{
    using mapped_type = Self;
    using value_type = Self;
};

The reason why we need helper types is because vector types don't define a mapped_type, so SFINAE is necessary to check if that even exists. value_type always exists for STL containers, but at that point I'd say it's a matter of consistency.

I'm using two different helper structs for mapped_type and value_type now and combining them via any_of, so there is no need at all for tie breaking:

// True if Container has a dependent type mapped_type that is equivalent
// to Container itself
// Actual implementation in the SFINAE specialization below
template <typename Container, typename MappedType = Container>
struct is_container_with_recursive_mapped_type : std::false_type {};

// This specialization is only valid if both conditions are fulfilled:
// 1) The mapped_type exists
// 2) And it is equivalent to Container
template <typename Container>
struct is_container_with_recursive_mapped_type<Container, typename Container::mapped_type>
    : std::true_type {};

// True if Container has a dependent type value_type that is equivalent
// to Container itself
// Actual implementation in the SFINAE specialization below
template <typename Container, typename MappedType = Container>
struct is_container_with_recursive_value_type : std::false_type {};

// This specialization is only valid if both conditions are fulfilled:
// 1) The value_type exists
// 2) And it is equivalent to Container
template <typename Container>
struct is_container_with_recursive_value_type<Container, typename Container::value_type>
    : std::true_type {};

// True constant if the type contains itself recursively.
// By default, this will check the mapped_type and value_type dependent types.
// In more complex recursion patterns, users can specialize this struct.
// The second template parameter SFINAE=void is for use of std::enable_if in specializations.
// An example is found in tests/test_stl_binders.cpp.
template <typename Container, typename SFINAE = void>
struct is_recursive_container_type : any_of<is_container_with_recursive_value_type<Container>,
                                            is_container_with_recursive_mapped_type<Container>> {};

franzpoeschel and others added 9 commits April 24, 2023 15:30
1) Apply to is_copy_assignable additionally
2) Check infinite recursion for map-like types
The bindings were previously in a block that was only activated if numpy
was available.
1) Renaming: is_recursive_container and
   MutuallyRecursiveContainerPair(MV|VM)
2) Avoid ambiguous specializations of is_recursive_container
@franzpoeschel
Copy link
Contributor Author

The master branch has newly introduced is_move_constructible, which does not do any kind of recursive checks, but caused a conflict with this branch.
I have rebased the branch now, leaving is_move_constructible untouched.

@rwgk
Copy link
Collaborator

rwgk commented Apr 24, 2023

The master branch has newly introduced is_move_constructible, which does not do any kind of recursive checks, but caused a conflict with this branch. I have rebased the branch now, leaving is_move_constructible untouched.

Oh, gee, sorry, I was totally sleeping on the wheel there. (Interesting coincidence!)

Could you please add the recursive checks to is_move_constructible as well?

Copy link
Collaborator

@rwgk rwgk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple small things I noticed glancing through.

I only have a tiny (laptop) screen right now and can look carefully only later today.

Overall, this looks really great already.

/*
* Pybind11 does not catch more complicated recursion schemes, such as mutual
* recursion.
* In that case, an alternative is to add a custom specialization to
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case custom is_recursive_container specializations need to be added.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

enable_if_t<container_traits<T>::is_vector &&
// Special case: The vector type is recursive
std::is_same<T, typename T::value_type>::value>> {
static constexpr const bool value = true;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true_type?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here it was me sleeping on the wheel:

This is now:

template <typename T>
struct is_comparable<T,
                     enable_if_t<container_traits<T>::is_vector &&
                                 // Special case: The vector type is recursive
                                 is_recursive_container<T>::value>> {
    static constexpr const bool value = container_traits<T>::is_comparable;
};

@rwgk rwgk requested a review from Skylion007 April 25, 2023 22:46
@@ -129,6 +163,12 @@ TEST_SUBMODULE(stl_binders, m) {
m, "VectorUndeclStruct", py::buffer_protocol());
});

// Bind recursive container types
py::bind_vector<RecursiveVector>(m, "RecursiveVector");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these tests cover the detail:is_move_constructable templates too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added static testing for the pybind11 type traits now (and coincidentally, uncovered a slight bug in pybind11::detail::is_copy_assignable)

@franzpoeschel
Copy link
Contributor Author

Very nice, thanks a lot!

Thank you too for your help in shaping this PR!

For context: We have been using pybind11 for a long time now in our openPMD-api project, and the larger share of our users likely use it via the Pybind11-shaped Python bindings.
I am currently experimenting with an extension that will allow arbitrary-depth user-defined hierarchical structures, where we will need support for recursive data structures in Pybind11.

Copy link
Collaborator

@rwgk rwgk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! The extra tests look like they were a lot of leg work, thanks for being so thorough!


/*
* Rest of the file:
* static_assert based tests for pybind11 adaptations of of
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

of of

* Need to remove the const qualifier from T1 here since the value_type in
* STL map types is std::pair<const Key, T>, and const types are never assignable
*/
: all_of<is_copy_assignable<typename std::remove_const<T1>::type>, is_copy_assignable<T2>> {};
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The asymmetry here seems odd, did you already think this through?

Concretely, would it make sense to apply std::remove_const also to T2?

Because for std::pair by itself (not considering how it's used elsewhere), the order of T1 and T2 doesn't have a function AFAIK.

Which leads me to think: It has to be either none or both.

Or we actually have a trait with subtly different semantics compared to the "pure" is_copy_assignable.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on this point, the asymmetry is a bit odd.

Copy link
Contributor Author

@franzpoeschel franzpoeschel May 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PLEASE DO NOT REVIEW YET, I need to go over this on my own again first.

Fixing this required changing the logic of this PR quite a bit.
The trouble of is_recursive_container<T> is that it only tells you that the type is recursive at all and then tells you to deal with it somehow. What I did now instead was replacing that with a recursive_container_traits that has a dependent type recursive_container_traits<Container>::type_to_check_recursively:

/*
 * Tag to be used for representing the bottom of recursively defined types.
 * Define this tag so we don't have to use void.
 */
struct recursive_bottom {};

template <typename Container, typename SFINAE = void>
struct impl_recursive_container_traits {
    using type_to_check_recursively = recursive_bottom;
};

// and its main logic in the specialization

The type_to_check_recursively is defined such:

  1. By default, it is recursive_bottom, so that the recursion is canceled.
  2. If the type is non-recursive and defines a value_type, then the value_type is used. If the value_type is a pair, the const` is removed from the first type.
  3. If the type is recursive and value_type is not a pair, then recursive_bottom is returned.
  4. If the type is recursive and value_type is a pair, then const is removed from the first type and the first type is returned.

Having this trait makes the definitions of custom traits such as is_move_constructible very straightforward and no longer requires SFINAE constructs there:

template <typename T>
struct is_move_constructible
    : all_of<std::is_move_constructible<T>,
             is_move_constructible<
                 typename recursive_container_traits<T>::type_to_check_recursively>> {};

template <>
struct is_move_constructible<recursive_bottom> : std::true_type {};

// Likewise for std::pair
// (after C++17 it is mandatory that the move constructor not exist when the two types aren't
// themselves move constructible, but this can not be relied upon when T1 or T2 are themselves
// containers).
template <typename T1, typename T2>
struct is_move_constructible<std::pair<T1, T2>>
    : all_of<is_move_constructible<T1>, is_move_constructible<T2>> {};

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixing this required changing the logic of this PR quite a bit.

Oh, wow. Thanks a lot for going the extra mile!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've fixed some comments now, should be ready for review :)

@franzpoeschel franzpoeschel force-pushed the fix-infinite-recursion-check branch 2 times, most recently from 46cfdbb to 517b48e Compare May 2, 2023 14:55
Instead of a trait `is_recursive_container`, use a trait
`recursive_container_traits` with dependent type
`recursive_container_traits::type_to_check_recursively`.
So, instead of just checking if a type is recursive and then trying to
somehow deal with it, recursively-defined traits such as
is_move_constructible can now directly ask this trait where the
recursion should proceed.
Copy link
Collaborator

@rwgk rwgk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really awesome now. I could only spend 15 minutes right now, I need to find a block of time to go through more carefully (asap).

Comment on lines 825 to 836
template <bool Condition, typename Then, typename Else>
struct if_then_else {};

template <typename T, typename SFINAE = void>
struct is_move_constructible : std::is_move_constructible<T> {};
template <typename Then, typename Else>
struct if_then_else<true, Then, Else> {
using type = Then;
};

template <typename Then, typename Else>
struct if_then_else<false, Then, Else> {
using type = Else;
};
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you reinvented std::conditional.

Also, we have conditional_t in pybind11/detail/common.h.

I like if_then_else as a name a lot better than conditional btw, but standard is standard :-)

1. Use std::conditional
2. Fix typo
Copy link
Collaborator

@rwgk rwgk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm running this through Google's global testing, to guard against surprises. That will take 6-8 hours to complete.

If you get a chance to update the PR title & description that would be great. Maybe the title could be "Introduce recursive_container_traits".

For the record: I wish recursive_container_traits lived in it's own include file, but looking through the changes, I got to think that's beyond of what can be achieved in one PR, given the current state of the code organization.

@@ -10,6 +10,7 @@
#include <pybind11/numpy.h>
#include <pybind11/stl_bind.h>

#include "pybind11/detail/type_caster_base.h"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please remove this include? To not encourage others to include a detail header. — The pybind11 code organization has me wishing a lot. Improving the organization will be harder if we implicitly suggest that the detail headers are part of the public interface.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I accidentally left it there before committing.

@franzpoeschel franzpoeschel changed the title type_caster_base.h: extend infinite recursion checks for is_copy_constructible and is_copy_assignable Introduce recursive_container_traits May 4, 2023
@franzpoeschel
Copy link
Contributor Author

franzpoeschel commented May 4, 2023

I'm running this through Google's global testing, to guard against surprises. That will take 6-8 hours to complete.

If you get a chance to update the PR title & description that would be great. Maybe the title could be "Introduce recursive_container_traits".

Done

For the record: I wish recursive_container_traits lived in it's own include file, but looking through the changes, I got to think that's beyond of what can be achieved in one PR, given the current state of the code organization.

As I did not really know the code structure of Pybind, I introduced new structs/traits where I needed to use them, which is probably not necessarily the best placement. If things should be moved within the scope of this PR, we can do it.
(Also, I think you should be able to push to the branch if you prefer doing this yourself)

@rwgk
Copy link
Collaborator

rwgk commented May 4, 2023

If things should be moved within the scope of this PR,

If you want to give it a try, I'l re-review.

When going through the motions just in my mind I got to think it's to difficult to untangle the new code from the existing traits (is_copy_constructible etc.). Maybe I'm too pessimistic. But then again, I think the current state is a great milestone to maybe improve from in follow-on work. Also, there are far bigger organizational problems (e.g. cast.h and type_caster_base.h far too monolithic), is a focus on recursive_container_traits actually the best starting point for improving the code organization? I doubt it.

@franzpoeschel
Copy link
Contributor Author

In that case, I'd leave things in the current state.

@rwgk
Copy link
Collaborator

rwgk commented May 5, 2023

Global testing found no issues (internal test ID is (OCL:529394047:BASE:529572152:1683249617215:288a60c7).

The one GHA failure here is a notorious test_iostream flake.

Thanks for the great work!

@rwgk rwgk merged commit f701654 into pybind:master May 5, 2023
@github-actions github-actions bot added the needs changelog Possibly needs a changelog entry label May 5, 2023
@rwgk
Copy link
Collaborator

rwgk commented May 5, 2023

Are these two sentences in the PR description still accurate?

Since mapped_type does not exist for all container-like types, this uses SFINAE in the helper struct map_self_referential.

More complicated self-referential patterns (such as mutually referential lists and maps) are not considered by this PR.

Maybe it's better to point to the now-merged test code?

@franzpoeschel
Copy link
Contributor Author

I have updated the PR description to the final state of the PR.
Thanks again for the help!

@rwgk
Copy link
Collaborator

rwgk commented May 5, 2023

I have updated the PR description to the final state of the PR. Thanks again for the help!

Awesome, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants