-
Notifications
You must be signed in to change notification settings - Fork 2.2k
feat: reapply fixed version of #3271 #3293
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@bmerry this should be a good starting point. Fixing this seems to be tripping up the new code, it's trying to make a copy of NonCopyableInt. Could you take a look? I can give you access to the branch if needed.
|
Fixed most of them by adding the double parentheses back in - should probably make a note that double parentheses are required there. Don't know why, would like to be able to say why - it was true across all compilers. Looks like at least ICC is unhappy now, but most are fine. Edit: ee0c5ee#r56899065 explains the parens. |
Ping @rainwoodman |
These are the three compilers that can't handle it:
PGI, NVCC, Intel ICC, which have had some historical problems with avoiding copies; and I expect my test is triggering valgrind. Also okay to revert #3271 then try again with this on top, if @bmerry is not available to fix this till next week and no one can fix these compilers. This is a pretty serous regression in master that was discovered <24 hours by a user after merging. (Note that a "fix" could simply mean not compiling this "non-copyable" test on those compilers, but some attempt should be made to understand/fix first. Can also not run the test breaking valgrind at runtime, as it doesn't really do anything). |
Reverting sounds good to me.
…On Wed, Sep 22, 2021 at 17:35 Henry Schreiner ***@***.***> wrote:
These are the three compilers that can't handle it:
X 🐍 3 • CentOS7 / PGI 20.9 • x64 15m19s https://github.com/pybind/pybind11/runs/3680311051
X 🐍 3 • ICC latest • x64 4m56s https://github.com/pybind/pybind11/runs/3680313285
X 🐍 3.8 • CUDA 11 • Ubuntu 20.04 5m23s https://github.com/pybind/pybind11/runs/3680312770
X 🐍 3.9-dbg (deadsnakes) • Valgrind • x64 14m39s https://github.com/pybind/pybind11/runs/3680309032
PGI, NVCC, Intel ICC, and I expect my test is triggering valgrind. Also
okay to revert #3271 <#3271> then
try again with this on top, if @bmerry <https://github.com/bmerry> is not
available to fix this till next week and no one can fix these compilers.
This is a pretty serous regression in master that was discovered <24 hours
after merging by a user.
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
<#3293 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFUZAEH62O2TMNLWWLMHH3UDJD3XANCNFSM5ESC7HXQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
f1e6b20
to
b7338ac
Compare
Going to revert, then move the rest of the changes into here. |
14e4f26
to
06a9632
Compare
Ideally, tests should be pulled out and applied to master first. |
Hi @henryiii. Thanks for working on this - seems like you have more time for it this week than I do. From a quick look you're taking a similar approach to what I was thinking of. However, I'd suggest you try not using the |
include/pybind11/pybind11.h
Outdated
template <typename Iterator, typename ResultType = decltype((*std::declval<Iterator>()))> | ||
struct iterator_access { | ||
using result_type = ResultType; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
template <typename Iterator, typename ResultType = decltype((*std::declval<Iterator>()))> | |
struct iterator_access { | |
using result_type = ResultType; | |
template <typename Iterator, typename SFINAE = decltype((*std::declval<Iterator>()))> | |
struct iterator_access { | |
// Note: can't use SFINAE template parameter to define result_type due to a bug in ICC | |
using result_type = decltype((*std::declval<Iterator>())); |
See if that works (will need to be applied similarly to iterator_key_access and iterator_value_access.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this fixes all three "special" compilers, but breaks every version of MSVC.
D:\a\pybind11\pybind11\include\pybind11/pybind11.h(1991,1): error C2440: 'return': cannot convert from 'const _Ty2' to 'const int &&' [D:\a\pybind11\pybind11\tests\pybind11_tests.vcxproj]
with
[
_Ty2=int
]
D:\a\pybind11\pybind11\include\pybind11/pybind11.h(1991,21): message : You cannot bind an lvalue to an rvalue reference [D:\a\pybind11\pybind11\tests\pybind11_tests.vcxproj]
D:\a\pybind11\pybind11\include\pybind11/pybind11.h(1990): message : while compiling class template member function 'const int &&pybind11::detail::iterator_value_access<Iterator,const _Ty2 &&>::operator ()(Iterator &) const' [D:\a\pybind11\pybind11\tests\pybind11_tests.vcxproj]
with
[
Iterator=const std::pair<int,int> *,
_Ty2=int
]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is extremely weird. Is this just by adding the SFINAE template parameter (relative to what was in #3271), or is it some interaction with the additional testing you added?
I see you've tried to make a workaround with add_lvalue_reference
. I suspect that is going to cause other problems if the iterator dereferencing doesn't produce a reference (such as an input iterator). The next thing I would try is changing std::declval<Iterator>()
to std::declval<Iterator &>()
(all over) since the former will have type Iterator &&
while the latter will have type Iterator &
. I don't see why applying dereferencing to an rvalue reference should give another rvalue reference though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's the additional testing, though I could be wrong. I think after adding the tests I've not been able to get MSVC to work. SFINAE seems to be working, it's just failing inside the correct overload. This would be easier if I set this up with MSVC locally on a Windows box. With nox it might be easier to setup than in the past. add_lvalue_reference
and adding a &
should be very similar except for void, so I'm expecting similar results, but let's see.
If we have an MSVC solution and an everyone-else solution, that would be doable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add_lvalue_reference and adding a & should be very similar except for void, so I'm expecting similar results, but let's see.
Not really - you were using add_lvalue_reference to modify the result type, whereas I'm proposing modifying the iterator type. But since I don't really understand what MSVC was doing I'm also going to wait and see. I'm trying to use compiler explorer to reproduce the issue (which was pretty handy for the ICC bug), but haven't managed it yet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, yes, I see that now. Looks like this is working a lot better! 95% complete with no failures yet.
06a9632
to
e850621
Compare
* Add make_value_iterator This is the counterpart to make_key_iterator, and will allow implementing a `value` method in `bind_map` (although doing so is left for a subsequent PR). I made a few design changes to reduce copy-and-paste boilerplate. Previously detail::iterator_state had a boolean template parameter to indicate whether it was being used for make_iterator or make_key_iterator. I replaced the boolean with a class that determines how to dereference the iterator. This allows for a generic implementation of `__next__`. I also added the ValueType and Extra... parameters to the iterator_state template args, because I think it was a bug that they were missing: if make_iterator is called twice with different values of these, only the first set has effect (because the state class is only registered once). There is still a potential issue in that the *values* of the extra arguments are latched on the first call, but since most policies are empty classes this should be even less common. * Add some remove_cv_t to appease clang-tidy * Make iterator_access and friends take reference For some reason I'd accidentally made it take a const value, which caused some issues with third-party packages. * Another attempt to remove remove_cv_t from iterators Some of the return types were const (non-reference) types because of the pecularities of decltype: `decltype((*it).first)` is the *declared* type of the member of the pair, rather than the type of the expression. So if the reference type of the iterator is `pair<const int, int> &`, then the decltype is `const int`. Wrapping an extra set of parentheses to form `decltype(((*it).first))` would instead give `const int &`. This means that the existing make_key_iterator actually returns by value from `__next__`, rather than by reference. Since for mapping types, keys are always const, this probably hasn't been noticed, but it will affect make_value_iterator if the Python code tries to mutate the returned objects. I've changed things to use double parentheses so that make_iterator, make_key_iterator and make_value_iterator should now all return the reference type of the iterator. I'll still need to add a test for that; for now I'm just checking whether I can keep Clang-Tidy happy. * Add back some NOLINTNEXTLINE to appease Clang-Tidy This is favoured over using remove_cv_t because in some cases a const value return type is deliberate (particularly for Eigen). * Add a unit test for iterator referencing Ensure that make_iterator, make_key_iterator and make_value_iterator return references to the container elements, rather than copies. The test for make_key_iterator fails to compile on master, which gives me confidence that this branch has fixed it. * Make the iterator_access etc operator() const I'm actually a little surprised it compiled at all given that the operator() is called on a temporary, but I don't claim to fully understand all the different value types in C++11. * Attempt to work around compiler bugs https://godbolt.org/ shows an example where ICC gets the wrong result for a decltype used as the default for a template argument, and CI also showed problems with PGI. This is a shot in the dark to see if it fixes things. * Make a test constructor explicit (Clang-Tidy) * Fix unit test on GCC 4.8.5 It seems to require the arguments to the std::pair constructor to be implicitly convertible to the types in the pair, rather than just requiring is_constructible. * Remove DOXYGEN_SHOULD_SKIP_THIS guards Now that a complex decltype expression has been replaced by a simpler nested type, I'm hoping Doxygen will be able to build it without issues. * Add comment to explain iterator_state template params
ada6b3e
to
760b442
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. May want to add a few comments explaining the black magic with the decltypes and how the & and the parens are needed.
Just to be sure, I'm going to verify that using ResultType still breaks those three compilers, now that this is fixed. I'll also add a comment as to why this is needed if it still is. I'll keep it in draft until then. |
@Skylion007 do we need a couple more |
760b442
to
755133b
Compare
I've managed to reproduce the MSVC bug in Compiler Explorer: https://godbolt.org/z/343qzxe6z. |
It looks like this is a known bug in MSVC: https://developercommunity2.visualstudio.com/t/decltypepointerToArray-produces-rva/291161?entry=problem&space=62 |
Doesn't that say fixed in 2019, specifically VS16.9, but it still seems to be present in VS16.11? |
755133b
to
93c0a38
Compare
eab5682
to
bde20cd
Compare
You're right. So presumably it's not exactly the same bug, but some variant of it. |
bde20cd
to
42ada15
Compare
42ada15
to
db6e0d3
Compare
Are you going to report it? Especially with a nice reproducer, should be easy to report? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was not able to follow the details, but with so much attention given to this work, by so many people, I'm sure this will be fine!
After it's merged I'll run the google testing again (tonight). I think it's safe to not block on waiting for the results.
Do any of you have any experience reporting ICC bugs? I tried searching for a bug reporting system, and the most useful thing I found was a forum rant from a user complaining that the only way to report bugs was on the forum and that the bug reports were usually ignored, so I didn't look any further. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks again for fixing this up for me @henryiii. Assuming the CI passes I reckon it's good to go. I've got some work in progress on wiring this up to bind_map
that I'll just need to rebase and write tests for next week.
They aren't all that great at responding to them, as they are leaving ICC behind for the new clang-based Intel compiler. I think @ax3l has reported before? |
Thanks all! |
Some reflection after the merge. I am astonished by the amount of C++ quirks we ran into for a relatively straightforward code reuse. Could it have been easier if we instead pursued three separate function bodies, one for each of make_iterator, make_key_iterator, and make_value_iterator? |
SFINAE would have still been broken (the reuse part actually worked seamlessly). Trying to support non-copyable on all compilers was where much of the effort came in. Now if there had simply been different names for the one that takes an iterator and a sentinel and the one that takes a object supporting iteration, it would have been really easy, as then SFINAE would not have been needed. But that was not the case. Welcome to C++. :D |
Thanks for the explanation! |
Description
SFINAE was broken, C++ does not perform SFINAE inside classes/structs. If someone wants to implement a better test (one that actually supports keep_alive), happy to take a contribution!
Activating SFINAE shows the new code is broken on some compilers.
Suggested changelog entry: