Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebuild for pypy37 #1

Conversation

regro-cf-autotick-bot
Copy link
Contributor

This PR has been triggered in an effort to update pypy37.

Notes and instructions for merging this PR:

  1. Please merge the PR only after the tests have passed.
  2. Feel free to push to the bot's branch to update this PR if needed.

Please note that if you close this PR we presume that the feedstock has been rebuilt, so if you are going to perform the rebuild yourself don't close this PR until the your rebuild has been merged.

This package has the following downstream children:

And potentially more.

If this PR was opened in error or needs to be updated please add the bot-rerun label to this PR. The bot will close this PR and schedule another one. If you do not have permissions to add this label, you can use the phrase @conda-forge-admin, please rerun bot in a PR comment to have the conda-forge-admin add it for you.

This PR was created by the regro-cf-autotick-bot.
The regro-cf-autotick-bot is a service to automatically track the dependency graph, migrate packages, and propose package version updates for conda-forge. If you would like a local version of this bot, you might consider using rever. Rever is a tool for automating software releases and forms the backbone of the bot's conda-forge PRing capability. Rever is both conda (conda install -c conda-forge rever) and pip (pip install re-ver) installable.
Finally, feel free to drop us a line if there are any issues!
This PR was generated by https://github.com/regro/autotick-bot/actions/runs/1316785350, please use this URL for debugging

@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@h-vetinari h-vetinari force-pushed the rebuild-pypy37-0-1_h36cb33 branch 3 times, most recently from 1a7d891 to 952c830 Compare October 7, 2021 21:59
@h-vetinari
Copy link
Member

@conda-forge/help-c-cpp @mattip
This fails with some template errors, but only for PyPy...? I'm at a loss what could cause such an interaction?

  src/encoding_decoding.cpp:77:48: error: no match for 'operator<<' (operand types are 'std::basic_ostream<char>' and 'pybind11::detail::accessor<pybind11::detail::accessor_policies::sequence_item>')
     77 |       ss << "not a valid type for conversion " << it << std::endl;
        |       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^~ ~~
        |          |                                        |
        |          std::basic_ostream<char>                 pybind11::detail::accessor<pybind11::detail::accessor_policies::sequence_item>
  In file included from /home/conda/feedstock_root/build_artifacts/pytomlpp_1633644044387/_build_env/x86_64-conda-linux-gnu/include/c++/9.4.0/istream:39,
                   from /home/conda/feedstock_root/build_artifacts/pytomlpp_1633644044387/_build_env/x86_64-conda-linux-gnu/include/c++/9.4.0/sstream:38,
                   from include/pytomlpp/pytomlpp.hpp:31,
                   from src/encoding_decoding.cpp:1:

@mattip
Copy link

mattip commented Oct 8, 2021

I wonder what it is at that point. It seems it cannot be printed.

@h-vetinari
Copy link
Member

Hey @mattip, thanks a lot for chiming in - I'm sorry, I somehow must have lost track of this notification. I also don't (yet) know what's happening here exactly...

@h-vetinari h-vetinari force-pushed the rebuild-pypy37-0-1_h36cb33 branch 2 times, most recently from 923ccbf to 2ea482e Compare November 18, 2021 00:32
@bobfang1992
Copy link
Contributor

@conda-forge/help-c-cpp @mattip This fails with some template errors, but only for PyPy...? I'm at a loss what could cause such an interaction?

  src/encoding_decoding.cpp:77:48: error: no match for 'operator<<' (operand types are 'std::basic_ostream<char>' and 'pybind11::detail::accessor<pybind11::detail::accessor_policies::sequence_item>')
     77 |       ss << "not a valid type for conversion " << it << std::endl;
        |       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^~ ~~
        |          |                                        |
        |          std::basic_ostream<char>                 pybind11::detail::accessor<pybind11::detail::accessor_policies::sequence_item>
  In file included from /home/conda/feedstock_root/build_artifacts/pytomlpp_1633644044387/_build_env/x86_64-conda-linux-gnu/include/c++/9.4.0/istream:39,
                   from /home/conda/feedstock_root/build_artifacts/pytomlpp_1633644044387/_build_env/x86_64-conda-linux-gnu/include/c++/9.4.0/sstream:38,
                   from include/pytomlpp/pytomlpp.hpp:31,
                   from src/encoding_decoding.cpp:1:

I am just wondering why this fails for pypy but not cpython?

@h-vetinari
Copy link
Member

I am just wondering why this fails for pypy but not cpython?

Yeah, so were we. To be honest, I have no idea. 😅

@mattip
Copy link

mattip commented Nov 18, 2021

This is coming from a long chain of if else. I imagine somehow PyPy is failing to match where cpython matches, falls through to the error clause, and fails to print the value wraps the class differently than CPython (seems to be a pybind11::detail::accessor<pybind11::detail::accessor_policies::sequence_item>). Here is the original code:

for (auto &&it : list) {
    if (py::isinstance<py::str>(it)) {
      std::string string_value = it.cast<py::str>();
      arr.push_back(std::move(string_value));
    } else if (py::isinstance<py::int_>(it)) {
      int64_t int_value = it.cast<py::int_>();
      arr.push_back(int_value);
    } else if (py::isinstance<py::float_>(it)) {
      double float_value = it.cast<py::float_>();
      arr.push_back(float_value);
    } else if (py::isinstance<py::bool_>(it)) {
      bool bool_value = it.cast<py::bool_>();
      arr.push_back(bool_value);
    } else if (py::isinstance<py::dict>(it)) {
      toml::table t = py_dict_to_toml_table(it.cast<py::dict>());
      arr.push_back(std::move(t));
    } else if (py::isinstance<py::list>(it)) {
      toml::array a = py_list_to_toml_array(it.cast<py::list>());
      arr.push_back(std::move(a));
    } else if (py::isinstance(it, datetime_class)) {
      // Order matters here.
      // isinstance(datetime_obj, datetime) --> true
      // isinstance(datetime_obj, date) --> true as well
      // so we need to test datetime first then date.
      toml::date_time date_time_value = it.cast<toml::date_time>();
      arr.push_back(date_time_value);
    } else if (py::isinstance(it, date_class)) {
      toml::date date_value = it.cast<toml::date>();
      arr.push_back(date_value);
    } else if (py::isinstance(it, time_class)) {
      toml::time time_value = it.cast<toml::time>();
      arr.push_back(time_value);
    } else {
      std::stringstream ss;
      ss << "not a valid type for conversion " << it << std::endl;
      throw py::type_error(ss.str());
    }
  }

Edit: this happens at compile, not runtime.

@mattip
Copy link

mattip commented Nov 18, 2021

If I do the following, all the tests pass. @bobfang1992 is this code path is hit in tests (so I can debug what is going on)?

iff --git a/src/encoding_decoding.cpp b/src/encoding_decoding.cpp
index 530dfa6..3241388 100644
--- a/src/encoding_decoding.cpp
+++ b/src/encoding_decoding.cpp
@@ -74,7 +74,7 @@ toml::array py_list_to_toml_array(const py::list &list) {
       arr.push_back(time_value);
     } else {
       std::stringstream ss;
-      ss << "not a valid type for conversion " << it << std::endl;
+      ss << "not a valid type for conversion " << std::endl;
       throw py::type_error(ss.str());
     }
   }

@bobfang1992
Copy link
Contributor

If I do the following, all the tests pass. @bobfang1992 is this code path is hit in tests (so I can debug what is going on)?

iff --git a/src/encoding_decoding.cpp b/src/encoding_decoding.cpp
index 530dfa6..3241388 100644
--- a/src/encoding_decoding.cpp
+++ b/src/encoding_decoding.cpp
@@ -74,7 +74,7 @@ toml::array py_list_to_toml_array(const py::list &list) {
       arr.push_back(time_value);
     } else {
       std::stringstream ss;
-      ss << "not a valid type for conversion " << it << std::endl;
+      ss << "not a valid type for conversion " << std::endl;
       throw py::type_error(ss.str());
     }
   }

yeah I think if you run pytest it will hit this code path for sure, I am not really entirely sure this is a code issue though, because everything works with cpython, I think this is more related to pypy tooling.

@mattip
Copy link

mattip commented Nov 18, 2021

I think this is more related to pypy tooling.

Right, I am a PyPy dev and want to get to the bottom of this.

@mattip
Copy link

mattip commented Nov 18, 2021

If I extend a test to test a list

def test_invalid_encode():
    class a:
        pass
    with pytest.raises(typeerror):
        pytomlpp.dumps({'a': a()})
    with pytest.raises(typeerror):
        pytomlpp.dumps({'a': [a()]})

and run the tests, I can stop on that line. CPython has this to say

Breakpoint 1, pytomlpp::py_list_to_toml_array (list=...) at src/encoding_decoding.cpp:77
77	      ss << "not a valid type for conversion " << it << std::endl;
(gdb) p it
$1 = (const pybind11::handle &&) @0x7fffffff6ad0: {<pybind11::detail::object_api<pybind11::handle>> = \
    {<pybind11::detail::pyobject_tag> = \
        {<No data fields>}, <No data fields>}, \
    m_ptr = <A at remote 0x7ffff6633670>}

Where PyPy has this to say:

77	      ss << "not a valid type for conversion " << std::endl;
(gdb) p it
$1 = (pybind11::detail::accessor<pybind11::detail::accessor_policies::sequence_item> &&) @0x7fffffff9070:
{<pybind11::detail::object_api<pybind11::detail::accessor<pybind11::detail::accessor_policies::sequence_item> >> = 
    {<pybind11::detail::pyobject_tag> = 
        {<No data fields>}, <No data fields>}, \
    obj = {<pybind11::detail::object_api<pybind11::handle>> = \
    {<pybind11::detail::pyobject_tag> = \
        {<No data fields>}, <No data fields>}, \
     m_ptr = 0xd599e0}, key = 0, \
     cache = {<pybind11::handle> = \
               {<pybind11::detail::object_api<pybind11::handle>> = \
                  {<pybind11::detail::pyobject_tag> = 
                     {<No data fields>}, <No data fields>}, \
        m_ptr = 0x7ec080}, <No data fields>}}

So pybind11 is wrapping the objects very differently.

If I go up one level in the debugger, CPython says

#1  0x00007ffff6a8f9e6 in pytomlpp::py_dict_to_toml_table (object=...) at src/encoding_decoding.cpp:123
123	      toml::array array_value = py_list_to_toml_array(value.cast<py::list>());
(gdb) p value
$1 = {<pybind11::detail::object_api<pybind11::handle>> = \
   {<pybind11::detail::pyobject_tag> = \
      {<No data fields>}, <No data fields>}, 
    m_ptr = [<A at remote 0x7ffff6635640>]}

and PyPy says

(gdb) up
#1  0x00007fffeffe9102 in pytomlpp::py_dict_to_toml_table (object=...) at src/encoding_decoding.cpp:123
123	      toml::array array_value = py_list_to_toml_array(value.cast<py::list>());
(gdb) p value
$2 = {<pybind11::detail::object_api<pybind11::handle>> = \
         {<pybind11::detail::pyobject_tag> = \
             {<No data fields>}, <No data fields>}, \
      m_ptr = 0xd599e0}

The difference seems to be that pybind11 "knows" that m_ptr contains a [A] on CPython and on PyPy it is an opaque pointer.

@bobfang1992
Copy link
Contributor

One thing I think we can do is simply revert pybind11 version back to 2.5, we upgraded from 2.5 to 2.8 in the 1.0.7 version but seems 1.0.6 is doing fine with all python distros including pypy. And I dont actually need 2.8 in pytomlpp. If you all agree I can publish another version (1.0.8) to fix this issue. We can then skip 1.0.7 entirely and conda user will have the latest code like pyp users.

@mattip @h-vetinari thoughts?

recipe/meta.yaml Outdated
@@ -29,7 +29,7 @@ requirements:
- python
- pip
- setuptools
- pybind11 >=2.5,<2.6
- pybind11 >=2.5
Copy link

@mattip mattip Nov 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could try setting this to a know "good" (for PyPy) version of pybind11

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... and add a comment `pybind11 ==2.5 # pypy fails to build with 2.8

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@h-vetinari can you add me as collaborator to this project? Seems I dont have the permission to push changes to this branch. Thanks!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There were no pypy builds for 2.5, AFAIR. There's definitely no cpython 3.10 build for pybind 2.5, so I'd be reluctant to go back (though we can, selectively, for pypy, if necessary)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah okay, then please ignore my suggestion 😢

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how this all works. If there is no conda package, is there a way to pip install? If I recall correctly pybind11 does not build any c-extensions, so the install should be cheap.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no no, we'll try. 🙃

And if it's really a specific pybind11-version that's bad for pypy, we could also rebuild the (missing) old pybind versions for pypy on the respective feedstock.

@h-vetinari
Copy link
Member

@h-vetinari can you add me as collaborator to this project? Seems I dont have the permission to push changes to this branch. Thanks!

Normally this goes through a normal PR, but I've pushed a commit to make you a collaborator directly (even though you now have the rights, please don't push to master directly - this is a very specific case).

In general, conda-forge is a zoo of automation that might take a little while to get used to. I'm happy to help if there are questions - best is to iterate on PRs, and hold off on quick merges until you get the hang of things.

@h-vetinari
Copy link
Member

Glad to have you onboard! :)

@bobfang1992
Copy link
Contributor

Glad to have you onboard! :)

Glad to be here! Thanks, it is a great to have pytomlpp on conda-forge, so what do I need to do now? I am not entirely sure how to:

rebuild the (missing) old pybind versions for pypy on the respective feedstock.

I see you just pushed something, shall I just wait and see its effect?

@h-vetinari
Copy link
Member

h-vetinari commented Nov 18, 2021

Thanks, it is a great to have pytomlpp on conda-forge, so what do I need to do now?

First off, cpython builds on linux/osx/windows are already there. This PR is about adding pypy builds as well, which I hope we'll be able to figure out, but it's not crazy urgent (and I really appreciate @mattip taking the time!). So right now, you don't need to do anything (but your help in debugging would be much appreciated!) 🙃

I am not entirely sure how to:

rebuild the (missing) old pybind versions for pypy on the respective feedstock.

One of the first things to get used to is that every package in conda-forge gets built on a feedstock; rarely one feedstock does a few related packages in one go, but mostly it's 1:1. For pybind11 it's here - you're not a maintainer there, and you'll become so less quickly than here (where you have the extraordinary headstart of being the upstream maintainer 😋) - but you can always raise PRs anywhere.

If we determine that things work with (for example) pybind 2.6, we'll have to discuss with the maintainers there to rebuild the required packages - generally the maintainers in conda-forge are very supportive, but everyone is doing this in their free time, so we should have a good reason, and cannot "demand" anything.

I see you just pushed something, shall I just wait and see its effect?

Now that I added a patch to unpin the upstream pybind11 requirement for pypy, you can wait and see what failures the CI produces, and if necessary, reduce the version of pybind11 again (I started by going from 2.8 -> 2.7). At some point there will not be C++ compilation errors as above, but environment resolution errors because the requisite pybind version might not have been built that far back for new-ish flavours like pypy (or cpython 3.10).

@h-vetinari
Copy link
Member

PS. Your upstream maintainership gives you certain "superpowers" that packagers don't have - you can decide what goes in the upstream code base. While it's possible to carry patches on the feedstocks (for example, it's unavoidable to use one for pointing to the conda-forge packaged toml++; which is packaged separately to avoid bloat & duplication), we obviously try to avoid that whenever possible, and to keep it minimal otherwise.

In other words, while you're in your rights to say that this should be solved outside of pytomlpp - if a patch like the above is not too painful upstream, it would solve a lot of the debugging straight away.

@h-vetinari
Copy link
Member

OK, it's getting late for me, TTYL

@bobfang1992
Copy link
Contributor

PS. Your upstream maintainership gives you certain "superpowers" that packagers don't have - you can decide what goes in the upstream code base. While it's possible to carry patches on the feedstocks (for example, it's unavoidable to use one for pointing to the conda-forge packaged toml++; which is packaged separately to avoid bloat & duplication), we obviously try to avoid that whenever possible, and to keep it minimal otherwise.

In other words, while you're in your rights to say that this should be solved outside of pytomlpp - if a patch like the above is not too painful upstream, it would solve a lot of the debugging straight away.

Let me have a think about this (i.e. removing it entirely here). I suspect I can do something smart there to still have the value printed out without breaking the pypy build, but not entirely sure yet. Thanks for the suggestion though. Will let you know if I can something working.

@h-vetinari
Copy link
Member

@mattip: You could try setting this to a know "good" (for PyPy) version of pybind11

I have now tested 2.8, 2.7, 2.6 (and previously 2.5), and they all fail in the same way, unfortunately.

@mattip
Copy link

mattip commented Nov 18, 2021

How about this patch, which

  • adds a test to hit the offending line
  • only skips printing it for PyPy
diff --git a/src/encoding_decoding.cpp b/src/encoding_decoding.cpp
index 530dfa6..b5e88dc 100644
--- a/src/encoding_decoding.cpp
+++ b/src/encoding_decoding.cpp
@@ -74,7 +74,14 @@ toml::array py_list_to_toml_array(const py::list &list) {
       arr.push_back(time_value);
     } else {
       std::stringstream ss;
+#ifdef PYPY_VERSION
+      // see https://github.com/conda-forge/pytomlpp-feedstock/pull/1#issuecomment-972738986
+      // and https://github.com/pybind/pybind11/issues/3408#issuecomment-972752210
+      ss << "not a valid type for conversion " << std::endl;
+#else
       ss << "not a valid type for conversion " << it << std::endl;
+#endif
       throw py::type_error(ss.str());
     }
   }
diff --git a/tests/python-tests/test_api.py b/tests/python-tests/test_api.py
index 8a65c1b..c2eadaf 100644
--- a/tests/python-tests/test_api.py
+++ b/tests/python-tests/test_api.py
@@ -117,6 +117,8 @@ def test_invalid_encode():
         pass
     with pytest.raises(TypeError):
         pytomlpp.dumps({'a': A()})
+    with pytest.raises(TypeError):
+        pytomlpp.dumps({'a': [A()]})

@bobfang1992
Copy link
Contributor

How about this patch, which

  • adds a test to hit the offending line
  • only skips printing it for PyPy
diff --git a/src/encoding_decoding.cpp b/src/encoding_decoding.cpp
index 530dfa6..b5e88dc 100644
--- a/src/encoding_decoding.cpp
+++ b/src/encoding_decoding.cpp
@@ -74,7 +74,14 @@ toml::array py_list_to_toml_array(const py::list &list) {
       arr.push_back(time_value);
     } else {
       std::stringstream ss;
+#ifdef PYPY_VERSION
+      // see https://github.com/conda-forge/pytomlpp-feedstock/pull/1#issuecomment-972738986
+      // and https://github.com/pybind/pybind11/issues/3408#issuecomment-972752210
+      ss << "not a valid type for conversion " << std::endl;
+#else
       ss << "not a valid type for conversion " << it << std::endl;
+#endif
       throw py::type_error(ss.str());
     }
   }
diff --git a/tests/python-tests/test_api.py b/tests/python-tests/test_api.py
index 8a65c1b..c2eadaf 100644
--- a/tests/python-tests/test_api.py
+++ b/tests/python-tests/test_api.py
@@ -117,6 +117,8 @@ def test_invalid_encode():
         pass
     with pytest.raises(TypeError):
         pytomlpp.dumps({'a': A()})
+    with pytest.raises(TypeError):
+        pytomlpp.dumps({'a': [A()]})

Looks great to me, do you mind submitting a PR to the upstream library? I will merge it and make another release.

@bobfang1992
Copy link
Contributor

bobfang1992/pytomlpp#61

The fix looks good and I plan to merge it once all CI finishes, so which way is better?

  • keep 1.0.7 as the current version and I force-move it point to the commit after fix
  • release a new version 1.0.8 instead
    Which one do you prefer? I don't mind either.

@mattip
Copy link

mattip commented Nov 18, 2021

If you know people are pinning to 1.0.7, and those people might be wanting to use PyPy, then we can add a patch to this repo that will be removed for the next release. Otherwise, we might as well go with a 1.0.8.

@bobfang1992
Copy link
Contributor

OK v1.0.8 is building and should be released to pypi soon. I expect a new PR will be automatically raised after that.

@h-vetinari
Copy link
Member

Really to cool to see you both arrived at a solution here! :)

The fix looks good and I plan to merge it once all CI finishes, so which way is better?

As Matti said, generally we don't need a new version (could carry the patch), but given that there's a new release already, I'll merge it and rebase this PR.

@h-vetinari
Copy link
Member

Now it builds 🥳

@h-vetinari h-vetinari added the automerge Merge the PR when CI passes label Nov 18, 2021
@github-actions github-actions bot merged commit 1a5f7be into conda-forge:master Nov 18, 2021
@github-actions
Copy link
Contributor

Hi! This is the friendly conda-forge automerge bot!

I considered the following status checks when analyzing this PR:

  • linter: passed
  • azure: passed

Thus the PR was passing and merged! Have a great day!

@regro-cf-autotick-bot regro-cf-autotick-bot deleted the rebuild-pypy37-0-1_h36cb33 branch November 18, 2021 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automerge Merge the PR when CI passes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants