update to pybind 2.10.1

[hellkite500]

To support python 3.11, as well as gather some important bug fixes, pybind11 needs to be updated. At the time of this PR 2.10.1 is the latest stable release.

A note on the changes in InterperterUtils.hpp, with this version of pybind, the module_ declarations had to be modified. ngen will compile just fine with the old definitions, but at runtime, you end up with this type of error:

libc++abi: terminating with uncaught exception of type pybind11::type_error: Object of type 'type' is not an instance of 'module_'

But updating them to generic py::object types seemed to work, and I tested this change on the existing 2.6.1 pybind as well and there were no issues.

I was curious about the members in Routing_Py_Adapter.hpp, since it stores py::module_ handles as well, but it seems like they weren't affected (I ran routing simulations with no runtime errors...).

Thanks to nmslib/nmslib#488 and microsoft/spacy-ann-linker#7 for pointing the way!

Changes

• update pybind submodule to commit for version 2.10.1
• change py::module_ types to py::object types

Testing

1. Ran through ngen and ngen+routing locally

Notes

• This PR alone does not address Parallel runs with routing crash #370 but some related work is inbound that should resolve that issue.

Checklist

• PR has an informative and human-readable title
• Changes are limited to a single goal (no scope creep)
• Code can be automatically merged (no conflicts)
• Code follows project standards (link if applicable)
• Passes all existing automated tests
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
• Reviewers requested with the Reviewers tool ➡️

Target Environment support

• Linux
• MacOS

[hellkite500]

Tests may not pass till this is merged, this is the same error referenced above:

[100%] Linking CXX executable test_bmi_python
terminate called after throwing an instance of 'pybind11::type_error'
  what():  Object of type 'type' is not an instance of 'module_'
CMake Error at /usr/local/share/cmake-3.24/Modules/GoogleTestAddTests.cmake:112 (message):
  Error running test executable.

    Path: '/home/runner/work/ngen/ngen/cmake_build/test/test_bmi_python'
    Result: Subprocess aborted
    Output:

[mattw-nws]

Tests may not pass till this is merged, this is the same error referenced above:

Yeah, this does not make sense... if this fixes that problem then the code run in the test should pass. Is this a cache issue again? Is the pybind11 submodule cached?

[mattw-nws]

@hellkite500 I have a feeling there may be other places that need the py::module_ -> py::type treatment... e.g. here?

And/or...

I noticed in the test output that it's failing to build a wheel for bmipy ... after that it says successfully installed, so it may have installed without building a wheel, but I'm not sure if this is new... maybe try installing wheel in the workflow?

Building wheels for collected packages: bmipy
  Building wheel for bmipy (setup.py): started
  Building wheel for bmipy (setup.py): finished with status 'error'
  Running setup.py clean for bmipy
  ERROR: Command errored out with exit status 1:
   command: /home/runner/work/ngen/ngen/.venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-4id0igpq/bmipy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-4id0igpq/bmipy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-16kf5fm3
       cwd: /tmp/pip-install-4id0igpq/bmipy/
  Complete output (8 lines):
  /home/runner/work/ngen/ngen/.venv/lib/python3.8/site-packages/setuptools/dist.py:473: UserWarning: Normalizing 'v2.0' to '2.0'
    warnings.warn(
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help
  
  error: invalid command 'bdist_wheel'
  ----------------------------------------
  ERROR: Failed building wheel for bmipy
Failed to build bmipy

[mattw-nws]

This whole thing seems odd because module_ is a thing and this should work... keep in mind for later, because we may very well be losing something by dropping the fact that these objects are module_s.

[hellkite500]

This whole thing seems odd because module_ is a thing and this should work... keep in mind for later, because we may very well be losing something by dropping the fact that these objects are module_s.

If you look at the usage of any of the module_ variables, we are just extracting .attr from them, which is a valid thing to do on a py::object.