Compile gtsam python for windows #1685

talregev · 2023-12-08T00:11:23Z

Compile gtsam python for windows

I manage to compile and run python tests on windows.
Some of the python tests are fails on windows (16 of 243).
Please take a look on my PR.

@dellaert @varunagrawal @ProfFan

Waiting for merge:

Cephes Updates #1704

talregev · 2023-12-08T11:13:23Z

I will disable the running of tests for windows. that you have a clean CI.

ProfFan · 2023-12-08T14:33:59Z

Thank you! Could you post which tests fail here?

talregev · 2023-12-08T19:39:03Z

I am post here the errors, also I am put a link for the run:
https://github.com/borglab/gtsam/actions/runs/7135793022/job/19433202623

======================================================================
ERROR: test_dogleg (test_NonlinearOptimizer.TestScenario.test_dogleg)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_NonlinearOptimizer.py", line 43, in setUp
    self.assertEqual(0.0, self.fg.error(self.optimal_values), 0.0)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "1", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,2,1,0,2,1>

======================================================================
ERROR: test_gauss_newton (test_NonlinearOptimizer.TestScenario.test_gauss_newton)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_NonlinearOptimizer.py", line 43, in setUp
    self.assertEqual(0.0, self.fg.error(self.optimal_values), 0.0)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "1", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,2,1,0,2,1>

======================================================================
ERROR: test_gnc_params (test_NonlinearOptimizer.TestScenario.test_gnc_params)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_NonlinearOptimizer.py", line 43, in setUp
    self.assertEqual(0.0, self.fg.error(self.optimal_values), 0.0)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "1", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,2,1,0,2,1>

======================================================================
ERROR: test_graduated_non_convexity (test_NonlinearOptimizer.TestScenario.test_graduated_non_convexity)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_NonlinearOptimizer.py", line 43, in setUp
    self.assertEqual(0.0, self.fg.error(self.optimal_values), 0.0)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "1", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,2,1,0,2,1>

======================================================================
ERROR: test_iteration_hook (test_NonlinearOptimizer.TestScenario.test_iteration_hook)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_NonlinearOptimizer.py", line 43, in setUp
    self.assertEqual(0.0, self.fg.error(self.optimal_values), 0.0)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "1", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,2,1,0,2,1>

======================================================================
ERROR: test_levenberg_marquardt (test_NonlinearOptimizer.TestScenario.test_levenberg_marquardt)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_NonlinearOptimizer.py", line 43, in setUp
    self.assertEqual(0.0, self.fg.error(self.optimal_values), 0.0)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "1", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,2,1,0,2,1>

======================================================================
ERROR: test_levenberg_marquardt_pcg (test_NonlinearOptimizer.TestScenario.test_levenberg_marquardt_pcg)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_NonlinearOptimizer.py", line 43, in setUp
    self.assertEqual(0.0, self.fg.error(self.optimal_values), 0.0)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "1", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,2,1,0,2,1>

======================================================================
ERROR: test_SFMExample (test_SFMExample.TestSFMExample.test_SFMExample)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_SFMExample.py", line 63, in test_SFMExample
    optimizer = gtsam.LevenbergMarquardtOptimizer(graph, initialEstimate)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "p0", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,3,1,0,3,1>

======================================================================
ERROR: test_StereoVOExample (test_StereoVOExample.TestStereoVOExample.test_StereoVOExample)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_StereoVOExample.py", line 71, in test_StereoVOExample
    optimizer = gtsam.LevenbergMarquardtOptimizer(graph, initialEstimate)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "l1", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,3,1,0,3,1>

======================================================================
ERROR: test_localToWorld (test_Utilities.TestUtilites.test_localToWorld)
Test localToWorld.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_Utilities.py", line 182, in test_localToWorld
    np.testing.assert_allclose(world.atPoint2(0), expected_point2)
                               ^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to at the key "0", which does not exist in the Values.

======================================================================
ERROR: test_perturbPoint2 (test_Utilities.TestUtilites.test_perturbPoint2)
Test perturbPoint2.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_Utilities.py", line 113, in test_perturbPoint2
    not np.allclose(values.atPoint2(1), gtsam.Point2(1, 1)))
                    ^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "1", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,2,1,0,2,1>

======================================================================
ERROR: test_perturbPoint3 (test_Utilities.TestUtilites.test_perturbPoint3)
Test perturbPoint3.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_Utilities.py", line 130, in test_perturbPoint3
    self.assertTrue(not np.allclose(values.atPoint3(1), point3))
                                    ^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "1", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,3,1,0,3,1>

======================================================================
ERROR: test_values (test_Values.TestValues.test_values)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_Values.py", line 62, in test_values
    self.gtsamAssertEquals(values.atPoint2(0), Point2(0,0), tol)
                           ^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "0", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,2,1,0,2,1>

======================================================================
ERROR: test_VisualISAMExample (test_VisualISAMExample.TestVisualISAMExample.test_VisualISAMExample)
Test to see if ISAM works as expected for a simple visual SLAM example.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_VisualISAMExample.py", line 48, in test_VisualISAMExample
    isam, result, nextPose = visual_isam.initialize(
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\gtsam\gtsam\build\python\gtsam\utils\visual_isam.py", line 67, in initialize
    batchOptimizer = gtsam.LevenbergMarquardtOptimizer(newFactors,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "l0", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,3,1,0,3,1>

======================================================================
ERROR: test_isam2_error (test_VisualISAMExample.TestVisualISAMExample.test_isam2_error)
Test for isam2 error() method.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_VisualISAMExample.py", line 67, in test_isam2_error
    isam, result, nextPose = visual_isam.initialize(
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\gtsam\gtsam\build\python\gtsam\utils\visual_isam.py", line 67, in initialize
    batchOptimizer = gtsam.LevenbergMarquardtOptimizer(newFactors,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "l0", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,3,1,0,3,1>

======================================================================
ERROR: test_isam2_update (test_VisualISAMExample.TestVisualISAMExample.test_isam2_update)
Test for full version of ISAM2::update method
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\gtsam\gtsam\build\python\gtsam\tests\test_VisualISAMExample.py", line 94, in test_isam2_update
    isam, result, nextPose = visual_isam.initialize(
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\gtsam\gtsam\build\python\gtsam\utils\visual_isam.py", line 67, in initialize
    batchOptimizer = gtsam.LevenbergMarquardtOptimizer(newFactors,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempting to retrieve value with key "l0", type stored in Values is class gtsam::GenericValue<class Eigen::Matrix<double,-1,1,0,-1,1> > but requested type was class Eigen::Matrix<double,3,1,0,3,1>

----------------------------------------------------------------------

Ran 243 tests in 0.543s



FAILED (errors=16, skipped=3)

ProfFan · 2023-12-14T22:01:00Z

It seems to be an RTTI issue, which actually makes the Python wrappers very cumbersome to use....

talregev · 2023-12-14T22:08:25Z

It seems to be an RTTI issue, which actually makes the Python wrappers very cumbersome to use....

I am not sure what are you suggesting.

talregev · 2023-12-29T02:02:57Z

I am not sure what do you mean by RTTI issue.
I did some debug.
The error we get from here:
https://github.com/borglab/gtsam/blob/develop/gtsam/nonlinear/Values.cpp#L297

That was thrown from here:
https://github.com/borglab/gtsam/blob/develop/gtsam/nonlinear/Values-inl.h#L141

Maybe is msvc bug?
The flow of the code is like that. start from here:
https://github.com/borglab/gtsam/blob/develop/gtsam/nonlinear/Values-inl.h#L213
For the handle class there is 2 option.
General case:
https://github.com/borglab/gtsam/blob/develop/gtsam/nonlinear/Values-inl.h#L134
Eigen::Matrix case (template specialization):
https://github.com/borglab/gtsam/blob/develop/gtsam/nonlinear/Values-inl.h#L193

On windows it choose the general case.
On Linux it choose the Eigen::Matrix case.

Also for linux:

storedTypeName: gtsam::GenericValue<Eigen::Matrix<double, -1, 1, 0, -1, 1> >
requestedTypeName: Eigen::Matrix<double, -1, 1, 0, -1, 1>

ProfFan · 2023-12-29T17:18:56Z

Yes, that is an RTTI issue, because dynamic_cast failed in the previous step

talregev · 2023-12-29T17:23:33Z

I suspect it before the dynamic cast. It from template specialization.
Because it choose the incorrect (or different from Linux) case for the template.

ProfFan · 2023-12-29T17:30:59Z

I fixed this problem before, don't know why it's coming up again. Culprit is likely type_info being inconsistent between different dynamic libraries. #218

ProfFan · 2023-12-29T17:33:01Z

Also borglab/GTDynamics#248

talregev · 2023-12-31T22:17:26Z

@ProfFan I added a workaround for windows template specialization.
I test it local. (I copy the gtsam code, and just change the template specialization):
https://github.com/borglab/gtsam/pull/1685/files#diff-2d6a3f78e983bc926a126f73463c25805ef90f9e838f91af83192f68b48353b9R200
Now it compile and run correctly.
It not a problem of dynamic_cast , nor a problem of type_info
It a problem of msbuild template substitute. I think it msbuild compiler bug. (when I compile this code on gcc, it create ambiguous error). msbuild not substitute correctly the template.
You can look at the code and see for yourself.

~~Now I have a new problem, from latest merge #1572:~~
~~It not found the cephes_igami~~

nonlinear.cpp.obj : error LNK2001: unresolved external symbol cephes_igami

@varunagrawal @ProfFan ~~Can you help me to add this symbol (or give an hint) to windows python?~~
Thank you in advanced, and Happy new year.

Edit:
I added workaround for export cephes_igami for windows.

ProfFan · 2024-01-02T20:42:26Z

~~What is cephes_igami?~~ nvm

talregev · 2024-01-02T20:46:26Z

What is cephes_igami?

cephes_igami is a define from 3rdparty cephes that was added here: #1572
https://github.com/borglab/gtsam/blob/develop/gtsam/3rdparty/cephes/cephes/cephes_names.h#L45
In my workaround, I export the igami function. then it also export the cephes_igami because of that define.

ProfFan

Thank you for the contributions! I'll leave the rest of review to @varunagrawal who is better suited as Windows reviewer. Also CC @mattking-smith @jlblancoc who may be more familiar with Windows.

gtsam/3rdparty/cephes/dllexport.h

gtsam/nonlinear/Values-inl.h

.github/scripts/python.sh

.github/workflows/build-python.yml

gtsam/3rdparty/cephes/CMakeLists.txt

talregev · 2024-01-03T21:09:24Z

Except the name changes, all the changes here are must and necessary to solve the problem of compiling windows gtsam python.

talregev · 2024-01-13T08:59:48Z

This branch is ready for review.

varunagrawal

Asking for one small change and then I can approve this PR.

talregev marked this pull request as ready for review December 8, 2023 00:33

talregev force-pushed the TalR/test_python_windows branch from edfe7ca to c9651e2 Compare December 8, 2023 00:38

talregev force-pushed the TalR/test_python_windows branch from c9651e2 to 72f6f4e Compare December 8, 2023 11:15

talregev force-pushed the TalR/test_python_windows branch from 3d1eb38 to 3c2ff14 Compare December 31, 2023 22:55

ProfFan reviewed Jan 2, 2024

View reviewed changes

gtsam/3rdparty/cephes/dllexport.h Outdated Show resolved Hide resolved

gtsam/nonlinear/Values-inl.h Show resolved Hide resolved

ProfFan mentioned this pull request Jan 2, 2024

GTSAM detect wrong data type with Point3 in *windows* #1693

Open

varunagrawal reviewed Jan 3, 2024

View reviewed changes

talregev force-pushed the TalR/test_python_windows branch 2 times, most recently from 369eff9 to 2a6d4ff Compare January 4, 2024 06:51

ProfFan mentioned this pull request Jan 10, 2024

gtsam doesn't build on i386 or armhf #1605

Open

talregev added 4 commits January 13, 2024 10:59

Compile gtsam python for windows

2971d9e

add windows template specialization

8023df4

revert lines according to review comments.

85cae70

Add comments explain the windows workaround.

d1ab94f

talregev force-pushed the TalR/test_python_windows branch from 95cb7aa to d1ab94f Compare January 13, 2024 08:59

talregev requested a review from varunagrawal January 13, 2024 08:59

varunagrawal reviewed Jan 16, 2024

View reviewed changes

fail-fast: true

b104fd6

varunagrawal approved these changes Jan 16, 2024

View reviewed changes

varunagrawal merged commit f191388 into borglab:develop Jan 17, 2024
31 checks passed

talregev deleted the TalR/test_python_windows branch January 17, 2024 06:02

hmaarten mentioned this pull request May 14, 2024

Installing GTSAM in combination with RoboStack ROS Noetic on Windows 10 conda-forge/gtsam-feedstock#25

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compile gtsam python for windows #1685

Compile gtsam python for windows #1685

talregev commented Dec 8, 2023 •

edited

Loading

talregev commented Dec 8, 2023

ProfFan commented Dec 8, 2023

talregev commented Dec 8, 2023 •

edited

Loading

ProfFan commented Dec 14, 2023

talregev commented Dec 14, 2023

talregev commented Dec 29, 2023 •

edited

Loading

ProfFan commented Dec 29, 2023

talregev commented Dec 29, 2023 •

edited

Loading

ProfFan commented Dec 29, 2023

ProfFan commented Dec 29, 2023

talregev commented Dec 31, 2023 •

edited

Loading

ProfFan commented Jan 2, 2024 •

edited

Loading

talregev commented Jan 2, 2024 •

edited

Loading

ProfFan left a comment

talregev commented Jan 3, 2024

talregev commented Jan 13, 2024

varunagrawal left a comment

Compile gtsam python for windows #1685

Compile gtsam python for windows #1685

Conversation

talregev commented Dec 8, 2023 • edited Loading

talregev commented Dec 8, 2023

ProfFan commented Dec 8, 2023

talregev commented Dec 8, 2023 • edited Loading

ProfFan commented Dec 14, 2023

talregev commented Dec 14, 2023

talregev commented Dec 29, 2023 • edited Loading

ProfFan commented Dec 29, 2023

talregev commented Dec 29, 2023 • edited Loading

ProfFan commented Dec 29, 2023

ProfFan commented Dec 29, 2023

talregev commented Dec 31, 2023 • edited Loading

ProfFan commented Jan 2, 2024 • edited Loading

talregev commented Jan 2, 2024 • edited Loading

ProfFan left a comment

Choose a reason for hiding this comment

talregev commented Jan 3, 2024

talregev commented Jan 13, 2024

varunagrawal left a comment

Choose a reason for hiding this comment

talregev commented Dec 8, 2023 •

edited

Loading

talregev commented Dec 8, 2023 •

edited

Loading

talregev commented Dec 29, 2023 •

edited

Loading

talregev commented Dec 29, 2023 •

edited

Loading

talregev commented Dec 31, 2023 •

edited

Loading

ProfFan commented Jan 2, 2024 •

edited

Loading

talregev commented Jan 2, 2024 •

edited

Loading