test_log_plotter fails locally

[keiyamamo]

Hi @johannesring

I just ran pytest locally on my Mac book and test_log_plotter failed. Here is diff image that was automatically generated. I’m guessing it is specific to my machine, but do you have any idea how to fix this?

[johannesring]

I think we can adjust the tolerance for matplotlib.testing.compare.compare_images. Currently we are using tol=0, which means that the images must be exactly equal. Try setting the tolerance to something like tol=1e-6.

[keiyamamo]

@johannesring Thanks for the suggestion! By setting tol=10, 9 tests passed but 21 still failed. I guess tol=10 might be too high to be used for the test. Do you think this could be related to font mismatch between different OS ? I found some documentation in matplotlib although it is not what we use.

Also, if the texts (labels, tick labels, etc) are not really part of what is tested, use remove_text=True as this will lead to smaller figures and reduce possible issues with font mismatch on different platforms.

https://matplotlib.org/devdocs/devel/testing.html#writing-an-image-comparison-test

[johannesring]

Yes, it looks like it is only a font mismatch. Does it look like a font mismatch for all the failed tests? The tests runs fine on the osx-64 CI test runner. Can you post the error message for one of the failed tests?

[keiyamamo]

Here is one of the error message from the test

>           assert result is None, f"Images differ: {result}"
E           AssertionError: Images differ: Error: Image files did not match.
E               RMS Value: 6.56455545588072
E               Expected:  
E                 /private/var/folders/02/f28j885s2wl9fm3thvk6x2kc0000gn/T/pytest-of-keiyamamoto/pytest-13/test_plot_all0/pressure.png
E               Actual:    
E                 tests/test_data/reference_images/test_all/pressure.png
E               Difference:
E                 tests/test_data/reference_images/test_all/pressure-failed-diff.png
E               Tolerance: 
E                 0
E           assert 'Error: Image files did not match.\n  RMS Value: 6.56455545588072\n  Expected:  \n    /private/var/folders/02/f28j885s2wl9fm3thvk6x2kc0000gn/T/pytest-of-keiyamamoto/pytest-13/test_plot_all0/pressure.png\n  Actual:    \n    tests/test_data/reference_images/test_all/pressure.png\n  Difference:\n    tests/test_data/reference_images/test_all/pressure-failed-diff.png\n  Tolerance: \n    0' is None

Most of them look like it’s just font mismatch (same as the image I posted first) expect probe points plot. Failed diff images have lines in it, so that’s the difference from other images.

[johannesring]

@keiyamamo - I finally found the reason for this problem. When I install matplotlib via pip, the tests fail and I see the same failed diff images as you. However, the tests runs fine when matplotlib is installed as a conda package. Can you confirm this? Do you have matplotlib installed via pip? Does it work if your run the tests in a new environment created from environment.yml?

[keiyamamo]

@johannesring Thanks for the update! I just created an environment with matplotlib in environment.yml and it made many of the tests pass, but it still failed with probes.

tests/test_log_plotter.py::test_plot_options[offset_stenosis_5_cycles.log-args0-cpu_time.png] PASSED [  2%]
tests/test_log_plotter.py::test_plot_options[offset_stenosis_5_cycles.log-args1-ramp_factor.png] PASSED [  5%]
tests/test_log_plotter.py::test_plot_options[offset_stenosis_5_cycles.log-args2-pressure.png] PASSED [  8%]
tests/test_log_plotter.py::test_plot_options[offset_stenosis_5_cycles.log-args3-newton_iteration_(atol).png] PASSED [ 11%]
tests/test_log_plotter.py::test_plot_options[offset_stenosis_5_cycles.log-args4-newton_iteration_(rtol).png] PASSED [ 14%]
tests/test_log_plotter.py::test_plot_options[offset_stenosis_5_cycles.log-args5-probe_points.png] FAILED [ 17%]
tests/test_log_plotter.py::test_plot_options[offset_stenosis_5_cycles.log-args6-probe_points_tke.png] FAILED [ 20%]
tests/test_log_plotter.py::test_plot_options[offset_stenosis_5_cycles.log-args7-flow_rate.png] PASSED [ 23%]
tests/test_log_plotter.py::test_plot_options[offset_stenosis_5_cycles.log-args8-velocity.png] PASSED [ 26%]
tests/test_log_plotter.py::test_plot_options[offset_stenosis_5_cycles.log-args9-cfl.png] PASSED [ 29%]
tests/test_log_plotter.py::test_plot_options[offset_stenosis_5_cycles.log-args10-reynolds_numbers.png] PASSED [ 32%]
tests/test_log_plotter.py::test_plot_options_compute_average[offset_stenosis_5_cycles.log-args0-cpu_time.png] PASSED [ 35%]
tests/test_log_plotter.py::test_plot_options_compute_average[offset_stenosis_5_cycles.log-args1-ramp_factor.png] PASSED [ 38%]
tests/test_log_plotter.py::test_plot_options_compute_average[offset_stenosis_5_cycles.log-args2-pressure.png] PASSED [ 41%]
tests/test_log_plotter.py::test_plot_options_compute_average[offset_stenosis_5_cycles.log-args3-newton_iteration_(atol).png] SKIPPED [ 44%]
tests/test_log_plotter.py::test_plot_options_compute_average[offset_stenosis_5_cycles.log-args4-newton_iteration_(rtol).png] SKIPPED [ 47%]
tests/test_log_plotter.py::test_plot_options_compute_average[offset_stenosis_5_cycles.log-args5-probe_points.png] FAILED                                                                                       [ 50%]
tests/test_log_plotter.py::test_plot_options_compute_average[offset_stenosis_5_cycles.log-args6-probe_points_tke.png] FAILED                                                                                   [ 52%]
tests/test_log_plotter.py::test_plot_options_compute_average[offset_stenosis_5_cycles.log-args7-flow_rate.png] PASSED                                                                                          [ 55%]
tests/test_log_plotter.py::test_plot_options_compute_average[offset_stenosis_5_cycles.log-args8-velocity.png] PASSED                                                                                           [ 58%]
tests/test_log_plotter.py::test_plot_options_compute_average[offset_stenosis_5_cycles.log-args9-cfl.png] PASSED                                                                                                [ 61%]
tests/test_log_plotter.py::test_plot_options_compute_average[offset_stenosis_5_cycles.log-args10-reynolds_numbers.png] PASSED                                                                                  [ 64%]
tests/test_log_plotter.py::test_plot_options_compare_cycles[offset_stenosis_5_cycles.log-args0-cpu_time.png] PASSED                                                                                            [ 67%]
tests/test_log_plotter.py::test_plot_options_compare_cycles[offset_stenosis_5_cycles.log-args1-ramp_factor.png] PASSED                                                                                         [ 70%]
tests/test_log_plotter.py::test_plot_options_compare_cycles[offset_stenosis_5_cycles.log-args2-pressure.png] PASSED                                                                                            [ 73%]
tests/test_log_plotter.py::test_plot_options_compare_cycles[offset_stenosis_5_cycles.log-args3-newton_iteration_(atol).png] SKIPPED                                                                            [ 76%]
tests/test_log_plotter.py::test_plot_options_compare_cycles[offset_stenosis_5_cycles.log-args4-newton_iteration_(rtol).png] SKIPPED                                                                            [ 79%]
tests/test_log_plotter.py::test_plot_options_compare_cycles[offset_stenosis_5_cycles.log-args5-probe_points.png] FAILED                                                                                        [ 82%]
tests/test_log_plotter.py::test_plot_options_compare_cycles[offset_stenosis_5_cycles.log-args6-probe_points_tke.png] PASSED                                                                                    [ 85%]
tests/test_log_plotter.py::test_plot_options_compare_cycles[offset_stenosis_5_cycles.log-args7-flow_rate.png] PASSED                                                                                           [ 88%]
tests/test_log_plotter.py::test_plot_options_compare_cycles[offset_stenosis_5_cycles.log-args8-velocity.png] FAILED                                                                                            [ 91%]
tests/test_log_plotter.py::test_plot_options_compare_cycles[offset_stenosis_5_cycles.log-args9-cfl.png] FAILED                                                                                                 [ 94%]
tests/test_log_plotter.py::test_plot_options_compare_cycles[offset_stenosis_5_cycles.log-args10-reynolds_numbers.png] FAILED                                                                                   [ 97%]
tests/test_log_plotter.py::test_plot_all FAILED                                                                                                                                                                [100%]

Diff images looked almost identical as the one I shared before. If this is very narrow OS or PC specific problem, i would say that we can leave it for now since it does not have a problem on github CI.

[johannesring]

I agree, let's leave it for now.

[keiyamamo]

Closing for now since it works on github workflow