Syntax error does not fail the test but stops the runner

[Chemaclass]

Summary

Avoid silent errors. Aim to fail fast and loud.

Current behavior

When encountering a syntax within a test, it should endup with a failure instead of a silence stop.
See the macos tests in this CI:

How to reproduce

See the current behaviour above, and see the example here from this fix PR so the previous code will reproduce it.

Expected behavior

A syntax error will endup in a failing test.