Unit tests for the grammar?

[PanAeon]

Hi,
I've written a small tool that allows to write assertions in a plaintext file against a textmate grammar, like so:

      def add(x:Int,y:Int) = x + y
          ^^^ entity.name.function.declaration

The tool uses vscode-texmate internally, so I'm pretty confident that it would work the same way as the visual studio code itself.

If maintainers are OK with this, I can include the tool in the project and write a few test cases for the Scala.tmLanguage.json.
What do you think?

[olafurpg]

This is a great idea! I was also thinking what would be the best approach to test changes to the syntax rules. A PR adding test cases are very much welcome 🙏

In vscode-tmgrammar-test, is it necessary to manually assertions like ^^^ entity.name.function.declaration? I'm wondering if it could be possible to use "expect tests" or "snapshot tests" (see https://jestjs.io/docs/en/snapshot-testing) where the input files are regular Scala files and output files are auto-generated and committed into the repo. The build would need a command to re-generate the output files when the expected behavior changes.

We use expect tests a lot in Metals, for example here is an auto-generated file for the result of calling "goto definition" on every token in the source file https://github.com/scalameta/metals/blob/f28c35ac855ed58bd5bf30fcea70e8497c49e697/tests/unit/src/test/resources/definition/example/Scalalib.scala When there is a regression then the tests fail with a nice diff error message

--- expected
+++ obtained
- val lst/*Scalalib.scala*/ = List/*List.scala*/[
+ val lst/*Scalalib.scala*/ = List/*Incorrect.scala*/[

To update the auto-generated output files in the Metals build we run the sbt save-expect command.

[PanAeon]

Well, "snpashot tests" is certainly a good idea in this context! The vscode-grammar-test doesn't support them now, but I don't think they are too difficult to add. If I got time next week I'll try to implement them.

As for the manual assertions they feel a bit more conservative to me. It's like when there's a massive change in the testcase sometimes it's easier to overlook some minor detail in a big diff or something.
So maybe I'll submit a set of manual assertions as well. They are not hard to write, I expect them to take just a few hours of work.

[olafurpg]

I agree manual assertions are helpful even if you have expect tests, they complement each other.

[olafurpg]

Fixed in #27 by @PanAeon 🎉