Tests failing locally

[Will-Sommers]

While working on #597 I had erroneously reported failing tests locally. The suite passed on the PR build action but locally I am having failing tests. I also checked on the main branch and also have similar failures. The test run hasn't finished yet so I'm not sure if they are deterministically failing but I'll update the issue with more information. Here's an example locally failing test from the PR:

  22) recorded test cases
       /Users/.../recorded/languages/typescript/ifElseWrapThis:

      AssertionError [ERR_ASSERTION]: Unexpected final state
      + expected - actual

       {
      -  "documentContents": "if () {\n  const foo = \"hello\";\n} else {\n  \n}"
      +  "documentContents": "if () {\n    const foo = \"hello\";\n} else {\n    \n}"
         "selections": [
           {
             "active": {
               "character": 4

Also, noted in Slack. I currently have to quit VSCode to run the suite. Probably a separate issue but worth noting.

[pokey]

@wenkokke was having this issue too. Do you remember if we got to the bottom of it?

[wenkokke]

We never managed to fix it, I simply ignored it and you didn't have this issue locally.

[pokey]

We never managed to fix it, I simply ignored it and you didn't have this issue locally.

Right ok that was my recollection but was hoping my recollection was wrong 😅

I believe this one has to do with VSCode space insertion. When you define a snippet in VSCode, it reinterprets tab characters in the snippet body as spaces according to your configuration for the given file / language. So I believe it has something to do with your indentation settings. We somehow need to get VSCode to ignore your custom indentation settings I believe

[Will-Sommers]

Hrm, got it. I'll think about this a bit. I wonder if it would make sense to give local developers the ability to run tests akin to the GH actions, in a local container.

[pokey]

oh that's a cool idea. I wonder if that would fix the insiders' thing too

[Will-Sommers]

This looks pretty cool, although new, and might obviate the need for a separate and possibly diverging config. https://github.com/nektos/act

[pokey]

Funny I just ran across that today for the first time

Interesting idea, and might simplify things, tho seems a bit heavy handed, as I believe it installs VSCode in the container every time. But prob not a showstopper; might not be too slow and could potentially be cached

Also wonder how it handles matrix, as our CI uses a matrix to run tests on 3 platforms

[wenkokke]

That probably means that it would download three separate images, and run the test three times… not a huge problem overall I suppose, but it might be nice to be able to limit it to a single platform only.

[pokey]

Windows and macOS based platforms are currently unsupported and won't work (see issue nektos/act#97)
- https://github.com/nektos/act#runners

So presumably it will only run Linux? Either that or try to run Mac and Windows and fail? 😅

[Will-Sommers]

Let me try and just run it with a docker. I'll look at checkout@v2 and the yarn action and replicate the set up steps.

[pokey]

Is this one still an issue?

[Will-Sommers]

Yes, it is still an issue, even after init sandbox

On main:

  1534 passing (46s)
  56 pending
  26 failing

Failing tests:

src/test/suite/fixtures/recorded/actions/indentVest:
&&
all of the if/else/ifElse wrap calls

[pokey]

Ok let's see if #672 helps once it's merged. I don't think it will help the wrap tests, but should help some local failures

[Will-Sommers]

My "tab size" was 2, I moved it to 4 and they're all passing now. I guess we should add notes within the README.md and maybe something

  1560 passing (42s)
  56 pending