Precommit failing intermittently

[elijah-potter]

Describe the bug

Precommit will sometimes fail on the untitled buffer VS Code integration test.

To Reproduce

It's hard to reproduce, since it doesn't happen deterministically. My best advice: run just precommit over and over again until it fails.

Screenshots

Platform

The VS Code integration tests.

Additional context

I think this started with #927. Any ideas @kiding?

[elijah-potter]

Here is a recent example of the problem: https://github.com/Automattic/harper/actions/runs/14204190745/job/39797911729

[kiding]

I think I know what the problem is. Originally in #927, it slept for 500ms after await openUntitled. #960 changed it to waitForUpdatesFromOpenedFile which only sleeps for 75ms. I remember in my testing it reliably failed when the sleep time was that short. I'll make a PR.

[mcecode]

I set out in #960 to standardize the waits we use for different things, that's why I changed it. How much we need to wait for diagnostics to change really depends on the machine, what's running on it, etc. I didn't want to use long sleep times every time diagnostics change so that the VS Code tests won't take forever to finish since the startup with the downloads already takes a while, but if it's really become flaky and non-deterministic, should we just set 500ms as our base time every time we need to wait for diagnostics? That should be enough time so we don't get false positives.

[kiding]

I’m okay with the idea to fix all delays to 500ms (precommit takes a LONG time anyways), but I think it’d be better to keep this for now and find a better solution. I’m hoping there has to be a decent way to make use of VS Code APIs to sense when the process is finished / the diagnostics are updated / harper-ls responded / etc.

[mcecode]

I’m hoping there has to be a decent way to make use of VS Code APIs to sense when the process is finished / the diagnostics are updated / harper-ls responded / etc.

I'm open to ideas, as far as I can remember, I couldn't really find an alternative to adding delays when I started writing the integration tests.

[elijah-potter]

Would you be able to send a command to harper-ls and use the response as an indication that it's ready?

[kiding]

Not entirely sure. I'll try to investigate more when I get a chance.