Skip to content

Fix flaky test#528

Merged
stefanvanburen merged 2 commits intomainfrom
svanburen/fix-flaky-test
Jan 21, 2026
Merged

Fix flaky test#528
stefanvanburen merged 2 commits intomainfrom
svanburen/fix-flaky-test

Conversation

@stefanvanburen
Copy link
Member

We seem to be hitting timeouts running some of the tests; bump them when we have to download the latest buf binary.

Ref: https://github.com/bufbuild/vscode-buf/actions/runs/21220252844/job/61052768068#step:11:109

We seem to be hitting timeouts running some of the tests; bump them when
we have to download the latest `buf` binary.

Ref: https://github.com/bufbuild/vscode-buf/actions/runs/21220252844/job/61052768068#step:11:109
@stefanvanburen
Copy link
Member Author

https://api.github.com/repos/bufbuild/buf/releases/latest 403 rate limit exceeded

holy moly, I guess we need a GH token to raise the rate limits.

Comment on lines 33 to 36
if (process.env.BUF_INSTALLED === "buf-not-on-path") {
// Increase timeout when downloading buf binary from GitHub
this.timeout(30000); // 30 seconds
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to adjust this timeout through the test runner, right? https://github.com/bufbuild/vscode-buf/blob/main/.vscode-test.mjs#L24-L35

https://code.visualstudio.com/api/working-with-extensions/testing-extension#quick-setup-the-test-cli

But also, this shouldn't be timing out at all, it should be calling out to the msw mock server: https://github.com/bufbuild/vscode-buf/blob/main/test/shared/shared.ts

Based on the CI output, it seems not to be doing that. Also, the PR before that adjusts the version check to be dynamic -- when we hit the mock server, it uses 1.54.0 from the npm install in the workspace, which is why it's hard coded, but I do think that making the check dynamic is nicer.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chatted offline, but tl;dr: msw doesn't work here, because MSW runs in the test process, but the VS Code extension runs in a separate extension host process. MSW can only intercept fetch calls in the same process where it's running.

Ended up plumbing down the auth token from the workflow instead to hopefully bypass rate limits: 8513f0f

@stefanvanburen stefanvanburen marked this pull request as draft January 21, 2026 18:57
For rate limits. Also, don't set up MSW in the tests, as it's confusing;
it doesn't actually mock the GitHub API
@stefanvanburen stefanvanburen marked this pull request as ready for review January 21, 2026 19:21
@stefanvanburen stefanvanburen merged commit 6c60844 into main Jan 21, 2026
8 checks passed
@stefanvanburen stefanvanburen deleted the svanburen/fix-flaky-test branch January 21, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants