Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Single-machine parallel execution occasionally results in "not a function" errors #19565

Closed
j1000 opened this issue Jan 5, 2022 · 5 comments
Closed

Comments

@j1000
Copy link

j1000 commented Jan 5, 2022

Current behavior

I have a weird intermittent failure that only happens when I run the tests in parallel on my local machine (using a Powershell script). Out of ~200 tests about 5 will fail, and it's never the same tests. The error is always "__ is not a function" where __ is one of my Cypress custom commands. It seems as if there's a race condition happening in Cypress's code, and the spec file is loading before /support/index.js has been loaded. (Maybe there's a file permission issue due to all the threads accessing the same stuff?)

image

Desired behavior

Ideally the bundling would work consistently and /support/index.js will always be properly loaded for each spec file. But alternatively it would be nice if test reruns reloaded (and rebundled) the spec file entirely so that the test doesn't fail upon rerun.

Test code to reproduce

I wish it were simple to reproduce. You need any large set of tests running in parallel locally. I am on a Windows 10 workstation. I'll give you my Powershell script in case that helps.

$processes = 5

# Provide an argument when invoking this script to target a specific environment
$bUrl = $args[0]
if ($bUrl) { $hostUrl = "baseUrl=$bUrl" } else { $hostUrl = "baseUrl=https://www.somewebsite.com" }

Write-Host "Running tests on $hostUrl"

$epoch = Get-Date -UFormat %s

$script = { param($pEpoch, $pHostUrl) cmd /c "npx cypress run --spec './cypress/integration/!(debug)/**' --record --key insertkeyhere123456789 --parallel --ci-build-id $pEpoch --browser chrome --config video=false,$pHostUrl" }

$loc = Get-Location

for ($i = 1; $i -le $processes; $i++) {
  Start-Job -ScriptBlock $script -WorkingDirectory $loc -ArgumentList $epoch, $hostUrl -Name "cypress"
}

Get-Job | Wait-Job

Remove-Job *

Cypress Version

9.2.0

Other

No response

@mjhenkes
Copy link
Member

Hi @j1000, we currently don't support running multiple instances of cypress in the same project on the same machine at the same time. It can cause weird interactions (as you've run into). You could take a look at splitting this up to run in several docker containers in parallel, but i can't promise it's a faster solution. Sorry.

@j1000
Copy link
Author

j1000 commented Jan 19, 2022

Gotcha, thanks. This is a shame, because it used to work without issue. I'm not sure if Cypress changed or if my tests changed.

@j1000
Copy link
Author

j1000 commented Feb 5, 2022

@mjhenkes I think I have a viable workaround for this. If you launch the tests with the configuration value numTestsKeptInMemory=0 the errors no longer happen. I'll continue monitoring this to make sure it's not a false correlation.

@j1000
Copy link
Author

j1000 commented Feb 17, 2022

@mjhenkes I think I have a viable workaround for this. If you launch the tests with the configuration value numTestsKeptInMemory=0 the errors no longer happen. I'll continue monitoring this to make sure it's not a false correlation.

After some time using this workaround, it's not a cure but it seems to make a huge difference. I was getting 3-6 "is not a function" failures with every run before; now I'm getting 0-1. Usually 0. I'll learn Docker eventually but this works for now ;)

@actuallymentor
Copy link

@mjhenkes I think I have a viable workaround for this. If you launch the tests with the configuration value numTestsKeptInMemory=0 the errors no longer happen. I'll continue monitoring this to make sure it's not a false correlation.

Thank you very much for this. Makes my tests far less flaky too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants