Module API doesn't fail the promise with Cypress failure messages

[NicholasBoll]

Forgive me if this issue exists, I wasn't able to find it.

Current behavior:

If using the Module API and Cypress fails for various reasons (baseUrl isn't reachable, spec isn't found) the module API resolves with a shape of { failures: 1, message: "Cypress error message" } instead of rejecting.

Desired behavior:

The module API should reject the promise with an error message. This is then easier to detect failure cases. There is a difference between failing to run because of some error and running successfully and returning results that could contain failed tests.

Steps to reproduce: (app code and test code)

You can run this from from any project that has Cypress in it:

node -e "\
  var cypress = require('cypress');\
  cypress.run({ spec: 'does-not-exist.js' })\
  .then(result => {\
    console.log('result', result);\
    process.exit(0);\
  })\
  .catch(error => {\
    console.log('error', error); /* we never get here */\
    process.exit(1);\
  })"

Running that will output a red error message from Cypress, but the process will exit with a 0. Even without an explicit process.exit and having no .catch will do the same thing. The issue can be mitigated by adding the following after the .then(result => line:

if (result.failures) {
  throw result
}

Versions

Cypress 3.1.4

[DamienCassou]

I confirm the problem for these cases:

• the ciBuildId already exists in the dashboard
• the record key is absent and --record was passed

For both of these cases, the promise resolves and I think it should be rejected.

Workaround:

cypress.run(...).then((testResults) => {
	if (testResults.status === "failed") {
		// For some reason, Cypress resolves its promise even when
		// it can't start the tests. I think rejection is better
		// suited. https://github.com/cypress-io/cypress/issues/3216
		return Promise.reject(testResults);
	}

	// When Cypress manages to run the tests (all of them
	// successful or not), we want to resolve the promise
	// successfully:
	return testResults;
})

[monkpit]

@jennifer-shehane Let me know if this should be a separate issue, but I have a related request:

When using the Cypress module API (I am using v9.7.0), it's possible to parse CLI arguments using cypress.cli.parseRunArguments. If there is anything wrong with the arguments, this function will call process.exit and shutdown the entire node process.

For my use case, I'd much rather be able to try/catch and trap the error. It won't work to add a process.on('exit') handler or something similar.

Maybe it would be possible to detect if the module was require'd and throw the error instead of calling process.exit? Or maybe it could be written to always throw and lift the process.exit call up higher in the design, allowing downstream users to call the function directly without triggering an exit event?

Thanks!

[cypress-app-bot]

This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided.

[cypress-app-bot]

This issue has been closed due to inactivity.