Can't visit some sites

[jhaynie]

I say some but I really mean https://github.com. My tests require me to visit github to login and I'm trying to script this. I have a portion of this working via OAuth and cy.request but I have an issue #391 which requires me to visit their logout page to attempt to clear their session.

This (reduced) test script just generally fails:

describe('Test', function(){
  it('test', function(){
    cy.visit('https://www.github.com')
    cy.title().should('include', 'github')
  })
})

It will flash the github homepage and then i get this (error) screen and it hangs:

If I change the domain to something else, say google, it works fine.

[brian-mann]

Github (in production) has a security mechanism in their JS that defeats Cypress.

It's basically something than this:

// this is true in Cypress
if (window.top !== window.self){
  // go back to the previous page
  window.history.back()
}

It's actually possible for Cypress to scan JS files and remove snippets like these, but it's a lot of work and adds a lot of overhead to the proxy layer instead of just piping the bytes to the browser.

We likely won't ever do this because we don't consider this an issue or a problem.

Why?

Cypress isn't for testing the web - it's for testing your application. If your application has security mechanisms enabled that make testing harder... then you should disable them.

When you control your own application it's easy, but if you're relying on another site, not only can you not control this behavior, but you're likely prone to write really flaky or inconsistent tests.

Here's a couple examples:

Facebook and Google both run random A/B experiments during oauth. What this means is that you run your tests from different locations you will get different pages. This breaks test scripts.

Another scenario we've seen is that Google will detect that you're running an automated script / robot and will send you to captcha or just disable responding to your IP Address altogether. So your first couple tests work, and then they will all immediately fail.

At the end of the day, you cannot consistently write automated tests to a 3rd party service.

Instead, we believe the sweet spot for Cypress isn't trying to replicate Selenium or do pure e2e testing - it's a hybrid approach where you use your application "like a user" but you often take shortcuts or "force things to happen" as a programmer. You can create escape hatches to bypass certain things that would otherwise go to 3rd party services. There is enormous upside to this - you stay within your application at all times, don't rely on 3rd party services, your tests will run orders of magnitude faster, and be much less flaky. The downside is that it takes a bit more work to setup and get right - but then again all code that's written without testing in mind is always painful to become testable.

In this scenario - I understand why you're trying to visit Github, in order to clear the session cookies. But the way you have this implemented wouldn't work anyways (even if Github did not have these security restrictions). There is another built in constraint to Cypress whereby you cannot visit two different super domains in a single test.

These constraints are all aligned with each other - they're all sort of telling the same story - you shouldn't visit external services.

Instead you could:

• Stub out the oauth call to Github to trick your application into think it's logged in. We are days away from releasing our new cy.stub and cy.spy API's which will be fully baked in and documented. So you could simply force the calls into your application to return what you want them to return.
• You could expose a backend endpoint which automatically "creates" a user and logs them in, and hands you back the cookie. All an outh call really does is eventually send data back to your server to identify you, and you could bypass all of this by asking your server to do the last leg of this for you up front.
• You can use cy.request to hit the external service directly but this may fail if the 3rd party service has rate limits or robot detection.

As for why the 3rd option isn't working for you - I'm not sure - my guess is that you're hitting an endpoint that doesn't have the Set-Cookie header. If you can paste some code, that would be helpful. You might be having a legitimate Cypress bug (because this should work).

Finally, a whole other feature that would sidestep this issue entirely is giving you the ability to clear all cookies on all domains irrespective of your current one. That would alleviate you ever having to forcibly log out of the 3rd party service, but it would completely incompatible with cross browsers in the future. Webdriver has no support for clearing cookies outside the current document domain's context.

[brian-mann]

Related to #408.

We are going to upgrade our API's to clear all cookies of all domains.