experimentalSourceRewriting performance issue

[laurentenhoor]

Current behavior:

{
	"experimentalSourceRewriting": true
}

After enabling experimentalSourceRewriting, a single VISIT takes about 5-10 times as long. An unacceptable increase for our pipelines. HTTP requests to GET the page's resources seems to be significantly slower.

Desired behavior:

We want the HTTP requests to be as performant as without experimentalSourceRewriting.

Test code to reproduce

I created this very simple PR: cypress-io/cypress-test-tiny#59
Please try to disable/re-enable experimentalSourceRewriting.
Have a look at the difference in time that the following test takes:

describe('page', () => {
  it('works', () => {
    cy.visit('https://example.cypress.io')
  })
})

if experimentalSourceRewriting: true it takes ~1583ms
if experimentalSourceRewriting: false it takes ~300ms

Versions

Cypress 4.6.0

[jennifer-shehane]

There does appear to be ~1000ms increase in the test time for visiting this specific URL when experimentalSourceRewriting: true is set.

spec.js

Cypress.on('test:after:run', (attributes) => {
  console.log('Test "%s" has finished in %dms', attributes.title, attributes.duration)
})

describe("Testing", () => {
  Array(15).fill(0).forEach((val, i) => {
    it(`test ${i}`, () => {
      cy.visit('https://example.cypress.io')
    })
  })
})

With experimentalSourceRewriting: true

With experimentalSourceRewriting: false

[a8trejo]

I am experiencing a similar Issue, a little worse,
Context,

• For my Test Suite I'm using Cucumber features
• There is a specific test with a login which only works when I set "experimentalSourceRewriting" as true
• When I set "experimentalSourceRewriting" as true, in all my other Scenarios (same feature file) other cy.visit never load the page, I already tried increasing the pageLoadTimeOut up to ridiculous 80 seconds and it just never loads.

Now I am trying to set Cypress.config('experimentalSourceRewriting',true) before that specific test where I need it, and the value supposedly changes (if I perform a cy.log it comes as true) but then I will get the HTTP 404 error as if it was not configured.

Is there a way to set Single test configuration for a specific cucumber Scenario? (Setting 'experimentalSourceRewriting',true in this case)

[a8trejo]

In case it helps anyone, the workaround I found for Cucumber Tests was Tagging that specific Scenario with something like @experimental and then use the command line to run only that Scenario with that configuration:

npx cypress run --spec **/Name.feature -e TAGS='@experimental' --config experimentalSourceRewriting=true

And for all my other tests (cypress.json has "experimentalSourceRewriting" : false ):
npx cypress run -e TAGS='not @experimental'

For normal Cypress mocha tests I guess they could pass the single test Configuration like this? (Haven't tried it)
https://docs.cypress.io/es/guides/references/configuration.html#Single-test-configuration

My workaound however gives me a new Issue as the cucumberJSON report generation only creates one report per feature file, and as far as I know still has no timestamp option, hence I only get one report out of my 2 executions of that feature and I need cucumberJSON format for my CI/CD cloud integration with JiraXray

[modern-sapien]

Issue 8128 is similar and has a workaround that may be worth referencing here.

[cypress-app-bot]

This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided.

[cypress-app-bot]

This issue has been closed due to inactivity.

[JoostK]

Coming from #29438, I dug into it with a profiler and found my case to be primarily due to cloning the AST in recast, where converting some linear searches into binary searches reduces the time to 5s (from 100s+, although I don't have exact numbers at the moment because I profiled using a different laptop and didn't capture the original duration on this machine)

Even with that addressed—I want have to look into it further to better understand why this makes such a difference here—there's more room for improvement, where reusing the original JS instead of printing a potentially unmodified AST is super wasteful. Potential saving of ~1.8s, bringing the total down to 3.2s for the file I was seeing 100s+ rewrite times for. Still not great and room for improvement, but getting somewhere.

Update: after some more optimizations, we're down to ~1.3s. Mostly in an experimental state still.

[jennifer-shehane]

@JoostK It'd be helpful, if possible, to get those optimizations back in as a PR to update this logic.

[JoostK]

@jennifer-shehane yeah definitely, I will have to carve out some more time for it though. Hopefully somewhere this month.

[JoostK]

Okay, I have two PRs out:

• perf: avoid excessive linear scanning for large functions benjamn/recast#1399
  • This offers the biggest improvement for the issue I was seeing, going from ~120s to 1.2s.
• perf: improve performance of experimentalSourceRewriting #29540
  • This implements some optimizations in Cypress' use of recast to allow my case to go from 4.4s to 2.3s (provided that the recast fix has been patched in).