Enabling javascript causes images to double in height

[rsimha]

We're using percy/capybara to generate snapshots in our visual tests for the AMP project. We recently switched from PhantomJS to Chrome / Selenium. Today, I noticed that our snapshots were missing certain portions due to the fact that JavaScript wasn't enabled on the Percy side. (The content showed up on my local machine when the page was first rendered prior to snapshotting, but was missing in the final snapshots because Percy doesn't enable JavaScript by default.)

When I repeated the snapshots with JavaScript enabled (using Percy::Capybara.snapshot(page, name: name, enable_javascript: true)), I noticed that the mere act of enabling JavaScript is causing all our image assets to double in height in the snapshots. (They appear normal when rendered locally, prior to snapshotting.) This is mysterious behavior and I'm not sure what we can do to prevent this. See the snapshots in https://percy.io/ampproject/amphtml/builds/538714 for example. The baselines do not have JavaScript enabled, while the new snapshots do.

FYI, here is how we are setting up Capybara to use Chrome instead of PhantomJS.

Left: with JS, Right: without JS

[rsimha]

/to @fotinakis @timhaines

[timhaines]

Hi @rsimha. In one snapshot, the images sources are img/hero@1x.jpg and img/sea@1x.jpg and in the other they're img/hero@2x.jpg and img/sea@2x.jpg

Do you have some JS in place that's switching the img source?

It looks like the css you have setup maintains the image height, but fixes the width?

Is this enough info for you to resolve this?

[rsimha]

@timhaines I didn't have visibility into that behavior, so I wasn't aware that the img source was at play. I'll investigate on our side. Thanks!

[rsimha]

@timhaines Unfortunately, after investigating, this appears to be a bug. I re-ran the visual tests locally, and when img/hero@2x.jpg is rendered locally, it appears in the correct aspect ratio. However, the Percy snapshots are displaying the wrong aspect ration (way taller than it is supposed to be).

The original images are here: https://github.com/ampproject/amphtml/tree/master/examples/visual-tests/article-access.amp/img

Any ideas?

[rsimha]

A more illustrative example of what's going wrong is here: https://percy.io/ampproject/amphtml/builds/538856/view/25118149/375?mode=head&snapshot=25118149

Scroll down to see that the image that's supposed to have a 1x1 aspect ratio is being rendered with double the height. When I ran this test locally with Percy / Capybara / Chrome, I noticed that the images were showing up with the correct aspect ratio.

[rsimha]

@timhaines I did a visual diff with another test page, which does more stuff with JS to see if this is an overall JS problem with the Percy backend. This test page is designed to display the first paragraph of an article, with a "Login to read more" message to indicate that the user hasn't subscribed to the service.

Here is what I'm seeing on my machine when Percy / Capybara / Chrome are in action. Notice that the image is showing in the correct aspect ratio, and there are no JS errors. Also notice that only the first paragraph of the article is displayed, followed by the "Login to read more" message, as expected.

However, when this same page is rendered by the Percy backend, here's the snapshot that is generated (I've cropped just the top portion). Notice that the image is double the height, the entire article is displayed, and the "Login to read more" message doesn't show up, but instead, we see the "Oops, something broke" message. Any idea what's going on here? (link to snapshot)

[rsimha]

@fotinakis @timhaines We're seeing the same issue with our real-world page visual tests that use the Python library. See https://percy.io/ampproject/danielrozenberg/builds/539438

Since this is breaking all our tests right now, would it be possible to prioritize the investigation of this issue? Note that all these pages render perfectly fine on a local Chrome browser, so I'm quite sure this has something to do with how the Percy backend renders AMP pages from DOM snapshots with JS enabled.

/cc @danielrozenberg @cramforce

[timhaines]

@rsimha We will spend some time looking into this. In the interim, are builds with javascript disabled still rendering the images correctly for you?

[danielrozenberg]

I think we might be using the API in an unexpected way. I used a Selenium driver to load the page on a local server, then I send the snapshot with enable_javascript=True. What I think is happening is that it takes the DOM snapshot, then on your server side since js is enable it re-does all the loading behaviours of AMP, which screw up things in many ways.

I tried a different approach right now where I skip the local server and Selenium driver, and directly send the raw HTML of each tested page with js enabled. This seems to produce the expected results for me (but I'm still investigating whether this is the approach we want to take)

[fotinakis]

@rsimha that is an odd one, we can help you look into this. Digging through the code briefly, we recommend removing enable_javascript: true in the snapshot call because the JS is already being executed locally when you're running through in Chrome headless and generating the DOM snapshot. — ie. you shouldn't mix Capybara with javascript enabled and Percy with javascript enabled, it should be either or.

What content is missing from the pages when you don't have javascript enabled? If you run the tests without --headless or use capybara's save_and_open_screenshot locally (https://github.com/teamcapybara/capybara#debugging), do you see the content there?

[rsimha]

@fotinakis thanks for pointing out how enabling JS for both Capybara and Percy isn't the right thing to do. That part now makes sense to me.

Re: missing content, first off, I'm not using --headless to generate DOM snapshots (I see a real Chrome window). Yet, we're missing these items:

• Certain kinds of content like ads render fine on the local Chrome window, but not when the page is rendered by Percy
• Other AMP components like amp-access just don't work on Percy. The article is supposed to be collapsed, and say "Login to read more", like in the first screenshot in Enabling javascript causes images to double in height #51 (comment). Instead, we're seeing this with "Oops, something broke".
• We're seeing blank content in the test pages for several of our AMP components. For example, this snapshot is supposed to have a GitHub gist in the blank frame. (It does have it when rendered locally.)