e2e tests: `framer-motion` animations are not disabled by plugin disabling CSS animations

[ciampo]

What

The disable-animation plugin runs during e2e tests in order to make them "more stable" (see original PR).

But this set of styles don't have an impact on animations happening via the framer-motion library (as noted by @talldan in #43186 (review)) — this is because framer-motion "manually" sets the animated values as inline styles for each frame (instead of relying on native CSS animations and transforms).

Proposed next steps

The best solution that comes to my mind is:

• add a script to the disable-animation plugin, setting a global variable that can be read in app code
• add the MotionConfig component (see docs): when the global variable from the point before is defined, the reducedMotion option can be set to always, while normally it would be set to user

It should be noted that this is only a suboptimal solution:

• From the docs: "When motion is reduced, transform and layout animations will be disabled. Other animations, like opacity and backgroundColor, will persist." — so this wouldn't fix entirely the issue (disabling all transitions/animations)
• this is not great, since our tests wouldn't be running the same code that our user see

An alternative approach could be to manually check for that global "e2e_disable_css_animations" variable in the React code, and conditionally render an alternative to framer-motion components that doesn't animate (i.e. a normal div instead of a motion.div) — although this approach would also mean that our e2e tests would not actually test the same exact components that are rendered in our user's browsers

[talldan]

Puppeteer does have a way to emulate reduced motion, but prior attempts to use it didn't work - #18453.

There have been quite a few updates of puppeteer since, so things may be different.

[ciampo]

Should we be try to emulate reduced motion in addition to "force-disabling" all CSS animations, or as an alternative?

The original intention of the e2e tests plugin was to disable all animations in order to make e2e tests "more stable" — if that intention still holds true today, simulating reduced motion likely won't give the same results (some animations may still be happening) and we may need to find alternative options.

I also wonder if this plugin is necessary — as state before, by disabling all CSS animations we're making our e2e tests less realistic (compared to what the final user will interact with)

[talldan]

Should we be try to emulate reduced motion in addition to "force-disabling" all CSS animations, or as an alternative?

Possibly both? My main concern is test reliability, which is what makes me think both.

I also wonder if this plugin is necessary — as state before, by disabling all CSS animations we're making our e2e tests less realistic (compared to what the final user will interact with)

I think I've seen this discussion before somewhere (but I wasn't participating). For Puppeteer, the tests are already regularly flakey, and historically written in a way where they expect elements to be visible instantly. It may not be realistic to turn all animations on, a lot of tests would need to be updated to use waitForSelector.

For Playwright, the argument is a bit simpler. Animations being active does make tests run more slowly, but given Playwright does auto-waiting there should really be no difference between having animations on and off in terms of whether tests pass.

I think the best thing for the long term is to continue facilitating the migration to Playwright so that this is a simpler choice.

[WunderBart]

Unfortunately, some components still seem to fail with Playwright, even when the reduced motion is set and animations are disabled (natively). I recorded those components during the tests and noticed that they are animating into their default/initial state. I've checked if the transitionend event is fired for those components, and it is, just not instantly. This probably means that Playwright can't fast-forward those animations to completion as it claims in its documentation. I wouldn't say it's Playwright's fault, though – probably not even Framer-Motion's. At this point, I think it might have something to do with how we initialize those components that fail the screenshot test, as I don't think we want them to be animating into their initial states. Here's the list of those that I found to be problematic, along with the recordings of their initialization:

• Components/FontSizePicker: Default

  video.webm

• Components/FontSizePicker: With Custom Sizes Disabled

  video.webm

• Components/FontSizePicker: With Complex CSS Values

  video.webm

• Components/FontSizePicker: With Units

  video.webm

• Components/Popover: Dynamic Height

  video.webm

• Components/Popover: With Slot Outside Iframe

  video.webm

• Components/Popover: All Placements

  video.webm

• Components (Experimental)/Navigator: Default

  video.webm

Hope it helps! 🙌

[ciampo]

Thank you @WunderBart !

We can look into removing initial animation from these components:

• Popover
• ToggleGroupControl (responsible for the FontSizePicker animations)
• Navigator

During your tests, were the rest of the framer-motion animations (i.e. the ones happening as the result of user interaction) detected correctly by Playwright ?

[mirka]

Sorry, I was under the impression that ToggleGroupControl was migrated to framer-motion but I guess it's not? It does its own transition handling.

[ciampo]

Sorry, I was under the impression that ToggleGroupControl was migrated to framer-motion but I guess it's not? It does its own transition handling.

I had started working on it — I opened a first attempt, which was then superseded by a second attempt using shared layout animations, which couldn't be merged because of a bug in framer-motion

[ciampo]

I also had another thought — if we wanted to disable framer-motion animations, what if we mocked framer-motion components?(motion.div mocked to render a div, ...)

[WunderBart]

During your tests, were the rest of the framer-motion animations (i.e. the ones happening as the result of user interaction) detected correctly by Playwright ?

@ciampo I haven't tested any user interactions yet - just the initial part. Which components do you think could be problematic? I'd be happy to do some more testing to learn more about FM behavior 👍

edit: I guess I can start from the FontSizePicker as it seems to be a good starting point 😄

[ciampo]

@ciampo I haven't tested any user interactions yet - just the initial part. Which components do you think could be problematic? I'd be happy to do some more testing to learn more about FM behavior 👍

edit: I guess I can start from the FontSizePicker as it seems to be a good starting point 😄

You could test FontSizePicker as it's an interesting one (even if it doesn't use framer-motion for animations)

Of the ones you listed, Popover would be a good candidate, since it features often in the editor and is often animated