[Feature]: Option to fail test on `webServer` stderr

[karlhorky]

🚀 Feature Request

Currently, if a webServer configuration emits stderr, it is only piped to the logs by default and the test execution continues.

In our test runs for our apps, we have encountered server exceptions which have been silently ignored for weeks, because the Playwright tests pass.

It would be useful to be able to configure webServer[number].stderr = 'fail' or a similar API, which would fail the currently running test if stderr is written to by the webServer process.

Possibly this should be also made the default value for webServer[number].stderr

Example

This configuration would result in test failure upon any stderr from either of the Frontend server or the Backend server:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  webServer: [
    {
      command: 'npm run start',
      url: 'http://localhost:3000',
      name: 'Frontend',
      timeout: 120 * 1000,
      reuseExistingServer: !process.env.CI,
+     stderr: 'fail',
    },
    {
      command: 'npm run backend',
      url: 'http://localhost:3333',
      name: 'Backend',
      timeout: 120 * 1000,
      reuseExistingServer: !process.env.CI,
+     stderr: 'fail',
    }
  ],
  use: {
    baseURL: 'http://localhost:3000',
  },
});

Motivation

Described in the first paragraphs of Feature Request

Alternatives Considered

1. A wrapper script which can be used around all servers which crashes the process of stderr is emitted (downsides: increased complexity of test runner script and setup, possible unknown problems with Playwright test execution when crash occurs)

[karlhorky]

Workaround

Use the exit-on-stderr.ts wrapper script to exit server on stderr, described on Playwright Tricks:

https://github.com/karlhorky/playwright-tricks#fail-test-on-webserver-stderr

Downsides:

1. Because this isn't integrated with the Playwright test lifecycle, the test may still succeed in certain cases - eg. if the stderr occurs at the end of the test
2. Exiting the server causes a weird disconnect in Playwright UI Mode, where all tests re-run afterwards will fail

[agg23]

Your workaround is one option. You could also use an automatic fixture to start and manage your server.

However, I am interested in how your server is generating errors. Our expectation is that any kind of server errors resulting from browser requests would be represented in the corresponding responses (for example a 500 error). In situations like that, the errors are visible to Playwright, thus you can respond to them in your test as necessary.

[karlhorky]

Our expectation is that any kind of server errors resulting from browser requests would be represented in the corresponding responses (for example a 500 error). In situations like that, the errors are visible to Playwright, thus you can respond to them in your test as necessary.

Ideally the server will respond with the error in the response. However, that will not always be the case, eg. for unexpected exceptions (possibly also caused by library behavior).

In our case, the Next.js rendering of MDX emitted stderr for a component which errored but delivered the rest of the page.

---

When the server is able to respond, HTTP 500 errors would be indeed in the status code of the response - we handle these separately, with page.on('response', (response) => { ... }).

• [Question] Fail on unhandled exception? #16355 (comment)

Arguably this should also be a built-in option for Playwright, so that this boilerplate code of the throwOnErrorsOrConsoleLogging() function doesn't need to be in every Playwright project.

[agg23]

What does throwOnErrorsOnConsoleLogging() have to do with this scenario at all? One is for failing tests on errors emitted in the page, one is for failing tests based on some unknown native process that the test runner spawned.

Given that there are clear workarounds for this problem, we are unlikely to take any additional action at this time. We don't necessarily agree with the premise; stderr is used often to log things that are not errors or are not total failures; it's up to the application to choose. By that logic, it's up to the test author to decide if they want to do something with stderr, and we provide them the tools with which to do that. It isn't something that should be a first party Playwright feature.

[karlhorky]

What does throwOnErrorsOnConsoleLogging() have to do with this scenario at all? One is for failing tests on errors emitted in the page, one is for failing tests based on some unknown native process that the test runner spawned.

If there is an HTTP 500 status code from one of the requests on the page, this can lead to console messages, which causes the test to fail with throwOnErrorsOnConsoleLogging().

But as I mentioned, this is not helpful always - for example, it's not helpful in the MDX scenario that affected us.

By that logic, it's up to the test author to decide if they want to do something with stderr, and we provide them the tools with which to do that. It isn't something that should be a first party Playwright feature.

What tools are provided by Playwright to hook into stderr from the server? I wasn't aware there were any hooks to manage the servers in webServer.

[dgozman]

We discussed this with the team, and don't see a clear solution that will work. Web server is a global component, and any automatic wiring of the error to a particular test is impossible. Playwright focuses on testing in the browser, and wiring up all kinds of backend tools is not practical. I am closing this issue, as no action item is planned. Thank you for filing!

[karlhorky]

Oh, so Playwright doesn't have a way of throwing "global"-level errors, failing the currently-running test?

No need to wire the error to a particular test or something.

I thought I've seen this behavior with Playwright already somewhere, where a more global-level error would cause the test to fail, instead of logic in the test 🤔

---

What would be the suggestion of the Playwright team to handle such a use case in userland? My current workaround solution has some unfortunate downsides...

If there would be a way of connecting the webServer server to Playwright itself (with some kind of API to fail the current test), that would make a userland option much easier to implement.