[Bug]: Running npx playwright test --list creates test-results folder (and also allure-results)

[urpizza]

Version

1.44.0

Steps to reproduce

Run
Running npx playwright test --list

Expected behavior

Not to create folders such as test-results, or allure-results (if added to config as reporter)

Actual behavior

Does create those files.

Additional context

No response

Environment

System:
    OS: Linux 6.5 Ubuntu 22.04.4 LTS 22.04.4 LTS (Jammy Jellyfish)
    CPU: (12) x64 13th Gen Intel(R) Core(TM) i7-1355U
    Memory: 11.02 GB / 31.01 GB
    Container: Yes
  Binaries:
    Node: 18.9.0 - ~/.nvm/versions/node/v18.9.0/bin/node
    npm: 8.19.1 - ~/.nvm/versions/node/v18.9.0/bin/npm
    pnpm: 8.15.4 - ~/.nvm/versions/node/v18.9.0/bin/pnpm
  Languages:
    Bash: 5.1.16 - /usr/bin/bash
  npmPackages:
    @playwright/test: 1.44.0 => 1.44.0 
    playwright: 1.44.0 => 1.44.0 
    playwright-core: 1.44.0 => 1.44.0

[lgarron]

Agreed on this. I just updated Playwright while updating all dependencies across an entire project and running npx playwright test to check on the tests. I might have accidentally committed test-results/.last-run.json if I had done the updates in a slightly different order.

I've already configured Playwright to store output in an ignored directory, so this was an unfortunate surprise. In our case it happens that I had only configured the outputFolder for a reporter, but that kind of detail is little consolation if my mental model is "I've already configured Playwright to write into an ignored dir." and I'm just maintaining a project. This kind of change is also a maintenance hurdle — I now have to look up:

• Why was this this directory/file created?
• What Playwright change caused a change in behaviour? Are there other recent changes to Playwright could create more? (Is it possible that Playwright could write a huge file that requires a hard git push if someone accidentally commits it?)
• How do I avoid committing these? Can I specify it in the config, do I need to pass a commandline flag, or do I need to add them to .gitignore?
• How do I prevent these files from appearing? If there is no config/commandline flag, do I have to manually delete the files afterwards. Could that cause any race conditions or test failures with Playwright if I do it incorrectly?
• Will this be an issue in any of another dozen projects where I use Playwright?

I'm not saying the feature is good or bad, just that the introduction of a new file artifact added significant cognitive overhead to my maintenance work. I happen to be experienced enough that I can probably resolve this whole issue in 15 minutes (ignoring the time spent on filing this issue), but presumably a bunch of other people are all going through this process separately.

If it's important to be able to keep this file and/or introduce new output files in point releases, I would suggest offering an output directory config that is inherited by all others, such as defaultOutputFolderRoot (or maybe something more concise). This would give me the confidence that Playwright can adjust its behaviour as needed without further intervention by me over something like this.

(Again, this isn't a criticism of the feature — I'm glad Playwright is actively maintained, and this kind of thing happens. I'm just trying to advocate caution for a sharp UX issue.)

[lgarron]

If it's important to be able to keep this file and/or introduce new output files in point releases, I would suggest offering an output directory config that is inherited by all others, such as defaultOutputFolderRoot (or maybe something more concise). This would give me the confidence that Playwright can adjust its behaviour as needed without further intervention by me over something like this.

On closer inspection, it appears the outputDir option meant to cover this:

playwright/packages/playwright/src/runner/runner.ts

Line 160 in c906448

const outputDir = project.project.outputDir;

However:

• Specifying an outputDir does not prevent test-results/.last-run.json from being created in our project.
• The outputDir doesn't seem to be used by the test reporters — do they all need outputFolder/outputFile to be specified separately?
  • EDIT: actually, I got an error implying these directories need to be separate. This is not at all obvious from the documentation and Playwright's functionality. It also means I'm back to the drawing board on a clear way to tell Playwright to place all files created by npx playwright test under specific directory.

[lgarron]

Okay, in case anyone else runs into something like this, here's what I've learned by now:

• Why was this this directory/file created?

A feature was added in Playwright v1.44.0 to rerun just the failed tests from last run. (Definitely a nice feature!)

This feature writes a .last-run.json file to keep track of relevant information. Notably, this file is not the output of the "json" test reporter. As far as I can tell, this is an unrelated feature that also involves writing test results to disk in JSON format.

• What Playwright change caused a change in behaviour?

• This feature was requested in: [Feature]: Option to run only failed cases from the previous run #30506 and [Command line flag that runs all "failures" from the last run] #10500
• This feature was added in: feat(runner): allow running last failed tests #30533
• This feature was updated to write .last-run.json into the project's outputDir in: fix(last-run): remove globalOutputDir #30571

Are there other recent changes to Playwright could create more? (Is it possible that Playwright could write a huge file that requires a hard git push if someone accidentally commits it?)

No, this a single file specific to this feature. It so happens that this feature:

1. runs by default,
2. uses the default output folder for test files, and
3. newly creates this folder as of v1.44.0 if you weren't writing files to the outputDir in any of your tests.

There is nothing in the implementing PRs or associated issues that implies other files could unexpectedly be written. If new files are written that are related to this feature, I presume they would also go into the outputDir.

• How do I avoid committing these? Can I specify it in the config, do I need to pass a commandline flag, or do I need to add them to .gitignore?
• How do I prevent these files from appearing?

This file is written on every run by default. This is definitely convenient if you unexpectedly encounter some failing tests, and want to pass --last-failed on the next run even if you didn't plan ahead to do so.

As far as I can tell, there is no way to prevent this file from being written. The only way to control where it's written is to specify an outputDir for the project: https://playwright.dev/docs/api/class-testproject#test-project-output-dir

Again, this feature is unrelated to the JSON reporter, so this must be a separate directory from any outputDir or outputFile specified by a reporter. However, they can both share a common parent directory. For example:

import { defineConfig, devices } from "@playwright/test";

export default defineConfig({
  // …
  outputDir: ".temp/playwright/test-results",
  reporter: [
    ["html", { outputFolder: "./.temp/playwright/report" }],
    ["json", { outputFile: "./.temp/playwright/report.json" }],
  ],
});

If there is no config/commandline flag, do I have to manually delete the files afterwards. Could that cause any race conditions or test failures with Playwright if I do it incorrectly?

It looks like .last-run.json is not used by any other feature. I think removing it will cause npx playwright test --last-failed to report Error: No tests found as if npx playwright test had never been invoked, but no other issues.

• Will this be an issue in any of another dozen projects where I use Playwright?

That depends. The options seem to be:

• Live with this file being committed accidentally.
• .gitignore it, or the test-results folder.
• Add an outputDir config to each project to place the file in a different folder, as in the example above.

[lgarron]

Also, I'm not certain if any of my issues actually have anything to do with @urpizza's specific issues, as all my issues are with npx playwright test without specifying --list.

This issue was filed two days after v1.44.0 was released, though. So it's almost certainly the same root cause for the test-results folder.

I have no idea about the allure-results folder, though. The Playwright codebase doesn't include allure-results, so presumably that folder is entirely the responsibility of https://allurereport.org/docs/playwright/ This does make me wish for a simple way to specify a root folder under which all output folders created by Playwright and its ecosystem are placed unless individually overridden, so I can "set and forget".

[baev]

Allure is yet another reporter. The issue can be reproduced using the Playwright HTML report:

$ npx playwright test --list

generates playwright-report/index.html file. The workaround is to override reporter configuration, e.g. by specifying dot reporter:

$ npx playwright test --list --reporter dot

will not produce any reporters' output.

The issue with the test-results/.last-run.json file still exists, though.

@pavelfeldman is there any reason to execute reporters when the --list option is used?