Skip to content

Get rid of legacy tests #3896

Closed
Closed
@smarter

Description

@smarter

Our legacy test infrastructure is a maintenance burden and uses up resources on our CI. To get rid of it I think we need:

  • Proper integration meta-tests. We already have unit meta-tests in https://github.com/lampepfl/dotty/blob/master/compiler/test/dotty/tools/vulpix/VulpixTests.scala but these don't run vulpix end-to-end. We should set up a directory of test files designed to stress vulpix in some ways (compiler crash, infinite loop at runtime, excessive memory usage, ...) and exercise common features (proper diff between check and run files, correct number of errors reported, ...). Then vulpix should be run on this directory as it is currently being run on tests/, and its output should be compared against some expected output.
  • Optional: a test mode where the list of files passed to the compiler is sorted in a random order (controlled by a seed). We sometimes catch bugs just because the legacy tests happen to pass the files to the compiler in a different order than vulpix. We should have a way to intentionally try different orders. While it's impractical to try every possible order all the time, we could do nightly tests using a few different random orders each time.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions