Skip to content

Support ingredients, affecting order of tests #392

@Bodigrim

Description

@Bodigrim

Currently there is no simple way to write a tasty ingredient (ingredient transformer, to be precise), which affects order of test execution. Two common applications are:

  • Randomize order of tests to detect implicit dependencies between them.
  • Execute the longest running tests first, maximizing multithreaded CPU utilization.

I do not propose to implement such mechanisms in tasty itself, we should keep the core minimalistic. The missing part, which inhibits third parties from providing such functionality, is to provide a way to configure the order of [TestAction] in launchTestTree:

launchTestTree opts tree k0 = do
(testActions, fins) <- createTestActions opts tree
let NumThreads numTheads = lookupOption opts
(t,k1) <- timed $ do
abortTests <- runInParallel numTheads (testAction <$> testActions)
(do let smap = IntMap.fromDistinctAscList $ zip [0..] (testStatus <$> testActions)

One could extend launchTestTree with an argument (OptionSet -> [TestAction] -> [TestAction]), but that would require us to expose data TestAction, which I deem undesirable:

data TestAction act = TestAction
{ testAction :: act
-- ^ Some action, typically 'UnresolvedAction', 'ResolvedAction', or 'Action'.
, testPath :: Path
-- ^ Path pointing to this action (a series of group names + a test name)
, testDeps :: Seq Dependency
-- ^ Dependencies introduced by AWK-like patterns
, testStatus :: TVar Status
-- ^ Status var that can be used to monitor test progress
}

Instead I suggest adding an argument of type (OptionSet -> (a -> Path) -> [a] -> [a]: "if you tell me how to extract Path from a, I'll be able to reorder [a]".

How does it sound?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions