Currently there is no simple way to write a tasty ingredient (ingredient transformer, to be precise), which affects order of test execution. Two common applications are:
- Randomize order of tests to detect implicit dependencies between them.
- Execute the longest running tests first, maximizing multithreaded CPU utilization.
I do not propose to implement such mechanisms in tasty itself, we should keep the core minimalistic. The missing part, which inhibits third parties from providing such functionality, is to provide a way to configure the order of [TestAction] in launchTestTree:
|
launchTestTree opts tree k0 = do |
|
(testActions, fins) <- createTestActions opts tree |
|
let NumThreads numTheads = lookupOption opts |
|
(t,k1) <- timed $ do |
|
abortTests <- runInParallel numTheads (testAction <$> testActions) |
|
(do let smap = IntMap.fromDistinctAscList $ zip [0..] (testStatus <$> testActions) |
One could extend launchTestTree with an argument (OptionSet -> [TestAction] -> [TestAction]), but that would require us to expose data TestAction, which I deem undesirable:
|
data TestAction act = TestAction |
|
{ testAction :: act |
|
-- ^ Some action, typically 'UnresolvedAction', 'ResolvedAction', or 'Action'. |
|
, testPath :: Path |
|
-- ^ Path pointing to this action (a series of group names + a test name) |
|
, testDeps :: Seq Dependency |
|
-- ^ Dependencies introduced by AWK-like patterns |
|
, testStatus :: TVar Status |
|
-- ^ Status var that can be used to monitor test progress |
|
} |
Instead I suggest adding an argument of type (OptionSet -> (a -> Path) -> [a] -> [a]: "if you tell me how to extract Path from a, I'll be able to reorder [a]".
How does it sound?
Currently there is no simple way to write a
tastyingredient (ingredient transformer, to be precise), which affects order of test execution. Two common applications are:I do not propose to implement such mechanisms in
tastyitself, we should keep the core minimalistic. The missing part, which inhibits third parties from providing such functionality, is to provide a way to configure the order of[TestAction]inlaunchTestTree:tasty/core/Test/Tasty/Run.hs
Lines 605 to 610 in f73ce42
One could extend
launchTestTreewith an argument(OptionSet -> [TestAction] -> [TestAction]), but that would require us to exposedata TestAction, which I deem undesirable:tasty/core/Test/Tasty/Run.hs
Lines 304 to 313 in f73ce42
Instead I suggest adding an argument of type
(OptionSet -> (a -> Path) -> [a] -> [a]: "if you tell me how to extractPathfroma, I'll be able to reorder[a]".How does it sound?