Mocking support idea

[spion]

Hello 👋

I really like this project, especially because it separates the test execution engine completely from the test code. Incidentally, it seems like this separation would also make it possible to implement mocking support in nutest!

A little aside on why: I've found that as complex scripts grow, it gets more difficult to rely on using aliases to redefine external commands only; additionally, most external commands return unstructured data and are therefore significantly more inconvenient to mock directly.

I started sketching out a proof of concept here: https://github.com/spion/numock

The main idea

• Test files that declare the need for mocking (e.g. through exporting a constant specifying which modules under test will need mocks) will be copied to a temporary directory, together with the specified modules
• The specified module-under-tests' non-mocked dependencies are also copied (PoC uses NU_LIB_DIRS but i realized later this will be error-prone, so the other dependencies need to be copied too)
• The mocked dependencies are not copied - instead, they're replaced with proxies that call out to the original modules if there is nothing in the "mocking registry", but call a closure in the mocking registry if its present instead.
• A "numock" module provides API to reset the mocks or redefine them

What this allows is to swap the implementation at run-time and test a variety of situations for commands that do complex orchestration of multiple side-effects!

Potential improvements

• Dependencies can be autodetected (parsing use instrcutions?), and all dependencies could be proxied back to the original directory.
• Individual functions within a module can also be mocked by replacing the implementation in its source (view source is generally accurate after the function signature block)
• The current approach needs to recreate the entire tree containing the common root of the module under test, mocked dependencies and the test itself. This might be a problem, and rewriting use instructions to different paths might end up being easier instead. (Note that the current PoC implementation is simplistic and doesn't do that - it simply assumes the test, module under test and mocks are in the same directory)

Implementation idea

In terms of impementing this for nutest, I don't think it would be too hard. Here is one potential idea I've been looking into:

• The discover phase will discover the type of tests in addition to their other attributes
• A new "preparation" phase will exist between discover and the orchestrator, which will prepare the tests that request mocks into a new temporary directory, and enrich the test description objects with a new attribute (e.g. execution_path),
• The orchestrator will prefer execution_path over path when executing the tests

Only tests that explicitly declare they need mocking would be processed by the processor - the rest would work normally.

---

What do you think? Also, do you know if there a simpler way to achieve mocking in nushell that I've missed?

[vyadh]

An intriguing idea. I'll admit I've not put much thought into how to do mocking. So far I've got away with aliasing/wrapping for externals or using WireMock for stuff that is just HTTP. I can't think of a better way than you propose. Your method sounds like a bit like dependency injection, but for mocks!

I guess ideally this would be simpler to achieve. You mention aliasing so a question on that. Are you saying that because of the unstructured data, you'd actually want to wrap an external command with a better interface via a Nushell internal command, and then mock that?

[spion]

More or less. One example is terraform: often executed as a multi-stage command making it hard to mock with aliases; produces very complex output (binary plans or large JSON description of plans), and if mocked with wiremock would require a huge amount of effort mocking every single API for every single provider to talk to.

I think def --wrapped might have enough power to achieve some of the things to deal with external commands, but the fact remains that once you build multiple modules with dependencies against each other, its convenient to not need to mock the whole orchestration (of multiple external command executions) end-to-end and only mock the direct dependencies. For example with terraform, sometimes you don't even need the whole plan: all you might need is a boolean result whether there are any changes or not.

I do wish there was an easier way to do this in nushell