Add test mode to run and dev

[freakboy3742]

Take 3. See #949 and #961 for the previous 2 attempts.

Adds a --test option to briefcase run and briefcase dev. When in --test mode:

• the app entry point is changed from <app_name> to tests.<app_name>
• the list of application sources is extended to include test_sources
• the list of application dependencies is extended to include test_requires
• The log runner then filters the output stream of the running app, looking for a sentinel value indicating test success/failure.

This approach has the the following advantages:

• It is much faster on iOS. I don't know what Xcode is doing when it starts a test suite, but this approach is up to 2 minutes faster to start the test suite.
• It requires minimal platform-specific modifications. We don't need to work out how to drive native unit test frameworks on every platform; we can just use "normal" app execution
• It can be used on "app" targets, not just project targets.

The downside is that:

• We lose access to any platform-specific test automation tooling. e.g., Xcode has frameworks for taking screenshots etc during a test; we won't be able to use those (at least, not easily)
• We need to manually manage the process of starting an application in the background, if we want to programatically drive an application.

Implemented for:

• iOS (requires the Test framework v2 briefcase-iOS-Xcode-template#13 template updates)
• macOS Xcode (requires the Test framework v2 briefcase-macOS-Xcode-template#15 template updates)
• macOS app (requires the Test framework v2 briefcase-macOS-app-template#11 template updates)
• Linux AppImage (requires the Add support for the BRIEFCASE_MAIN_MODULE testing override. briefcase-linux-appimage-template#23 template updates)
• Linux FlatPak (requires the Add support for the BRIEFCASE_MAIN_MODULE testing endpoint. briefcase-linux-flatpak-template#8 template updates)
• Windows VisualStudio (requires the Add support for BRIEFCASE_MAIN_MODULE testing entry point. briefcase-windows-VisualStudio-template#5 template updates)
• Android (requires the Add dynamic entry point for Android code. briefcase-android-gradle-template#56 template updates)

If you specify a template_branch='testcase2' at the app level, these templates will be picked up.

The most common example of usage will be a pytest suite, where all the test code is contained in a tests folder. This would be configured as:

test_sources = ['tests']
test_requires = ['pytest']

the code copied in as part of the test_sources should contain an app_name.py that will start the test suite. This script should contain:

import os
from pathlib import Path

import pytest


if __name__ == "__main__":
    os.chdir(Path(__file__).parent.parent)
    result = pytest.main(["-vv", "--color=no", "test"])

Alternatively, a unittest suite with automatic discovery would look like:

import unittest
from pathlib import Path

if __name__ == "__main__":
    loader = unittest.TestLoader()
    suite = loader.discover(Path(__file__).parent.parent / "tests")
    runner = unittest.TextTestRunner()
    runner.run(suite)

And a unittest suite based on explicit code imports would look like:

import unittest
from pathlib import Path

from .test_module1 import *
from .test_module2 import *

if __name__ == "__main__":
    unittest.main()

Other notes:

• run --test will implicitly do a full update and build on every execution. This will resulting the test code and dependencies always being available when running as a test; however, it also means that if you do run directly after run --test, the test dependencies will be in the app. Mobile platforms will also end up in a situation where they run as the test app. So - it is advisable to do a run -u the first time after running run --test.

Fixes #973

PR Checklist:

• All new features have been tested
• All new features have been documented
• I have read the CONTRIBUTING.md file
• I will abide by the code of conduct

[mhsmith]

For Android, the easiest way I can think of to set the entry point is for Briefcase to create a resource file, e.g. app/src/main/res/values/briefcase.xml, containing a string resource, and for MainActivity to get this resource using getString.

The name of the file isn't significant: all files in the values directory are simply combined when building the app. But using a separate file would make it easier for Briefcase to manage it without conflicting with anything else.

[rmartin16]

I don't have any large concerns.

The necessary configuration is a bit opaque to me; will the default templates be updated to perhaps provide a usable example of implementing tests?

I don't know if this would have changed things but Rich was already recording all the captured output. The same Rich API we use to create the log file can be used to obtain a copy of past output.

I haven't been able to get my test suite to run on Android but I'm presuming that user error for now.

Finally, I'm particularly disappointed at all the missed opportunities to use the walrus operator in this PR ;)

[rmartin16]

To help draw the difference from macOS, I'd rename all these Popen objects from log_popen to app_popen. It gets confusing to me what the Popen objects really represent in the different codepaths.

[rmartin16]

This same change also probably makes sense for flatpak and windows.

[rmartin16]

What's the purpose of sending SIGINT here? Why shouldn't the process finish normally?

[freakboy3742]

Primarily because on iOS and Android, it won't ever finish. iOS and Android apps don't ever truly exit - it's assumed that the app will start, and will persist indefinitely; on macOS, the log stream will persist.

However, we might be able to use the stop_func in the Android/iOS case; we're already using the stop func on the macOS case. Let me dig into this a bit.

[freakboy3742]

I taken another look at this, and I think I've found an alternate approach. Instead of sending SIGINT, we can tell the streamer to stop. The process will then fall back on the normal subprocess.cleanup(). We'll still potentially be terminating the process, but the logic is all contained in the subprocess module.

In the process of working this out, I was also able to build a stop_func implementation for iOS, so we now have a safety mechanism there, too. In normal operation, that safety mechanism won't be triggered; but if someone manages to build an app that fully crashes, the log will terminate along with the app, which brings iOS in alignment with every other platform.

[rmartin16]

The user experience for briefcase dev --test for an app that's improperly configured is a little rough.

Do you think some guardrails here and exception masking with more obvious feedback to the user is appropriate?

❯ briefcase dev --test

[testmode] Running test suite in dev environment...
Traceback (most recent call last):
  File "/home/russell/.pyenv/versions/3.10.8-debug/lib/python3.10/runpy.py", line 129, in _get_module_details
    spec = importlib.util.find_spec(mod_name)
  File "/home/russell/.pyenv/versions/3.10.8-debug/lib/python3.10/importlib/util.py", line 94, in find_spec
    parent = __import__(parent_name, fromlist=['__path__'])
ModuleNotFoundError: No module named 'tests'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/russell/.pyenv/versions/3.10.8-debug/lib/python3.10/runpy.py", line 220, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/home/russell/.pyenv/versions/3.10.8-debug/lib/python3.10/runpy.py", line 138, in _get_module_details
    raise error(msg.format(mod_name, type(ex).__name__, ex)) from ex
ImportError: Error while finding module specification for 'tests.testmode' (ModuleNotFoundError: No module named 'tests')

[freakboy3742]

Hrm - I agree this isn't a great UX. However, there are some limits on what we can do here, because there's functionally no difference between user-space code raising an ImportError, and the main execution script raising an ImportError. We could catch the ImportError, and then interrogate to see if the underlying cause is the module that was used to start the app; however, that kind of interrogation makes me really nervous - being "smart" in this kind of way has burned me in the past.

However - I noticed that I'm inadvertently eating the error condition; The mainline essentially ignores BriefcaseTestSuiteError, assuming the method that raised it has done all the error handling, so when your test suite raises an error, that's not caught as a "couldn't start test suite" error. Catching that error at least gives us the chance to raise a slightly more meaningful "Unable to start test suite 'tests.myapp'" message, and capture the log; test suite success/failure is a "successful" run.

Does that look any better?

[freakboy3742]

I don't have any large concerns.

The necessary configuration is a bit opaque to me; will the default templates be updated to perhaps provide a usable example of implementing tests?

That's the idea. Adding a default test suite to the briefcase template is on my todo list for today.

I don't know if this would have changed things but Rich was already recording all the captured output. The same Rich API we use to create the log file can be used to obtain a copy of past output.

It would be - except for the iOS/Android case. Since those test apps never truly exit, we need to have a streaming log filter to identify the failure conditions. Plus, it gives us the opportunity to strip off some of the excessive logging noise added by the macOS log stream.

I haven't been able to get my test suite to run on Android but I'm presuming that user error for now.

Yeah - that needs some hand-holding for now, plus a manual modification to the gradle template. @mhsmith pushed some updates last night which provide a temporary workaround for the manual modifications. If you want a working example, beeware/Python-support-testbed#6 is the best point of reference.

Finally, I'm particularly disappointed at all the missed opportunities to use the walrus operator in this PR ;)

:-P

[mhsmith]

I think all of my comments have been addressed, except for this one about the --no-auto-update argument.

[freakboy3742]

@rmartin16 FYI - I've added beeware/briefcase-template#39; this will bootstrap a pytest test suite into a new project. The template contains an option for a unittest suite; for now, you need to manually roll out the template to get that option. We could add the extra question to this PR; but (a) I don't think it hurts to strongly encourage pytest, and (b) we really need the longer term fix in the form of a fix for #392.

[freakboy3742]

I've had a quick word with @rmartin16 on Discord, and he's given the go-ahead to merge this. We're certain to have tweaks as we fine tune the testing behavior as we gain experience, but it will be easier to handle those as PRs on this base.

If nothing else: when beeware/briefcase-template#39 lands, we'll be able to extend CI to run the minimal test suite, which will give us the first proof that Briefcase is always producing an executable that can execute.