Integration test framework

[rodja]

This pull request introduces a test framework which allows to write high-level integration tests which run as fast as unit tests.

---

Note: Partly based on PR #1931.

---

Open tasks:

• __str__: We need to clarify and test the general layout of the generated string.
• user.should_not_see: We need to handle ui.notify?
• user.gather_elements: Do we really need nested list initializers? And should we work with sets to avoid duplicates?
• Documentation for ElementFilter: Maybe "Styling & Appearance" is a better place for it?
• pycheck nicegui isn't happy yet (mainly unused fixure arguments, T vs. Element, missing _iterator, missing docstrings)
• Why do we need response.encoding = 'utf-8'? --> We don't.
• What is the purpose of test_run.py?
• Can we run (most of) test_element_filter.py without Screen?
• test_multiple_pages: What do we test by calling activate()? It doesn't seem to make any difference.
• test_navigation: The "back" and "forward" buttons are currently unused. But the following lines breaks the test.

   user.find('back').click()
   await user.should_see('Main page')
   user.find('forward').click()
   await user.should_see('Other page')

• user interaction: We don't seem to support typing in ui.number, ui.editor, ui.codemirror etc.
• user interaction: We seem to replace text rather appending it.
• I haven't reviewed fixtures.py, since I could use some guidance with the overall idea.
• I haven't reviewed element_filter.py in detail, since there are still so many type and linter errors.
• I'm sceptical about the way we handle different element_contents. It allows a test like this to succeed:

  async def test_xxx(user: User) -> None:
      ui.input(placeholder='x', value='y')
  
      await user.open('/')
      await user.should_see('x y')

• ElementFilter.within_kinds has no test and does not seem to work
• The first parameter of ElementFilter.exclude may be an element and not kind...
• Add filter rules to docstring of ElementFilter.
• Fix the rule for ancestor kinds.
• Simplify ancestor marker rules?
• Ensure pytest plugin is working.
• Is it safe to provide "user" and "screen" as fixture names in a pytest plugin? -> Yes, without defining an entry point.
• Remove pytest entry point from pyproject.toml.
• Add conftest.py for backward compatibility.
• Documentation (new section about screen, user, create_user, link to examples) -> PR Add documentation of pytest fixtures and pytest configuration #3413
• Remove user.activate and user.deactivate?

[rodja]

Is it safe to provide "user" and "screen" as fixture names in a pytest plugin?

True. The magic fixture loading from plugins occasionally leads to unpredictable name clashes. See pytest-dev/pytest#3834 and pytest-dev/pytest#3966 for a lengthly discussions. The example of test_client provided by pytest-sanic and pytest-aiohttp is often provided. As yunstanford/pytest-sanic#22, shows sanic changed their fixture (but still provide the old one as depreciated).

Still, I would really like to keep the simple fixture names screen and user because they are the used so much. For example:

@pytest.mark.module_under_test(main)
async def test_basic_startup_appearance(user: User) -> None:
    """Test basic appearance of the chat app."""
    await user.open('/')
    await user.should_see('simple chat app')
    await user.should_see('message')
    ...

I have not found any other plugins which use that name. But they plugin space is vast.
An alternative would be the explicit import of fixtures as we do it in 1.4.
But the pytest documentation explicitly mentions that this should not be done:

Sometimes users will import fixtures from other projects for use, however this is not recommended: importing fixtures into a module will register them in pytest as defined in that module. This has minor consequences, such as appearing multiple times in pytest --help, but it is not recommended because this behavior might change/stop working in future versions.

https://docs.pytest.org/en/6.2.x/fixture.html#use-fixtures-in-classes-and-modules-with-usefixtures:~:text=Sometimes%20users%20will,in%20future%20versions.

I have experimented with explicit activation of our fixtures and have a working prototype where you need to pass --with-nicegui-fixtures to pytest to enable user and screen. But that also is not very nice because every developer needs to include the option every time the tests should be run.

@falkoschindler came up with an interesting pattern which could solve the dillema: We would provide the fixtures nicegui_screen and nicegui_user which can then be shorthanded by developers in their own conftest.py like this:

@pytest.mark.fixture
def screen(nicegui_screen):
    return nicegui_screen

@pytest.mark.fixture
def user(nicegui_user):
    return nicegui_user

But that would make using nicegui tests a bit more complicated. All in all, I'm in favour of just keeping at as it is right now in this pull request:

1. provide a pytest plugin via entrypoint which directly offers the fixtures
2. use screen and user as fixture names
3. hope that virtual envs shield the users from the worst case
4. provide guidance on how to disable plugins which might have name clashes (-p no:name command line option and other techniques)
5. wait for pytest to come up with a better way to load/define fixtures in plugins

[falkoschindler]

@rodja I have an alternative suggestion based on @codingpaula's experiments from yesterday: She struggled to get our pytest plugin (provided via poetry entrypoint) to work in a separate test project. Even though it works in the NiceGUI example, it doesn't seem to be correctly registered when used outside of the NiceGUI workspace.

But it does work when using the pytest_plugins variable in the user project:

pytest_plugins = ['nicegui.testing.fixtures']

Strangely, this works even without defining the plugin via poetry in NiceGUI. We couldn't figure out why. But currently we have doubts that the magic registration of pytest plugins via poetry entrypoints actually works. Yes, RoSys also provides such a plugin. But do we have proof it actually works? We couldn't find any project that actually uses fixtures from the RoSys pytest plugin.

Maybe we are completely on the wrong track, and it would certainly help to chat about it together on Monday. Nevertheless, the pytest_plugins variable seems to allow for a rather nice solution:

1. Don't provide any pytest plugin via poetry entrypoints.
2. Remain with the short and handy fixture names screen and user.
3. Let the users put pytest_plugins = ['nicegui.testing.fixtures'] in their conftest.py to explicitly use our fixtures.
4. Add a conftest.py next to fixtures.py with from .fixtures import * and a deprecation warning for backward compatibility.

The latter still allows writing from nicegui.testing.conftest import * in the user's conftest.py, so we don't break anything.

[falkoschindler]

Regarding the filter rules: I noticed that "within(instance=)" is a bit nonsensical when used with multiple elements. Since it now only matches when the element's ancestors include all given instances, you could as well specify the one closest to the element. All others don't contribute anything. Example:

root > A > B > C > label

within(instance=[A, C])
within(instance=C)  # equivalent

But in favor of a very symmetric rule set, I think we can leave it as it is.

[rodja]

About the pytest plugin

But do we have proof it actually works? We couldn't find any project that actually uses fixtures from the RoSys pytest plugin

The Field Friend is using the integration fixture from RoSys to setup it's own system fixture.

Strangely, [using pyplugin = ...] works even without defining the plugin via poetry in NiceGUI.

Oh wow. I did not get that in their documentation. And also ChatGPT 4o and Claude Sonnet did not tell me about it when I asked how I could "not autoload a plugin, but do it explicitly". That is great!

A slightly modified strategy of what you are suggesting would be:

1. Provide a pytest plugin via poetry entrypoints which have the nicegui_screen and nicegui_user fixtures.
2. Provide another plugin which offers the the short and handy fixture names screen and user to let users put pytest_plugins = ['nicegui.testing.shorthands'] in their conftest.py.
3. Add a conftest.py next to fixtures.py with from .fixtures import * and a deprecation warning for backward compatibility.

While it would make the documentation a bit more lengthly, small projects like our chat_app example would not need a conftest.py at all (when they are ok with the longer fixture names).

About the within filter rule

I noticed that "within(instance=)" is a bit nonsensical when used with multiple elements

I'm not so sure. Consider the scenario where there is another C below root:

root > A > B > C > label 1
     > C > label 2

within(instance=[A, C]) must only yield label 1 where within(instance=C) provides access to label 1 and label 2. I just added a test to ensure this is possible.

[falkoschindler]

About the pytest plugin

Ok, that makes three ways to use nicegui.testing (poetry entry point, pytest_plugins, old conftest.py). But with a good documentation this might be optimal. Let's discuss it tomorrow with @codingpaula. I think she already implemented it (partly) in PR #3411.

About the within filter rule

Sure, there can be multiple elements with marker or content "C". But one C can't be the same instance like another C.

[falkoschindler]

I think this PR is finally ready to merge.
In 161c51f I removed the need for activate() and deactivate() by replacing ui.navigate whenever an attribute of a user is accessed. Since the slot stack manipulation was obsolete anyway, we could get rid of quite some code.
@rodja Feel free to merge if you agree with the latest changes. 🙂