Integration test framework (#3121)

* begin implementing ui.get to query elements * type hinting, getting all elements and __repr__ * set style, classes, props with ui.get() * introduce .within filter * more stable test * element.keys(..) + ui.get(key=...) * allow setting multiple keys for ui.get * allow filtering ui.get(..).within(key=..) * allow multiple keys in ui.get * testing ui.get with TextElement * allow filtering with `ui.get(text=...)` * ui.get(..).exclude(type=..., key=..., text=...) * introduce ui.get().not_within(...) * add documentation page and use_local_scope param * fixed default * ui.get within elements * Enhance test example to show how to test with routing. Also rename pytest example to integration_tests to not have name-collisions * first proof of concept for simulated screen * demo async test * fix click in simulated screen * simulated screen can now handle async pages * do module reloading through pytest marker * determine whether to use simulation screen or selenium screen * rename Screen to SeleniumScreen to better pair it with SimulatedScreen * docstrings * custom marker * first concept of detecting failures on event loop * checking exceptions in other thread * use same event loop for tests and app which simplifies things a lot * show content of simulated screen on failure * simulating sio handshake to test for connection * notes * provide pytest plugin and ensure SimulatedScreen does not load Selenium fixtures * remove pytest.ini * better description for pytest marker * ensure fixtures are run when using screen * ensure loop is closed * cleanup * fixed async warning by making selenium_screen fixture non-async * start testing the chat app example * test typing into chat app * fix test * cleanup * ignore test_ files when building link examples * python 3.8 compatibility * re-adding tests/conftest.py because nicegui itself does not benefit from the newly provided pytest plugin * for testing we need to manage multiple clients in the same asyncio task * use __init__.py and relative import to get right main module * element tests without screen need to explicitly reset globals * keep original stack behaviour if we do not simulate * rename ui.get to ui.find * rename element.tags to element.markers * refactored ui.find into ElementFilter which serves as foundation for new user.should_see(...) * better naming and fixed documentation * cleanup * increased is-displayed delay * do multiple checks for visibility * adapt tests to use SeleniumScreen * use configurable url * allow subclasses of screen fixtures * clear urls and reset on_air for each test * cleanup * registering signals does not work in pytests * user.should_see is now async * warn if its not a nicegui page * print markers * provide check if code is executed in pytest * support storage in simulated screen * dropped simulated screen concept to make one user visit multiple pages (for example via ui.navigate.to) * re-add typing capabilities * check for notifications * fix ui.navigate.to for simulation * renaming * fix excluding_text test * make conftest generally available (like the examples) * cleanup * fix usage of context * support for auto-index-pages * click checkbox toggles value in simulation * fix "Self" import * make fixtures available via pytest plugin * adapted new tests to new structure * ensure utf-8 * try latest pytest * use newly named screen class * update dependencies * restore original ui.navigate.to after we exit simulated user * fix logging and types * fix type hint for python 3.8 * fix users fixture in combination with ui.navigate.to * try with longer timeout * only prepare auto index client for simulation if the user fixture is chosen * analyze core.loop issue with failing tests * Revert "analyze core.loop issue with failing tests" This reverts commit 725965c. * only assert loop is closed if it has been created because some tests may exit without calling screen.open() * better naming * renaming function and files * renaming * cleanup * add user.should_not_see and user.trigger methods * add integration tests for todo list example * icon prop is content * fix typing * return self * add integration tests for authentication example this uncovered the need to get new context in every retry to handle page changes * allow clicking on links in user simulation * always stop server * add link to example * remove accidentally commited env file * renaming * improve check if we are running as pytest * add typehint * remove obsolete try/except * remove obsolete code (may have been here due to bad merge conflict resolving) * cleanup * reload run module between tests to fix it on GitHub Action * introduction of UserFocus to trigger events * using non-async user.focus to simplify API * add user.focus(...).click() * updated tests to use new user.focus for interaction * formatting * reset to main version * adapted auth tests * make Self available for lower python versions * provide overloaded methods to simplify API * renaming and more simplifications * better imports * fix import Self * renaming * code review * add back requirements.txt * code review * rework str output shown on a failing test * fix some typing issues * remove obsolete test * simplify test_multiple_pages * update element filter tests to use User instead of Screen * fix tests by resetting default attributes for elements * Use ui.notification for ui.notify to simplify user simulation tests (#3369) * use ui.notification for ui.notify to simplify user simulation tests * code review --------- Co-authored-by: Falko Schindler <falko@zauberzeug.com> * Use `Set` in user._gather_elements and improve typing (#3370) * use set in user._gather_elements and improve typing * code review --------- Co-authored-by: Falko Schindler <falko@zauberzeug.com> * code review for Element.__str__ * simplify mark() * Fix typing for ElementFilter and User (#3381) * improve typing and naming * fix typing * reverted "kind" and type T of ElementFilter * fix element filter type inference * fix typing in __next__ * ignore typing where it can not be matched * open page * code review --------- Co-authored-by: Falko Schindler <falko@zauberzeug.com> * Implement full `ui.navigate` functionality for user simulation (#3382) * make ui.navigate an object of a replaceable class to simplify and improve user simulation * provide ui.navigate.reload for user simulation * code review --------- Co-authored-by: Falko Schindler <falko@zauberzeug.com> * Make combining multiple content filters more robust (#3383) * make ui.navigate an object of a replaceable class to simplify and improve user simulation * provide ui.navigate.reload for user simulation * make combining multiple content filters more robust --------- Co-authored-by: Falko Schindler <falko@zauberzeug.com> * fix problem with `ui.navigate` on documentation page * move element filter documentation to styling and appearance * fix ui.open * pylint * try removing explicit response.encoding * add user interaction with ui.switch, ui.editor and ui.codemirror * Improve iteration of elements (#3385) * make ui.navigate an object of a replaceable class to simplify and improve user simulation * provide ui.navigate.reload for user simulation * make combining multiple content filters more robust * introduce element.ancestors and element.descendants * rewrite __iter__ (tests still failing) * fix typo * improve filter * fix another condition * fix deletion of elements * replace tee with list for efficiency * add some more tests and fix conditions * remove len and getitem --------- Co-authored-by: Rodja Trappe <rodja@zauberzeug.com> * improve type annotation * extend docstring * fix rule for ancestor kinds * refactor element filter tests * fix link to example folder * more consistent rules Co-authored-by: Falko Schindler <falko@zauberzeug.com> * update filter rules and re-organize element filter tests * add test which combines multiple "within" statements * demonstrate validity of having multiple elements for a singe within clause * pytest configuration & fixtures (#3411) * remove pytest-prefix from plugin specification * fix requirements and settings for pytests example * rename screen and user with prefix nicegui_ * add conftest with renaming to pytests example * add internal renaming of nicegui_screen to screen and nicegui_user to user * remove unused pylint ignore * fix rename in examples * remove entry point * remove renaming * remove renaming in examples * remove renaming fixtures and add example code * double quotes * reworked as described in docs (#3413) * add pytest plugin loading to examples * run the pytest examples as part of the test_startup.sh to avoid conflicts in setup * renamed the pytest plugin from fixtures.py to plugin.py * fix todo tests * rename folder --------- Co-authored-by: Rodja Trappe <rodja@zauberzeug.com> Co-authored-by: Falko Schindler <falko@zauberzeug.com> * remove redundant test * add multi-user navigation test * Add documentation of pytest fixtures and pytest configuration (#3413) * begin with documentation of fixtures and pytest setup * code review * also prepare simulation for screen fixture * replace hacky docs.pytest and Demo.raw with generic docs.part * better split of testing topics in the docs * improve user docs * add UserInteraction reference * review * simplify doc.part * fix storage test * web driver info * review --------- Co-authored-by: Falko Schindler <falko@zauberzeug.com> * add conftest.py for backward compatibility * remove activate() and deactivate() --------- Co-authored-by: Falko Schindler <falko@zauberzeug.com> Co-authored-by: Paula Kammler <paula@zauberzeug.com>
zauberzeug · Aug 3, 2024 · b77d9d9 · b77d9d9
1 parent c341f88
commit b77d9d9
Show file tree

Hide file tree

Showing 67 changed files with 2,248 additions and 277 deletions.
diff --git a/examples/authentication/__init__.py b/examples/authentication/__init__.py
diff --git a/examples/authentication/main.py b/examples/authentication/main.py
@@ -38,10 +38,13 @@ async def dispatch(self, request: Request, call_next):
 
 @ui.page('/')
 def main_page() -> None:
+    def logout() -> None:
+        app.storage.user.clear()
+        ui.navigate.to('/login')
+
     with ui.column().classes('absolute-center items-center'):
         ui.label(f'Hello {app.storage.user["username"]}!').classes('text-2xl')
-        ui.button(on_click=lambda: (app.storage.user.clear(), ui.navigate.to('/login')), icon='logout') \
-            .props('outline round')
+        ui.button(on_click=logout, icon='logout').props('outline round')
 
 
 @ui.page('/subpage')
@@ -67,4 +70,5 @@ def try_login() -> None:  # local function to avoid passing username and passwor
     return None
 
 
-ui.run(storage_secret='THIS_NEEDS_TO_BE_CHANGED')
+if __name__ in {'__main__', '__mp_main__'}:
+    ui.run(storage_secret='THIS_NEEDS_TO_BE_CHANGED')
diff --git a/examples/authentication/test_authentication.py b/examples/authentication/test_authentication.py
@@ -0,0 +1,37 @@
+import pytest
+
+from nicegui.testing import User
+
+from . import main
+
+# pylint: disable=missing-function-docstring
+
+pytest_plugins = ['nicegui.testing.plugin']
+
+
+@pytest.mark.module_under_test(main)
+async def test_login_logoff(user: User) -> None:
+    await user.open('/')
+    user.find('Username').type('user1')
+    user.find('Password').type('pass1')
+    user.find('Log in').click()
+    await user.should_see('Hello user1!')
+    user.find('logout').click()
+    await user.should_see('Log in')
+
+
+@pytest.mark.module_under_test(main)
+async def test_wrong_password(user: User) -> None:
+    await user.open('/')
+    user.find('Username').type('user1')
+    user.find('Password').type('wrong').trigger('keydown.enter')
+    await user.should_see('Wrong username or password')
+
+
+@pytest.mark.module_under_test(main)
+async def test_subpage_access(user: User) -> None:
+    await user.open('/subpage')
+    await user.should_see('Log in')
+    user.find('Username').type('user1')
+    user.find('Password').type('pass1').trigger('keydown.enter')
+    await user.should_see('This is a sub page.')
diff --git a/examples/chat_app/__init__.py b/examples/chat_app/__init__.py
diff --git a/examples/chat_app/main.py b/examples/chat_app/main.py
@@ -10,8 +10,11 @@
 
 @ui.refreshable
 def chat_messages(own_id: str) -> None:
-    for user_id, avatar, text, stamp in messages:
-        ui.chat_message(text=text, stamp=stamp, avatar=avatar, sent=own_id == user_id)
+    if messages:
+        for user_id, avatar, text, stamp in messages:
+            ui.chat_message(text=text, stamp=stamp, avatar=avatar, sent=own_id == user_id)
+    else:
+        ui.label('No messages yet').classes('mx-auto my-36')
     ui.run_javascript('window.scrollTo(0, document.body.scrollHeight)')
 
 
@@ -40,4 +43,6 @@ def send() -> None:
     with ui.column().classes('w-full max-w-2xl mx-auto items-stretch'):
         chat_messages(user_id)
 
-ui.run()
+
+if __name__ in {'__main__', '__mp_main__'}:
+    ui.run()
diff --git a/examples/chat_app/test_chat_app.py b/examples/chat_app/test_chat_app.py
@@ -0,0 +1,40 @@
+from typing import Callable
+
+import pytest
+
+from nicegui import ui
+from nicegui.testing import User
+
+from . import main
+
+pytest_plugins = ['nicegui.testing.plugin']
+
+
+@pytest.mark.module_under_test(main)
+async def test_basic_startup_appearance(user: User) -> None:
+    """Test basic appearance of the chat app."""
+    await user.open('/')
+    await user.should_see('simple chat app')
+    await user.should_see('https://robohash.org/')
+    await user.should_see('message')
+    await user.should_see('No messages yet')
+
+
+@pytest.mark.module_under_test(main)
+async def test_sending_messages(create_user: Callable[[], User]) -> None:
+    """Test sending messages from two different screens."""
+    userA = create_user()
+    userB = create_user()
+
+    await userA.open('/')
+    userA.find(ui.input).type('Hello from screen A!').trigger('keydown.enter')
+    await userA.should_see('Hello from screen A!')
+    await userA.should_see('message')
+
+    await userB.open('/')
+    await userB.should_see('Hello from screen A!')
+    userB.find(ui.input).type('Hello from screen B!').trigger('keydown.enter')
+    await userB.should_see('message')
+
+    await userA.should_see('Hello from screen A!')
+    await userA.should_see('Hello from screen B!')
diff --git a/examples/pytest/conftest.py b/examples/pytest/conftest.py
diff --git a/examples/pytest/main.py b/examples/pytest/main.py
diff --git a/examples/pytest/requirements.txt b/examples/pytest/requirements.txt
diff --git a/examples/pytest/test_main_page.py b/examples/pytest/test_main_page.py
diff --git a/examples/pytest/.gitignore → examples/pytests/.gitignore b/examples/pytest/.gitignore → examples/pytests/.gitignore
diff --git a/examples/pytests/app/__init__.py b/examples/pytests/app/__init__.py
diff --git a/examples/pytests/app/startup.py b/examples/pytests/app/startup.py
@@ -0,0 +1,21 @@
+from nicegui import Client, ui
+
+# pylint: disable=missing-function-docstring
+
+
+def startup() -> None:
+    @ui.page('/')
+    def main_page() -> None:
+        ui.markdown('Try running `pytest` on this project!')
+        ui.button('Click me', on_click=lambda: ui.notify('Button clicked!'))
+        ui.link('go to subpage', '/subpage')
+
+    @ui.page('/subpage')
+    def sub_page() -> None:
+        ui.markdown('This is a subpage')
+
+    @ui.page('/with_connected')
+    async def with_connected(client: Client) -> None:
+        ui.markdown('This is an async connection demo')
+        await client.connected()
+        ui.markdown('Connected!')
diff --git a/examples/pytests/main.py b/examples/pytests/main.py
@@ -0,0 +1,8 @@
+#!/usr/bin/env python3
+from app.startup import startup
+
+from nicegui import app, ui
+
+app.on_startup(startup)
+
+ui.run()
diff --git a/examples/pytests/pytest.ini b/examples/pytests/pytest.ini
@@ -0,0 +1,2 @@
+[pytest]
+asyncio_mode = auto
diff --git a/examples/pytests/requirements.txt b/examples/pytests/requirements.txt
@@ -0,0 +1,4 @@
+nicegui
+icecream
+pytest-asyncio
+pytest-selenium
diff --git a/examples/pytests/tests/__init__.py b/examples/pytests/tests/__init__.py
diff --git a/examples/pytests/tests/conftest.py b/examples/pytests/tests/conftest.py
@@ -0,0 +1,20 @@
+from typing import Generator
+
+import pytest
+from app.startup import startup
+
+from nicegui.testing import Screen, User
+
+pytest_plugins = ['nicegui.testing.plugin']
+
+
+@pytest.fixture
+def user(user: User) -> Generator[User, None, None]:
+    startup()
+    yield user
+
+
+@pytest.fixture
+def screen(screen: Screen) -> Generator[Screen, None, None]:
+    startup()
+    yield screen
diff --git a/examples/pytests/tests/test_with_screen.py b/examples/pytests/tests/test_with_screen.py
@@ -0,0 +1,31 @@
+from nicegui.testing import Screen
+
+# pylint: disable=missing-function-docstring
+
+
+def test_markdown_message(screen: Screen) -> None:
+    screen.open('/')
+    screen.should_contain('Try running')
+
+
+def test_button_click(screen: Screen) -> None:
+    screen.open('/')
+    screen.click('Click me')
+    screen.should_contain('Button clicked!')
+
+
+def test_sub_page(screen: Screen) -> None:
+    screen.open('/subpage')
+    screen.should_contain('This is a subpage')
+
+
+def test_with_connected(screen: Screen) -> None:
+    screen.open('/with_connected')
+    screen.should_contain('This is an async connection demo')
+    screen.should_contain('Connected!')
+
+
+def test_navigation(screen: Screen) -> None:
+    screen.open('/')
+    screen.click('go to subpage')
+    screen.should_contain('This is a subpage')
diff --git a/examples/pytests/tests/test_with_user.py b/examples/pytests/tests/test_with_user.py
@@ -0,0 +1,31 @@
+from nicegui.testing import User
+
+# pylint: disable=missing-function-docstring
+
+
+async def test_markdown_message(user: User) -> None:
+    await user.open('/')
+    await user.should_see('Try running')
+
+
+async def test_button_click(user: User) -> None:
+    await user.open('/')
+    user.find('Click me').click()
+    await user.should_see('Button clicked!')
+
+
+async def test_sub_page(user: User) -> None:
+    await user.open('/subpage')
+    await user.should_see('This is a subpage')
+
+
+async def test_with_connected(user: User) -> None:
+    await user.open('/with_connected')
+    await user.should_see('This is an async connection demo')
+    await user.should_see('Connected!')
+
+
+async def test_navigation(user: User) -> None:
+    await user.open('/')
+    user.find('go to subpage').click()
+    await user.should_see('This is a subpage')
diff --git a/examples/todo_list/__init__.py b/examples/todo_list/__init__.py
diff --git a/examples/todo_list/main.py b/examples/todo_list/main.py
@@ -37,7 +37,8 @@ def todo_ui():
         ui.label(f'Remaining: {sum(not item.done for item in todos.items)}')
     for item in todos.items:
         with ui.row().classes('items-center'):
-            ui.checkbox(value=item.done, on_change=todo_ui.refresh).bind_value(item, 'done')
+            ui.checkbox(value=item.done, on_change=todo_ui.refresh).bind_value(item, 'done') \
+                .mark(f'checkbox-{item.name.lower().replace(" ", "-")}')
             ui.input(value=item.name).classes('flex-grow').bind_value(item, 'name')
             ui.button(on_click=lambda item=item: todos.remove(item), icon='delete').props('flat fab-mini color=grey')
 
@@ -51,7 +52,9 @@ def todo_ui():
 with ui.card().classes('w-80 items-stretch'):
     ui.label().bind_text_from(todos, 'title').classes('text-semibold text-2xl')
     todo_ui()
-    add_input = ui.input('New item').classes('mx-12')
-    add_input.on('keydown.enter', lambda: (todos.add(add_input.value), add_input.set_value('')))
+    add_input = ui.input('New item').classes('mx-12').mark('new-item')
+    add_input.on('keydown.enter', lambda: todos.add(add_input.value))
+    add_input.on('keydown.enter', lambda: add_input.set_value(''))
 
-ui.run()
+if __name__ in {'__main__', '__mp_main__'}:
+    ui.run()
diff --git a/examples/todo_list/test_todo_list.py b/examples/todo_list/test_todo_list.py
@@ -0,0 +1,38 @@
+import pytest
+
+from nicegui.testing import User
+
+from . import main
+
+# pylint: disable=missing-function-docstring
+
+pytest_plugins = ['nicegui.testing.plugin']
+
+
+@pytest.mark.module_under_test(main)
+async def test_checking_items(user: User) -> None:
+    await user.open('/')
+    await user.should_see('Completed: 1')
+    await user.should_see('Remaining: 3')
+    user.find('checkbox-new-nicegui-release').click()
+    await user.should_see('Completed: 2')
+    await user.should_see('Remaining: 2')
+    user.find('checkbox-call-mom').click()
+    await user.should_see('Completed: 3')
+    await user.should_see('Remaining: 1')
+    user.find('checkbox-order-pizza').click()
+    await user.should_see('Completed: 2')
+    await user.should_see('Remaining: 2')
+
+
+@pytest.mark.module_under_test(main)
+async def test_adding_items(user: User) -> None:
+    await user.open('/')
+    user.find('new-item') \
+        .type('Buy milk').trigger('keydown.enter') \
+        .type('Buy eggs').trigger('keydown.enter')
+    await user.should_see('Buy milk')
+    await user.should_see('Buy eggs')
+    user.find('checkbox-buy-milk').click()
+    await user.should_see('Completed: 2')
+    await user.should_see('Remaining: 4')
diff --git a/nicegui/__init__.py b/nicegui/__init__.py
@@ -3,6 +3,7 @@
 from .app.app import App
 from .client import Client
 from .context import context
+from .element_filter import ElementFilter
 from .nicegui import app
 from .tailwind import Tailwind
 from .version import __version__
@@ -13,6 +14,7 @@
     'App',
     'Client',
     'context',
+    'ElementFilter',
     'elements',
     'run',
     'Tailwind',