Skip to content

Hotfix: Added celery bug report snippet to CI #260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/workflows/examples.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,39 @@ permissions:
contents: read # to fetch code (actions/checkout)

jobs:
celery_bug_report:
runs-on: ${{ matrix.os }}

strategy:
fail-fast: false
matrix:
python-version: ["3.12"]
os: ["ubuntu-latest"]

steps:
- name: Install apt packages
if: startsWith(matrix.os, 'ubuntu-')
run: |
sudo apt update
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: '**/setup.py'
- name: Install dependencies
working-directory: examples
run: |
python -m pip install --upgrade pip pytest-cov
pip install -U "pytest-celery[all]@git+https://github.com/celery/pytest-celery.git"

- name: Run tests
working-directory: examples
timeout-minutes: 5
run: |
pytest -xsv celery_bug_report.py --no-cov

myworker:
runs-on: ${{ matrix.os }}

Expand Down
122 changes: 11 additions & 111 deletions docs/userguide/celery-bug-report.rst
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,12 @@ Inline handlers can be used like this:
Injecting signal handlers is using a similar pattern to adding tasks and can be done according
to the :ref:`signal-handlers-modules-injection` section.

Tasks
~~~~~

The :ref:`default-tasks` can be used out-of-the-box, but you can also add new tasks to the worker by creating a new module
and injecting it into the environment. See :ref:`injecting-tasks` for more information.

Templates
=========

Expand All @@ -283,6 +289,8 @@ Standalone Test Snippet
The following snippet can be used as a starting point for a bug report script. To use it, just copy and paste it into a
new file and run it with pytest.

The snippet is also part of the `CI system <https://github.com/celery/pytest-celery/actions/workflows/examples.yml>`_.

RabbitMQ Management Broker
--------------------------

Expand All @@ -298,117 +306,9 @@ Built-in Worker

We'll use the :ref:`built-in-worker` to use a specific Celery release.

.. code-block:: python

# flake8: noqa

from __future__ import annotations

import pytest
from celery import Celery
from celery.canvas import Signature
from celery.result import AsyncResult

from pytest_celery import RABBITMQ_PORTS
from pytest_celery import CeleryBackendCluster
from pytest_celery import CeleryBrokerCluster
from pytest_celery import CeleryTestSetup
from pytest_celery import RabbitMQContainer
from pytest_celery import RabbitMQTestBroker
from pytest_celery import RedisTestBackend
from pytest_celery import ping

###############################################################################
# RabbitMQ Management Broker
###############################################################################


class RabbitMQManagementTestBroker(RabbitMQTestBroker):
def get_management_url(self) -> str:
"""Opening this link during debugging allows you to see the
RabbitMQ management UI in your browser.
"""
ports = self.container.attrs["NetworkSettings"]["Ports"]
ip = ports["15672/tcp"][0]["HostIp"]
port = ports["15672/tcp"][0]["HostPort"]
return f"http://{ip}:{port}"


@pytest.fixture
def default_rabbitmq_broker_image() -> str:
return "rabbitmq:management"


@pytest.fixture
def default_rabbitmq_broker_ports() -> dict:
# Expose the management UI port
ports = RABBITMQ_PORTS.copy()
ports.update({"15672/tcp": None})
return ports


@pytest.fixture
def celery_rabbitmq_broker(default_rabbitmq_broker: RabbitMQContainer) -> RabbitMQTestBroker:
broker = RabbitMQManagementTestBroker(default_rabbitmq_broker)
yield broker
broker.teardown()


@pytest.fixture
def celery_broker_cluster(celery_rabbitmq_broker: RabbitMQTestBroker) -> CeleryBrokerCluster:
cluster = CeleryBrokerCluster(celery_rabbitmq_broker)
yield cluster
cluster.teardown()


###############################################################################
# Redis Result Backend
###############################################################################


@pytest.fixture
def celery_backend_cluster(celery_redis_backend: RedisTestBackend) -> CeleryBackendCluster:
cluster = CeleryBackendCluster(celery_redis_backend)
yield cluster
cluster.teardown()


@pytest.fixture
def default_redis_backend_image() -> str:
return "redis:latest"


###############################################################################
# Worker Configuration
###############################################################################


@pytest.fixture(scope="session")
def default_worker_celery_log_level() -> str:
return "INFO"


@pytest.fixture(scope="session")
def default_worker_celery_version() -> str:
return "5.2.7"


@pytest.fixture
def default_worker_app(default_worker_app: Celery) -> Celery:
app = default_worker_app
# app.conf... # Add any additional configuration here
return app


###############################################################################
# Bug Reproduction
###############################################################################


def test_issue_1234(celery_setup: CeleryTestSetup):
sig: Signature = ping.s()
res: AsyncResult = sig.delay()
assert res.get() == "pong"
.. literalinclude:: ../../examples/celery_bug_report.py
:language: python
:caption: examples.celery_bug_report.py

Execute with Pytest
###################
Expand Down
14 changes: 13 additions & 1 deletion docs/userguide/default-tasks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,21 @@
The plugin provides a list of built-in celery tasks that can be used out of the box. This page will
list all the available tasks.

To import the tasks, you can use the following code:

.. code-block:: python

from pytest_celery import the, tasks, you, want

or

.. code-block:: python

from pytest_celery.vendors.worker import tasks

.. tip::

The tasks injected into the workers that use the default volume with:
The tasks are injected into the workers that use the default volume with:

.. code-block:: python

Expand Down
106 changes: 106 additions & 0 deletions examples/celery_bug_report.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
from __future__ import annotations

import pytest
from celery import Celery
from celery.canvas import Signature
from celery.result import AsyncResult

from pytest_celery import RABBITMQ_PORTS
from pytest_celery import CeleryBackendCluster
from pytest_celery import CeleryBrokerCluster
from pytest_celery import CeleryTestSetup
from pytest_celery import RabbitMQContainer
from pytest_celery import RabbitMQTestBroker
from pytest_celery import RedisTestBackend
from pytest_celery import ping

###############################################################################
# RabbitMQ Management Broker
###############################################################################


class RabbitMQManagementTestBroker(RabbitMQTestBroker):
def get_management_url(self) -> str:
"""Opening this link during debugging allows you to see the RabbitMQ
management UI in your browser."""
ports = self.container.attrs["NetworkSettings"]["Ports"]
ip = ports["15672/tcp"][0]["HostIp"]
port = ports["15672/tcp"][0]["HostPort"]
return f"http://{ip}:{port}"


@pytest.fixture
def default_rabbitmq_broker_image() -> str:
return "rabbitmq:management"


@pytest.fixture
def default_rabbitmq_broker_ports() -> dict:
# Expose the management UI port
ports = RABBITMQ_PORTS.copy()
ports.update({"15672/tcp": None})
return ports


@pytest.fixture
def celery_rabbitmq_broker(default_rabbitmq_broker: RabbitMQContainer) -> RabbitMQTestBroker:
broker = RabbitMQManagementTestBroker(default_rabbitmq_broker)
yield broker
broker.teardown()


@pytest.fixture
def celery_broker_cluster(celery_rabbitmq_broker: RabbitMQTestBroker) -> CeleryBrokerCluster:
cluster = CeleryBrokerCluster(celery_rabbitmq_broker)
yield cluster
cluster.teardown()


###############################################################################
# Redis Result Backend
###############################################################################


@pytest.fixture
def celery_backend_cluster(celery_redis_backend: RedisTestBackend) -> CeleryBackendCluster:
cluster = CeleryBackendCluster(celery_redis_backend)
yield cluster
cluster.teardown()


@pytest.fixture
def default_redis_backend_image() -> str:
return "redis:latest"


###############################################################################
# Worker Configuration
###############################################################################


@pytest.fixture(scope="session")
def default_worker_celery_log_level() -> str:
return "INFO"


@pytest.fixture(scope="session")
def default_worker_celery_version() -> str:
return "5.2.7"


@pytest.fixture
def default_worker_app(default_worker_app: Celery) -> Celery:
app = default_worker_app
# app.conf... # Add any additional configuration here
return app


###############################################################################
# Bug Reproduction
###############################################################################


def test_issue_1234(celery_setup: CeleryTestSetup):
sig: Signature = ping.s()
res: AsyncResult = sig.delay()
assert res.get() == "pong"
3 changes: 1 addition & 2 deletions src/pytest_celery/vendors/worker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,7 @@ ENV PYTHONDONTWRITEBYTECODE=1
RUN pip install --no-cache-dir --upgrade \
pip \
celery[redis,pymemcache]${WORKER_VERSION:+==$WORKER_VERSION} \
pytest-celery@git+https://github.com/Katz-Consulting-Group/pytest-celery.git@hotfix
# pytest-celery@git+https://github.com/celery/pytest-celery.git@hotfix
pytest-celery@git+https://github.com/celery/pytest-celery.git

# The workdir must be /app
WORKDIR /app
Expand Down