cmg: Implement CogStack Model Gateway & Scheduler

This commit introduces the CogStack Model Gateway and Scheduler
services, along with a common package for managing external components,
including a database, an object store, and a message queue. More
specifically, the project is deployed through a docker-compose file and
consists of the following core services:
* Gateway: A FastAPI server that receives requests from the user
    targetting a deployed model server. These are submitted as tasks to
    be executed by the downstream services. It also exposes endpoints
    for listing the available model services, as well as fetching
    information about previously submitted tasks.
* Database: A PostgreSQL database for storing information about the
    submitted tasks, most importantly their status, IDs for tracking
    them across different services (e.g. other stores deployed as part
    of the CogStack Model Gateway as well as the model tracking server),
    and references to their results.
* Queue: A RabbitMQ message queue for managing the tasks submitted to
    the system. The queue is used to distribute the tasks to the
    scheduler, which is in turn responsible for executing them.
* Object Store: A MinIO object store for serialising larger user
    requests (e.g. attached files or request bodies) that will be picked
    up an reconstructed by the scheduler, as well as for storing task
    results. The results are stored as objects with a prefix that
    corresponds to the corresponding task's UUID.
* Scheduler: A Python application that listens to the message queue for
    new tasks and executes them by forwarding them to the appropriate
    model server for execution. Currently, it is also responsible for
    monitoring the execution of long-running tasks and updating all
    relevant stores with the task's status and results.

The E2E workflow looks something like this:
* The user submits a task to the Gateway, specifying the desired model
  server and action as part of the request (i.e. path parameters)
* The Gateway receives the task and creates an entry in the database,
  generating a UUID that can be used to track the task across the CMG
  components and setting its status to "PENDING".
* The Gateway deconstructs the request, extracting information like the
  target endpoint and action, the content type, query params, body, and
  attached files. The request body and the attached files are serialised
  and stored in the object store.
* The Gateway creates a message containing all data necessary for
  reconstructing the request and sends it to the queue. For objects
  stored in the object store, it includes a reference instead, so that
  the queue isn't cluttered with large payloads. While all messages are
  assigned the same priority at the moment, we intend to extend the
  Gateway to calculate an appropriate value based on the information
  received.
* The Scheduler listens to the queue for new messages and picks them up
  as they come.
* The Scheduler reconstructs the request and forwards it to the
  appropriate model server for execution, setting its status in the
  database to "SCHEDULED" and acknowledging the message delivery.
* The model server executes the task and returns a response.
* If the server signifies the completion of the task (i.e. short-running
  task that completes synchronously), the Scheduler stores the result in
  the object store, updates the task's status to "SUCCEEDED" or "FAILED"
  depending on the response, and adds a reference to the results in a
  bucket in the object store.
* If the server signifies the task was accepted (i.e. long-running task)
  the Scheduler updates the task's status to "RUNNING" and starts
  periodically checking its status by monitoring the model tracking
  server (MLflow). Once the task is completed, the MLflow UI URL is
  serialised and uploaded to MinIO, while the task's status is updated
  to "SUCCEEDED" or "FAILED" depending on the response.
* In the future we will extend this service to send a notification to
  the user that requested the task execution. Monitoring the task
  execution will likely be moved out of th scheduler and into a separate
  service altogether. In the meantime, the user can query the Gateway
  for the task's status and results and even ask for a URL to directly
  download the file containing the results from the object store.

Signed-off-by: Phoevos Kalemkeris <phoevos.kalemkeris@ucl.ac.uk.com>

Author: phoevos

.github/workflows/ci.yaml

@@ -0,0 +1,32 @@
+name: Python tests
+
+on:
+  push:
+    branches:
+    - main
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v4
+
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: "3.12"
+
+    - name: Install Poetry
+      run: |
+        curl -sSL https://install.python-poetry.org | python3 -
+
+    - name: Install dependencies
+      run: |
+        poetry install --with dev
+
+    - name: Run tests
+      run:
+        poetry run pytest

.gitignore

@@ -0,0 +1,168 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
+.pdm.toml
+.pdm-python
+.pdm-build/
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# ruff
+.ruff_cache/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# VS Code
+.vscode/

.pre-commit-config.yaml

@@ -0,0 +1,27 @@
+default_install_hook_types:
+  - pre-commit
+  - post-checkout
+  - post-merge
+
+repos:
+- repo: https://github.com/pre-commit/pre-commit-hooks
+  rev: v5.0.0
+  hooks:
+  - id: trailing-whitespace
+  - id: end-of-file-fixer
+  - id: check-yaml
+  - id: check-added-large-files
+  - id: check-json
+- repo: https://github.com/astral-sh/ruff-pre-commit
+  rev: v0.8.0
+  hooks:
+    - id: ruff
+      entry: poetry run ruff check --force-exclude --fix
+    - id: ruff-format
+      entry: poetry run ruff format --force-exclude
+-   repo: https://github.com/python-poetry/poetry
+    rev: 1.8.4
+    hooks:
+    -   id: poetry-check
+    -   id: poetry-lock
+    -   id: poetry-install

CONTRIBUTING.md

@@ -0,0 +1,47 @@
+# Contributing
+
+Thank you for your interest in contributing to the project! Here are some useful instructions for
+getting you started.
+
+## Installation
+
+1. Clone the repository:
+
+    ```shell
+    git clone https://github.com/CogStack/CogStack-ModelGateway.git cms-gateway
+    cd cms-gateway
+    ```
+
+2. Install Poetry if you haven't already:
+
+    ```shell
+    curl -sSL https://install.python-poetry.org | python3 -
+    ```
+
+3. Install the project dependencies:
+
+    ```shell
+    poetry install --with dev
+    ```
+
+## Pre-commit Hooks
+
+We use pre-commit hooks to ensure code quality. To set them up, follow these steps:
+
+1. Install pre-commit if you haven't already:
+
+    ```shell
+    pip install pre-commit
+    ```
+
+2. Install the pre-commit hooks to your local Git repository:
+
+    ```shell
+    pre-commit install
+    ```
+
+3. (optional) To run the pre-commit hooks manually on all files, use:
+
+    ```shell
+    pre-commit run --all-files
+    ```