readme-fastapi-redis.md

ASYNC-ML-INFERENCE

Empowering ML with Async Magic!

_Built with:_

Table of Contents

1. Overview
2. Features
3. Project Structure 3.1. Project Index
4. Getting Started 4.1. Prerequisites 4.2. Installation 4.3. Usage 4.4. Tests
5. Roadmap
6. Contributing
7. License
8. Acknowledgments

---

Overview

async-ml-inference orchestrates distributed tasks for audio processing and Euro results retrieval. It streamlines asynchronous processing, enhancing efficiency and scalability. Ideal for developers seeking seamless task management in machine learning applications.

---

Features

	Feature	Summary
⚙️	Architecture	Orchestrates Docker services using docker-compose.yaml for a Celery-based distributed system. Utilizes RabbitMQ and Redis containers for task queuing and backend storage. Spins up Celery workers for audio and euro tasks, along with an API service and a client for interaction.
🔩	Code Quality	Defines FastAPI endpoints in src/api/api.py for creating and retrieving audio and Euro task results. Implements Celery workers in src/workers/audio/worker.py and src/workers/euro/worker.py for efficient task processing. Utilizes flake8 for code linting and pydocstyle for docstring checking.
📄	Documentation	Primary language: Python Package managers: pipenv and pip Contains detailed documentation in various file formats like .yaml, .lock, .txt, and .py.
🔌	Integrations	Integrates with Redis and RabbitMQ for backend and broker connections. Utilizes requests for making HTTP requests and joblib for parallel processing in the client module. Uses Celery for asynchronous task execution.
🧩	Modularity	Separates concerns into API, workers, and client modules for clear functionality. Configures Celery settings in separate files like config.py for Audio Length and Euromillions workers. Ensures dependencies are managed separately for each module using requirements.txt files.
🧪	Testing	Tests the project using pytest with commands provided for both pipenv and pip installations. Includes unit tests for API endpoints, worker functions, and client interactions. Ensures code quality with testing and linting tools like mypy and pycodestyle.
⚡️	Performance	Optimizes task processing efficiency by configuring Celery settings in config.py files. Utilizes parallel processing in the client module for enhanced performance. Calculates audio length efficiently using librosa in the audio worker.
🛡️	Security	Secures backend and broker connections using environment variables and connection validation. Ensures secure communication between components within the distributed system. Follows best practices for handling sensitive data and user inputs.

---

Project Structure

└── async-ml-inference/
    ├── Pipfile
    ├── Pipfile.lock
    ├── README.md
    ├── docker-compose.yaml
    ├── docs
    │   └── diagram
    ├── src
    │   ├── api
    │   ├── client
    │   └── workers
    └── tests
        └── README.md

Project Index

Async-ml-inference Index

__root__

docker-compose.yaml	- Orchestrates Docker services for a Celery-based distributed system - Manages RabbitMQ and Redis containers for task queuing and backend storage - Spins up Celery workers for audio and euro tasks, along with an API service and a client for interaction - Establishes network connections for seamless communication between components.
Pipfile	Define project dependencies and scripts for development and deployment using the Pipfile.

src

api

requirements.txt	- Facilitates dependency management for the project's API by specifying required packages and versions in the 'requirements.txt' file located at 'src/api/' - This file ensures that the necessary libraries like Celery, FastAPI, uvicorn, and pydantic are installed to support the API functionality within the project architecture.
Dockerfile	- Facilitates containerized deployment of the API service by defining necessary dependencies and configurations - Sets up a Python environment, installs required packages, exposes specified ports, and launches the API service using Uvicorn - Streamlines the deployment process within the project architecture.
api.py	- Defines FastAPI endpoints for creating and retrieving audio and Euro task results using Celery for asynchronous processing - Handles task creation, status retrieval, and result sending - Utilizes Redis and RabbitMQ for backend and broker connections - Implements background tasks for result notifications.

workers

backend.py	- Defines functions to retrieve Redis connection details and check if the backend is running - The code constructs a Redis URL based on environment variables and attempts a connection to the Redis instance - If successful, it confirms the backend is operational; otherwise, it logs an error.
requirements.txt	- Manages project dependencies for the workers module, ensuring the required libraries are available for seamless execution - This file specifies the versions of libraries like BeautifulSoup, Celery, librosa, and numba needed by the workers to perform tasks efficiently within the project architecture.
Dockerfile	Builds a Docker image for worker processes, installing dependencies and exposing necessary ports.
broker.py	- Defines functions to retrieve RabbitMQ connection details and check if the broker is running - The code constructs a broker URL using environment variables and attempts to connect to RabbitMQ with retries - If successful, it confirms the broker is running; otherwise, it logs an error.

audio

worker.py	- Implements a Celery worker for extracting audio length - Validates backend and broker connections before processing audio URL - Downloads audio data, calculates duration using librosa, and simulates task processing time - Handles exceptions and returns the audio length upon completion.
config.py	Configure Celery settings for the Audio Length worker to optimize task processing efficiency and ensure reliable task handling within the project architecture.

euro

worker.py	- The code file in src/workers/euro/worker.py serves as a Celery worker for fetching Euromillions results - It ensures the backend and broker services are running before scraping the results from a specified URL - The worker extracts numbers and stars from the webpage based on the draw date provided, returning them as a tuple.
config.py	- Configure Celery settings for the Euromillions Results worker, including task acknowledgment, prefetch multiplier, task queues, and result expiration time - This module centralizes the configurations necessary for efficient task processing within the project's architecture.

client

requirements.txt	Manage external dependencies for the client-side application using the specified versions of requests, retrying, and joblib.
Dockerfile	Builds a lightweight Python container for the client module, installing dependencies and exposing port 5000.
client.py	- Generates audio and date tasks, sends them to the API, retrieves results, and displays successful outputs - Utilizes parallel processing for efficiency - Key functionalities include posting audio URLs and dates, checking task statuses, and handling retries - The script orchestrates the entire process seamlessly, enhancing performance through parallel execution.

---

Getting Started

Prerequisites

Before getting started with async-ml-inference, ensure your runtime environment meets the following requirements:

• Programming Language: Python
• Package Manager: Pipenv, Pip
• Container Runtime: Docker

Installation

Install async-ml-inference using one of the following methods:

Build from source:

1. Clone the async-ml-inference repository:

❯ git clone https://github.com/FerrariDG/async-ml-inference

2. Navigate to the project directory:

❯ cd async-ml-inference

3. Install the project dependencies:

Using pipenv  

❯ pipenv install

Using pip  

❯ pip install -r src/api/requirements.txt, src/workers/requirements.txt, src/client/requirements.txt

Using docker  

❯ docker build -t FerrariDG/async-ml-inference .

Usage

Run async-ml-inference using the following command:

Using pipenv  

❯ pipenv shell
❯ pipenv run python {entrypoint}

Using pip  

❯ python {entrypoint}

Using docker  

❯ docker run -it {image_name}

Testing

Run the test suite using the following command:

Using pipenv  

❯ pipenv shell
❯ pipenv run pytest

Using pip  

❯ pytest

Roadmap

• Task 1: Implement feature one.
• Task 2: Implement feature two.
• Task 3: Implement feature three.

---

Contributing

Contributions are welcome! Here are several ways you can contribute:

• Report Issues: Submit bugs found or log feature requests for the async-ml-inference project.
• Submit Pull Requests: Review open PRs, and submit your own PRs.
• Join the Discussions: Share your insights, provide feedback, or ask questions.

Contributing Guidelines

1. Fork the Repository: Start by forking the project repository to your github account.
2. Clone Locally: Clone the forked repository to your local machine using a git client.

   git clone https://github.com/FerrariDG/async-ml-inference

3. Create a New Branch: Always work on a new branch, giving it a descriptive name.

   git checkout -b new-feature-x

4. Make Your Changes: Develop and test your changes locally.
5. Commit Your Changes: Commit with a clear message describing your updates.

   git commit -m 'Implemented new feature x.'

6. Push to github: Push the changes to your forked repository.

   git push origin new-feature-x

7. Submit a Pull Request: Create a PR against the original project repository. Clearly describe the changes and their motivations.
8. Review: Once your PR is reviewed and approved, it will be merged into the main branch. Congratulations on your contribution!

Contributor Graph

---

License

This project is protected under the SELECT-A-LICENSE License. For more details, refer to the LICENSE file.

---

Acknowledgments

• List any resources, contributors, inspiration, etc. here.

---