FlowTracer

FlowTracer is a comprehensive Django-based application for tracking, tracing, and executing system workflows. It allows you to define complete application flows with components and connections, execute them with custom retry strategies, and recover from failures by retrying specific components.

Features

• Visual Workflow Builder: Create complex application workflows with an intuitive UI
• Component Types: Support for Kafka, Message Queue, Database, Service, and API components
• Connection Types: Define relationships like Kafka-to-Kafka, Kafka-to-MQ, MQ-to-MQ, MQ-to-Kafka, and DB operations
• Customizable Retry Strategies: Define fixed interval, exponential backoff, or custom retry strategies for components
• YAML Generation: Automatically generate YAML files from workflow definitions
• Execution Management: Execute full workflows or retry specific components
• Validation Support: Enable validation during execution with different modes
• Authentication: Built-in user authentication and authorization

Installation

Prerequisites

• Python 3.8 or higher
• pip (Python package installer)
• git (optional, for cloning the repository)

Setup

1. Clone the repository (or download and extract the zip file):

   git clone https://github.com/yourusername/flowtracer.git
   cd flowtracer

2. Create a virtual environment:

   python -m venv venv

3. Activate the virtual environment:

   • On Windows:

     venv\Scripts\activate

   • On macOS/Linux:

     source venv/bin/activate

4. Install the required dependencies:

   pip install -r requirements.txt

5. Apply database migrations:

   python manage.py migrate

6. Create a superuser:

   python manage.py createsuperuser

7. Run the development server:

   python manage.py runserver

8. Access the application at http://127.0.0.1:8000/

Usage Guide

Creating Workflows

1. Log in: Start by logging in with your credentials
2. Create Workflow: Click "Create Workflow" and provide a name and description
3. Add Components: Navigate to the Components tab and add application components
4. Define Connections: Create connections between components from the Connections tab
5. Add Retry Strategies: Configure retry strategies for critical components
6. Visualize: View the workflow diagram in the Visual Editor tab
7. Generate YAML: Download the YAML definition for use in other systems

Executing Workflows

1. Navigate to Execution: Click "Execute" for your workflow
2. Configure Validation: Enable or disable validation during execution
3. Start Execution: Click "Execute Workflow" to start the execution
4. Monitor Progress: View the component status and execution logs
5. Recover from Failures: Retry failed components if needed

Example Scenario: Kafka to MQ Replay

1. Create a workflow named "Order Processing"
2. Add components:
   • "OrderTopic" (Kafka component)
   • "OrderQueue" (MQ component)
   • "OrderService" (Service component)
3. Create connections:
   • "OrderTopic" to "OrderQueue" (Kafka to MQ connection)
   • "OrderQueue" to "OrderService" (Processing connection)
4. Add retry strategies to components
5. Execute the workflow
6. If a component fails, retry it individually

Project Structure

flowtracer/
├── flowtracer/                 # Main Django project
├── authentication/             # Authentication app
├── workflow/                   # Workflow builder app
│   ├── models.py               # Data models for workflows
│   ├── views.py                # View functions for UI
│   ├── services/               # Business logic services
│   │   ├── yaml_generator.py   # YAML generation service
│   │   └── workflow_executor.py # Workflow execution engine
├── static/                     # Static files (CSS, JS)
├── templates/                  # HTML templates
├── manage.py                   # Django management script
└── requirements.txt            # Python dependencies

Integration

FlowTracer is designed to be integrated with your existing systems. The YAML files generated by the system can be used by your application components to understand the workflow structure. You can extend the connectors in the workflow/services/connectors/ directory to integrate with your specific Kafka, MQ, or database systems.

Extension Points

1. Component Types: Add new component types in the WorkflowComponent model
2. Connection Types: Define new connection types in the WorkflowConnection model
3. Connectors: Implement real connectors in the workflow/services/connectors/ directory
4. Validation Rules: Extend the validation logic in the executor service

Production Deployment

For production deployment, make sure to:

1. Set DEBUG = False in settings.py
2. Configure a proper database (PostgreSQL recommended)
3. Set up static files serving with whitenoise
4. Use a production-ready web server like Gunicorn
5. Consider using Docker for containerization
6. Set up proper monitoring and logging

Sample Docker configuration:

FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000

CMD ["gunicorn", "--bind", "0.0.0.0:8000", "flowtracer.wsgi"]

Troubleshooting

If you encounter any issues:

1. NoReverseMatch Error: Check that all templates correctly handle workflows with empty/null IDs
2. Database Issues: Reset migrations if necessary
3. Static Files Not Loading: Check settings.py configuration and template references
4. Missing Templates: Ensure all template files are in the correct locations

For detailed troubleshooting steps, see the Troubleshooting Guide.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Django - The web framework used
• Bootstrap - Frontend framework
• jsPlumb - For workflow diagram visualization
• YAML - For workflow definitions