Kartevonmorgen Workflows (kvmflows)

Automated subscription and notification service for OpenFairDB entries

---

Overview

Kartevonmorgen Workflows (kvmflows) is a backend service for managing user subscriptions to OpenFairDB entries. Users can subscribe to updates or new entries in specific geographic areas and receive periodic email notifications. The system includes:

• API Server: Accepts subscription requests, handles activation/unsubscription, and manages user preferences.
• Email Service: Sends activation emails and periodic batch notifications using Mailgun and Liquid templates.
• Sync Flows: Two-tier data synchronization strategy:
  • Full Sync: Comprehensive scraping of all OpenFairDB entries for complete data coverage
  • Recent Sync: Hourly updates fetching only recently changed entries for real-time responsiveness
• Optimized Email Timing: Subscription emails are scheduled for 8:00 AM UTC to maximize open rates and user engagement, with hourly notifications at minute 5 to ensure recent sync completion.

---

Features

• Subscription API: Create, activate, and unsubscribe from area-based entry notifications.
• Email Activation: Users must activate subscriptions via a secure email link.
• Batch Notifications: Periodic emails with new/updated entries in the user's area.
• Configurable Intervals: Daily, weekly, and monthly notification options.
• Robust Error Handling: Retries, rate limiting, and logging for email delivery.

---

Architecture

• FastAPI: RESTful API for subscription management.
• Mailgun: Email delivery for activation and batch notifications.
• Liquid Templates: Customizable email content.
• PostgreSQL: Persistent storage for subscriptions and entries.
• APScheduler: Cron jobs for scraping and notifications.
• Docker Compose: Containerized deployment for all services.

---

Getting Started

Prerequisites

• Python 3.12+
• Docker & Docker Compose
• Poetry for dependency management
• Mailgun account (for email delivery)
• PostgreSQL database

Installation

1. Clone the repository:

   git clone https://github.com/your-org/kvmflows.git
   cd kvmflows

2. Configure environment:
   • Edit config.yaml for API keys, database, and email settings.
   • Ensure Mailgun credentials and template paths are set.
3. Install dependencies:

   poetry install

4. Run database migrations: (If needed, apply migrations or run init_pg_dbs.sql)

Running the Project

Local Development

• Start API server:

  poetry run python src/kvmflows/apis/server.py

• Run sync flows manually:

  # Full synchronization of all entries
  poetry run python src/kvmflows/crons/sync_all_entries.py
  # Recent entries synchronization (hourly updates)
  poetry run python src/kvmflows/crons/sync_recent_entries.py
  # Send subscription emails
  poetry run python src/kvmflows/crons/send_subscription_emails.py

Docker Compose

• Build and start all services:

  docker-compose up --build

  This will start:
  • API server (server.py)
  • Full entry synchronizer (sync_all_entries.py)
  • Recent entry synchronizer (sync_recent_entries.py) - runs hourly
  • Subscription email service (send_subscription_emails.py)

---

API Endpoints

• POST /v1/subscriptions — Create a new subscription
• GET /v1/subscriptions/{id}/activate — Activate a subscription
• GET /v1/subscriptions/{id}/unsubscribe — Unsubscribe

See OpenAPI docs for full endpoint details.

---

Email Flow

1. User subscribes via API.
2. Activation email sent with secure link.
3. User activates subscription.
4. Sync flows continuously update entry data:
   • Full sync scrapes all OpenFairDB entries for comprehensive coverage
   • Recent sync runs hourly to capture newly changed entries
5. Batch emails sent to users with relevant entries at 8:00 AM UTC for optimal engagement, with hourly notifications scheduled at minute 5 to ensure recent sync completion.
6. User can unsubscribe at any time.

Note: Subscription emails are scheduled for 8:00 AM UTC as this timing significantly increases the likelihood of emails being read and acted upon by users. Hourly notifications are sent at minute 5 of each hour to allow the recent entries sync process to complete. All service times operate in UTC timezone.

---

Configuration

• All settings are managed in config.yaml:
  • API keys, email templates, database credentials, sync schedules, area definitions, etc.
  • Email timing is configured for 8:00 AM UTC delivery to maximize user engagement and read rates.
  • Hourly notifications are scheduled at minute 5 of each hour to ensure recent entries sync completion.
  • All service operations run in UTC timezone for consistency across deployments.
• Example email templates are in src/kvmflows/templates/.

---

Contributing

1. Fork the repository
2. Create your feature branch (git checkout -b feature/foo)
3. Commit your changes (git commit -am 'Add feature')
4. Push to the branch (git push origin feature/foo)
5. Create a new Pull Request

---

License

This project is licensed under the MIT License.

---

Maintainers

• Navid Kalaei (navidkalaei@gmail.com)