CoverCraft-AI

A high-performance, multi-threaded book description generator that uses the Open router API to generate detailed descriptions for books. This project offers multiple processing modes optimized for different use cases, from development testing to production-scale processing.

🚀 Features

• Multiple Processing Modes: Choose between rate-limited (production-safe) and high-performance generators
• Async & Multi-threading Support: Utilizes multiple CPU cores for maximum performance
• Rate Limiting Compliance: Built-in rate limiting to respect API limits (40 requests/minute)
• Real-time Progress Tracking: Live progress updates with completion rates and timing
• Comprehensive Error Handling: Detailed error tracking with separate failed items logging
• Flexible Configuration: Adjustable worker counts, batch sizes, and processing modes
• Dual Output System: Successful results and failed items saved to separate JSON files
• Incremental Batch Saving: Saves processed data after each batch to batch_processed.json for resilience.

📋 Quick Start

1. Installation

# Clone the repository
git clone git@github.com:kabir0st/CoverCraft-AI.git
cd CoverCraft-AI

# Install dependencies
uv sync

2. Environment Setup

Create a .env file with your Open Router API key:

echo "KEY=your_open_router_api_key_here" > .env

3. Prepare Your Data

Ensure you have an items.json file in the src/ directory with your book data:

[
  {
    "id": 13625,
    "name": "Book Title",
    "barcode": "123456789"
  }
]

4. Run the Generator

For Production (Recommended):

cd src
python run_rate_limited.py --demo

For Development/Testing:

cd src
python run_generator.py --demo

🎯 Which Generator Should You Use?

🚦 Rate-Limited Generator (Production)

• File: src/rate_limited_generator.py
• Rate Limit: Strict 40 requests per minute
• Workers: 6-8 recommended
• Use Case: Production, large datasets, shared API keys
• Safety: Guaranteed API compliance

python src/run_rate_limited.py --workers 8 --batch-size 10

⚡ High-Performance Generator (Development)

• File: src/libs/async_generator.py
• Rate Limit: None (system/API limited)
• Workers: Up to 32 workers
• Use Case: Development, testing, small datasets
• Speed: Maximum performance

python src/run_generator.py --workers 16 --batch-size 25

📝 Simple Generator (Learning)

• File: src/app.py
• Type: Sequential processing
• Use Case: Understanding the basic workflow
• Speed: Slow (one at a time)

python src/app.py

📚 Documentation

This project includes comprehensive documentation for different aspects:

Core Documentation

• Generator Options Guide - Complete comparison of all available generators
• Usage Guide - Detailed usage instructions with examples and best practices
• Rate Limiting Guide - Specialized guide for production-safe rate-limited processing
• Async Generator Guide - High-performance async processing documentation

Quick Reference Links

• Installation & Setup
• Command Line Options
• Performance Tuning
• Error Handling
• Python API Usage

🔧 Configuration Examples

Production Processing (Recommended)

# Safe, rate-limited processing
python src/run_rate_limited.py --workers 8 --batch-size 10

# Test first with demo
python src/run_rate_limited.py --demo

Development/Testing

# Fast processing for development
python src/run_generator.py --demo --workers 4

# High performance testing
python src/run_generator.py --workers 16 --batch-size 20

System-Specific Recommendations

High-End Workstation (32+ cores, 64+ GB RAM)

python src/run_generator.py --workers 32 --batch-size 50

Standard Laptop (8-16 cores, 16+ GB RAM)

python src/run_rate_limited.py --workers 8 --batch-size 10

Budget System (4-8 cores, 8+ GB RAM)

python src/run_rate_limited.py --workers 4 --batch-size 5

📊 Performance Comparison

Generator	Speed	Safety	Workers	Best For
Rate-Limited	40/min	High	6-8	Production
High-Performance	System Max	Medium	16-32	Development
Simple	~1/min	High	1	Learning

📁 Output Files

Successful Results

• Rate-Limited: items_with_descriptions.json
• High-Performance: items_with_descriptions.json
• Simple: items_with_desc.json

Failed Items

• Rate-Limited: failed_items.json
• High-Performance: failed_items.json

🚨 Common Issues & Solutions

API Rate Limiting

Error: Rate limit exceeded
Solution: Use rate-limited generator
Command: python src/run_rate_limited.py

Memory Issues

Error: Out of memory
Solution: Reduce batch size and workers
Command: python src/run_rate_limited.py --workers 4 --batch-size 5

API Key Issues

Error: KEY not found in .env
Solution: Create .env file with KEY=your_api_key

🛡️ Best Practices

1. Always Start with Demo

python src/run_rate_limited.py --demo

2. Use Rate-Limited for Production

• ✅ Production processing
• ✅ Large datasets (100+ items)
• ✅ Shared/company API keys
• ✅ Unattended processing

3. Monitor Progress

Watch for the rate indicator: 🚦 38/40/min

4. Handle Interruptions

Both generators handle Ctrl+C gracefully and save progress.

5. Backup Important Data

Always backup your original items.json before processing.

🔍 Project Structure

CoverCraft-AI/
├── README.md                     # This file - main project overview
├── GENERATOR_OPTIONS.md          # Complete generator comparison guide
├── USAGE_GUIDE.md                # Detailed usage instructions
├── RATE_LIMITING_GUIDE.md        # Production-safe rate limiting guide
├── README_ASYNC.md               # High-performance async guide
├── requirements.txt              # Python dependencies
├── pyproject.toml                # Project configuration
└── src/                          # Source code directory
    ├── rate_limited_generator.py   # Production-safe rate-limited generator
    ├── run_generator.py          # CLI for async generator
    ├── run_rate_limited.py       # CLI for rate-limited generator (deprecated, use run_generator.py with options)
    ├── demo_async.py             # Demo script for async generator
    ├── items.json                # Input data file
    ├── system_prompt.txt         # System prompt for the generator
    └── libs/                     # Core library files
        ├── __init__.py
        ├── agent.py              # Core API interaction logic
        ├── async_app.py          # Async application (if applicable)
        ├── async_generator.py    # High-performance async generator
        ├── gen.py                # General generation utilities (if applicable)
        ├── run_async.py          # Runner for async operations (if applicable)
        ├── run_rate_limited.py   # Runner for rate-limited operations (if applicable, might be deprecated)
        └── utils.py              # Helper functions

📈 Processing Time Estimates

Rate-Limited Generator (40/min)

• 50 items: ~2-3 minutes
• 100 items: ~5-6 minutes
• 500 items: ~25-30 minutes
• 1000 items: ~45-50 minutes

High-Performance Generator (varies)

• 50 items: ~30 seconds - 2 minutes
• 100 items: ~1-4 minutes
• 500 items: ~5-20 minutes
• 1000 items: ~10-40 minutes

Times depend on system specs and API response times

🎯 Recommended Workflow

1. Start with Demo: python src/run_rate_limited.py --demo
2. Test Small Batch: python src/run_rate_limited.py --workers 4 --batch-size 5
3. Scale Up Gradually: python src/run_rate_limited.py --workers 8 --batch-size 10
4. Full Production Run: python src/run_rate_limited.py

🔧 System Requirements

• Python 3.12+
• 4+ GB RAM (8+ GB recommended)
• Multi-core CPU (4+ cores recommended)
• Stable internet connection
• Valid Open Router API key

📖 Getting Help

1. Read the Documentation: Start with the Usage Guide
2. Check Common Issues: Review error messages and solutions above
3. Examine Failed Items: Check failed_items.json for specific failures
4. Try Conservative Settings: Use fewer workers and smaller batches
5. Test with Demo Mode: Always test with demo mode first

🚀 Next Steps

1. New Users: Start with the Usage Guide
2. Production Users: Read the Rate Limiting Guide
3. Developers: Check the Async Generator Guide
4. Comparison Shopping: Review Generator Options

---

Remember: It's better to process slowly and successfully than to fail fast! Always prioritize API compliance and data integrity over speed.