Instagram Caption Generator

An AI-powered application that generates creative Instagram captions for images using fine-tuned BLIP (Bootstrapped Language-Image Pretraining) model.

🎯 Features

• Caption Generation: Generate multiple caption variations with customizable length
• Hashtag Generation: Automatic relevant hashtag suggestions
• Model Training: Fine-tune BLIP on Instagram dataset (5 epochs completed)
• Model Evaluation: Comprehensive metrics (BLEU, CIDEr, METEOR, ROUGE-L)
• Web Interfaces: Streamlit frontend and Vue.js dashboard
• REST API: FastAPI backend with full documentation
• Data Analysis: Jupyter notebooks for exploration and evaluation

🛠️ Technology Stack

• Model: BLIP (Salesforce)
• Backend: FastAPI, PyTorch, HuggingFace Transformers
• Frontend: Streamlit, Vue.js with Chart.js
• Deployment: Docker, Docker Compose
• Hardware: NVIDIA RTX 3060 (12GB VRAM)
• Analysis: Jupyter Notebooks, Pandas, Matplotlib, Seaborn

📁 Project Structure

instagram-caption-generator/
├── backend/
│   ├── main.py              # FastAPI application
│   ├── model.py             # BLIP model wrapper
│   ├── train.py             # Training script
│   ├── evaluate.py          # Evaluation script
│   ├── config.py            # Configuration
│   ├── utils.py             # Utility functions
│   ├── data_loader.py       # Data loading utilities
│   ├── preprocessing.py     # Data preprocessing
│   ├── caption_enhancer.py  # Caption enhancement
│   ├── Dockerfile
│   ├── requirements.txt
│   └── tests/               # Unit tests
├── frontend/
│   ├── streamlit/
│   │   ├── app.py           # Streamlit application
│   │   └── requirements.txt
│   └── vue-app/             # Vue.js dashboard
│       ├── index.html
│       ├── package.json
│       ├── vite.config.js
│       └── src/
├── data/
│   ├── raw/
│   │   ├── instagram_data/
│   │   │   ├── img/         # Images folder 1
│   │   │   └── captions.csv # Captions file 1
│   │   └── instagram_data2/
│   │       ├── img2/        # Images folder 2
│   │       └── captions2.csv # Captions file 2
│   └── processed/           # Train/val/test splits
│       ├── train.csv
│       ├── val.csv
│       ├── test.csv
│       └── metadata.json
├── models/
│   ├── checkpoints/         # Trained model checkpoints
│   │   ├── best_model/      # Best performing checkpoint
│   │   ├── final_model/     # Final epoch checkpoint
│   │   └── epoch_*/         # Individual epoch checkpoints
│   ├── evaluation/          # Evaluation results
│   │   ├── baseline_metrics.json
│   │   ├── baseline_predictions.json
│   │   └── baseline_results.csv
│   └── models--Salesforce--blip-image-captioning-base/
├── configs/                 # Configuration files
│   ├── model_config.yaml    # Model hyperparameters
│   ├── data_config.yaml     # Data processing config
│   └── training_config.yaml # Training parameters
├── notebooks/               # Jupyter notebooks
│   ├── 01_data_exploration.ipynb    # Dataset analysis
│   ├── 02_baseline_evaluation.ipynb # Baseline model eval
│   ├── 03_results_analysis.ipynb    # Results analysis
│   └── 04_error_analysis.ipynb      # Error analysis
├── scripts/                 # Utility scripts
│   ├── preprocess_data.py   # Data preprocessing
│   ├── download_dataset.py  # Dataset download
│   ├── evaluate_test_set.py # Test set evaluation
│   ├── error_analysis.py    # Error analysis
│   ├── fix_csv_headers.py   # CSV fixing utilities
│   ├── image_base64.py      # Image encoding
│   ├── split_dataset.py     # Dataset splitting
│   └── test_improvements.py # Testing improvements
├── logs/                    # Training and evaluation logs
│   ├── training/            # Training logs
│   ├── evaluation/          # Evaluation logs
│   ├── api/                 # API logs
│   └── csv/                 # CSV metrics logs
├── docs/                    # Documentation
│   ├── API.md              # API documentation
│   ├── DEPLOYMENT.md       # Deployment guide
│   └── USER_GUIDE.md       # User guide
├── tests/                   # Performance tests
│   └── performance/
├── template/                # HTML templates
└── docker-compose.yml

🚀 Quick Start

Prerequisites

• Docker and Docker Compose
• NVIDIA GPU with CUDA support
• NVIDIA Container Toolkit

1. Setup Data

Place your dataset in the following structure:

data/raw/
├── instagram_data/
│   ├── img/              # Images folder 1
│   └── captions.csv      # Captions file 1
└── instagram_data2/
    ├── img2/             # Images folder 2
    └── captions2.csv     # Captions file 2

2. Deploy with Script

The easiest way to deploy the entire application:

# Make scripts executable (Linux/Mac)
chmod +x scripts/deploy.sh scripts/build_docker.sh

# Deploy the complete application
./scripts/deploy.sh

This will:

• Check prerequisites (Docker, Docker Compose)
• Build Docker images
• Start all services
• Wait for services to be healthy
• Show access URLs and management commands

3. Manual Docker Deployment

If you prefer manual control:

# Build Docker images
docker-compose build

# Or use the build script
./scripts/build_docker.sh

# Start services
docker-compose up -d

# Check service status
docker-compose ps

# View logs
docker-compose logs -f

4. Explore Data (Optional)

# Enter the backend container
docker exec -it caption_backend bash

# Inside container: Preprocess data (if not done)
python scripts/preprocess_data.py

# Start Jupyter for data exploration
jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root

# Open notebooks/01_data_exploration.ipynb in your browser

5. Evaluate Baseline Model

# Run baseline evaluation
docker exec -it caption_backend bash
python backend/evaluate.py

6. Access the Application

After deployment, access the application at:

• Backend API: http://localhost:8000
• API Documentation: http://localhost:8000/docs
• Vue.js Dashboard: http://localhost:3000
• Streamlit App: http://localhost:8501 (if started separately)

Streamlit App

# In a new terminal (outside container)
cd frontend/streamlit

# Install dependencies
pip install -r requirements.txt

# Run Streamlit
streamlit run app.py

# Or use the script
bash scripts/run_streamlit.sh

Access the app at: http://localhost:8501

Vue.js Dashboard

# In a new terminal
cd frontend/vue-app

# Install dependencies
npm install

# Run development server
npm run dev

Access the dashboard at: http://localhost:5173

📊 Dataset Information

• Total images: ~29,899 photos
• Captions: ~29,899 captions (processed)
• Caption length: 3-2176 characters (mean: 50, median: 33)
• Word count: 1-402 words (mean: 8.6, median: 6)
• Emoji usage: 31.4% of captions contain emojis (avg 0.39 per caption)
• Hashtag usage: 21.9% of captions contain hashtags (avg 0.43 per caption)
• Style: Short, casual, emoji-inclusive Instagram captions
• Split: 70% train, 15% validation, 15% test

🎓 Training Status

The model has been trained for 5 epochs with the following results:

Training Metrics (Epoch 5)

• Validation Loss: 1.09
• BLEU-4: 0.018
• CIDEr: 0.149

Baseline Performance (Pretrained BLIP)

• BLEU-1: 0.032
• BLEU-2: 0.006
• BLEU-3: 0.000
• BLEU-4: 0.000
• CIDEr: 0.030
• METEOR: 0.012
• ROUGE-L: 0.027

Target Performance (Fine-tuned)

• BLEU-4: >0.030
• CIDEr: >1.000

📓 Jupyter Notebooks

The project includes comprehensive analysis notebooks:

01_data_exploration.ipynb

• Dataset loading and structure analysis
• Caption length and word count distributions
• Emoji and hashtag usage analysis
• Data quality assessment

02_baseline_evaluation.ipynb

• Baseline model evaluation setup
• Performance metrics calculation (BLEU, CIDEr, METEOR, ROUGE-L)
• Sample predictions analysis
• Model loading and inference testing

03_results_analysis.ipynb

• Training results visualization
• Performance metrics over epochs
• Model comparison and analysis

04_error_analysis.ipynb

• Error analysis framework
• Common failure patterns identification
• Model improvement suggestions

Logs and Metrics

The project maintains comprehensive logs for monitoring and analysis:

Training Logs (logs/training/)

• TensorBoard event files for visualization
• Training progress and loss curves
• GPU memory usage and performance metrics

CSV Metrics (logs/csv/)

• val-val_loss.csv - Validation loss over epochs
• val-bleu_4.csv - BLEU-4 scores over epochs
• val-cider.csv - CIDEr scores over epochs
• train-loss.csv - Training loss
• train-lr.csv - Learning rate schedule

Evaluation Logs (logs/evaluation/)

• Model evaluation results
• Performance comparisons
• Baseline vs trained model metrics

🔧 API Endpoints

• POST /generate - Generate captions from uploaded image
• POST /generate/base64 - Generate captions from base64 image

Training

• POST /train/start - Start model training
• GET /train/status - Get training status
• GET /train/logs - Get training logs
• GET /train/checkpoints - List available checkpoints

Evaluation

• POST /evaluate - Evaluate model on test set
• GET /evaluate/metrics - Get latest evaluation metrics

Model Management

• GET /models - List available models
• POST /models/load - Load specific checkpoint

Health

• GET / - API information
• GET /health - Health check

Full API documentation: http://localhost:8000/docs

🎓 Usage Examples

Generate Captions via API

import requests

# Upload image
with open('image.jpg', 'rb') as f:
    files = {'file': f}
    data = {
        'num_captions': 3,
        'length': 'medium',
        'generate_hashtags': True,
        'num_hashtags': 5
    }
    
    response = requests.post(
        'http://localhost:8000/generate',
        files=files,
        data=data
    )
    
    result = response.json()
    print("Captions:", result['captions'])
    print("Hashtags:", result['hashtags'])

Start Training via API

import requests

data = {
    'batch_size': 8,
    'epochs': 10,
    'learning_rate': 5e-5,
    'model_name': 'my-instagram-model'
}

response = requests.post(
    'http://localhost:8000/train/start',
    json=data
)

print(response.json())

Generate Captions Programmatically

from model import CaptionModel

# Load model
model = CaptionModel()

# Generate captions
captions = model.generate_with_length(
    'path/to/image.jpg',
    length='medium',
    num_captions=3
)

# Generate with hashtags
result = model.generate_with_hashtags(
    'path/to/image.jpg',
    num_captions=3,
    num_hashtags=5
)

print("Captions:", result['captions'])
print("Hashtags:", result['hashtags'])

📈 Evaluation Metrics

• BLEU-1 to BLEU-4: Measures n-gram overlap with reference captions
• CIDEr: Consensus-based metric specifically for image captioning
• METEOR: Measures semantic similarity and synonymy
• ROUGE-L: Measures longest common subsequence
• Higher scores indicate better caption quality

Current Performance (5 epochs trained):

• BLEU-4: 0.018
• CIDEr: 0.149

Baseline Performance (Pretrained BLIP):

• BLEU-1: 0.032, BLEU-2: 0.006, BLEU-3: 0.000, BLEU-4: 0.000
• CIDEr: 0.030, METEOR: 0.012, ROUGE-L: 0.027

Target Performance (Fine-tuned):

• BLEU-4: >0.030
• CIDEr: >1.000

⚙️ Configuration

Configuration is managed through YAML files in the configs/ directory:

• configs/model_config.yaml - Model architecture and generation parameters
• configs/data_config.yaml - Data processing and loading settings
• configs/training_config.yaml - Training hyperparameters and settings

You can also edit backend/config.py for additional customization:

• Model parameters
• Training hyperparameters
• Data paths
• Generation settings

🐳 Docker Commands

Using Deployment Script

# Deploy complete application
./scripts/deploy.sh

# Stop services
./scripts/deploy.sh stop

# Restart services
./scripts/deploy.sh restart

# View logs
./scripts/deploy.sh logs

# Check status
./scripts/deploy.sh status

# Clean up (removes containers, images, volumes)
./scripts/deploy.sh clean

# Show help
./scripts/deploy.sh help

Manual Docker Commands

# Build images
docker-compose build
# or
./scripts/build_docker.sh

# Start services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop services
docker-compose down

# Rebuild after code changes
docker-compose up -d --build

# Access backend shell
docker exec -it caption_backend bash

# Access frontend shell
docker exec -it caption_frontend sh

🔍 Troubleshooting

Issue: CUDA Out of Memory

Solution:

• Reduce batch size in config.py
• Enable gradient accumulation
• Use smaller image size (224x224 instead of 384x384)

Issue: API Connection Refused

Solution:

• Check if backend is running: docker ps
• Verify port mapping: docker-compose ps
• Check API health: curl http://localhost:8000/health

Issue: Training Not Starting

Solution:

• Verify preprocessed data exists: ls data/processed/
• Run preprocessing: python scripts/preprocess_data.py
• Check training logs: GET /train/logs

Issue: Images Not Found

Solution:

• Verify dataset structure matches expected paths
• Check config.py paths are correct
• Ensure Docker volume mounts are correct in docker-compose.yml

📝 Development Workflow

1. Data Preparation

   python scripts/preprocess_data.py
   jupyter notebook notebooks/01_data_exploration.ipynb

2. Baseline Evaluation

   jupyter notebook notebooks/02_baseline_evaluation.ipynb

3. Training (Completed - 5 epochs)

   # Training completed with checkpoints in models/checkpoints/
   # View training logs in logs/training/ and logs/csv/
   # Best model: models/checkpoints/best_model/
   # Monitor metrics via notebooks/03_results_analysis.ipynb

4. Evaluation

   python backend/evaluate.py --checkpoint models/checkpoints/*/best_model.pt

5. Deployment

   # Deploy complete application
   ./scripts/deploy.sh
   
   # Or manually:
   # docker-compose up -d
   # cd frontend/streamlit && streamlit run app.py

🚀 Future Enhancements

Short-term

• Style-conditioned caption generation
• Enhanced hashtag generation with trending tags
• Caption editing interface
• Batch processing for multiple images

Medium-term

• Multi-language support
• User authentication and accounts
• Caption history and favorites
• Advanced model fine-tuning options
• A/B testing framework for caption variants

Long-term

• Mobile application
• Instagram API integration
• Video caption generation
• Community features and ratings

📄 License

This is a college project for educational purposes.

🙏 Acknowledgments

• BLIP model by Salesforce Research
• HuggingFace Transformers
• FastAPI and Streamlit communities
• Instagram dataset contributors

📧 Contact

For questions or issues, please open an issue on GitHub.

---

Made using BLIP, FastAPI, and Streamlit