QuantEcon Repository Backup Workflow

A centralized workflow for backing up QuantEcon repositories to AWS S3.

Overview

This workflow automatically backs up GitHub repositories to AWS S3 for disaster recovery and compliance. It runs from this single repository and backs up all matching repositories across the organization. It supports pattern-based repository selection, allowing you to backup specific repositories from an organization.

Features

• Pattern-based selection: Use regex patterns to select which repositories to backup
• Mirror backups: Complete repository backups including all branches, tags, and history
• S3 storage: Secure storage in AWS S3 with upload verification
• Skip existing: Avoid redundant backups with automatic duplicate detection
• Backup reporting: Generate reports on backup status and storage usage

Quick Start

1. Configure AWS IAM for OIDC Authentication (Recommended)

OIDC authentication is more secure than static credentials—no long-lived secrets to manage.

Create IAM Identity Provider

aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com

Create IAM Role with Trust Policy

Create a file trust-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:YOUR_ORG/workflow-backups:*"
        }
      }
    }
  ]
}

Create the role:

aws iam create-role \
  --role-name GitHubActionsBackupRole \
  --assume-role-policy-document file://trust-policy.json

Attach S3 Permissions

Create s3-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:HeadObject"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR_BUCKET_NAME",
        "arn:aws:s3:::YOUR_BUCKET_NAME/*"
      ]
    }
  ]
}

aws iam put-role-policy \
  --role-name GitHubActionsBackupRole \
  --policy-name S3BackupAccess \
  --policy-document file://s3-policy.json

2. Configure GitHub Repository

Add the following secret to your repository:

• AWS_ROLE_ARN: The ARN of the IAM role (e.g., arn:aws:iam::123456789012:role/GitHubActionsBackupRole)

Optionally add a variable:

• AWS_REGION: AWS region (default: us-east-1)

GitHub Token Permissions

Repository Type	Token Required	Notes
Public repos	GITHUB_TOKEN (default)	Works automatically, no setup needed
Private repos	Personal Access Token (PAT)	Requires repo scope

For public repositories only, the default GITHUB_TOKEN works out of the box.

To backup private repositories, create a PAT with repo scope and add it as a secret:

• REPO_BACKUP_TOKEN: PAT with repo scope for private repository access

The workflow automatically uses REPO_BACKUP_TOKEN if available, falling back to GITHUB_TOKEN.

3. Create Configuration File

Create config.yml in your repository:

backup:
  enabled: true
  organization: "your-org"
  
  # Skip archived repositories (default: false)
  exclude_archived: true
  
  # Exact repository names (simple, no regex needed)
  repositories:
    - "my-important-repo"
    - "another.repo"
  
  # Regex patterns for matching multiple repos
  patterns:
    - "lecture-.*"      # Backup repos starting with "lecture-"
    - "quantecon-.*"    # Backup repos starting with "quantecon-"
  
  # Exclude specific repos by exact name
  exclude_repositories:
    - "testing-repo"
    - "codespaces-test"
  
  # Exclude repos matching patterns (applied after include matching)
  exclude_patterns:
    - ".*-test$"        # Exclude repos ending in -test
    - "test-.*"         # Exclude repos starting with test-
  
  # Metadata backup (GitHub issues, releases, etc.)
  # WARNING: Issues backup requires many API calls - see issue #3
  backup_metadata:
    issues: false       # Export issues with comments (disabled by default)
  
  s3:
    bucket: "your-backup-bucket"
    region: "us-east-1"
    prefix: "backups/"

4. Use the Workflow

The included workflow (.github/workflows/backup.yml) runs weekly and can be triggered manually:

name: Repository Backup
on:
  schedule:
    - cron: '0 2 * * 0'  # Weekly on Sunday at 2 AM UTC
  workflow_dispatch:

jobs:
  backup:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -r requirements.txt
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: us-east-1
      - run: python -m src.main --config config.yml --task backup
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Local Development

# Clone repository
git clone https://github.com/quantecon/workflow-backups.git
cd workflow-backups

# Set up environment
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Configure AWS credentials locally
export AWS_ACCESS_KEY_ID="your_key"
export AWS_SECRET_ACCESS_KEY="your_secret"
export GITHUB_TOKEN="your_token"

# Run backup
python -m src.main --config config.yml --task backup

# Run tests
pytest tests/

CLI Options

python -m src.main --help

Options:
  --config PATH       Path to configuration file (default: config.yml)
  --task {backup,report}  Task to run (default: backup)
  --organization ORG  Override organization from config
  --force            Force backup even if today's backup exists
  --verbose          Enable debug logging

Backup Storage Structure

Backups are stored in S3 with the following structure:

s3://bucket-name/
├── repo-name/
│   ├── repo-name-20251127.tar.gz
│   ├── repo-name-20251120.tar.gz
│   ├── repo-name-issues-20251127.json   # If issues backup enabled
│   └── repo-name-20251113.tar.gz
└── another-repo/
    └── another-repo-20251127.tar.gz

Each backup includes metadata:

• Repository full name
• Backup timestamp
• Default branch
• Archive size

Restoring a Backup

To restore a repository from a backup:

# 1. Download the backup from S3
aws s3 cp s3://bucket-name/repo-name/repo-name-20251127.tar.gz .

# 2. Extract the archive (contains a bare git mirror)
tar -xzf repo-name-20251127.tar.gz

# 3. Clone from the bare repo to create a working repository
git clone repo-name restored-repo

# 4. You now have a full working repository
cd restored-repo
git branch -a   # View all branches
git tag         # View all tags

The backup is a complete git mirror including all branches, tags, and full commit history.

Security

Read-Only Operations

This workflow is designed to be completely read-only with respect to GitHub repositories:

• No GitHub write operations: The code only uses read operations (get_organization(), get_repos(), get_issues(), repository/issue property access)
• No git push/commit: Only git clone --mirror is used (downloads only, never pushes)
• No repository modifications: Source repositories are never modified in any way

The only write operations are:

• S3 uploads: Backup archives are uploaded to your S3 bucket
• Issue updates: Backup reports are posted to issues in this repository (workflow-backups)

Token Permissions

Token	Required Scopes	Purpose
GITHUB_TOKEN	contents: read	Clone public repos
REPO_BACKUP_TOKEN (PAT)	Contents: Read-only	Clone private repos
AWS Role	s3:PutObject, s3:GetObject, s3:ListBucket	S3 backup storage

Best Practices

• Use OIDC authentication for AWS (no static credentials)
• Use fine-grained PATs with minimal scopes
• Store all tokens/credentials as GitHub Secrets
• Never commit credentials to the repository

Technology Stack

• Language: Python 3.9+
• Cloud Storage: AWS S3
• GitHub API: PyGithub
• AWS SDK: boto3
• Testing: pytest (88% code coverage)

License

MIT License - see LICENSE for details.

Support

• Open an issue in this repository
• Contact the QuantEcon development team