ON_DEMAND_BUNDLES.md

On-Demand Bundle Generation Setup

This guide explains how to set up and use the on-demand bundle generation feature.

Overview

The on-demand bundle generation feature allows users to:

• Enter any public GitHub repository URL on your website
• Automatically trigger indexing and bundle creation via GitHub Actions
• Download the generated .cgc bundle file

Architecture

User (Website) → API Endpoint → GitHub Actions → Bundle Created → GitHub Release

Setup Instructions

1. GitHub Personal Access Token

You need a GitHub Personal Access Token (PAT) with the following permissions:

Required Scopes:

• repo (Full control of private repositories)
• workflow (Update GitHub Action workflows)

Steps to create:

1. Go to GitHub Settings → Developer settings → Personal access tokens → Tokens (classic)
2. Click "Generate new token (classic)"
3. Select scopes: repo and workflow
4. Generate and copy the token

2. Configure Environment Variables

For Local Development:

Create /website/.env:

GITHUB_TOKEN="ghp_your_token_here"
GITHUB_REPOSITORY="CodeGraphContext/CodeGraphContext"

For Vercel Deployment:

1. Go to your Vercel project settings
2. Navigate to "Environment Variables"
3. Add the following variables:
   • GITHUB_TOKEN = your GitHub PAT
   • GITHUB_REPOSITORY = CodeGraphContext/CodeGraphContext

3. Test the Workflow

Manual Test (GitHub UI):

1. Go to your repository on GitHub
2. Click "Actions" tab
3. Select "Generate Bundle On-Demand" workflow
4. Click "Run workflow"
5. Enter a repository URL (e.g., https://github.com/psf/requests)
6. Click "Run workflow"
7. Wait 5-10 minutes
8. Check the "on-demand-bundles" release for the generated bundle

API Test (Local):

# Start the website locally
cd website
npm run dev

# In another terminal, trigger bundle generation
curl -X POST http://localhost:5173/api/trigger-bundle \
  -H "Content-Type: application/json" \
  -d '{"repoUrl": "https://github.com/psf/requests"}'

4. Deploy to Vercel

cd website
vercel --prod

Make sure environment variables are set in Vercel dashboard before deploying.

Usage

For Users:

1. Visit your website
2. Scroll to "Generate Custom Bundle" section
3. Enter a GitHub repository URL
4. Click "Generate Bundle"
5. Wait for completion (5-10 minutes)
6. Download the .cgc file
7. Run: cgc load bundle.cgc

API Endpoints:

Trigger Bundle Generation

POST /api/trigger-bundle
Content-Type: application/json

{
  "repoUrl": "https://github.com/owner/repo"
}

Response (Success):

{
  "status": "triggered",
  "message": "Bundle generation started",
  "repository": "owner/repo",
  "repo_size_mb": "15.23",
  "estimated_time": "5-10 minutes",
  "run_id": "12345",
  "run_url": "https://github.com/.../actions/runs/12345",
  "status_url": "/api/bundle-status?repo=owner/repo"
}

Response (Already Exists):

{
  "status": "exists",
  "message": "Bundle already exists",
  "bundle": {
    "repo": "owner/repo",
    "bundle_name": "repo-v1.0.0-abc123.cgc",
    "download_url": "https://github.com/.../releases/download/..."
  }
}

Check Bundle Status

GET /api/bundle-status?run_id=12345
# or
GET /api/bundle-status?repo=owner/repo

Response:

{
  "status": "completed",
  "conclusion": "success",
  "progress": 100,
  "run_url": "https://github.com/.../actions/runs/12345"
}

Troubleshooting

Issue: "Failed to trigger workflow"

Cause: Invalid GitHub token or insufficient permissions

Solution:

1. Verify GITHUB_TOKEN is set correctly
2. Ensure token has repo and workflow scopes
3. Check token hasn't expired

Issue: "Repository too large"

Cause: Repository exceeds 1GB size limit

Solution:

• This is a safety limit to prevent long-running workflows
• Consider increasing the limit in /website/api/trigger-bundle.ts if needed
• Or add the repository to the weekly pre-indexed list instead

Issue: "Bundle generation failed"

Cause: Indexing error or unsupported language

Solution:

1. Check the GitHub Actions workflow logs
2. Verify the repository has supported languages (Python, JavaScript, etc.)
3. Check if the repository has unusual structure

Issue: Workflow doesn't start

Cause: GitHub Actions quota exceeded or workflow disabled

Solution:

1. Check GitHub Actions usage in repository settings
2. Ensure workflow file is in main branch
3. Verify workflow is enabled in Actions settings

Limitations

• Repository Size: Max 1GB (configurable)
• Processing Time: 5-30 minutes depending on size
• GitHub Actions Quota: 2000 minutes/month (free tier)
• Public Repositories Only: Private repos require user's own token
• Bundle Retention: 30 days (configurable)

Future Enhancements

• Email notifications when bundle is ready
• Priority queue for premium users
• Support for private repositories
• Batch generation (multiple repos at once)
• Automatic cleanup of old bundles
• Bundle analytics and popularity tracking

Security Notes

• Never commit .env file with real tokens
• Use Vercel environment variables for production
• GitHub token should have minimal required permissions
• Consider implementing rate limiting to prevent abuse
• Monitor GitHub Actions usage to avoid quota exhaustion

Support

If you encounter issues:

1. Check the GitHub Actions workflow logs
2. Review the API response error messages
3. Open an issue on GitHub with details

---

Last Updated: 2026-01-21