Creatures of Habit

Developers tend to re-use endpoints in different projects, this Python script fetches all public repositories from a GitHub user or organization, lists all files with specific extensions, and examines their contents to detect web routes using configurable regular expressions.

Features

• Fetches all public repositories from a GitHub user or organization
• Automatically detects whether the input is a user or organization
• Recursively searches all directories in repositories
• Lists files with configurable extensions (defined in settings.yaml)
• Examines file contents to detect web routes
• Configurable regex patterns for different frameworks (Flask, Laravel, Express.js, etc.)
• Handles pagination for users/organizations with many repositories
• Supports GitHub personal access tokens for higher rate limits
• Error handling for API requests
• Docker support for easy deployment

Installation

Option 1: Python Installation

1. Clone or download this repository
2. Install the required dependencies:

pip install -r requirements.txt

Option 2: Docker Installation

1. Clone or download this repository
2. Build the Docker image:

docker build -t github-route-detector .

Configuration

The script uses a settings.yaml file to configure file extensions and route detection patterns:

# File extensions to search for in repositories
file_extensions:
  - py
  - php
  - rs
  - java
  - js
  - aspx
  - cs
  - go
  - rb

# Regular expressions to detect web routes in files
route_patterns:
  # Python Flask routes
  - name: "Flask Route"
    pattern: "\\.route\\(\\s*(\\'|\\\")([^\\'\\\"]{1,})(\\'|\\\")\\)"
    framework: "Flask"
    language: "python"
  
  # Laravel routes
  - name: "Laravel Route"
    pattern: "Route::([a-z]{1,})\\((\\s*(\\'|\\\")([^\\'\\\"]{1,})(\\'|\\\"))"
    framework: "Laravel"
    language: "php"
  
  # Express.js routes
  - name: "Express.js Route"
    pattern: "\\.(get|post|put|delete|patch)\\(\\s*(\\'|\\\")([^\\'\\\"]{1,})(\\'|\\\")"
    framework: "Express.js"
    language: "javascript"

# GitHub API settings
github:
  # Maximum files per repository to fetch (to avoid rate limits)
  max_files_per_repo: 1000
  # Maximum repositories to process (0 = unlimited)
  max_repos: 0
  # Maximum file size to download and analyze (in bytes, 0 = unlimited)
  max_file_size: 1048576  # 1MB

You can modify this file to:

• Search for different file extensions
• Add custom regex patterns for other frameworks
• Adjust API limits and file size restrictions

Usage

Python Usage

Basic Usage

python github_org_contributors.py <username_or_organization>

Examples:

python github_org_contributors.py microsoft
python github_org_contributors.py octocat
python github_org_contributors.py torvalds

Advanced Usage

# With GitHub personal access token (recommended for large users/organizations)
python github_org_contributors.py microsoft --token YOUR_GITHUB_TOKEN

# Use custom settings file
python github_org_contributors.py microsoft --settings custom_settings.yaml

# Combine options
python github_org_contributors.py microsoft --token YOUR_GITHUB_TOKEN --detailed

Docker Usage

Basic Usage

# Build and run in one command
docker run --rm github-route-detector microsoft

# Or if you've already built the image
docker run --rm github-route-detector octocat

Advanced Usage

# Show detailed route information
docker run --rm github-route-detector microsoft --detailed

# With GitHub token (pass as environment variable)
docker run --rm -e GITHUB_TOKEN=your_token_here github-route-detector microsoft --token $GITHUB_TOKEN

# Use custom settings file (mount volume)
docker run --rm -v $(pwd)/custom_settings.yaml:/app/settings.yaml github-route-detector microsoft --settings settings.yaml

Using Docker Compose

# Basic usage
docker-compose run --rm github-repo-lister microsoft

# With detailed output
docker-compose run --rm github-repo-lister microsoft --detailed

# With GitHub token
docker-compose run --rm -e GITHUB_TOKEN=your_token_here github-repo-lister microsoft --token $GITHUB_TOKEN

Command Line Arguments

• name: GitHub username or organization name (required)
• --token: GitHub personal access token (optional, for higher rate limits)
• --detailed: Show detailed route information (optional)
• --settings: Path to settings file (default: settings.yaml)

GitHub Personal Access Token

For users/organizations with many repositories, it's recommended to use a GitHub personal access token to avoid rate limiting:

1. Go to GitHub Settings → Developer settings → Personal access tokens
2. Generate a new token with public_repo scope (for public repositories)
3. Use the token with the --token argument

Output

The script provides:

1. Endpoints - Endpoints that you might be able to use against other targets

The script includes regex patterns for detecting routes in:

• Flask (Python) - @app.route() decorators
• Laravel (PHP) - Route::get(), Route::post(), etc.
• Express.js (JavaScript) - app.get(), app.post(), etc.
• Django (Python) - path() URL patterns
• ASP.NET (C#) - [Route()] attributes
• Ruby on Rails (Ruby) - get route definitions
• Go HTTP (Go) - HandleFunc() and Handle() methods

You can easily add more patterns by editing the settings.yaml file.

Error Handling

The script handles various error scenarios:

• Invalid usernames or organization names
• Network connectivity issues
• API rate limiting (with appropriate delays)
• Missing repositories or files
• Invalid settings file format
• File decoding errors
• Invalid regex patterns

Rate Limiting

• Without token: 60 requests per hour (GitHub API limit)
• With token: 5000 requests per hour

For users/organizations with many repositories, using a personal access token is highly recommended.

Dependencies

• requests: HTTP library for API calls
• PyYAML: YAML file parsing
• argparse: Command line argument parsing
• Standard library modules: sys, typing, datetime, os, urllib, re, base64, collections

Docker Files

• Dockerfile: Container definition
• docker-compose.yml: Docker Compose configuration
• .dockerignore: Files to exclude from Docker build

License

This script is provided as-is for educational and utility purposes.