Streaming Service

A high-performance, production-ready video/audio streaming service built in Go, designed for 100M+ RPS with HLS adaptive bitrate streaming.

🎯 Overview

This streaming platform provides enterprise-grade video delivery with:

• HLS Adaptive Streaming - Automatic quality adjustment based on viewer bandwidth
• Multi-Bitrate Encoding - 1080p, 720p, 480p, 360p renditions
• Audio Support - Standalone audio streaming and video audio extraction
• AWS Native - S3 storage, DynamoDB metadata, CloudFront CDN
• Kubernetes Ready - Terraform-managed infrastructure for auto-scaling

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                              CLIENT LAYER                                    │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐                     │
│  │   Web    │  │  Mobile  │  │ Smart TV │  │   OTT    │                     │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘                     │
└───────┼─────────────┼─────────────┼─────────────┼───────────────────────────┘
        │             │             │             │
        └─────────────┴──────┬──────┴─────────────┘
                             ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           CDN / EDGE LAYER                                   │
│                        ┌─────────────────┐                                   │
│                        │   CloudFront    │  ◄── Cached HLS Segments         │
│                        └────────┬────────┘                                   │
└─────────────────────────────────┼───────────────────────────────────────────┘
                                  ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         API GATEWAY LAYER                                    │
│  ┌───────────────┐  ┌───────────────┐  ┌───────────────┐                    │
│  │ Load Balancer │──│  API Server   │──│     Auth      │                    │
│  └───────────────┘  └───────┬───────┘  └───────────────┘                    │
│                             │                                                │
└─────────────────────────────┼───────────────────────────────────────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                        MICROSERVICES LAYER                                   │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │   Upload    │  │  Transcode  │  │   Stream    │  │    Audio    │         │
│  │   Service   │  │   Service   │  │   Service   │  │   Service   │         │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘         │
└─────────┼────────────────┼────────────────┼────────────────┼────────────────┘
          │                │                │                │
          ▼                ▼                ▼                ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         DATA & PROCESSING LAYER                              │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │     S3      │  │   FFMPEG    │  │  DynamoDB   │  │    Redis    │         │
│  │  (Storage)  │  │  (Encoding) │  │ (Metadata)  │  │   (Queue)   │         │
│  └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘         │
└─────────────────────────────────────────────────────────────────────────────┘

📁 Project Structure

streaming-service/
├── cmd/
│   ├── api/                 # API server entrypoint
│   └── worker/              # Transcoding worker entrypoint
├── internal/
│   ├── api/                 # HTTP handlers & Chi router
│   ├── config/              # Viper configuration management
│   ├── domain/              # Business entities (Media, Video, Audio)
│   ├── media/
│   │   ├── ffmpeg/          # FFMPEG video/audio processors
│   │   └── processor/       # Factory & Strategy pattern implementations
│   ├── queue/               # Redis job queue with priority support
│   ├── repository/
│   │   ├── dynamodb/        # Metadata CRUD operations
│   │   └── s3/              # Object storage with presigned URLs
│   └── service/
│       ├── audio/           # Audio extraction & processing
│       ├── stream/          # Playback URL generation
│       ├── transcode/       # HLS transcoding pipeline
│       └── upload/          # File upload handling
├── pkg/
│   └── logger/              # Zap structured logging
├── deployments/
│   ├── docker/              # Multi-stage Dockerfiles
│   ├── kubernetes/          # K8s manifests
│   └── terraform/           # Infrastructure as Code
└── config.yaml              # Application configuration

🎨 Design Patterns

Factory Pattern

Creates appropriate media processors based on content type:

processor := factory.CreateProcessor(domain.MediaTypeVideo) // or MediaTypeAudio

Strategy Pattern

Interchangeable transcoding strategies for different output formats:

executor.AddStrategy(NewHLSTranscodeStrategy(profile1080p))
executor.AddStrategy(NewHLSTranscodeStrategy(profile720p))
executor.Execute(ctx, input, outputDir, cmdExecutor)

Worker Pool Pattern

Concurrent job processing with configurable parallelism:

worker := transcode.NewWorker(queue, service, concurrency, logger)
worker.Start(ctx)

🚀 Quick Start

Prerequisites

• Go 1.22+
• Docker & Docker Compose
• FFMPEG (for local development)
• AWS CLI (configured)

Local Development

# Clone repository
git clone https://github.com/ichbingautam/streaming-service.git
cd streaming-service

# Install dependencies
go mod download

# Start infrastructure (Redis + LocalStack)
docker-compose up -d redis localstack

# Run API server
make run-api

# Run transcoding worker (in another terminal)
make run-worker

Docker Compose (Full Stack)

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f api worker

📡 API Endpoints

Method	Endpoint	Description
GET	/health	Health check
GET	/ready	Readiness probe
POST	/api/v1/upload	Upload media file (multipart)
POST	/api/v1/upload/presign	Get presigned upload URL
POST	/api/v1/upload/{id}/confirm	Confirm presigned upload
GET	/api/v1/media	List user's media
GET	/api/v1/media/{id}	Get media details
DELETE	/api/v1/media/{id}	Delete media
GET	/api/v1/media/{id}/playback	Get HLS playback URL

Example: Upload Video

curl -X POST http://localhost:8080/api/v1/upload \
  -H "X-User-ID: user123" \
  -F "file=@video.mp4" \
  -F "title=My Video" \
  -F "description=Sample video"

Example: Get Playback URL

curl http://localhost:8080/api/v1/media/{media_id}/playback \
  -H "X-User-ID: user123"

⚙️ Configuration

Configuration via config.yaml or environment variables (prefix: STREAM_):

app:
  name: streaming-service
  version: 1.0.0
  environment: production

server:
  port: 8080
  readtimeout: 30s
  writetimeout: 30s

aws:
  region: us-east-1
  s3rawbucket: streaming-raw-media
  s3processedbucket: streaming-processed-media
  dynamodbtable: video-metadata
  cloudfrontdomain: d1234.cloudfront.net

redis:
  host: redis
  port: 6379

ffmpeg:
  binarypath: ffmpeg
  segmentduration: 6
  profiles:
    - name: "1080p"
      width: 1920
      height: 1080
      videobitrate: "5000k"
      audiobitrate: "192k"

worker:
  concurrency: 4
  jobtimeout: 30m

log:
  level: info
  format: json

☸️ Kubernetes Deployment

Using Terraform

cd deployments/terraform

# Initialize Terraform
terraform init

# Review plan
terraform plan -var="environment=production"

# Apply infrastructure
terraform apply -var="environment=production"

Using kubectl

# Create namespace
kubectl create namespace streaming

# Apply manifests
kubectl apply -f deployments/kubernetes/ -n streaming

# Check deployment
kubectl get pods -n streaming

🔧 Development

# Run tests
make test

# Run with coverage
make test-coverage

# Lint code
make lint

# Build binaries
make build

# Build Docker images
make docker-build

📊 Performance Targets

Metric	Target
API Latency (p99)	< 50ms
Transcode Time (1min video)	< 60s
CDN Cache Hit Ratio	> 95%
Concurrent Streams	100M+

🛠️ Technology Stack

Component	Technology
Language	Go 1.22+
HTTP Router	Chi
Configuration	Viper
Logging	Zap
Media Processing	FFMPEG
Storage	AWS S3
Metadata	AWS DynamoDB
CDN	AWS CloudFront
Queue	Redis
Containers	Docker
Orchestration	Kubernetes
IaC	Terraform

📄 License

MIT License - see LICENSE for details.

🤝 Contributing

1. Fork the repository
2. Create feature branch (git checkout -b feature/amazing)
3. Commit changes (git commit -m 'Add amazing feature')
4. Push to branch (git push origin feature/amazing)
5. Open Pull Request