High-Availability Cluster Setup

This repository contains the configuration and setup scripts for a high-availability (HA) cluster implementation using multiple technologies including HAProxy, Corosync/Pacemaker, GlusterFS, and RAID storage.

Architecture Overview

The cluster setup consists of the following components:

Load Balancing & High Availability

• HAProxy: Load balancer with round-robin distribution
• Corosync/Pacemaker: Cluster resource manager for failover
• Virtual IP: Floating IP address (172.20.32.1) for transparent failover

Storage

• RAID 1: Software RAID for disk redundancy using mdadm
• GlusterFS: Distributed file system for data replication across nodes

Monitoring & Web Services

• Ganglia: Cluster monitoring and performance metrics
• Nginx: Reverse proxy for Ganglia web interface
• Docker/Docker Compose: Container orchestration for services

Network Configuration

• Backend Servers:
  • Server1: 192.168.32.121:80
  • Server2: 192.168.32.122:80
• Virtual IP: 172.20.32.1/24
• Ganglia Web Interface: Port 82

Directory Structure

setup/
├── docker-compose.yml          # Container services configuration
├── glusterfs-setup.sh         # GlusterFS installation and setup
├── glusterfs.sh               # GlusterFS operations script
├── haproxy-setup.sh           # HAProxy and Pacemaker setup
├── mount-glusterfs.sh         # GlusterFS mounting script
├── raid-setup.sh              # RAID 1 configuration
├── web-setup.sh               # Docker and web services setup
├── web.sh                     # Web services management
├── conf/
│   ├── corosync-haproxy.conf  # Corosync cluster configuration
│   ├── haproxy.cfg            # HAProxy load balancer config
│   └── nginx.conf             # Nginx reverse proxy config
└── ganglia/
    ├── ganglia-vhost.conf     # Ganglia virtual host
    ├── gmetad.conf            # Ganglia metadata daemon config
    ├── gmond.conf             # Ganglia monitoring daemon config
    └── monitor-setup.sh       # Ganglia monitoring setup

Installation Guide

Prerequisites

• Ubuntu/Debian-based Linux systems
• Root or sudo access
• Multiple nodes for cluster setup
• Additional storage devices for RAID (sdb, sdc)

1. RAID Setup

Set up RAID 1 for storage redundancy:

sudo bash setup/raid-setup.sh

This script will:

• Install mdadm
• Create RAID 1 array using /dev/sdb and /dev/sdc
• Format with ext4 filesystem
• Mount to /raid1
• Add to /etc/fstab for persistent mounting

2. GlusterFS Setup

Install and configure distributed storage:

sudo bash setup/glusterfs-setup.sh

This will:

• Install GlusterFS server
• Enable and start glusterd service
• Create /raid1 directory for GlusterFS mount

3. HAProxy and Pacemaker Setup

Configure load balancing and cluster management:

sudo bash setup/haproxy-setup.sh

This script configures:

• HAProxy load balancer
• Corosync cluster communication
• Pacemaker resource management
• Virtual IP resource (172.20.32.1)
• Resource colocation and constraints

4. Web Services Setup

Install Docker and prepare container services:

sudo bash setup/web-setup.sh

5. Monitoring Setup

Install Ganglia monitoring on each node:

sudo bash setup/ganglia/monitor-setup.sh

Service Management

Starting Services

# Start containerized services
docker-compose -f setup/docker-compose.yml up -d

# Check cluster status
sudo pcs status

# Check RAID status
cat /proc/mdstat

# Check GlusterFS status
sudo gluster volume status

Monitoring

• Ganglia Web Interface: http://your-server:82/ganglia/
• HAProxy Stats: Configure stats in haproxy.cfg if needed
• Cluster Status: sudo pcs status

Configuration Details

HAProxy Load Balancer

• Frontend: Binds to 0.0.0.0:80
• Backend: Round-robin between Server1 (192.168.32.121) and Server2 (192.168.32.122)
• Health Checks: Enabled for both backend servers

Pacemaker Resources

• haproxy: HAProxy service resource
• virtualip: Virtual IP resource (172.20.32.1)
• Constraints: HAProxy and Virtual IP are colocated
• STONITH: Disabled (stonith-enabled=false)

Storage Configuration

• RAID 1: /dev/md0 mounted at /raid1
• GlusterFS: Distributed across cluster nodes
• Filesystem: ext4 for RAID volume

Troubleshooting

Common Issues

1. Cluster Communication Problems

   sudo systemctl restart corosync pacemaker
   sudo pcs cluster start --all

2. RAID Array Issues

   # Check RAID status
   cat /proc/mdstat
   # Detailed array info
   sudo mdadm --detail /dev/md0

3. GlusterFS Mount Issues

   # Check GlusterFS processes
   sudo systemctl status glusterd
   # Restart if needed
   sudo systemctl restart glusterd

4. HAProxy Not Starting

   # Check configuration syntax
   sudo haproxy -c -f /etc/haproxy/haproxy.cfg
   # Check logs
   sudo journalctl -u haproxy

Log Locations

• HAProxy: /var/log/haproxy.log
• Pacemaker: /var/log/pacemaker/pacemaker.log
• Corosync: /var/log/corosync/corosync.log
• Docker: docker-compose logs

Maintenance

Regular Tasks

1. Monitor cluster health: sudo pcs status
2. Check RAID status: cat /proc/mdstat
3. Review logs: Check service logs regularly
4. Update containers: docker-compose pull && docker-compose up -d

Backup Procedures

1. Configuration backup: Copy all config files from setup/conf/
2. Data backup: Backup GlusterFS volumes
3. RAID backup: Consider external backup of /raid1

Security Considerations

• Ensure proper firewall rules for cluster communication
• Secure Ganglia web interface access
• Regular security updates for all components
• Consider SSL/TLS termination at HAProxy level

Performance Tuning

• Adjust HAProxy timeout values based on application needs
• Monitor Ganglia metrics for performance bottlenecks
• Tune GlusterFS volume settings for workload
• Optimize RAID chunk size if needed

License

This project is intended for educational and system administration purposes.