Skip to content

Commit 969fd12

Browse files
mattmattoxclaude
mattmattox
and
claude
committed
Complete multiple high-priority tasks: metadata recovery, performance optimization, and multi-server docs
- ST-02: Metadata Recovery Implementation - Created metadata-recovery command-line tool for reconstructing metadata from backups - Supports scanning both local filesystem and S3 storage - Handles duplicate detection and reconciliation - Added comprehensive test suite - Integrated into build process (Makefile and Dockerfile) - ST-04: Metadata Performance Optimization - Added comprehensive database indexes for common query patterns - Implemented paginated API endpoint with configurable page sizes - Created optimized query methods using database aggregations - Added performance benchmarks demonstrating improvements - Documented optimization strategies and best practices - ST-07: Multi-Server Documentation - Created comprehensive multi-server configuration guide - Added three example configurations (production, sharded, geographic) - Developed extensive troubleshooting guide with solutions - Documented best practices for security and performance Also includes: - Enhanced error logging with modal dialogs in UI - Server statistics dashboard and management page - MySQL dump options configuration UI - Enhanced filtering with date ranges and search - Metadata persistence testing with comprehensive test suite 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent cbea93b commit 969fd12

File tree

117 files changed

+24553
-1
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

117 files changed

+24553
-1
lines changed

‎.github/workflows/pipeline.yml

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
name: GoSQLGuard CI/CD
2+
3+
on:
4+
push:
5+
branches: [ main, master ]
6+
pull_request:
7+
branches: [ main, master ]
8+
9+
jobs:
10+
Test:
11+
runs-on: ubuntu-latest
12+
13+
steps:
14+
- name: Checkout code
15+
uses: actions/checkout@v4
16+
17+
- name: Set up Go
18+
uses: actions/setup-go@v4
19+
with:
20+
go-version: "1.22"
21+
22+
- name: Cache Go modules
23+
uses: actions/cache@v4
24+
with:
25+
path: |
26+
~/.cache/go-build
27+
~/go/pkg/mod
28+
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
29+
restore-keys: |
30+
${{ runner.os }}-go-
31+
32+
- name: Install static analysis tools
33+
run: |
34+
go install golang.org/x/lint/golint@latest
35+
go install honnef.co/go/tools/cmd/staticcheck@latest
36+
go install github.com/securego/gosec/v2/cmd/gosec@latest
37+
go install github.com/psampaz/go-mod-outdated@latest
38+
go install github.com/remyoudompheng/go-misc/deadcode@latest
39+
40+
- name: Go static analysis
41+
run: |
42+
golint ./...
43+
staticcheck ./...
44+
go vet ./...
45+
deadcode .
46+
47+
- name: Dependency management
48+
run: |
49+
go mod vendor
50+
go mod verify
51+
go mod tidy
52+
53+
- name: Security scanning
54+
run: |
55+
gosec ./...
56+
57+
Build:
58+
needs: Test
59+
runs-on: ubuntu-latest
60+
61+
steps:
62+
- uses: actions/checkout@v4
63+
64+
- name: Set up Docker Buildx
65+
uses: docker/setup-buildx-action@v3
66+
67+
- name: Login to DockerHub
68+
uses: docker/login-action@v3
69+
with:
70+
username: ${{ secrets.DOCKER_USERNAME }}
71+
password: ${{ secrets.DOCKER_PASSWORD }}
72+
73+
- name: Docker build and push
74+
run: |
75+
docker buildx build \
76+
--platform linux/amd64 \
77+
--pull \
78+
--build-arg VERSION=v${{ github.run_number }} \
79+
--build-arg GIT_COMMIT=${{ github.sha }} \
80+
--build-arg BUILD_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ") \
81+
--cache-from supporttools/gosqlguard:latest \
82+
-t supporttools/gosqlguard:"0.1.${{ github.run_number }}" \
83+
-t supporttools/gosqlguard:latest \
84+
--push \
85+
-f Dockerfile .
86+
87+
Publish:
88+
runs-on: ubuntu-latest
89+
needs:
90+
- Build
91+
92+
steps:
93+
- name: Checkout repository
94+
uses: actions/checkout@v4
95+
96+
- name: Set up Helm
97+
uses: azure/setup-helm@v4.2.0
98+
99+
- name: Helm Lint
100+
run: helm lint charts/gosqlguard/
101+
102+
- name: Package Helm chart
103+
run: |
104+
export CHART_VERSION="0.1.${{ github.run_number }}"
105+
export APP_VERSION="0.1.${{ github.run_number }}"
106+
export IMAGE_TAG="0.1.${{ github.run_number }}"
107+
echo "CHART_VERSION=${CHART_VERSION}"
108+
echo "APP_VERSION=${APP_VERSION}"
109+
envsubst < charts/gosqlguard/Chart.yaml.template > charts/gosqlguard/Chart.yaml
110+
envsubst < charts/gosqlguard/values.yaml.template > charts/gosqlguard/values.yaml
111+
helm package charts/gosqlguard --destination helm/repo
112+
113+
- name: Checkout helm-chart repository
114+
uses: actions/checkout@v4
115+
with:
116+
repository: supporttools/helm-chart
117+
path: helm-chart
118+
token: ${{ secrets.BOT_TOKEN }}
119+
120+
- name: Configure Git
121+
run: |
122+
git config --global user.email "github-action@users.noreply.github.com"
123+
git config --global user.name "GitHub Action"
124+
125+
- name: Update Helm repository
126+
run: |
127+
mkdir -p helm/repo
128+
cp helm/repo/gosqlguard-*.tgz helm-chart/
129+
cd helm-chart
130+
helm repo index . --url https://charts.support.tools/
131+
git add .
132+
git commit -m "Update Helm chart for gosqlguard"
133+
git push

‎.gitignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,8 @@ go.work.sum
2323

2424
# env file
2525
.env
26+
27+
# Backup directory for testing
28+
data/*
29+
backups/*
30+
configs/*

‎.rc-version

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
5

‎CLAUDE.md

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Project Overview
6+
7+
GoSQLGuard is a Go-based database backup management tool designed for Kubernetes environments. It supports MySQL and PostgreSQL databases with flexible scheduling, retention policies, and multi-destination storage (local filesystem and S3).
8+
9+
## Architecture
10+
11+
The codebase follows a modular architecture with clear separation of concerns:
12+
13+
- **Entry Point**: `main.go` initializes configuration, starts the scheduler, and launches the admin server
14+
- **Core Components**:
15+
- `pkg/backup/`: Database backup logic with provider-based architecture for MySQL/PostgreSQL
16+
- `pkg/scheduler/`: Cron-based scheduling system for automated backups
17+
- `pkg/storage/`: Storage providers (local filesystem and S3)
18+
- `pkg/metadata/`: Metadata tracking (file-based or MySQL-backed)
19+
- `pkg/adminserver/`: Web UI and API endpoints
20+
- `pkg/config/`: Configuration parsing with environment variable substitution
21+
22+
## Development Commands
23+
24+
### Build and Release
25+
```bash
26+
make build # Build Docker image with auto-incrementing RC version
27+
make push # Push to registry
28+
make release # Build and push (full release)
29+
make increment-rc # Just increment RC number
30+
make help # View current version info
31+
```
32+
33+
### Local Development
34+
```bash
35+
# Start development environment
36+
docker-compose up -d
37+
38+
# Run all tests
39+
./scripts/run-tests.sh
40+
41+
# Run specific database tests
42+
./scripts/run-mysql-tests.sh
43+
./scripts/run-postgres-tests.sh
44+
45+
# Access services
46+
# Admin UI: http://localhost:8888
47+
# MinIO Console: http://localhost:9001 (minioadmin/minioadmin)
48+
```
49+
50+
### Testing
51+
The project uses Go's built-in testing framework. Integration tests are located in `pkg/test/integration/` and require a running database instance.
52+
53+
## Configuration Patterns
54+
55+
The application uses YAML configuration with environment variable substitution (`${VAR}` syntax). Key configuration sections:
56+
57+
- `mysql`/`postgresql`: Database connection settings
58+
- `local`: Local storage configuration
59+
- `s3`: S3-compatible storage configuration
60+
- `backupTypes`: Defines backup schedules and retention policies
61+
- `metadata_database`: Optional MySQL-based metadata storage
62+
63+
## API Design
64+
65+
The admin server exposes:
66+
- Web UI at `/` with server-side rendered HTML templates
67+
- API endpoints under `/api/` for backup operations
68+
- Prometheus metrics at `/metrics`
69+
- Health check at `/health`
70+
71+
## Key Development Patterns
72+
73+
1. **Provider Pattern**: Database operations use a provider interface allowing easy extension for new database types
74+
2. **Configuration Validation**: All configuration is validated at startup with clear error messages
75+
3. **Error Handling**: Consistent error wrapping with context for debugging
76+
4. **Metrics**: All operations emit Prometheus metrics for monitoring
77+
5. **Logging**: Structured logging with configurable debug mode
78+
79+
## Important Files
80+
81+
- `pkg/config/config.go`: Central configuration structure and validation
82+
- `pkg/backup/backup.go`: Core backup orchestration logic
83+
- `pkg/scheduler/scheduler.go`: Cron job management
84+
- `pkg/pages/*.go`: HTML template rendering for admin UI
85+
- `example-configs/`: Reference configurations for common scenarios

‎Dockerfile

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Use golang alpine image as the builder stage
2+
FROM golang:1.24.2-alpine AS builder
3+
4+
# Install git and other necessary tools
5+
RUN apk update && apk add --no-cache git bash mysql-client postgresql-client
6+
7+
# Set the Current Working Directory inside the container
8+
WORKDIR /src
9+
10+
# Copy go.mod and go.sum files first to leverage Docker cache
11+
COPY go.mod go.sum ./
12+
13+
# Download dependencies with module and build cache
14+
RUN --mount=type=cache,target=/go/pkg/mod \
15+
go mod download
16+
17+
# Copy the rest of the application source code
18+
COPY . .
19+
20+
# Build arguments for versioning
21+
ARG VERSION
22+
ARG GIT_COMMIT
23+
ARG BUILD_DATE
24+
25+
# Build the Go app with static linking and versioning information
26+
RUN --mount=type=cache,target=/go/pkg/mod \
27+
--mount=type=cache,target=/root/.cache/go-build \
28+
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
29+
-ldflags "-w -s -X github.com/supporttools/GoSQLGuard/pkg/version.Version=${VERSION} \
30+
-X github.com/supporttools/GoSQLGuard/pkg/version.GitCommit=${GIT_COMMIT} \
31+
-X github.com/supporttools/GoSQLGuard/pkg/version.BuildTime=${BUILD_DATE}" \
32+
-o /gosqlguard
33+
34+
# Build the metadata recovery tool
35+
RUN --mount=type=cache,target=/go/pkg/mod \
36+
--mount=type=cache,target=/root/.cache/go-build \
37+
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
38+
-ldflags "-w -s -X github.com/supporttools/GoSQLGuard/pkg/version.Version=${VERSION} \
39+
-X github.com/supporttools/GoSQLGuard/pkg/version.GitCommit=${GIT_COMMIT} \
40+
-X github.com/supporttools/GoSQLGuard/pkg/version.BuildTime=${BUILD_DATE}" \
41+
-o /metadata-recovery ./cmd/metadata-recovery/main.go
42+
43+
# Use Ubuntu as the runtime base image
44+
FROM ubuntu:22.04
45+
46+
# Install required packages for backup operations including MySQL 8.0 client
47+
# Set noninteractive to avoid tzdata configuration prompts
48+
ENV DEBIAN_FRONTEND=noninteractive
49+
50+
RUN apt-get update && apt-get install -y \
51+
mysql-client-core-8.0 \
52+
postgresql-client \
53+
ca-certificates \
54+
openssl \
55+
&& rm -rf /var/lib/apt/lists/*
56+
57+
# Container metadata
58+
LABEL org.opencontainers.image.title="GoSQLGuard" \
59+
org.opencontainers.image.description="A tool for automating database backups and retention with support for MySQL and PostgreSQL." \
60+
org.opencontainers.image.source="https://github.com/supporttools/GoSQLGuard" \
61+
org.opencontainers.image.vendor="SupportTools" \
62+
org.opencontainers.image.licenses="MIT"
63+
64+
# Copy the binaries from the builder stage
65+
COPY --from=builder /gosqlguard /usr/local/bin/gosqlguard
66+
COPY --from=builder /metadata-recovery /usr/local/bin/metadata-recovery
67+
68+
# Set working directory
69+
WORKDIR /app
70+
71+
# Create directories for configuration and backups
72+
RUN mkdir -p /app/config /app/data/backups/hourly
73+
74+
# Copy configuration file
75+
COPY config.yaml /app/config/
76+
77+
# Set permissions on directories
78+
RUN chmod -R 755 /app/data
79+
80+
# Set the entrypoint
81+
ENTRYPOINT ["/usr/local/bin/gosqlguard"]

0 commit comments

Comments
 (0)