Skip to content

Commit cb1d0a2

Browse files
owineowine
committed
Merge branch 'feature/enhanced-stack-removal-detection'
2 parents c60bd90 + 9763380 commit cb1d0a2

File tree

3 files changed

+430
-18
lines changed

3 files changed

+430
-18
lines changed

‎.github/workflows/deploy.yml‎

Lines changed: 206 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,7 @@ jobs:
105105
outputs:
106106
previous_sha: ${{ steps.backup.outputs.previous_sha }}
107107
deployment_needed: ${{ steps.backup.outputs.deployment_needed }}
108+
deleted_files: ${{ steps.changed-files.outputs.deleted_files }}
108109
deploy_status: ${{ steps.deploy.outcome }}
109110
health_status: ${{ steps.health.outcome }}
110111
cleanup_status: ${{ steps.cleanup.outcome }}
@@ -362,6 +363,44 @@ jobs:
362363
363364
echo "✅ SSH connection optimization configured"
364365
366+
- name: Checkout repository for change detection
367+
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
368+
with:
369+
fetch-depth: 0 # Fetch full history for accurate change detection
370+
371+
- name: Determine previous deployment SHA
372+
id: determine-previous
373+
run: |
374+
# Use retry mechanism for SSH connection
375+
source /tmp/retry.sh
376+
377+
# Get current deployment SHA with error handling
378+
echo "🔍 Checking current deployment SHA for change detection..."
379+
if CURRENT_SHA=$(ssh_retry 3 5 "ssh -o 'StrictHostKeyChecking no' ${{ secrets.SSH_USER }}@${{ secrets.SSH_HOST }} 'cd /opt/compose && git rev-parse HEAD 2>/dev/null'"); then
380+
# Validate SHA format
381+
if [[ "$CURRENT_SHA" =~ ^[a-fA-F0-9]{40}$ ]]; then
382+
echo "✅ Current deployed SHA: $CURRENT_SHA"
383+
echo "previous_sha=$CURRENT_SHA" >> "$GITHUB_OUTPUT"
384+
else
385+
echo "⚠️ Invalid SHA format from server: $CURRENT_SHA"
386+
echo "⚠️ Using HEAD^ as fallback for change detection"
387+
echo "previous_sha=HEAD^" >> "$GITHUB_OUTPUT"
388+
fi
389+
else
390+
echo "⚠️ Could not retrieve current deployment SHA - using HEAD^ for change detection"
391+
echo "previous_sha=HEAD^" >> "$GITHUB_OUTPUT"
392+
fi
393+
394+
- name: Get changed files for removal detection
395+
id: changed-files
396+
# Pin to commit SHA instead of tag for security (CVE-2025-30066)
397+
# This is v47.0.0 released on 2024-09-13
398+
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62
399+
with:
400+
json: true
401+
sha: ${{ inputs.target-ref }}
402+
base_sha: ${{ steps.determine-previous.outputs.previous_sha }}
403+
365404
- name: Store current deployment for rollback
366405
id: backup
367406
run: |
@@ -414,18 +453,29 @@ jobs:
414453
fi
415454
echo "::endgroup::"
416455
456+
# ================================================================
417457
# STACK REMOVAL DETECTION AND CLEANUP
418-
# This step runs before repository update to clean up Docker containers
419-
# for stacks that have been completely removed from the repository.
458+
# ================================================================
459+
# Automatically detect and clean up Docker stacks that have been
460+
# removed from the repository using three independent detection methods.
461+
#
462+
# Detection Methods:
463+
# 1. Git Diff: Compares current deployed SHA vs target SHA
464+
# 2. Tree Comparison: Compares target commit tree vs server filesystem
465+
# (catches removals from previous undeployed commits)
466+
# 3. Discovery Analysis: Analyzes tj-actions/changed-files output
467+
# (validates removals from GitHub perspective)
420468
#
421469
# Process:
422-
# 1. Compare current deployed SHA with target SHA using git diff
423-
# 2. Identify deleted */compose.yaml files (one level deep only)
424-
# 3. Run 'docker compose down' for each removed stack
425-
# 4. Fail deployment if any cleanup fails (fail-safe)
426-
# 5. Send Discord notification listing removed stacks
470+
# 1. Run all three detection methods independently on deployment server
471+
# 2. Fail deployment if ANY detection method encounters errors (fail-safe)
472+
# 3. Aggregate results using union approach (remove anything found by any method)
473+
# 4. Deduplicate and validate stack names
474+
# 5. Run 'docker compose down' for each removed stack
475+
# 6. Fail deployment if any cleanup fails
476+
# 7. Send Discord notification listing removed stacks
427477
#
428-
# Design: docs/plans/2025-12-03-stack-removal-detection-design.md
478+
# Design: docs/plans/2025-12-06-enhanced-stack-removal-detection-design.md
429479

430480
- name: Detect and clean up removed stacks
431481
id: cleanup-removed
@@ -435,12 +485,12 @@ jobs:
435485
# Source retry functions
436486
source /tmp/retry.sh
437487
438-
# === DETECTION FUNCTION ===
439-
# Purpose: Detect removed stacks by comparing git commits
488+
# === DETECTION FUNCTION: GIT DIFF ===
489+
# Purpose: Detect stacks removed between two git commits
440490
# Inputs: $1=current_sha, $2=target_ref
441491
# Output: Newline-separated list of stack names (stdout)
442492
# Returns: 0 on success, 1 on error
443-
detect_removed_stacks() {
493+
detect_removed_stacks_gitdiff() {
444494
local current_sha="$1"
445495
local target_ref="$2"
446496
@@ -487,6 +537,109 @@ jobs:
487537
echo "$detect_script" | ssh_retry 3 5 "ssh -o \"StrictHostKeyChecking no\" deployment-server /bin/bash -s \"$current_sha\" \"$target_ref\""
488538
}
489539
540+
# === DETECTION FUNCTION: TREE COMPARISON ===
541+
# Purpose: Detect stacks on server filesystem missing from target commit tree
542+
# Inputs: $1=target_ref
543+
# Output: Newline-separated list of stack names (stdout)
544+
# Returns: 0 on success, 1 on error
545+
detect_removed_stacks_tree() {
546+
local target_ref="$1"
547+
548+
# Build detection script
549+
local detect_script
550+
detect_script=$(cat << 'DETECT_TREE_EOF'
551+
set -e
552+
TARGET_REF="$1"
553+
554+
cd /opt/compose
555+
556+
# Fetch target ref to ensure we have it
557+
if ! git fetch origin "$TARGET_REF" 2>/dev/null; then
558+
echo "⚠️ Failed to fetch target ref, trying general fetch..." >&2
559+
if ! git fetch 2>/dev/null; then
560+
echo "::error::Failed to fetch repository updates" >&2
561+
exit 1
562+
fi
563+
fi
564+
565+
# Resolve target ref to SHA
566+
TARGET_SHA=$(git rev-parse "$TARGET_REF" 2>/dev/null || echo "$TARGET_REF")
567+
568+
# Validate target SHA exists
569+
if ! git cat-file -e "$TARGET_SHA" 2>/dev/null; then
570+
echo "::error::Target SHA $TARGET_SHA not found in repository" >&2
571+
exit 1
572+
fi
573+
574+
# Get directories in target commit (one level deep, directories only)
575+
COMMIT_DIRS=$(git ls-tree --name-only "$TARGET_SHA" 2>/dev/null | sort)
576+
577+
# Get directories on server filesystem (exclude .git and hidden dirs)
578+
SERVER_DIRS=$(find /opt/compose -maxdepth 1 -mindepth 1 -type d ! -name '.*' -exec basename {} \; 2>/dev/null | sort)
579+
580+
# Find directories on server but not in commit
581+
MISSING_IN_COMMIT=$(comm -13 <(echo "$COMMIT_DIRS") <(echo "$SERVER_DIRS"))
582+
583+
# Filter for directories with compose.yaml files
584+
for dir in $MISSING_IN_COMMIT; do
585+
if [ -f "/opt/compose/$dir/compose.yaml" ]; then
586+
echo "$dir"
587+
fi
588+
done
589+
DETECT_TREE_EOF
590+
)
591+
592+
# Execute detection script on remote server
593+
echo "$detect_script" | ssh_retry 3 5 "ssh -o \"StrictHostKeyChecking no\" deployment-server /bin/bash -s \"$target_ref\""
594+
}
595+
596+
# === DETECTION FUNCTION: DISCOVERY ANALYSIS ===
597+
# Purpose: Analyze deleted files from tj-actions/changed-files output
598+
# Inputs: $1=deleted_files_json (JSON array from tj-actions/changed-files)
599+
# Output: Newline-separated list of stack names (stdout)
600+
# Returns: 0 on success, 1 on error
601+
detect_removed_stacks_discovery() {
602+
local deleted_files_json="$1"
603+
604+
# Build detection script
605+
local detect_script
606+
detect_script=$(cat << 'DETECT_DISCOVERY_EOF'
607+
set -e
608+
DELETED_FILES_JSON="$1"
609+
610+
# Parse JSON array and filter for compose.yaml deletions
611+
# Pattern: one level deep only (stack-name/compose.yaml)
612+
echo "$DELETED_FILES_JSON" | jq -r '.[]' 2>/dev/null | \
613+
grep -E '^[^/]+/compose\.yaml$' | \
614+
sed 's|/compose\.yaml||' || echo ""
615+
DETECT_DISCOVERY_EOF
616+
)
617+
618+
# Execute detection script on remote server
619+
echo "$detect_script" | ssh_retry 3 5 "ssh -o \"StrictHostKeyChecking no\" deployment-server /bin/bash -s \"$deleted_files_json\""
620+
}
621+
622+
# === AGGREGATION FUNCTION ===
623+
# Purpose: Merge and deduplicate results from all three detection methods
624+
# Inputs: $1=gitdiff_stacks, $2=tree_stacks, $3=discovery_stacks (newline-separated lists)
625+
# Output: Deduplicated newline-separated list of stack names (stdout)
626+
# Returns: 0 on success (empty string if all inputs empty, not an error)
627+
aggregate_removed_stacks() {
628+
local gitdiff_stacks="$1"
629+
local tree_stacks="$2"
630+
local discovery_stacks="$3"
631+
632+
# Concatenate all three lists, remove empty lines, sort and deduplicate
633+
{
634+
echo "$gitdiff_stacks"
635+
echo "$tree_stacks"
636+
echo "$discovery_stacks"
637+
} | \
638+
grep -v '^$' | \
639+
sort -u | \
640+
grep -E '^[a-zA-Z0-9_-]+$' || echo ""
641+
}
642+
490643
# === CLEANUP FUNCTION ===
491644
# Purpose: Clean up a single removed stack using docker compose down
492645
# Inputs: $1=stack_name
@@ -555,14 +708,51 @@ jobs:
555708
echo " Target: $TARGET_REF"
556709
echo "🔍 Checking for removed stacks..."
557710
558-
# Execute detection
559-
REMOVED_STACKS=$(detect_removed_stacks "$CURRENT_SHA" "$TARGET_REF")
560-
DETECTION_EXIT=$?
711+
# Read deleted files from discover-stacks step
712+
DELETED_FILES='${{ steps.changed-files.outputs.deleted_files }}'
713+
714+
echo "🔍 Running three detection methods..."
715+
716+
# Execute all three detection methods independently
717+
echo " 1. Git diff detection (commit comparison)..."
718+
GITDIFF_STACKS=$(detect_removed_stacks_gitdiff "$CURRENT_SHA" "$TARGET_REF") || GITDIFF_EXIT=$?
719+
720+
echo " 2. Tree comparison detection (filesystem vs commit)..."
721+
TREE_STACKS=$(detect_removed_stacks_tree "$TARGET_REF") || TREE_EXIT=$?
722+
723+
echo " 3. Discovery analysis detection (changed files)..."
724+
DISCOVERY_STACKS=$(detect_removed_stacks_discovery "$DELETED_FILES") || DISCOVERY_EXIT=$?
561725
562-
if [ $DETECTION_EXIT -ne 0 ]; then
563-
echo "::error::Failed to detect removed stacks (exit code: $DETECTION_EXIT)"
726+
# Fail deployment if any detection method failed (fail-safe)
727+
if [ "${GITDIFF_EXIT:-0}" -ne 0 ]; then
728+
echo "::error::Git diff detection failed (exit code: $GITDIFF_EXIT)"
564729
exit 1
565730
fi
731+
if [ "${TREE_EXIT:-0}" -ne 0 ]; then
732+
echo "::error::Tree comparison detection failed (exit code: $TREE_EXIT)"
733+
exit 1
734+
fi
735+
if [ "${DISCOVERY_EXIT:-0}" -ne 0 ]; then
736+
echo "::error::Discovery analysis detection failed (exit code: $DISCOVERY_EXIT)"
737+
exit 1
738+
fi
739+
740+
echo "✅ All detection methods completed successfully"
741+
742+
# Aggregate results (union of all three methods)
743+
echo "📊 Aggregating results..."
744+
REMOVED_STACKS=$(aggregate_removed_stacks "$GITDIFF_STACKS" "$TREE_STACKS" "$DISCOVERY_STACKS")
745+
746+
# Debug logging
747+
if [ -n "$GITDIFF_STACKS" ]; then
748+
echo " Git diff found: $(echo "$GITDIFF_STACKS" | tr '\n' ', ' | sed 's/,$//')"
749+
fi
750+
if [ -n "$TREE_STACKS" ]; then
751+
echo " Tree comparison found: $(echo "$TREE_STACKS" | tr '\n' ', ' | sed 's/,$//')"
752+
fi
753+
if [ -n "$DISCOVERY_STACKS" ]; then
754+
echo " Discovery analysis found: $(echo "$DISCOVERY_STACKS" | tr '\n' ', ' | sed 's/,$//')"
755+
fi
566756
567757
# Process results
568758
if [ -z "$REMOVED_STACKS" ]; then

‎CLAUDE.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,9 +30,9 @@ This repository provides two main reusable workflows:
3030
- `target-ref`: Git reference to checkout (default: main)
3131
- Various GitHub event parameters for context
3232

33-
#### 2. Deploy Workflow (`/.github/workflows/deploy.yml`)
33+
#### 2. Deploy Workflow (`/.github/workflows/deploy.yml`)
3434
- **Purpose**: Handles deployment, health checks, rollback, and cleanup with comprehensive monitoring
35-
- **Features**:
35+
- **Features**:
3636
- Input validation and sanitization for security
3737
- Enhanced error handling with retry logic and exponential backoff
3838
- Parallel stack deployment with detailed logging
@@ -42,6 +42,11 @@ This repository provides two main reusable workflows:
4242
- SSH connection optimization with multiplexing
4343
- Tailscale integration with cached state
4444
- Rich Discord notifications with deployment metrics
45+
- **Enhanced Stack Removal Detection**: Three-method detection system
46+
- Git diff: Commit comparison (existing)
47+
- Tree comparison: Filesystem vs commit tree (catches undeployed removals)
48+
- Discovery analysis: tj-actions/changed-files validation
49+
- Union-based aggregation with fail-safe error handling
4550
- **Key Input Parameters**:
4651
- `stacks`: JSON array of stack names to deploy
4752
- `webhook-url`: 1Password reference to Discord webhook

0 commit comments

Comments
 (0)