(fix) Overhaul teardown logic

Summary of Teardown Workflow Overhaul

  ✅ Phase 1: Backend State Validation & Pre-flight Checks

  - Replaced setup-backend job with new validate-backend-state job
  - Comprehensive validation: Checks if state bucket exists, validates remote state access, and confirms resource inventory
  - Smart bucket naming: Auto-determines bucket name if not provided
  - Resource counting: Provides accurate count of resources to be destroyed
  - Fail-fast approach: Exits early if backend doesn't exist or state is invalid

  ✅ Phase 2: Enhanced State Management Integration

  - New prepare-teardown job with advanced capabilities:
    - State drift detection: Identifies resources modified outside Terraform
    - Resource inventory creation: Detailed breakdown of resources by type
    - State stability validation: Ensures state hasn't changed unexpectedly
    - Comprehensive resource listing: JSON inventory with metadata

  ✅ Phase 3: Streamlined Teardown Logic

  - Enhanced teardown job with robust error handling:
    - Retry logic: Automatic retries for transient failures
    - Better progress reporting: Shows exactly what will be destroyed
    - Partial failure recovery: Handles incomplete teardowns gracefully
    - State verification: Confirms state matches validation before proceeding

  ✅ Phase 4: Improved Error Recovery & Reporting

  - Comprehensive post-teardown reporting:
    - Detailed status analysis: Includes all job results and resource counts
    - Smart issue creation: Only creates issues when actually needed
    - Actionable cleanup guidance: Specific steps for manual intervention
    - Rich artifact retention: 30-day retention for troubleshooting

  ✅ Phase 5: Supporting Script Updates

  - Updated scripts/lifecycle/teardown.sh:
    - Remote state awareness: Proper handling of backend.tf vs local state
    - Improved validation: Better state accessibility checks
    - Enhanced cleanup: Proper archival of remote state
    - Better error handling: More robust failure detection and reporting

Key Benefits Achieved

  1. 🛡️ Reliability: Proper state validation prevents failed teardowns due to missing or invalid backends
  2. ⚡ Efficiency: Streamlined workflow reduces complexity and runtime through better job dependencies
  3. 🔒 Safety: Enhanced validation prevents accidental operations on wrong environments
  4. 🔧 Recovery: Comprehensive error handling and detailed cleanup guidance
  5. 💰 Cost Control: More reliable destruction significantly reduces orphaned resource risks

Workflow Improvements Summary

  | Aspect             | Before                              | After                                             |
  |--------------------|-------------------------------------|---------------------------------------------------|
  | Backend Handling   | Creates new backend during teardown | Validates existing backend state                  |
  | State Verification | Basic existence check               | Comprehensive validation with drift detection     |
  | Error Recovery     | Limited retry logic                 | Multi-level retries with partial failure handling |
  | Reporting          | Basic status reporting              | Comprehensive reports with actionable guidance    |
  | Resource Tracking  | Simple resource counting            | Detailed inventory with categorization            |
  | Issue Creation     | Always on failure                   | Smart detection of actual cleanup needs           |

The overhauled workflow is now much more robust, reliable, and provides excellent visibility into the teardown process. It properly
  handles edge cases, provides comprehensive reporting, and ensures cost-effective resource cleanup.

Author: pacphi