Fix Phase 1 Critical ETL Transformation Issues

[devin-ai-integration]

Fix Phase 1 Critical ETL Transformation Issues

Summary

This PR addresses four critical issues identified in the ETL pipeline's transformation logic:

1. AWS Secret Key Bug: Fixed incorrect environment variable name aws_secret_access_key_id → aws_secret_access_key (affects S3 uploads)
2. Data Duplication Root Cause: Changed SQL query to aggregate service records instead of creating duplicate rows per service record
3. Database Connection Leak: Added conn.close() to prevent connection pool exhaustion
4. Silent S3 Failures: Added raise after S3 exception logging to propagate errors properly

⚠️ Breaking Change: The output schema has changed - individual service records are now aggregated into latest_service_date, service_count, and total_service_cost columns.

Review & Testing Checklist for Human

Risk Level: 🟡 Medium - Schema changes and incomplete S3 testing require careful validation

• Verify data aggregation logic meets business requirements - Review the new GROUP BY query in src/extract.py and confirm that aggregating service records (vs. individual records) aligns with downstream use cases
• Test full pipeline end-to-end with valid AWS credentials - I was unable to test S3 upload functionality; verify the AWS variable fix works and data uploads correctly
• Check for downstream impact from schema changes - The output now has latest_service_date, service_count, total_service_cost instead of individual service_date, service_type, service_cost - ensure no systems break
• Validate edge cases in aggregated data - Test vehicles with: no service records, single service record, multiple service records to ensure aggregation behaves correctly

Test Plan Recommendations

1. Run pipeline locally with valid AWS credentials and verify S3 upload succeeds
2. Compare output before/after for a vehicle with multiple service records to confirm aggregation is correct
3. Check any downstream dashboards/reports that consume this data still work

Notes

• Local testing confirmed extraction and transformation work correctly (4 rows extracted, no duplicates)
• Database connection leak was verified as fixed (no connection warnings)
• This work was requested by @catchthewave99 in session: https://app.devin.ai/sessions/efd30dba69a74bb7b04d145d69c267d8

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring