Feat: Historical Reports and FSE Testing Build #3509
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
AlexZorkin
commented
Nov 18, 2025
AlexZorkin
commented
- updated migrations and data transfer scripts for TFRS Shutdown work
Prepares LCFS database schema for future TFRS historical data migration. This PR contains ONLY schema changes - no data migration or ETL execution. Database Migrations (5): - a3290902296b: Add renewable diesel fuel type (fuel_type_id 24) - 54d55e878dad: Update compliance_report_summary model * Add historical_snapshot JSONB column for flexible TFRS data storage * Remove unused credits_offset_a/b/c columns * Recreate dependent metabase views - 5c3de27f158b: Add BETA_TESTER role to role_enum - 87592f5136b3: Add historical target carbon intensities (2013-2023) * Gasoline and diesel target CI values * Legacy fuel type mappings (fuel_type 21, 22, 23) * Energy effectiveness ratios for compliance periods 4-14 - 8e530edb155f: Populate missing organization snapshots * Backfills organization snapshots for compliance reports * Uses current org data to create historical snapshots Model Changes: - ComplianceReportSummary: Add historical_snapshot, remove credits_offset columns - Role: Add BETA_TESTER enum value Risk Assessment: LOW - All changes are schema preparation only - No breaking changes to existing functionality - historical_snapshot column is nullable (empty until ETL runs) - Removed columns (credits_offset_a/b/c) are unused in production Testing: - All migrations tested on dev database - Migrations have proper downgrade paths - No impact to existing compliance reports - Views recreated successfully Follow-up: Phase 2 will include ETL data migration scripts to populate the historical_snapshot field with actual TFRS data. Refs: LCFS-XXX (TFRS Shutdown Migration)
Adds comprehensive Python-based data migration framework for transferring historical compliance reporting data from legacy TFRS (Transportation Fuels Reporting System) to LCFS (Low Carbon Fuel Standard) system. These scripts do not execute automatically and are safe to merge into develop. They provide the tooling needed to perform the actual data migration when ready. ## Python Migration Framework (etl/python_migration/) ### Core Infrastructure - config.py: Database connection management for TFRS and LCFS - database.py: SQLAlchemy session handling and connection pooling - utils.py: Shared utilities for data transformation and logging ### Migration Scripts (10 scripts) 1. migrate_allocation_agreements.py: Allocation agreement transfers 2. migrate_compliance_report_history.py: Historical report versioning 3. migrate_compliance_summaries.py: Summary data transformation 4. migrate_compliance_summary_updates.py: Update history tracking 5. migrate_data_cleanup.py: Data quality fixes and corrections 6. migrate_fuel_supply.py: Fuel supply records with versioning 7. migrate_notional_transfers.py: Notional transfer records 8. migrate_orphaned_allocation_agreements.py: Edge case handling 9. migrate_other_uses.py: Other uses fuel records 10. run_all_migrations.py: Orchestration and execution order ### Setup & Orchestration - database_manager.py: Database health checks and state management - docker_manager.py: Container orchestration for testing - migration_orchestrator.py: Migration sequencing and error handling - validation_runner.py: Comprehensive validation suite runner ### Validation Suite - validation_base.py: Base validation framework - validate_allocation_agreements.py: Allocation agreement data validation - validate_compliance_summaries.py: Summary data integrity checks - validate_fuel_supply.py: Fuel supply record validation - validate_notional_transfers.py: Transfer data validation - validate_other_uses.py: Other uses validation - validate_2019_fixes.py: Specific data quality validation ### Documentation - README.md: Quick start guide and usage instructions - README_SETUP.md: Detailed setup and configuration guide - VERSIONING_REVIEW.md: Data versioning strategy and considerations ### Configuration & Deployment - docker-compose.yml: Local testing environment setup - Makefile: Common tasks and commands - requirements.txt: Python dependencies - .env.example: Environment configuration template - quick-start.sh: Automated setup script - setup-paths.sh: Path configuration helper ## NiFi Integration Updates Enhanced existing NiFi Groovy scripts with improved error handling and data transformation logic: - allocation_agreement.groovy - compliance_report.groovy - compliance_summary.groovy - compliance_summary_update.groovy - fuel_supply.groovy - notional_transfer.groovy - other_uses.groovy - orphaned_allocation_agreement.groovy (new) ## Deployment Scripts - data-transfer.sh: Standard data transfer execution - data-transfer-enhanced.sh: Enhanced transfer with validation - manual-deploy-to-dev.sh: Manual deployment to dev environment ## Key Features ✅ Does not auto-execute on deployment (safe to merge) ✅ Comprehensive validation suite ensures data integrity ✅ Rollback capability via Alembic migrations ✅ Detailed logging and error reporting ✅ Docker-based testing environment ✅ Handles edge cases and data quality issues ✅ Preserves historical data accuracy (2013-2023) ## Next Steps After merging: 1. Review and test migration scripts in isolated environment 2. Validate against production TFRS data snapshot 3. Schedule maintenance window for actual migration execution 4. Execute migrations with monitoring and validation Relates to TFRS shutdown and LCFS historical data preservation initiative.
Adds comprehensive UI and API support for viewing and managing historical compliance reports migrated from legacy TFRS (Transportation Fuels Reporting System) to LCFS (Low Carbon Fuel Standard). This PR contains all the frontend and backend code changes needed to work with the TFRS schema preparation (PR #1) and ETL migration scripts (PR #2). ## Frontend Changes ### New Legacy Compliance Report Views - **ViewLegacyComplianceReport.jsx**: Full legacy report viewer with historical data - **ComplianceReportViewSelector.jsx**: Router to select modern vs legacy report view - **LegacyAssessmentCard.jsx**: Assessment display for historical reports - **NewComplianceReportButton.jsx**: Enhanced button supporting legacy context ### Legacy Report Detail Components (legacy-deprecated/) - **LegacyReportSummary.jsx**: Historical report summary display - **LegacyReportDetails.jsx**: Detailed report information - **ScheduleASummary.jsx**: Schedule A (Renewable fuel supply) - **ScheduleBSummary.jsx**: Schedule B (Other uses & exemptions) - **ScheduleCSummary.jsx**: Schedule C (Notional transfers) - **ExclusionAgreementSummary.jsx**: Allocation agreements display - **_schema.jsx**: Legacy data schemas and formatters ### Enhanced Existing Components - **AddEditFuelSupplies.jsx**: Support for legacy fuel supply data - **AddEditChargingSite.jsx**: Charging site management improvements - **BCGridBase.jsx**: Grid enhancements for historical data display - **Login.jsx**: Beta tester role support ### Test Coverage Added comprehensive tests for all new components: - ViewLegacyComplianceReport.test.jsx - ComplianceReportViewSelector.test.jsx - LegacyReportSummary.test.jsx - ScheduleASummary.test.jsx - ScheduleBSummary.test.jsx - ScheduleCSummary.test.jsx - AddEditFuelSupplies.test.jsx - AddEditOrg.test.jsx - AddEditUser.test.jsx ### Configuration & Localization - **constants/roles.js**: Added BETA_TESTER role - **constants/common.js**: Legacy data constants - **constants/config.js**: Feature flags and configuration - **locales/en/legacy.json**: Legacy UI text translations - **locales/en/chargingSite.json**: Charging site translations - **_legacySchema.jsx**: Legacy fuel supply schema definitions ## Backend Changes ### API Services Enhanced **Compliance Reports**: - **summary_service.py**: Updated for historical_snapshot JSONB field - **summary_calculations.md**: Documentation for legacy calculations - **constants.py**: Constants for TFRS compatibility period mappings **Fuel Supply**: - **legacy_repo.py**: New repository for legacy fuel supply queries - **services.py**: Enhanced with legacy data handling - **schema.py**: Extended schemas for historical data **Allocation Agreements**: - **repo.py**: Enhanced queries for historical allocation agreements - **services.py**: Business logic for legacy agreement processing - **schema.py**: Schema updates for TFRS compatibility **Notional Transfers**: - **services.py**: Historical notional transfer handling **Other Uses**: - **services.py**: Legacy other uses fuel record support **Organizations**: - **repo.py**: Organization snapshot queries for historical context - **schema.py**: Extended organization schemas **Charging Sites**: - **validation.py**: Enhanced validation rules - **views.py**: API endpoints for charging infrastructure ### Infrastructure & Utilities - **application.py**: Application configuration updates - **exception_handler.py**: Enhanced error handling for legacy data - **calculations.py**: Utility functions for legacy calculations ### Database Updates - **metabase.sql**: Updated analytics views for historical reporting - Additional migration updates for down_revision chains ### Test Suite Enhancements - **test_summary_service.py**: Tests for historical snapshot handling - **test_fuel_supplies_services.py**: Legacy fuel supply tests - **test_allocation_agreement_views.py**: Allocation agreement API tests - **test_alllocation_agreement_repo.py**: Repository layer tests - **conftest.py**: Test fixtures for allocation agreements ## CI/CD Updates - **.github/workflows/pr-build.yaml**: Updated build workflow ## Key Features ✅ Seamless viewing of historical TFRS compliance reports (2013-2023) ✅ Modern UI components for legacy data display ✅ Backward compatibility maintained for current LCFS reports ✅ Comprehensive test coverage for all new functionality ✅ Role-based access control with BETA_TESTER role ✅ Enhanced error handling for historical data edge cases ✅ Metabase analytics support for historical reporting ✅ Charging infrastructure management integration ## Dependencies This PR depends on: 1. **feat/tfrs-schema-preparation**: Database schema must be in place 2. **feat/tfrs-etl-migration-scripts**: ETL scripts must be available (but don't need to be executed yet) ## Testing Notes - All frontend components have unit tests - Backend services have integration tests - Legacy data display tested with mock historical snapshots - Backward compatibility verified with current LCFS reports ## Migration Path 1. Merge schema preparation (PR #1) ✓ 2. Merge ETL scripts (PR #2) ✓ 3. Merge this PR (frontend/backend integration) 4. Execute ETL migration during maintenance window 5. Historical TFRS reports become visible in UI Relates to TFRS shutdown and LCFS historical data preservation initiative.
TFRS data is migrated into existing LCFS tables and displayed using standard LCFS compliance report views. Custom legacy views are not needed. Removed: - 11 legacy-deprecated frontend components and their tests - ViewLegacyComplianceReport and LegacyAssessmentCard components - Legacy fuel supply schema (_legacySchema.jsx) - Legacy translations (legacy.json) - Legacy fuel supply repository (backend) Updated: - ComplianceReportViewSelector now always uses EditViewComplianceReport - Removed feature flag logic for legacy view switching - AddEditFuelSupplies simplified to use standard schema for all periods - Tests updated to reflect unified view approach All historical TFRS-migrated reports will display using the existing LCFS compliance report interface.
…gration-rebuild-251117
…tfrs-migration-rebuild-251117
Backend Test Results1 615 tests 1 614 ✅ 13m 11s ⏱️ Results for commit 178acdd. ♻️ This comment has been updated with latest results. |
Frontend Test Results 1 files 305 suites 3m 14s ⏱️ Results for commit 178acdd. ♻️ This comment has been updated with latest results. |
Hotfix Release v1.2.1.1
Release v1.2.2
…ifications being sent
Hotfix: Fuel code expiry - 3682
Release v1.2.3
Hotfix: 2026 Reporting Fix
…nt TCI/EER/RCI instead of historical values
…3719 Fix: Historical TFRS Bugs.
Release v1.2.4
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.