Skip to content

Auto-trigger data sync when archive contains newer data than deployed#537

Merged
mmathieum merged 23 commits intomasterfrom
copilot/update-data-trigger-workflow
Feb 3, 2026
Merged

Auto-trigger data sync when archive contains newer data than deployed#537
mmathieum merged 23 commits intomasterfrom
copilot/update-data-trigger-workflow

Conversation

Copy link
Contributor

Copilot AI commented Jan 30, 2026

Summary

Updated check_data_outdated.sh to parse XML files and extract the last departure timestamp to properly determine if data is outdated. Implemented comprehensive error handling and validation with a conservative approach to avoid unnecessary sync triggers.

Changes Made

Script: shared-opt-dir/agency-parser/check_data_outdated.sh

The script now:

  • ✅ Parses *_gtfs_rts_values_gen.xml files to extract *_gtfs_rts_last_departure_in_sec timestamp
  • ✅ Prefers "next" file if available, otherwise falls back to "current"
  • ✅ Checks if deployed data has expired (last departure < current time)
  • ✅ Compares archive end dates with deployed last departure timestamp
  • ✅ Returns exit code 1 if data is definitively outdated, 0 if current or uncertain
  • ✅ Conservative approach: avoids triggering sync when data freshness cannot be determined
  • ✅ Improved error messages to distinguish between missing files and missing timestamps
  • ✅ Added validation for archive filename format
  • ✅ Enhanced regex pattern with $ anchor for precise matching
  • ✅ Added to MT.gitignore for proper deployment

Workflow: shared-overwrite/.github/workflows/mt-download-data.yml

  • Merged data check and workflow trigger into a single step
  • Runs check_data_outdated.sh and triggers mt-sync-code-data.yml if data is outdated
  • Uses continue-on-error: true for fault tolerance
  • Triggers workflow on same repo and default branch

Implementation Details

XML Parsing:

  • Uses grep -E "<integer name=\"[^\"]*_gtfs_rts_last_departure_in_sec\">[0-9]+</integer>$" | tr -dc '0-9'
  • $ anchor ensures end-of-line match (consistent with existing codebase patterns)
  • Matches both current_gtfs_rts_last_departure_in_sec and next_gtfs_rts_last_departure_in_sec

Data Freshness Logic:

  1. Extract last departure timestamp from deployed data (prefer "next", fallback to "current")
  2. If timestamp cannot be extracted: return exit 0 (conservative - avoid triggering sync when uncertain)
  3. Check if deployed data expired: DEPLOYED_LAST_DEPARTURE_SEC < NOW_TIMESTAMP_SEC
  4. Check if archive has newer data: ARCHIVE_END_TIMESTAMP > DEPLOYED_LAST_DEPARTURE_SEC
  5. Trigger sync only when data is definitively outdated

Workflow Logic:

  • Single step checks data freshness and conditionally triggers sync workflow
  • If script exits 0 (success): data is up-to-date or uncertain, no action taken
  • If script exits 1 (failure): data is definitively outdated, triggers mt-sync-code-data.yml

Error Handling:

  • Conservative approach: when in doubt, don't trigger sync
  • Distinguishes between "no files found" and "files found but timestamp missing"
  • Both uncertain cases return exit 0 to avoid unnecessary sync operations
  • Validates archive filename format (YYYYMMDD-YYYYMMDD.zip)
  • Provides clear warning messages for debugging

Archive Date Comparison:

  • Converts archive end date (YYYYMMDD) to timestamp at end of day: date -d "${ARCHIVE_END_DATE} 23:59:59" +%s
  • Compares archive end timestamp with deployed last departure timestamp

Deployment:

  • Added check_data_outdated.sh to MT.gitignore in alphabetical order under "# Shared files" section
  • Ensures proper deployment alongside other shared agency-parser scripts

This implementation takes a conservative approach to avoid triggering unnecessary syncs, only acting when data is definitively outdated based on parsed timestamp content.

Original prompt

This section details on the original issue you should resolve

<issue_title>mt-download-data.yml > trigger mt-sync-code-data.yml if new data and data available outdated</issue_title>
<issue_description>- Change this CI workflow: shared-overwrite/.github/workflows/mt-download-data.yml

  • Add step(s) in the job to check if current data in app-android/src/main/(res-*/raw/*_gtfs_schedule_service_dates|values/*_gtfs_rts_values_gen.xml) is outdated and current data available in agency-parser/archive.
  • If yes, triggers gh workflow run mt-sync-code-data.yml on same repo.

Look into these files to get the last departure in seconds:

  • app-android/src/main/res-current/raw/current_gtfs_schedule_service_dates
  • app-android/src/main/res-current/values/current_gtfs_rts_values_gen.xml
  • app-android/src/main/res-next/raw/next_gtfs_schedule_service_dates
  • app-android/src/main/res-next/values/next_gtfs_rts_values_gen.xml
    </issue_description>

Comments on the Issue (you are @copilot in this section)


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 3 commits January 30, 2026 13:01
Co-authored-by: mmathieum <177998+mmathieum@users.noreply.github.com>
Co-authored-by: mmathieum <177998+mmathieum@users.noreply.github.com>
Co-authored-by: mmathieum <177998+mmathieum@users.noreply.github.com>
Copilot AI changed the title [WIP] Update mt-download-data.yml to trigger mt-sync-code-data.yml Auto-trigger data sync when archive contains newer data than deployed Jan 30, 2026
Copilot AI requested a review from mmathieum January 30, 2026 13:10
Copilot AI and others added 2 commits January 30, 2026 20:15
Co-authored-by: mmathieum <177998+mmathieum@users.noreply.github.com>
Co-authored-by: mmathieum <177998+mmathieum@users.noreply.github.com>
Copilot AI requested a review from mmathieum January 30, 2026 20:20
@mmathieum mmathieum marked this pull request as ready for review January 30, 2026 20:44
Co-authored-by: mmathieum <177998+mmathieum@users.noreply.github.com>
Co-authored-by: mmathieum <177998+mmathieum@users.noreply.github.com>
Co-authored-by: mmathieum <177998+mmathieum@users.noreply.github.com>
Copilot AI requested a review from mmathieum February 2, 2026 14:58
montransit added a commit to mtransitapps/ca-gatineau-sto-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
mmathieum added a commit to mtransitapps/ca-richelieu-citvr-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-calgary-transit-train-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-edmonton-ets-train-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-brampton-transit-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-chambly-richelieu-carignan-citcrc-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-regina-transit-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-l-assomption-mrclasso-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/us-anchorage-people-mover-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-grand-river-transit-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-squamish-transit-system-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/us-snohomish-county-community-transit-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-hamilton-hsr-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/us-clark-county-c-tran-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-oakville-transit-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-ste-julie-omitsju-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-durham-region-transit-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-gtha-go-transit-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-montreal-stm-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-york-region-yrt-viva-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-sunshine-coast-regional-transit-system-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-mississauga-miway-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-ottawa-oc-transpo-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-edmonton-ets-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-whistler-transit-system-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-quebec-rtc-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-strathcona-county-transit-bus-android that referenced this pull request Feb 3, 2026
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
montransit added a commit to mtransitapps/ca-toronto-ttc-bus-android that referenced this pull request Feb 3, 2026
- commons: mt-sync-code-data.yml > fix skip on main repo
- commons: CI: add frequently used gh command with inputs in comments
- commons: Auto-trigger data sync when archive contains newer data than deployed mtransitapps/commons#537
- parser: Fix compat with service IDs only used for data changed
- parser: fix compat with no last service calendar dates
- parser: Remove unused trip IDs ints. mtransitapps/parser#31
- parser: Schedule > clear head-sign instead of setting empty string mtransitapps/parser#30
- parser: Data Changed > compat with service IDs regex mtransitapps/parser#32
mmathieum added a commit that referenced this pull request Feb 9, 2026
…ta is outdated & newer data vailable (#553)

Following:
- #537

Solving:
- #482
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mt-download-data.yml > trigger mt-sync-code-data.yml if new data and data available outdated

2 participants