DataLad Maint-to-Master Sync Workflow

Automated workflow to synchronize changes from maint branch to master branch in DataLad.

Quick Start

For DataLad Repository

1. Add PAT Secret (one-time setup):

   # Create a classic PAT at: https://github.com/settings/tokens
   # Scopes needed: repo, workflow
   # Add as repository secret named GH_PAT
   gh secret set GH_PAT --repo datalad/datalad

2. Deploy Workflows:

   # Copy workflows to datalad/datalad
   cp .github/workflows/sync-maint-to-master.yml /path/to/datalad/datalad/.github/workflows/
   cp .github/workflows/cleanup-sync-branch.yml /path/to/datalad/datalad/.github/workflows/
   
   # Commit and push to both branches
   cd /path/to/datalad/datalad
   git add .github/workflows/
   git commit -m "Add automated maint-to-master sync"
   git push origin master
   
   git checkout maint
   git merge master --no-edit
   git push origin maint

3. Enable Auto-Merge (optional):

   • Go to Settings → General → Pull Requests
   • Enable "Allow auto-merge"

How It Works

Two Operating Modes

Fast-Forward Mode (90% of cases)

• When: Master hasn't diverged from maint
• Action: Direct push to master (no PR)
• Speed: ~20 seconds
• Example: Post-release, only bug fixes on maint

PR Mode (10% of cases)

• When: Master has diverged (has features not in maint)
• Action: Creates/updates PR for review
• Speed: 5-15 minutes (CI + auto-merge)
• Example: Master has new features + maint has fixes

Workflow Triggers

• On push to maint: Automatically syncs to master
• Daily at 2 AM UTC: Checks for any missed syncs (silent if nothing to do)
• Manual: Via workflow_dispatch

Testing

Run the automated test suite:

./test-maint-to-master-workflow.sh

This creates a test repository and validates:

• ✅ Fast-forward mode (works with default GITHUB_TOKEN)
• ✅ PR creation mode (requires GH_PAT secret)
• ✅ Conflict detection
• ✅ Auto-merge enablement

Test Repository Setup: The test script needs a PAT to test PR creation. For yarikoptic-test org testing:

# PAT should be in: /home/yoh/proj/datalad/datalad-maint-enh-maint-to-master/secret2
# Or create new one and save to that path

Files

• .github/workflows/sync-maint-to-master.yml - Main synchronization workflow
• .github/workflows/cleanup-sync-branch.yml - Cleans up sync branch after PR merge
• .github/workflows/ci.yml - Example CI workflow
• test-maint-to-master-workflow.sh - Automated test script
• docs/ - Additional documentation

Documentation

• SETUP.md - Detailed deployment instructions
• ARCHITECTURE.md - Design decisions and workflow logic
• TROUBLESHOOTING.md - Common issues and solutions

Monitoring

# View recent workflow runs
gh run list --workflow=sync-maint-to-master.yml --repo datalad/datalad --limit 5

# Check for open sync PRs
gh pr list --head maint-to-master --repo datalad/datalad

# View specific workflow run
gh run view <run-id> --log --repo datalad/datalad

Typical Scenarios

Scenario 1: Post-Release (Fast-Forward)

State:  maint == master (both at v1.2.0)
Action: Push fix to maint
Result: Master automatically fast-forwards to maint
Time:   ~20 seconds, no PR needed

Scenario 2: Active Development (PR Mode)

State:  master has feature, maint has fix
Action: Push fix to maint
Result: PR created "Sync maint → master"
Time:   ~5-15 minutes (CI + auto-merge)

Scenario 3: Merge Conflict

State:  Same file edited differently in both branches
Action: Push to maint
Result: PR created with conflict notice
Next:   Manual resolution on maint-to-master branch

Future Enhancement: Reusable Workflow

If multiple repositories need this workflow, it can be converted to a reusable workflow:

Advantages of Reusable Workflow

• Single source of truth for multiple repos
• Centralized maintenance and updates
• Consistent behavior across repositories

Current Status

Not implemented - Current design uses direct workflow files because:

• Simpler to understand and debug
• More transparent (all logic in one file)
• Easier to customize per repository
• Only one repository (datalad/datalad) needs it currently

If/When Needed

To convert to reusable workflow:

1. In this repo - Create .github/workflows/sync-reusable.yml:

   name: Reusable Maint-to-Master Sync
   on:
     workflow_call:
       secrets:
         GH_PAT:
           required: true
           description: 'Personal Access Token with repo and workflow scopes'
   # ... (rest of current sync workflow)

2. In consuming repos - Use it:

   name: Sync maint → master
   on:
     push:
       branches: [maint]
     schedule:
       - cron: '0 2 * * *'
   jobs:
     sync:
       uses: datalad/maint-to-master/.github/workflows/sync-reusable.yml@main
       secrets:
         GH_PAT: ${{ secrets.GH_PAT }}

Decision Point: Implement reusable workflow when:

• 3+ DataLad-related repositories need this pattern
• Centralized updates become more valuable than per-repo customization

Maintenance

Updating the Workflow

1. Make changes in this repository
2. Test with ./test-maint-to-master-workflow.sh
3. Copy updated workflow to datalad/datalad:

   cp .github/workflows/sync-maint-to-master.yml \
      /path/to/datalad/datalad/.github/workflows/

4. Commit and push to both maint and master

Monitoring Health

The workflow is self-healing but monitor for:

• Persistent PR conflicts (indicates divergence issues)
• Failed workflow runs (check permissions or GitHub API issues)
• Unopened PRs when divergence exists (check PAT validity)

Design Philosophy

1. Optimize for the common case: Fast-forward happens 90% of the time
2. Fail safe: Conflicts pause automation, require manual review
3. Transparent: All actions visible in PRs and workflow logs
4. Idempotent: Safe to run multiple times
5. Silent when successful: No notifications for routine syncs

Credits

• Designed: 2026-01-17
• Implementation: Claude Code (Sonnet 4.5)
• Testing: Validated in yarikoptic-test-dandisets organization
• Target: datalad/datalad repository

---

Status: ✅ Production Ready Fast-Forward Mode: ✅ Fully Functional PR Mode: ✅ Fully Functional (requires GH_PAT) Last Updated: 2026-01-17