[Phase 7.1] Integrate pipeline with checkpoint hooks [PARTIALLY COMPLETE]

[zhexuany]

Summary

Integrate the existing pipeline with checkpoint hooks for distributed mode. Add distributed context and checkpoint triggers at stage boundaries.

Status: PARTIALLY COMPLETE ✅🔧

✅ Completed (via #72, #73)

• DistributedPipelineContext equivalent - Worker.process_job() now integrates with TiKV
• Source stage hook - ProgressCallback with byte offset tracking (via messages_processed)
• Parser stage hook - Frame index tracking via frame counting
• Checkpoint trigger - Configurable by frame count and time interval
• Resume from checkpoint - Worker checks for existing checkpoints on job start

🔄 Remaining Enhancements (Optional)

These are optional refinements to the current implementation:

1. Processor stage hook (video encoder state)

   • Current: Checkpoints track frame progress
   • Enhancement: Track video encoder state for GPU encoder resume
   • Priority: Low - only needed if using GPU encoders ([Phase 8] Add NVENC GPU video encoding [PARALLEL ⏳] #49)

2. Uploader stage hook (detailed multipart state)

   • Current: Upload coordinator tracks completion at episode level
   • Enhancement: Track in-progress multipart upload IDs for part-by-part resume
   • Priority: Medium - useful for large video uploads
   • Note: See TODO in WorkerCheckpointCallback about episode-level tracking

Recommendation

This issue can be closed with the understanding that:

1. Frame-level checkpointing works ([Phase 1] Add checkpoint save during pipeline processing #73)
2. Resume from checkpoint works
3. Upload state tracking is simplified (completion-based, not multipart-detailed)

If detailed multipart upload resume is needed, create a new issue specifically for that feature.

Original Issue Content Below

---

Integrate the existing pipeline with checkpoint hooks for distributed mode. Add distributed context and checkpoint triggers at stage boundaries.

Pipeline Context

Add DistributedPipelineContext to carry:

• TiKV client reference
• Job ID and file hash
• Pod ID
• Checkpoint manager
• Initial checkpoint state (for resume)

Checkpoint Hooks

Inject at stage boundaries:

Source → [checkpoint: byte_offset] → Parser
Parser → [checkpoint: frame_idx] → Processor  
Processor → [checkpoint: encoder_state] → Uploader
Uploader → [checkpoint: parts[]] → Complete

Acceptance Criteria

• DistributedPipelineContext defined
• Pipeline wrapper with hooks
• Source stage tracks byte offset
• Parser stage tracks frame index
• Processor stage tracks encoder state (optional)
• Uploader stage tracks parts (optional)
• Checkpoint trigger at intervals
• Resume works from any checkpoint
• Completion handler works
• No data loss or duplication
• Integration test: interrupt and resume (basic test passes)

[zhexuany]

Status Update: Partially Complete

This issue's core functionality has been implemented through #72 and #73. The remaining items are enhancements:

✅ Already Completed (via #72, #73)

• ✅ DistributedPipelineContext equivalent - Worker.process_job() now integrates with TiKV
• ✅ Source stage hook - ProgressCallback with byte offset tracking
• ✅ Parser stage hook - Frame index tracking via messages_processed
• ✅ Checkpoint trigger - Configurable by frame count and time interval
• ✅ Resume from checkpoint - Worker checks for existing checkpoints on job start

🔄 Remaining Enhancements

These are optional refinements to the current implementation:

1. Processor stage hook (video encoder state)

   • Current: Checkpoints track frame progress
   • Enhancement: Track video encoder state for GPU encoder resume
   • Priority: Low - only needed if using GPU encoders

2. Uploader stage hook (detailed multipart state)

   • Current: Upload coordinator tracks completion at episode level
   • Enhancement: Track in-progress multipart upload IDs for part-by-part resume
   • Priority: Medium - useful for large video uploads
   • Note: See TODO in WorkerCheckpointCallback about episode-level tracking

Recommendation

This issue can be closed as completed with the understanding that:

1. Frame-level checkpointing works ([Phase 1] Add checkpoint save during pipeline processing #73)
2. Resume from checkpoint works
3. Upload state tracking is simplified (completion-based, not multipart-detailed)

If detailed multipart upload resume is needed, create a new issue specifically for that feature.