docs: update project tracking files for Week 3 completion

poldrack · claude · poldrack · commit 7d7753c3aa88 · 2025-08-27T15:34:02.000-07:00
- Updated SCRATCHPAD.md to reflect 100% Week 3 completion - Updated progress statistics: 38% overall (85/221 tasks) - Updated next steps to focus on Week 4 CPU backend - Cleaned up tracking state for completed Data I/O Layer 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
diff --git a/SCRATCHPAD.md b/SCRATCHPAD.md
@@ -8,16 +8,16 @@
 ## Project State Summary (2025-08-27 Refresh)
 
 ### Current Status
-**Branch**: dev/week3-data-io  
-**Phase**: Week 3 - Data I/O Layer Implementation  
-**Overall Progress**: 85/221 tasks completed (38%)  
-**Week 3 Progress**: 42/42 tasks completed (100%)  
+**Branch**: dev/week3-data-io
+**Phase**: Week 3 - Data I/O Layer Implementation
+**Overall Progress**: 85/221 tasks completed (38%)
+**Week 3 Progress**: 42/42 tasks completed (100%)
 
 ### Current Work in Progress
 **Data I/O Layer Implementation: ✅ WEEK 3 FULLY COMPLETED**
 - ✅ **NiftiLoader class COMPLETED** - All functionality implemented and tested
   - Full-featured class with lazy loading, chunking, masking support
-  - Memory optimization features implemented  
+  - Memory optimization features implemented
   - Data integrity validation included
   - All validation methods (_check_data_integrity, _validate_dimensions, etc.)
   - 14 tests all passing (100% pass rate), 90.22% code coverage
@@ -62,7 +62,7 @@
 **Week 2 Completion:**
 - ✅ **Complete Phase 1 Week 2 core architecture** (fully implemented)
   - Backend abstraction layer with comprehensive ABC interface
-  - Core orchestrator for workflow coordination and pipeline management  
+  - Core orchestrator for workflow coordination and pipeline management
   - Logging framework with colored output, progress reporting, and system info
   - Configuration management using Pydantic with TOML/env variable support
   - Error handling framework with hierarchical exceptions and recovery suggestions
@@ -101,7 +101,7 @@
 - ✅ Backend abstraction layer (ABC with is_available, compute_glm)
 - ✅ Core orchestrator (workflow coordination)
 - ✅ Logging framework (colored output, progress reporting)
-- ✅ Configuration management (Pydantic with TOML support)  
+- ✅ Configuration management (Pydantic with TOML support)
 - ✅ Error handling framework (hierarchical exceptions)
 - ✅ NIfTI I/O layer (full NiftiLoader implementation)
 
diff --git a/TASKS.md b/TASKS.md
@@ -86,11 +86,16 @@
   - [x] Add utility functions: load_design_matrix, validate_design_matrix, create_contrast_matrix (2025-08-27)
   - [x] Add FSL and SPM format compatibility (2025-08-27)
   - [x] Implement Gram-Schmidt orthogonalization for correlated regressors (2025-08-27)
-- [ ] Implement contrast file loader
-  - [ ] Parse contrast matrices from text files
-  - [ ] Validate contrast compatibility with design
-  - [ ] Support multiple contrasts
-  - [ ] Write unit tests
+- [x] Implement contrast file loader (2025-08-27)
+  - [x] Implement src/accelperm/io/contrast.py with ContrastLoader class (2025-08-27)
+  - [x] Parse contrast matrices from text files (FSL .con, CSV, TXT formats) (2025-08-27)
+  - [x] Validate contrast compatibility with design matrices (2025-08-27)
+  - [x] Support multiple contrast files and batch loading (2025-08-27)
+  - [x] Add contrast validation (rank deficiency, zero contrasts, data integrity) (2025-08-27)
+  - [x] Add format compatibility (FSL, SPM styles) (2025-08-27)
+  - [x] Add programmatic contrast creation (t-contrasts, F-contrasts, polynomial) (2025-08-27)
+  - [x] Write comprehensive unit tests with 21 tests, 100% pass rate (2025-08-27)
+  - [x] Add utility functions: load_contrast_matrix, validate_contrast_compatibility, create_t_contrast, create_f_contrast (2025-08-27)
 - [x] Create output writer module (2025-08-27)
   - [x] Implement src/accelperm/io/output.py with OutputWriter class (2025-08-27)
   - [x] Support statistical map output to NIfTI format (2025-08-27)
@@ -545,12 +550,12 @@
 - Blocked: 0
 - **Progress: 0%**
 
-### Week 3 Progress (Data I/O Layer)
+### Week 3 Progress (Data I/O Layer) - **COMPLETE!** 🎉
 - NIfTI handling: **COMPLETE** (10/10 subtasks)
 - Design matrix loader: **COMPLETE** (10/10 subtasks)
-- Contrast file loader: **PENDING** (0/4 subtasks) 
+- Contrast file loader: **COMPLETE** (10/10 subtasks) 
 - Output writer module: **COMPLETE** (12/12 subtasks)
-- **Week 3 Progress: 91%** (32/36 subtasks complete)
+- **Week 3 Progress: 100%** (42/42 subtasks complete)
 
 ### Overall Project
 - **Total tasks: 221**
diff --git a/src/accelperm/io/contrast.py b/src/accelperm/io/contrast.py
@@ -42,28 +42,34 @@ def load(
             elif filepath.suffix.lower() in [".txt", ".mat"]:
                 # Tab or space-separated format
                 # Handle inconsistent row lengths manually
-                with open(filepath, "r") as f:
+                with open(filepath) as f:
                     lines = f.readlines()
-                
+
                 # Parse lines manually to handle inconsistent lengths
                 rows = []
                 max_cols = 0
-                
+
                 for line in lines:
                     line = line.strip()
                     if line:  # Skip empty lines
                         if "\t" in line:
                             values = line.split("\t")
                         else:
                             values = line.split()
-                        
+
                         # Check for inconsistent number of columns
-                        if self.validate_contrasts and max_cols > 0 and len(values) != max_cols:
-                            raise ValueError("Inconsistent number of regressors in contrast file")
-                        
+                        if (
+                            self.validate_contrasts
+                            and max_cols > 0
+                            and len(values) != max_cols
+                        ):
+                            raise ValueError(
+                                "Inconsistent number of regressors in contrast file"
+                            )
+
                         max_cols = max(max_cols, len(values))
                         rows.append([float(v) for v in values])
-                
+
                 # Convert to DataFrame
                 data = pd.DataFrame(rows)
             else:
@@ -86,7 +92,9 @@ def load(
             if self.format_style == "fsl":
                 contrast_names = [f"C{i+1}" for i in range(contrast_matrix.shape[0])]
             else:
-                contrast_names = [f"contrast_{i+1}" for i in range(contrast_matrix.shape[0])]
+                contrast_names = [
+                    f"contrast_{i+1}" for i in range(contrast_matrix.shape[0])
+                ]
 
         # Validate contrasts
         if self.validate_contrasts:
@@ -101,7 +109,9 @@ def load(
         # Check design compatibility if provided
         design_compatible = True
         if design_info is not None:
-            design_compatible, _ = validate_contrast_compatibility(contrast_matrix, design_info)
+            design_compatible, _ = validate_contrast_compatibility(
+                contrast_matrix, design_info
+            )
 
         # Prepare result
         result = {
@@ -148,38 +158,36 @@ def create_standard_contrasts(self, column_names: list[str]) -> dict[str, Any]:
             if col_name.lower() != "intercept":
                 contrast = np.zeros(len(column_names))
                 contrast[i] = 1
-                contrasts["main_effects"].append({
-                    "name": f"{col_name}_effect",
-                    "contrast": contrast
-                })
+                contrasts["main_effects"].append(
+                    {"name": f"{col_name}_effect", "contrast": contrast}
+                )
 
         # Create interaction contrasts (for columns containing 'x' or '_x_')
         for i, col_name in enumerate(column_names):
             if "x" in col_name.lower() or "_x_" in col_name.lower():
                 contrast = np.zeros(len(column_names))
                 contrast[i] = 1
-                contrasts["interactions"].append({
-                    "name": f"{col_name}_interaction",
-                    "contrast": contrast
-                })
+                contrasts["interactions"].append(
+                    {"name": f"{col_name}_interaction", "contrast": contrast}
+                )
 
         return contrasts
 
     def create_polynomial_contrasts(self, n_levels: int) -> dict[str, np.ndarray]:
         """Create polynomial contrasts for ordered factors."""
         contrasts = {}
-        
+
         if n_levels >= 2:
             # Linear contrast
             linear = np.linspace(-1, 1, n_levels)
             contrasts["linear"] = linear
-            
+
         if n_levels >= 3:
             # Quadratic contrast
             x = np.linspace(-1, 1, n_levels)
             quadratic = x**2 - np.mean(x**2)
             contrasts["quadratic"] = quadratic
-            
+
         if n_levels >= 4:
             # Cubic contrast
             cubic = x**3 - np.mean(x**3)
@@ -226,13 +234,17 @@ def validate_contrast_compatibility(
         # For orthogonal designs, contrasts should sum to zero for proper interpretation
         for i, contrast_row in enumerate(contrast_matrix):
             if not np.isclose(np.sum(contrast_row), 0, atol=1e-10):
-                issues.append(f"Contrast {i+1} does not sum to zero (may not be interpretable)")
+                issues.append(
+                    f"Contrast {i+1} does not sum to zero (may not be interpretable)"
+                )
 
     is_compatible = len(issues) == 0
     return is_compatible, issues
 
 
-def create_t_contrast(column_names: list[str], contrast_spec: dict[str, float]) -> np.ndarray:
+def create_t_contrast(
+    column_names: list[str], contrast_spec: dict[str, float]
+) -> np.ndarray:
     """Create t-contrast vector from column names and specification."""
     n_regressors = len(column_names)
     contrast = np.zeros(n_regressors)
@@ -258,4 +270,4 @@ def create_f_contrast(
     for i, contrast_spec in enumerate(contrast_specs):
         f_contrast[i, :] = create_t_contrast(column_names, contrast_spec)
 
-    return f_contrast
+    return f_contrast
diff --git a/src/accelperm/io/nifti.py b/src/accelperm/io/nifti.py
@@ -1,7 +1,7 @@
 """NIfTI file I/O operations for AccelPerm."""
 
 from pathlib import Path
-from typing import Any, Dict, Iterator, Optional, Tuple, Union
+from typing import Any
 
 import nibabel as nib
 import numpy as np
@@ -10,16 +10,18 @@
 class NiftiLoader:
     """Loader for NIfTI files with memory optimization and validation."""
 
-    def __init__(self, lazy_loading: bool = False, chunk_size: Optional[int] = None) -> None:
+    def __init__(
+        self, lazy_loading: bool = False, chunk_size: int | None = None
+    ) -> None:
         self.lazy_loading = lazy_loading
         self.chunk_size = chunk_size
 
-    def load(self, filepath: Path, mask: Optional[Path] = None) -> Dict[str, Any]:
+    def load(self, filepath: Path, mask: Path | None = None) -> dict[str, Any]:
         """Load NIfTI file and return data with metadata."""
         try:
             img = nib.load(str(filepath))
         except FileNotFoundError:
-            raise FileNotFoundError(f"NIfTI file not found: {filepath}")
+            raise FileNotFoundError(f"NIfTI file not found: {filepath}") from None
         except Exception as e:
             raise ValueError(f"Invalid NIfTI file: {filepath}") from e
 
@@ -32,7 +34,7 @@ def load(self, filepath: Path, mask: Optional[Path] = None) -> Dict[str, Any]:
         else:
             data = img.get_fdata()
             self._check_data_integrity(data)
-            
+
             result = {
                 "data": data,
                 "affine": img.affine,
@@ -50,41 +52,35 @@ def load(self, filepath: Path, mask: Optional[Path] = None) -> Dict[str, Any]:
             mask_data = mask_img.get_fdata().astype(bool)
             result["mask"] = mask_data
             result["n_voxels"] = np.sum(mask_data)
-            
+
             if not self.lazy_loading:
                 # Apply mask to data
-                if data.ndim == 4:
-                    masked_data = data[mask_data, :]
-                else:
-                    masked_data = data[mask_data]
+                masked_data = data[mask_data, :] if data.ndim == 4 else data[mask_data]
                 result["masked_data"] = masked_data
 
         return result
 
-    def load_chunked(self, filepath: Path) -> Dict[str, Any]:
+    def load_chunked(self, filepath: Path) -> dict[str, Any]:
         """Load NIfTI file in chunks for memory efficiency."""
         try:
             img = nib.load(str(filepath))
         except FileNotFoundError:
-            raise FileNotFoundError(f"NIfTI file not found: {filepath}")
+            raise FileNotFoundError(f"NIfTI file not found: {filepath}") from None
         data = img.get_fdata()
-        
+
         # Calculate number of chunks based on chunk_size
         total_voxels = np.prod(data.shape[:3])
-        if self.chunk_size is None:
-            chunk_size = 1000
-        else:
-            chunk_size = self.chunk_size
-            
+        chunk_size = 1000 if self.chunk_size is None else self.chunk_size
+
         total_chunks = int(np.ceil(total_voxels / chunk_size))
-        
+
         def chunk_iterator():
             for i in range(total_chunks):
                 start_idx = i * chunk_size
                 end_idx = min((i + 1) * chunk_size, total_voxels)
                 # Return a chunk of data
                 yield data.flat[start_idx:end_idx]
-        
+
         return {
             "chunk_iterator": chunk_iterator(),
             "total_chunks": total_chunks,
@@ -110,7 +106,7 @@ def _validate_spatial_match(self, data1: np.ndarray, data2: np.ndarray) -> None:
             raise ValueError("Spatial dimensions do not match")
 
 
-def load_nifti(filepath: Path) -> Tuple[np.ndarray, np.ndarray, Any]:
+def load_nifti(filepath: Path) -> tuple[np.ndarray, np.ndarray, Any]:
     """Load NIfTI file and return data, affine, and header."""
     img = nib.load(str(filepath))
     data = img.get_fdata()
@@ -123,20 +119,19 @@ def save_nifti(data: np.ndarray, affine: np.ndarray, filepath: Path) -> None:
     """Save data as NIfTI file."""
     # Create output directory if it doesn't exist
     filepath.parent.mkdir(parents=True, exist_ok=True)
-    
+
     # Create NIfTI image and save
     img = nib.Nifti1Image(data, affine)
     img.to_filename(str(filepath))
 
 
-def validate_nifti_compatibility(img1_info: Dict[str, Any], img2_info: Dict[str, Any]) -> bool:
+def validate_nifti_compatibility(
+    img1_info: dict[str, Any], img2_info: dict[str, Any]
+) -> bool:
     """Validate that two NIfTI images are compatible for processing."""
     # Check spatial dimensions match
     if img1_info["spatial_shape"] != img2_info["spatial_shape"]:
         return False
-    
+
     # Check affine matrices are similar (allowing for small floating point differences)
-    if not np.allclose(img1_info["affine"], img2_info["affine"], rtol=1e-6):
-        return False
-    
-    return True
+    return np.allclose(img1_info["affine"], img2_info["affine"], rtol=1e-6)
diff --git a/tests/unit/test_io_contrast.py b/tests/unit/test_io_contrast.py
diff --git a/tests/unit/test_io_nifti.py b/tests/unit/test_io_nifti.py
