refactor: Improve skimming and metadata code organization, naming, and S3 support

[MoAly98]

Overview

This PR refactors the skimming and metadata extraction subsystems to improve code maintainability, discoverability, and user experience. The changes focus on clearer naming, better documentation, logical code organization, and robust S3 storage support for distributed processing.

Key Improvements

1. Naming & Documentation

• Replaced vague function names with clear, action-oriented names
• Enhanced docstrings with comprehensive parameter/return documentation
• Added module-level constants to eliminate magic strings
• Improved inline comments for complex logic

2. Code Organization

• Reorganized utils/skimming.py (1094 lines) into 7 logical sections with clear headers
• Moved main entry point to end of file for better discoverability
• Grouped related helper functions by responsibility

3. S3 Storage Support

• Added WorkerEval class for worker-side environment variable evaluation
• Implemented _resolve_lazy_values() for recursive lazy evaluation
• Updated S3 configuration to support distributed credentials
• Added comprehensive S3 example configuration in example_cms/configs/skim.py

4. Interactive Demo

• Added demo_workflow.ipynb with full workflow demonstration
• Configurable DIST/S3 flags for flexible deployment
• Clear documentation of configuration combinations

Breaking Changes

Function Renamings

utils/metadata_extractor.py:

• _parse_dataset() → parse_dataset_key()
• summarise_nanoaods() → summarize_event_counts()

utils/skimming.py:

• workitem_analysis() → process_workitem()
• reduce_results() → merge_results()
• _build_output_suffix() → _build_output_path()
• process_workitems_with_skimming() → process_and_load_events()

New Constants

# utils/metadata_extractor.py
DATASET_DELIMITER = "__"
DEFAULT_VARIATION = "nominal"

# utils/skimming.py
COUNTER_DELIMITER = "::"
ENTRY_RANGE_DELIMITER = "_"

Files Changed

Core utilities:

• utils/schema.py - Added WorkerEval class
• utils/skimming.py - Complete reorganization + lazy evaluation support
• utils/metadata_extractor.py - Improved naming and documentation
• utils/datasets.py - Minor updates for consistency

Configuration:

• example_cms/configs/skim.py - New file with S3 storage configuration
• example_cms/configs/configuration.py - New consolidated config
• example_opendata/configs/*.py - Updated configs for opendata example

Documentation:

• demo_workflow.ipynb - New interactive demonstration notebook
• README.md - Updated to reflect new structure

Entry points:

• analysis.py - Updated to use new function names
• dev/dev_test_skimming*.py - Updated to use new function names

Migration Guide

For Users

Update function calls in your code:

# Before
from utils.skimming import process_workitems_with_skimming
datasets = process_workitems_with_skimming(workitems, config, ...)

# After
from utils.skimming import process_and_load_events
datasets = process_and_load_events(workitems, config, ...)

For S3 Storage

Configure worker-side credentials using WorkerEval:

from utils.schema import WorkerEval
import os

skimming_config = {
    "output": {
        "format": "parquet",
        "protocol": "s3",
        "to_kwargs": {
            "storage_options": {
                "key": WorkerEval(lambda: os.environ['AWS_ACCESS_KEY_ID']),
                "secret": WorkerEval(lambda: os.environ['AWS_SECRET_ACCESS_KEY']),
                "client_kwargs": {
                    "endpoint_url": "https://your-s3-endpoint.com"
                }
            }
        }
    }
}

---

[alexander-held]

Let's make sure we squash when merging as there are some fairly big files in the history here.