Skip to content

Conversation

@mike-ferguson
Copy link
Member

Add Local Packaging and Upload Validator

New Features

Local Packaging Functions

  • Added package_stimulus_set_locally() and package_data_assembly_locally() to save data locally instead of uploading to S3
  • Functions save to ~/Downloads/brainscore_packages/ by default with custom path support
  • Maintains same validation and file processing as original S3 functions
  • Returns file paths and SHA1 hashes for data integrity

Upload Validator

  • Created upload_validator.py with comprehensive validation for stimulus sets and assemblies
  • Validates required columns (stimulus_id + at least one other), data types, and compatibility
  • Integrated validation into local packaging functions with clear error handling
  • Prevents packaging of invalid or incompatible data

Usage

# Instead of S3 upload
package_stimulus_set_locally(proto_stimulus_set, "identifier")
package_data_assembly_locally(proto_data_assembly, "identifier", "stimulus_id")

Benefits

  • No S3 dependencies for local development/testing
  • Same validation and processing as production functions
  • Clear error messages for invalid data
  • Working examples for plugin development

@mike-ferguson mike-ferguson requested a review from KartikP October 21, 2025 13:47
@KartikP KartikP closed this Oct 21, 2025
@KartikP KartikP reopened this Oct 21, 2025
@KartikP KartikP closed this Oct 21, 2025
@KartikP KartikP reopened this Oct 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants