Skip to content

feat: use dynamic foraging library to build qc#2

Open
arjunsridhar12345 wants to merge 6 commits into
mainfrom
1-use-dynamic-foraging-library-to-build-qc
Open

feat: use dynamic foraging library to build qc#2
arjunsridhar12345 wants to merge 6 commits into
mainfrom
1-use-dynamic-foraging-library-to-build-qc

Conversation

@arjunsridhar12345

@arjunsridhar12345 arjunsridhar12345 commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Uses functions from this PR: AllenNeuralDynamics/dynamic-foraging-processing#25

Attempts to close #1. Uses the dynamic foraging processing library to build the qc and the rest is boilerplate. I pinned the library to a commit, this will need to be updated before releasing the capsule.

I'm working in code ocean dev environment right now, so for releasing, there will be a separate PR in code ocean production environment.

Added logging. Sample log output is below:

{"timestamp": "2026-07-02T22:31:53.605921Z", "level": "INFO", "message": "Begin QC ...", "acquisition_name": "845023_2026-06-17_19-41-02", "process_name": "dynamic-foraging-qc", "pipeline_name": "", "event_type": "stage_start"}
{"timestamp": "2026-07-02T22:31:53.606338Z", "level": "INFO", "message": "Found session 845023_2026-06-17_19-41-02. Running QC", "acquisition_name": "845023_2026-06-17_19-41-02", "process_name": "dynamic-foraging-qc", "pipeline_name": ""}
{"timestamp": "2026-07-02T22:32:17.727111Z", "level": "INFO", "message": "Finished QC. Written to results", "acquisition_name": "845023_2026-06-17_19-41-02", "process_name": "dynamic-foraging-qc", "pipeline_name": ""}
{"timestamp": "2026-07-02T22:32:17.727309Z", "level": "INFO", "message": "Pipeline stage completed", "acquisition_name": "845023_2026-06-17_19-41-02", "process_name": "dynamic-foraging-qc", "pipeline_name": "", "event_type": "stage_complete"}

Did a run on real data uploaded to test env, sample output is shown below with the qc.json file, and then folder for plots:

{842A5CF6-8810-48C8-AF4F-713BBCEBC08E}

@arjunsridhar12345 arjunsridhar12345 linked an issue Jul 2, 2026 that may be closed by this pull request
Use dynamic foraging library to build QC #1
Open
arielleleon
arielleleon requested changes Jul 7, 2026
View reviewed changes

@arielleleon arielleleon left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two small things to remove

Comment thread code/run_capsule.py Outdated
from pydantic_settings import BaseSettings

if __name__ == "__main__": run() No newline at end of file
logger = logging.getLogger(__name__)

@arielleleon arielleleon Jul 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this

@seanmcculloch seanmcculloch Jul 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is necessary after calling log_schema.setup_logging.

Also, Jessy is discussing the shape/scope of log schema with team on Thursday 7/8

@arielleleon arielleleon Jul 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, that doesn't sound correct. Can you check and see why? Can you elaborate on shape/scope? Are they changing the log data structure?

Comment thread code/run_capsule.py Outdated
Entrypoint for executing
"""
settings = InputSettings()
logging.basicConfig(

@arielleleon arielleleon Jul 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this

seanmcculloch
seanmcculloch reviewed Jul 7, 2026
View reviewed changes

@seanmcculloch seanmcculloch left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, after addressing logging comments

arielleleon
arielleleon requested changes Jul 8, 2026
View reviewed changes

@arielleleon arielleleon left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of comments

Comment thread code/run_capsule.py
"Running QC"
)
raw_data_loader = RawDataLoader(raw_data_path)
pipeline_runner = Pipeline(raw_data_loader)

@arielleleon arielleleon Jul 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This Pipeline object should be removed. It's redundant to load the raw data here unless I am missing something. This capsule should only call run_qc_from_nwb

@arjunsridhar12345 arjunsridhar12345 Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's needed for running the harp qc suite which uses the raw data

Comment thread code/run_capsule.py
)

nwb_file_path = nwb_path[0] / "behavior.nwb.zarr"
with NWBZarrIO(nwb_file_path, "r") as io:

@arielleleon arielleleon Jul 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about remove the NWB IO from the capsule and putting it in the library? It seems like a processing library concern

@arjunsridhar12345 arjunsridhar12345 Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a preference. I can move it there if that's desired. It's only a few lines of code so should be good either way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@seanmcculloch seanmcculloch seanmcculloch left review comments
@arielleleon arielleleon arielleleon requested changes
@dougollerenshaw dougollerenshaw Awaiting requested review from dougollerenshaw

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Use dynamic foraging library to build QC

3 participants

@arjunsridhar12345 @arielleleon @seanmcculloch