Skip to content

Idea: parallelize multi-datasource processing (profile first) #55

Description

@AlexisJanin

Datasources are processed sequentially in the wrapper. Each is independent, so there may be wall-clock to gain on patients with many devices.

Worth a look someday, but not obviously worth doing — a few things to check before committing:

  • Profile first. Confirm datasource loading is actually the bottleneck (vs. rendering/resampling) on a real, full-size patient.
  • Threads vs processes depends on where the time goes (GIL): I/O / pyarrow reads favour threads; Python-heavy formatting would need processes. Note processes are the riskier option here because the app ships as a PyInstaller frozen bundle.
  • Touches the sequential progress_callback, output/snapshot ordering, and per-datasource error reporting — so it's not a drop-in.

Park for now; revisit if startup feels slow on real data.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions