Skip to content

use multiprocessing in validation and analysis for speedup #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 3, 2025

Conversation

AurelienJaquier
Copy link
Collaborator

Validation running time went from 100s to 20s on my machine (10s for threshold computation and 10s for validation)

Each validation now runs in parallel in a separate pool.
Each simulation in iv_curve and fi_curve also run in parallel in a separate pool

@AurelienJaquier AurelienJaquier self-assigned this Jun 3, 2025
Copy link

codecov bot commented Jun 3, 2025

Codecov Report

Attention: Patch coverage is 93.87755% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
tests/test_utils.py 90.90% 2 Missing ⚠️
tests/test_analysis/test_analysis.py 75.00% 1 Missing ⚠️
Files with missing lines Coverage Δ
bluecellulab/analysis/analysis.py 31.60% <100.00%> (-2.49%) ⬇️
bluecellulab/utils.py 97.67% <100.00%> (+1.00%) ⬆️
tests/test_analysis/test_analysis.py 94.02% <75.00%> (-4.48%) ⬇️
tests/test_utils.py 96.36% <90.90%> (-3.64%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AurelienJaquier
Copy link
Collaborator Author

pinging @pgetta since he recommended to parallelize validation for speedup.

@AurelienJaquier AurelienJaquier requested a review from ilkilic June 3, 2025 12:19
Copy link
Collaborator

@ilkilic ilkilic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Thanks a lot for the boost!

@AurelienJaquier AurelienJaquier merged commit 4debf4a into main Jun 3, 2025
8 checks passed
@AurelienJaquier AurelienJaquier deleted the multiprocessing branch June 3, 2025 12:38
@pgetta
Copy link

pgetta commented Jun 5, 2025

Thank you, @AurelienJaquier, that is awesome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants