Skip to content

Conversation

@paulz
Copy link
Contributor

@paulz paulz commented Mar 25, 2025

This pull request includes several changes to the examples/team_recommender/tests suite, focusing on statistical significance testing and sample size calculations. The most important changes include the addition of new functions, modifications to existing test functions, and the removal of a test file.

New functions and modifications:

Test function updates:

File removals and dependency updates:

@paulz paulz marked this pull request as ready for review March 25, 2025 18:47
carl added 3 commits March 25, 2025 11:47
Signed-off-by: Paul Zabelin <paulzabelin@artium.ai>
Signed-off-by: Paul Zabelin <paulzabelin@artium.ai>
Signed-off-by: Paul Zabelin <paulzabelin@artium.ai>
@paulz paulz force-pushed the test-examples-on-wiki branch from d66d649 to 879b9e8 Compare March 25, 2025 18:47
@tkersey tkersey requested a review from Copilot March 25, 2025 18:48
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

A refactoring to correct statistical calculations in calculate_ztest and improve test clarity for sample size functions.

  • Corrected the parameters passed to proportions_ztest in calculate_ztest.
  • Renamed and introduced new sample size functions to handle both one failure and no failure cases with accompanying tests.
  • Improved error messages and added a test case based on a wiki example.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
examples/team_recommender/tests/test_helpers.py Refactored sample size helper functions and updated tests with consistent naming.
examples/team_recommender/tests/test_proportions_ztest.py Corrected calculate_ztest parameters and updated significance tests with a wiki example.
Comments suppressed due to low confidence (3)

examples/team_recommender/tests/test_proportions_ztest.py:126

  • The error message does not match the test parameters; it refers to '0 out of 3' while the sample size is 1000. Please update the message to accurately reflect the test scenario.
assert is_statistically_significant(0.7, 0, 1000), "not significant result for 0 out of 3"

examples/team_recommender/tests/test_proportions_ztest.py:131

  • The assertion expects a statistically significant result, but the error message suggests otherwise. Please align the error message with the expected outcome.
assert is_statistically_significant(0.7, 0, 10), "no improvement detected at 10"

examples/team_recommender/tests/test_proportions_ztest.py:139

  • The error message implies a lack of improvement despite the assertion requiring statistical significance. Please update the message to match the intended check.
assert is_statistically_significant(0.97, 0, 100), "no improvement detected at 100"

carl added 5 commits March 25, 2025 12:05
Signed-off-by: Paul Zabelin <paulzabelin@artium.ai>
Signed-off-by: Paul Zabelin <paulzabelin@artium.ai>
test_no_failures_always_cause_insignificance

Signed-off-by: Paul Zabelin <paulzabelin@artium.ai>
Signed-off-by: Paul Zabelin <paulzabelin@artium.ai>
as they generate stats only for 95% confidence and we use 90%

Signed-off-by: Paul Zabelin <paulzabelin@artium.ai>
@carl
Copy link
Contributor

carl commented Mar 25, 2025

🐻 approved

@paulz paulz merged commit 2329c15 into thisisartium:main Mar 25, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants