metrics.py: fix inconsistent interface 3/n #41
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t.py, thresholding.py, and torchscript_modules.py
…calibration import [...]'
…st, which is not included in this release
…th Apple M1/M2 hardware, resulting in segmentation faults. We ignore affected tests.
…ting_file_exists Adding Contributing file
…conduct_file_exists Adding Code of Conduct file
… is not release yet on pypi. Here we undo the tuning change to make it compatible with 1.1.2.
- Fix spelling and grammar issues throughout documentation - Enhance clarity with concrete examples and expanded acronyms - Correct MCE definition from 'Maximum Calibration Error' to 'Multicalibration Error' - Fix all code examples to match actual DataFrame-based API - Add academic references for main paper (KDD 2026), metrics paper, and applications paper - Update all documentation files with accurate method signatures and parameter names Files modified: - README.md: Add proper citations, fix code example API - pyproject.toml: Fix subject-verb agreement - website/docs/: Update all doc pages with correct API usage - website/src/pages/index.js: Update homepage with correct citations and code example
The internal and external repositories are out of sync. This Pull Request attempts to brings them back in sync by patching the GitHub repository. Please carefully review this patch. You must disable ShipIt for your project in order to merge this pull request. DO NOT IMPORT this pull request. Instead, merge it directly on GitHub using the MERGE BUTTON. Re-enable ShipIt after merging.
The internal and external repositories are out of sync. This Pull Request attempts to brings them back in sync by patching the GitHub repository. Please carefully review this patch. You must disable ShipIt for your project in order to merge this pull request. DO NOT IMPORT this pull request. Instead, merge it directly on GitHub using the MERGE BUTTON. Re-enable ShipIt after merging.
The internal and external repositories are out of sync. This Pull Request attempts to brings them back in sync by patching the GitHub repository. Please carefully review this patch. You must disable ShipIt for your project in order to merge this pull request. DO NOT IMPORT this pull request. Instead, merge it directly on GitHub using the MERGE BUTTON. Re-enable ShipIt after merging.
Summary: Pull Request resolved: #24 We're starting to see some "running out of disk space" errors on the machines that build Sphinx. We have plenty of budget, so let's just throw a bigger machine. #18 Reviewed By: Lorenzo-Perini Differential Revision: D88486737 fbshipit-source-id: 3ab67c816a15a6d54bc330106e962b3e1f315e70
facebook-github-bot
pushed a commit
that referenced
this pull request
Dec 9, 2025
Summary: Some of our methods using `predicted_labels` (for discrete predictions): - recall - precision - dcg_score - ndcg_score etc. Other methods used `predicted_scores` (for continuous functions): - expected_calibration_error - proportional_expected_calibration_error - adaptive_calibration_error - proportional_adaptive_calibration_error - calibration_ratio - kuiper_calibration However, `normalized_entropy` and `calibration_free_normalized_entropy` used `predictions`, which is inconsistent. Let's switch to `predicted_scores` like the rest. Differential Revision: D88746323
… internal only module (#36) Summary: Pull Request resolved: #36 * MCNet is not part of the OSS release * Create a new module for internal only implementations * For backward compatibility re-export MCNet in `methods.py` with `oss-disable` directive * To avoid circular import due to `BaseCalibrator` move the abstract class to a new private module Reviewed By: TaXxER Differential Revision: D88473570 fbshipit-source-id: 96f52388e4d9055db020418b8b44703c5bac2466
…ng -> PlattScalingWithFeatures (#34) Summary: Pull Request resolved: #34 SwissCheese is not a public name so we should rename this to something more generic. * Rename the implementation `SwissCheesePlattScaling` -> `PlattScalingWithFeatures` * Create a wrapper class of `SwissCheesePlattScaling` for backward compatibility (oss-disabled) Reviewed By: TaXxER Differential Revision: D88475946 Privacy Context Container: L1334583 fbshipit-source-id: 0a2120d75ab42d1ee8f4965383ff43157e70b3fc
facebook-github-bot
pushed a commit
that referenced
this pull request
Dec 10, 2025
Summary: Some of our methods using `predicted_labels` (for discrete predictions): - recall - precision - dcg_score - ndcg_score etc. Other methods used `predicted_scores` (for continuous functions): - expected_calibration_error - proportional_expected_calibration_error - adaptive_calibration_error - proportional_adaptive_calibration_error - calibration_ratio - kuiper_calibration However, `normalized_entropy` and `calibration_free_normalized_entropy` used `predictions`, which is inconsistent. Let's switch to `predicted_scores` like the rest. Differential Revision: D88746323
facebook-github-bot
pushed a commit
that referenced
this pull request
Dec 10, 2025
Summary: Some of our methods using `predicted_labels` (for discrete predictions): - recall - precision - dcg_score - ndcg_score etc. Other methods used `predicted_scores` (for continuous functions): - expected_calibration_error - proportional_expected_calibration_error - adaptive_calibration_error - proportional_adaptive_calibration_error - calibration_ratio - kuiper_calibration However, `normalized_entropy` and `calibration_free_normalized_entropy` used `predictions`, which is inconsistent. Let's switch to `predicted_scores` like the rest. Differential Revision: D88746323
facebook-github-bot
force-pushed
the
export-D88746323
branch
from
0ddd339 to
3cee263
Compare
…/private methods naming (#43) Summary: Pull Request resolved: #43 We have several methods that are just used internally and not meant to be exposed to users / documented in our website / docs. Let's make this consistent and only expose a selection of methods that are core functionality and we want to fully document. Reviewed By: TaXxER Differential Revision: D88750778 fbshipit-source-id: b125880f51d516b0d246cd93c79489dca50ff2eb
Summary: Pull Request resolved: #45 Latest sklearn version no longer accepts empty input, and raises an error. This breaks our unit test, and our Github CI. This change keeps our behavior the same as it was. See error: https://github.com/facebookincubator/MCGrad/actions/runs/20099929745/job/57667810962 Reviewed By: Lorenzo-Perini Differential Revision: D88847925 fbshipit-source-id: 57e50b0deaa6da4a36af5d180ea4e65309d5b482
Summary: Pull Request resolved: #39 This diff just does the low-hanging fruit: removing references to internal wikis, documentation, code, etc # Discussion point - There are still a bunch of metrics that have `adjust_unjoined` argument, and a description of what unjoined data is. It will be somewhat non-trivial to fully disentangle this from the main metrics. Do want to: 1) Fully remove all mentions of `adjust_unjoined`, disentangle all unjoined versions of the metrics from the main metric implementation, and move those to the `internal` folder, or 2) Simply just remove the references of where unjoined data is used (previously one comment said that it is used a lot in the Ads or), and be OK with `adjust_unjoined` existing in the codebase. My feeling is that this will come across at worst as "a bit silly" to outsiders, and they will wonder "why the hell would anyone ever have data in that format", but as I foresee, the harm of leaving it in won't really go beyond that. Reviewed By: Lorenzo-Perini Differential Revision: D88739146 fbshipit-source-id: e229d5d056d45a43e3c963d33def69d3915e8478
Summary: Pull Request resolved: #40 Changes: - Fixed typo min_segmens_size -> min_segments_size - One of the tests missed the `test_` prefix and therefore never ran. It was actually a test to check that a mutable input argument was never modified, so given recent SEV, it is an important type of test - In one unit test, I switched to using an RNG seed, to prevent that this test might potentially be flaky / non-deterministic - List comprehension was unnecessarily complex with brackets and parantheses Reviewed By: Lorenzo-Perini Differential Revision: D88744756 fbshipit-source-id: f4a0d8228149b943b32b4ecf0373d923e61bb271
Summary: Pull Request resolved: #46 Providing arguments to pass df_val into tuning/final fit. Likely not something everyone wants to do - especially if people want to use cross validation. But this provides an option for those usecases where it's needed. Reviewed By: flinder Differential Revision: D88816384 fbshipit-source-id: 80da8accc544027222bae7cc8052b02650eb6fad
…ith `ValueErrors` (#48) Summary: Pull Request resolved: #48 Replaced 3 assert statements used for input validation with proper raise ValueError exceptions. Asserts can be disabled with Python's -O flag, making them unsuitable for production validation. This ensures validation always runs regardless of Python optimization settings. Reviewed By: leepface Differential Revision: D89060673 fbshipit-source-id: 8eb8c199b24135b23a3eb1643d9c0277ed835382
…or EARLY_STOPPING_ESTIMATION_METHOD (#49) Summary: Pull Request resolved: #49 Replaced a deeply nested ternary expression with a clear if/elif/else block when determining the early stopping estimation method. Added explicit type annotation before the conditional to satisfy Pyre. Reviewed By: leepface Differential Revision: D89060683 fbshipit-source-id: ebb4b7b3f4b6cb74eff03975d4379f7e54e0a641
…n expressions (#50) Summary: Pull Request resolved: #50 Simplified two redundant True if condition else False patterns to direct boolean expressions. The comparison already returns a boolean, so wrapping in a ternary is unnecessary. Reviewed By: leepface Differential Revision: D89060702 fbshipit-source-id: 4c18f4edc177509582dabfff0ce63e7c53e610a6
…> lower case class attributes (start with MONTONE_T) (#54) Summary: Pull Request resolved: #54 Dynamically set class attributes should be lower case according to PEP8 / Google style guide. I introduce a backward compatibility module `_compat.py` (python convention) to avoid cluttering the main class. How it works: It implements a Mixinclass which adds deprecated alias descriptors to the class. This allows retaining the old attribute name and produce a deprecation warning whenever it is set or accessed. Since we don't have open source users yet, backward compatibility is not a concern there, therefore I `oss-disable` the mixin. Reviewed By: Lorenzo-Perini Differential Revision: D89061588 fbshipit-source-id: e7564e6d6437c31234b381a73b2b5c6c7a32aa93
Summary: Some of our methods using `predicted_labels` (for discrete predictions): - recall - precision - dcg_score - ndcg_score etc. Other methods used `predicted_scores` (for continuous functions): - expected_calibration_error - proportional_expected_calibration_error - adaptive_calibration_error - proportional_adaptive_calibration_error - calibration_ratio - kuiper_calibration However, `normalized_entropy` and `calibration_free_normalized_entropy` used `predictions`, which is inconsistent. Let's switch to `predicted_scores` like the rest. Differential Revision: D88746323
facebook-github-bot
pushed a commit
that referenced
this pull request
Dec 19, 2025
Summary: Some of our methods using `predicted_labels` (for discrete predictions): - recall - precision - dcg_score - ndcg_score etc. Other methods used `predicted_scores` (for continuous functions): - expected_calibration_error - proportional_expected_calibration_error - adaptive_calibration_error - proportional_adaptive_calibration_error - calibration_ratio - kuiper_calibration However, `normalized_entropy` and `calibration_free_normalized_entropy` used `predictions`, which is inconsistent. Let's switch to `predicted_scores` like the rest. Differential Revision: D88746323
facebook-github-bot
pushed a commit
that referenced
this pull request
Dec 19, 2025
Summary: Some of our methods using `predicted_labels` (for discrete predictions): - recall - precision - dcg_score - ndcg_score etc. Other methods used `predicted_scores` (for continuous functions): - expected_calibration_error - proportional_expected_calibration_error - adaptive_calibration_error - proportional_adaptive_calibration_error - calibration_ratio - kuiper_calibration However, `normalized_entropy` and `calibration_free_normalized_entropy` used `predictions`, which is inconsistent. Let's switch to `predicted_scores` like the rest. Differential Revision: D88746323
facebook-github-bot
force-pushed
the
export-D88746323
branch
2 times, most recently
from
d39f7fc to
142d2b4
Compare
facebook-github-bot
pushed a commit
that referenced
this pull request
Dec 19, 2025
Summary: Some of our methods using `predicted_labels` (for discrete predictions): - recall - precision - dcg_score - ndcg_score etc. Other methods used `predicted_scores` (for continuous functions): - expected_calibration_error - proportional_expected_calibration_error - adaptive_calibration_error - proportional_adaptive_calibration_error - calibration_ratio - kuiper_calibration However, `normalized_entropy` and `calibration_free_normalized_entropy` used `predictions`, which is inconsistent. Let's switch to `predicted_scores` like the rest. Differential Revision: D88746323
facebook-github-bot
force-pushed
the
main
branch
from
d3f884a to
126ba8b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Some of our methods using
predicted_labels(for discrete predictions):etc.
Other methods used
predicted_scores(for continuous functions):However,
normalized_entropyandcalibration_free_normalized_entropyusedpredictions, which is inconsistent. Let's switch topredicted_scoreslike the rest.Differential Revision: D88746323