add support for fine-grained tolerance settings #6921
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
datumbox
approved these changes
Nov 7, 2022
pmeier
commented
Nov 7, 2022
| class InfoBase: | ||
| def __init__(self, *, id, test_marks=None, closeness_kwargs=None): | ||
| def __init__( | ||
| self, |
There was a problem hiding this comment.
This change just moves the comments to the signature as it is done in all other related classes.
Comment on lines
+64
to
+67
| DEFAULT_PIL_REFERENCE_CLOSENESS_KWARGS = { | ||
| (("TestKernels", "test_against_reference"), torch.float32, "cpu"): dict(atol=1e-5, rtol=0, agg_method="mean"), | ||
| (("TestKernels", "test_against_reference"), torch.uint8, "cpu"): dict(atol=1e-5, rtol=0, agg_method="mean"), | ||
| } |
There was a problem hiding this comment.
Two lessons here:
- We only need the tolerances for the reference tests, but they were applied everywhere. There was no harm here, but this could have hidden bugs before.
- Some of our kernels like
resizeproduce quite large differences for some pixels, which are hidden by aggregating them. Maybe we should review and use stricter "default" tolerances and just increase it for the few that need more.
facebook-github-bot
pushed a commit
that referenced
this pull request
Nov 14, 2022
Summary: * add support for fine-grained tolerance settings * fix test_cuda_vs_cpu Reviewed By: NicolasHug Differential Revision: D41265204 fbshipit-source-id: 8309f5d957f0ff351277684349b6926c8f2c1775
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #6750. Instead of having a fixed
info.closeness_kwargs, this PR adds support forinfo.get_closeness_kwargs(test_id, dtype=dtype, device=device). Here,test_idis the same concept we are already using for@pytest.mark'svision/test/prototype_common_utils.py
Lines 605 to 607 in 4f3a000
Meaning, with this PR we have the ability to set the tolerances on a test, dtype, device level. IMO that should be enough. If someone needs even finer control, you are probably better off writing a custom test rather than using the common test infrastructure.
cc @vfdev-5 @datumbox @bjuncek