Update robust loss #502

xphsean12 · 2023-04-27T05:30:26Z

Motivation and Context

Current Robust Loss cannot deal with outlinear existing in a optimization problem

How Has This Been Tested

Types of changes

Docs change / refactoring / dependency upgrade
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have completed my CLA (see CONTRIBUTING)
I have added tests to cover my changes.
All new and existing tests passed.

Modification of Robust Loss. Original one function on the whole norm square, while the new one function on individual residuals. And the radius is changed from exp() to square()

Because I do not want to mess up the current environment of Theseus, it would be more convenient to test the new Robust Loss Class defined in the Notebook

luisenp

Thanks a lot for your contribution! I left some comments that need to be addressed.

Also CI is failing due to some lint errors. Can you please install pip install -r requirements/dev.txt and also pre-commit hooks, so that they take care of lint errors automatically.

Thanks!

luisenp · 2023-04-28T03:50:05Z

theseus/core/robust_cost_function.py

@@ -143,4 +186,4 @@ def _supports_masking(self) -> bool:
    @_supports_masking.setter
    def _supports_masking(self, val: bool):
        self.cost_function._supports_masking = val
-        self.__supports_masking__ = val
+        self.__supports_masking__ = val


Missing end line at end of file.

luisenp · 2023-04-28T03:58:14Z

theseus/core/robust_cost_function.py

@@ -52,6 +52,7 @@ def __init__(
        cost_function: CostFunction,
        loss_cls: Type[RobustLoss],
        log_loss_radius: Variable,
+        RobustSum: bool = False,


Let's keep python naming conventions using lower_case for variables instead of RobustSum. Not sure what a good name for this should be; let's start with flatten_dims: bool = False, which defaults to the old version, and True means the new version.

luisenp · 2023-04-28T03:58:26Z

theseus/core/robust_cost_function.py

@@ -70,6 +71,7 @@ def __init__(
        self.log_loss_radius = log_loss_radius
        self.register_aux_var("log_loss_radius")
        self.loss = loss_cls()
+        self.RobustSum = RobustSum


self.flatten_dims = flatten_dims

luisenp · 2023-04-28T04:01:14Z

theseus/core/robust_cost_function.py

-        error_loss = self.loss.evaluate(squared_norm, self.log_loss_radius.tensor)
-
+
+        if self.RobustSum:


I think this can be done much more simply. If I'm not mistaken, you can keep all of the old code and just add the following lines at the beginning of weighted_error()

if self.flatten_dims: weighted_error = weighted_error.view(-1, 1)

and then reshape the output back to (-1, self.dim()) before returning.

A similar procedure can be followed for weighted_jacobians.

I am not sure what you mean. And why I need to reshape it. Because what I want is given any shape of error, like (Batch, Dim) and the corresponding radius with the shape of (Batch, 1), they will all be passed to the evaluation function self.loss.evaluate and return the scaled loss.

It's a bit hard to explain, but it works, I just tested it. I'll put another PR soon.

Please take a look at #503. The code for the jacobians part was a bit tricker than I suggested above, but I was still able to mostly use all of the old computation. Also look at the unit test I added and let me know if covers the expected behavior. Thanks!

It's a bit hard to explain, but it works, I just tested it. I'll put another PR soon.

Oh! I see what you mean! You treat every element of the original cost as the original "square norm"

luisenp · 2023-04-28T04:01:57Z

theseus/core/robust_cost_function.py

+            return (
+              (error_loss  + RobustCostFunction._EPS).sqrt()
+            )
+        # Inside will compare squared_norm with exp(radius)


Please remove any debug comments.

luisenp · 2023-04-28T04:02:23Z

theseus/core/robust_loss.py

    @classmethod
    def evaluate(cls, x: torch.Tensor, log_radius: torch.Tensor) -> torch.Tensor:
-        return cls._evaluate_impl(x, log_radius.exp())
-
+        return cls._evaluate_impl(x, log_radius.square()) 


We should keep this as exp().

Sure. Could I know the initiative behind it?

Since square() is not monotonous, I personally prefer 'exp()', which is more useful when we try to learn a robust loss radius.

Oh! That's great! Because at a very early time, I tried to let the system automatically learn the Radius, and I failed at that time (performance was not good). But with the new version, the training outcome may change.

luisenp · 2023-04-28T04:02:31Z

theseus/core/robust_loss.py

@@ -49,4 +50,4 @@ def _evaluate_impl(x: torch.Tensor, radius: torch.Tensor) -> torch.Tensor:

    @staticmethod
    def _linearize_impl(x: torch.Tensor, radius: torch.Tensor) -> torch.Tensor:
-        return torch.sqrt(radius / torch.max(x, radius) + _LOSS_EPS)
+        return torch.sqrt(radius / torch.max(x, radius) + _LOSS_EPS)


Missing line at end of file.

luisenp · 2023-04-28T04:04:33Z

tutorials/test_RobustLoss.ipynb

@@ -0,0 +1,731 @@
+{


For the unit test, the right place to do it is this file. There is no need to write a full example. We only need two versions of a small objective and compare that the both versions lead to the same error and jacobians tensors. We can remove this notebook.

I use notebook to test to avoid the old version will mess up with the new one. Because I have installed Theseus 0.1.4 and do not want to make any changes to it. And any of the function that call th.xx will call the original Theseus function. Do you have any good ideas? Or I can just replace my new script file with the one in the environment site-package?

luisenp · 2023-04-28T04:10:23Z

theseus/core/robust_cost_function.py

-        return [
-            rescale.view(-1, 1, 1) * jacobian for jacobian in weighted_jacobians
-        ], rescale * weighted_error
+        #squared_norm = torch.sum(weighted_error**2, dim=1, keepdim=True)


See comment above on a suggestion for implementing this more easily.

fantaosha

Add some more comments in #503.

fantaosha · 2023-04-28T19:51:44Z

theseus/core/robust_loss.py

    @classmethod
    def evaluate(cls, x: torch.Tensor, log_radius: torch.Tensor) -> torch.Tensor:
-        return cls._evaluate_impl(x, log_radius.exp())
-
+        return cls._evaluate_impl(x, log_radius.square()) 


Since square() is not monotonous, I personally prefer 'exp()', which is more useful when we try to learn a robust loss radius.

xphsean12 added 3 commits April 27, 2023 13:16

Add files via upload

f7b1e7e

Modification of Robust Loss. Original one function on the whole norm square, while the new one function on individual residuals. And the radius is changed from exp() to square()

Test of Robust Loss

594ecbf

Because I do not want to mess up the current environment of Theseus, it would be more convenient to test the new Robust Loss Class defined in the Notebook

Add files via upload

a7a1173

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 27, 2023

xphsean12 added 2 commits April 27, 2023 14:48

Add files via upload

335363c

Add files via upload

3095ea9

mhmukadam changed the title ~~Updata robust loss~~ Update robust loss Apr 27, 2023

mhmukadam requested review from luisenp and fantaosha April 27, 2023 17:35

mhmukadam assigned xphsean12 Apr 27, 2023

mhmukadam added the enhancement New feature or request label Apr 27, 2023

mhmukadam linked an issue Apr 27, 2023 that may be closed by this pull request

Huber Loss Not Correct #497

Closed

xphsean12 added 2 commits April 28, 2023 11:28

Add files via upload

c787ad4

Add files via upload

c78a18f

luisenp requested changes Apr 28, 2023

View reviewed changes

xphsean12 requested a review from luisenp April 28, 2023 05:20

fantaosha reviewed Apr 28, 2023

View reviewed changes

luisenp closed this May 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update robust loss #502

Update robust loss #502

xphsean12 commented Apr 27, 2023

luisenp left a comment

luisenp Apr 28, 2023

luisenp Apr 28, 2023

luisenp Apr 28, 2023

luisenp Apr 28, 2023

xphsean12 Apr 28, 2023

luisenp Apr 28, 2023

luisenp Apr 28, 2023

xphsean12 Apr 29, 2023

luisenp Apr 28, 2023

luisenp Apr 28, 2023

xphsean12 Apr 28, 2023

fantaosha Apr 28, 2023

xphsean12 Apr 29, 2023

luisenp Apr 28, 2023

luisenp Apr 28, 2023 •

edited

Loading

xphsean12 Apr 28, 2023

luisenp Apr 28, 2023

fantaosha left a comment

fantaosha Apr 28, 2023

		error_loss = self.loss.evaluate(squared_norm, self.log_loss_radius.tensor)


		if self.RobustSum:

Update robust loss #502

Update robust loss #502

Conversation

xphsean12 commented Apr 27, 2023

Motivation and Context

How Has This Been Tested

Types of changes

Checklist

luisenp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luisenp Apr 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fantaosha left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luisenp Apr 28, 2023 •

edited

Loading