Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Use a better formula to calculate sigmoid_bce and logistic_loss #9404

Merged
merged 2 commits into from
Jan 15, 2018

Conversation

sxjscience
Copy link
Member

@sxjscience sxjscience commented Jan 12, 2018

Description

Use the same formula to calculate the BCE as TF https://github.com/tensorflow/tensorflow/blob/r1.1/tensorflow/python/ops/nn_impl.py#L128

The formula is simpler than the old one and is more accurate.

import mxnet as mx
import mxnet.ndarray as nd

def sigmoid_bce1(pred, label):
    max_val = nd.relu(-pred)
    loss = pred - pred*label + max_val + nd.log(nd.exp(-max_val)+nd.exp(-pred-max_val))
    return loss.asscalar()

def sigmoid_bce2(pred, label):
    loss = nd.relu(pred) - pred * label + nd.Activation(-nd.abs(pred), act_type='softrelu')
    return loss.asscalar()

for x, label in [(-100, 0), (-50, 0), (-20, 0), (-10, 0), (-1, 0), (1, 1), (10, 1), (20, 1), (50, 1), (100, 1)]:
    x_nd = nd.array([x], ctx=mx.gpu())
    label_nd = nd.array([label], ctx=mx.gpu())
    val1 = sigmoid_bce1(x_nd, label_nd)
    val2 = sigmoid_bce2(x_nd, label_nd)
    print("Formula1=", val1, "Formula2=", val2)
Formula1= 0.0 Formula2= 3.78351e-44
Formula1= 0.0 Formula2= 1.92875e-22
Formula1= 0.0 Formula2= 2.06115e-09
Formula1= 4.54177e-05 Formula2= 4.53989e-05
Formula1= 0.313262 Formula2= 0.313262
Formula1= 0.313262 Formula2= 0.313262
Formula1= 4.54177e-05 Formula2= 4.53989e-05
Formula1= 0.0 Formula2= 2.06115e-09
Formula1= 0.0 Formula2= 1.92875e-22
Formula1= 0.0 Formula2= 3.78351e-44

Checklist

Essentials

  • Passed code style checking (make lint)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new C++ functions in header files, their functionalities and arguments are documented.
  • For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

  • Use max(x, 0) - x * z + log(1 + exp(-abs(x))) to calculate BCE + Test

@szha
Copy link
Member

szha commented Jan 14, 2018

Could you also update the docstring for Input in LogisticLoss? It should have been updated when the label_format flag was introduced.

- **label**: truth tensor with values -1 or 1. Must have the same size
as pred.
- **label**: truth tensor with values -1/1 (label_format is 'signed')
or 0/1 (label_format is 'binary'). Must have the same size as pred.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@szha I've updated the description of 'label'.

@szha szha merged commit 134b205 into apache:master Jan 15, 2018
CodingCat pushed a commit to CodingCat/mxnet that referenced this pull request Jan 16, 2018
…he#9404)

* use a more stable formula to calculate sigmoid_bce and logistic_loss

* fix docstring
yuxiangw pushed a commit to yuxiangw/incubator-mxnet that referenced this pull request Jan 25, 2018
…he#9404)

* use a more stable formula to calculate sigmoid_bce and logistic_loss

* fix docstring
rahul003 pushed a commit to rahul003/mxnet that referenced this pull request Jun 4, 2018
…he#9404)

* use a more stable formula to calculate sigmoid_bce and logistic_loss

* fix docstring
zheng-da pushed a commit to zheng-da/incubator-mxnet that referenced this pull request Jun 28, 2018
…he#9404)

* use a more stable formula to calculate sigmoid_bce and logistic_loss

* fix docstring
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants