Potential conflict with sklearn metrics?

[ThilinaRajapakse]

Any updated, it seems that evaluation has some confilt with sklearn metrics.

On Mon, 7 Oct 2019, 10:02 pm Thilina Rajapakse, notifications@github.com
wrote:

@ThilinaRajapakse commented on this pull request.

I just noticed this and fixed it before I saw your PR. Thanks!

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#3,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPJIZ5DGVH3O5IFCFYWWRTQNM6QVANCNFSM4I6EUXDQ
.

Originally posted by @hawktang in #3 (comment)

[ThilinaRajapakse]

What is the conflict? I didn't get any on my end.

[hawktang]

hi, Raj, Sorry, it is just a warning. /home/hawktang/anaconda3/envs/nlp/lib/python3.7/site-packages/sklearn/metrics/classification.py:872: RuntimeWarning: invalid value encountered in double_scalars mcc = cov_ytyp / np.sqrt(cov_ytyt * cov_ypyp) {'mcc': 0.0, 'tp': 0, 'tn': 1, 'fp': 0, 'fn': 1} Best, Peter Ze TANG (汤赜)

…

On Mon, Oct 7, 2019 at 10:13 PM Thilina Rajapakse ***@***.***> wrote: What is the conflict? I didn't get any on my end. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4?email_source=notifications&email_token=AAPJIZZQOE5VRWD6FF3PPXDQNM72LA5CNFSM4I6FDW2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAQPUSQ#issuecomment-539032138>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPJIZ4MLAGXEUNXHUAYPALQNM72LANCNFSM4I6FDW2A> .

[ThilinaRajapakse]

Ah, that is likely due to the fact that there are no true positives. It shouldn't be an issue on a real dataset.

[MrRobot2211]

I am geting a similar issue. The model is predicting all negatives on a real dataset. after 20 epochs

/home/ubuntu/anaconda3/lib/python3.7/site-packages/sklearn/metrics/classification.py:872: RuntimeWarning: invalid value encountered in double_scalars
mcc = cov_ytyp / np.sqrt(cov_ytyt * cov_ypyp)

result={'mcc': 0.0, 'tp': 0, 'tn': 6000, 'fp': 0, 'fn': 4679}

[rabeehkarimimahabadi]

Hi
the same issue, if the predictions are invalid, can we compute the correct mcc metrics? thanks

[ThilinaRajapakse]

You can't compute an MCC score when the predictions are invalid as a number divided by zero (which is what is happening here) is undefined.

[higopires]

This problem is happening to me only when I do hyperparameterization. In other problems, with the same dataset, this problem does not occur ...
This message appears when the evaluation is done during the training.

[ThilinaRajapakse]

It's not a "problem". It simply means that the MCC score is undefined at that particular moment.

If any of the terms in the denominator evaluates to 0, the MCC score is undefined.

[higopires]

Thanks, @ThilinaRajapakse.
What could be causing, in my situation, any of these values to be zero?

[ThilinaRajapakse]

It tends to happen in the early stages of training because the model is basically doing random predictions at that stage. It can also happen if the model is being trained with too large learning rates because the model will end up predicting the same label for all inputs.

[Cameron-Nguyen1]

In case new users come to this post like I did searching for answers as to why you received

"RuntimeWarning: invalid value encountered in double_scalars
mcc = cov_ytyp / np.sqrt(cov_ytyt * cov_ypyp)"

My reason was that my dataset was portioned into invalid training, validation, and testing allocations.
This means that my dataset was not properly split, try rearranging your dataset through a splitter function to remove the error. Once I did that, my model performed fantastically.