MaximumLikelihoodEstimationModel and NaNs

[FranzHahn]

It seems like the MaximumLikelihoodEstimationModel is introducing NaNs somewhere. When trying it out with my data that has no NaN values anywhere I get an error:

ValueErrorTraceback (most recent call last)
<ipython-input-1-cf2cd2ac15d7> in <module>()
     22     sub_mod_denoise_mle = MaximumLikelihoodEstimationModel(dataset_reader=dataset_reader)
     23     sub_mod_denoise_mle_rest = MaximumLikelihoodEstimationModel(dataset_reader=dataset_reader_rest)
---> 24     result_denoise_mle = sub_mod_denoise_mle.run_modeling()
     25     result_denoise_mle_rest = sub_mod_denoise_mle_rest.run_modeling()
     26     corrs_denoised_mle.append(np.corrcoef(result_denoise_mle['quality_scores'],result_denoise_mle_rest['quality_scores']))

/home/fh/miniconda2/envs/keras_py27up/lib/python2.7/site-packages/sureal/subjective_model.py in run_modeling(self, **kwargs)
     61 
     62     def run_modeling(self, **kwargs):
---> 63         model_result = self._run_modeling(self.dataset_reader, **kwargs)
     64         self._postprocess_model_result(model_result, **kwargs)
     65         self.model_result = model_result

/home/fh/miniconda2/envs/keras_py27up/lib/python2.7/site-packages/sureal/subjective_model.py in _run_modeling(cls, dataset_reader, **kwargs)
    816             print(np.isnan(x_e).any())
    817 
--> 818             delta_x_e = linalg.norm(x_e_prev - x_e)
    819 
    820             likelihood = np.sum(cls.loglikelihood_fcn(x_es, x_e, b_s, v_s, a_c, dataset_reader.content_id_of_dis_videos, 1, numerical_pdf))

/home/fh/miniconda2/envs/keras_py27up/lib/python2.7/site-packages/scipy/linalg/misc.pyc in norm(a, ord, axis, keepdims)
    135     """
    136     # Differs from numpy only in non-finite handling and the use of blas.
--> 137     a = np.asarray_chkfinite(a)
    138 
    139     # Only use optimized norms if axis and keepdims are not specified.

/home/fh/miniconda2/envs/keras_py27up/lib/python2.7/site-packages/numpy/lib/function_base.pyc in asarray_chkfinite(a, dtype, order)
   1231     if a.dtype.char in typecodes['AllFloat'] and not np.isfinite(a).all():
   1232         raise ValueError(
-> 1233             "array must not contain infs or NaNs")
   1234     return a
   1235 

ValueError: array must not contain infs or NaNs

Notice I did change the model to check for NaN values in x_e_prev and x_e, and only x_e contains them. I am not sure what causes this bug, but I'd love to see if someone knows what's the cause.

Edit: Also, I get these additional warning messages:

/home/fh/miniconda2/envs/keras_py27up/lib/python2.7/site-packages/sureal/subjective_model.py:711: RuntimeWarning: divide by zero encountered in divide
  num = - np.tile(a_c_e, (S, 1)).T / vs2_add_ace2 + np.tile(a_c_e, (S, 1)).T * a_es**2 / vs2_add_ace2**2
/home/fh/miniconda2/envs/keras_py27up/lib/python2.7/site-packages/sureal/subjective_model.py:711: RuntimeWarning: invalid value encountered in divide
  num = - np.tile(a_c_e, (S, 1)).T / vs2_add_ace2 + np.tile(a_c_e, (S, 1)).T * a_es**2 / vs2_add_ace2**2
/home/fh/miniconda2/envs/keras_py27up/lib/python2.7/site-packages/sureal/subjective_model.py:717: RuntimeWarning: divide by zero encountered in divide
  den = - vs2_minus_ace2 / vs2_add_ace2**2 + a_es**2 * poly_term / vs2_add_ace2**4
/home/fh/miniconda2/envs/keras_py27up/lib/python2.7/site-packages/sureal/subjective_model.py:717: RuntimeWarning: invalid value encountered in divide
  den = - vs2_minus_ace2 / vs2_add_ace2**2 + a_es**2 * poly_term / vs2_add_ace2**4
/home/fh/miniconda2/envs/keras_py27up/lib/python2.7/site-packages/sureal/subjective_model.py:725: RuntimeWarning: divide by zero encountered in divide
  -vs2_minus_ace2 / vs2_add_ace2**2 + a_es**2 * poly_term / vs2_add_ace2**4
/home/fh/miniconda2/envs/keras_py27up/lib/python2.7/site-packages/sureal/subjective_model.py:725: RuntimeWarning: invalid value encountered in divide
  -vs2_minus_ace2 / vs2_add_ace2**2 + a_es**2 * poly_term / vs2_add_ace2**4
/home/fh/miniconda2/envs/keras_py27up/lib/python2.7/site-packages/sureal/subjective_model.py:730: RuntimeWarning: divide by zero encountered in divide
  a_c_std = 1.0 /np.sqrt(np.maximum(0., -lpp))
/home/fh/miniconda2/envs/keras_py27up/lib/python2.7/site-packages/sureal/subjective_model.py:779: RuntimeWarning: divide by zero encountered in divide
  num = pd.DataFrame(num_num / num_den).sum(axis=1) # sum over s
/home/fh/miniconda2/envs/keras_py27up/lib/python2.7/site-packages/sureal/subjective_model.py:782: RuntimeWarning: divide by zero encountered in divide
  den = pd.DataFrame(den_num / den_den).sum(axis=1) # sum over s

[FranzHahn]

Maybe I should clarify. My dataset is not annotated by each user entirely. So I have set up my data as you suggest in the Readme, and the SR models work just fine (and improve the mos), but the MLE models don't seem to work. None of the tests you supply seem to handle partially-annotated data in the sense that observers have not annotated all contents.

[FranzHahn]

The problem seems to be in the gradient computations of the model parameters. Depending on the data I input I get division by zero warnings at different places. I have created a sample dataset where all stimuli are annotated by all participants. If I remove one single vote anywhere I get the errors. I'd be super glad, if you could help me figure out where the problem lies.

[li-zhi]

Hi @FranzHahn I'd love to take a look if I can reproduce the issue, if you could share with me your dataset.

[FranzHahn]

Hi @li-zhi , I have a json file that holds the data on a pastebin. Here, specifically, the last stimulus is missing a vote from "44796579" (which is 3). Using this dataset gives me the above mentioned errors. Specifically, it seems that variables such as vs2_add_ace2, or num_den seem to take the value 0, subsequently cause a division by 0, and introduce NaNs into the calculations. This happens regardless of the gradient methods.

Edit: Additionally, there are some problematic statements in your code:

a_c_std = 1.0 /np.sqrt(np.maximum(0., -lpp))

Shouldn't this be

a_c_std = 1.0 /np.sqrt(np.maximum(EPSILON**2, -lpp))

such that a division by 0 can never occur?

[li-zhi]

@FranzHahn I can reproduce your issue. With "MLE" I ended up with the same "NaN" result. If I switch to "DMOS_MLE" model (MLE running on the DMOS instead of MOS scores), things work better.

I don't think forcing a_c_std = 1.0 /np.sqrt(np.maximum(EPSILON**2, -lpp)) would fundamentally solve the issue (have you tried?).

The current MLE solver is known to have the issue of lacking of uniqueness in its solution. Some discussions can be found here. This topic has been discussed quite extensively in the VQEG SAM meeting group.

Since you have a small dataset, it seems more likely it would invoke its unstable behavior, thus the division by 0 result.

We don't have a good solution at this moment, and we are lack of theoretically analysis. You are more than welcomed to help look into possible solutions.

[FranzHahn]

I have tried adding EPSILONs anywhere a division occurs to potentially catch division by 0 but that alone didn't solve the issues. I suspect a better initial guess for some of the parameters would help. I have some ideas in this regard. I will see if I can implement them and inform you if it helps.

Why do you think the small size of the start should have a negative impact on this issue? I have much bigger variants of this dataset where the issue occurs all the same. In fact, on none my of partially annotated datasets of tiny to huge sizes does the mle method work. And I would have assumed that smaller datasets with fewer stimuli and fewer subjects would be less complex, as there are fewer parameters?

[FranzHahn]

@FranzHahn I can reproduce your issue. With "MLE" I ended up with the same "NaN" result. If I switch to "DMOS_MLE" model (MLE running on the DMOS instead of MOS scores), things work better.

Could you enlighten me as to how to use your model to do this?

Are the resulting scores meaningful?

I've tried using dataset_reader.to_pc_dataset but I'm not sure what to do with this then?

Could you supply me a short code snippet how to switch to DMOS MLE model?

[li-zhi]

Hi you can run it in the command line:

python -m sureal DMOS_MLE dataset.py

See the README.md file for more information.

The dataset can be prepared in the following format (I converted your json into the following .py file):

dataset_name = 'FranzTest'
ref_score = 5

ref_videos = [
    {
        "path": "orig_10181314856_540_5s.mp4",
        "content_id": 0,
        "content_name": "10181314856"
    },
    {
        "path": "orig_10259212155_540_5s.mp4",
        "content_id": 1,
        "content_name": "10259212155"
    },
    {
        "path": "orig_10320058616_540_5s.mp4",
        "content_id": 2,
        "content_name": "10320058616"
    },
    {
        "path": "orig_10332507295_540_5s.mp4",
        "content_id": 3,
        "content_name": "10332507295"
    },
    {
        "path": "orig_10410965174_540_5s.mp4",
        "content_id": 4,
        "content_name": "10410965174"
    },
    {
        "path": "orig_10420562595_540_5s.mp4",
        "content_id": 5,
        "content_name": "10420562595"
    },
    {
        "path": "orig_10932852506_540_5s.mp4",
        "content_id": 6,
        "content_name": "10932852506"
    }
]

dis_videos = [
    {
        "asset_id": 0,
        "path": "orig_10181314856_540_5s.mp4",
        "content_id": 0,
        "os": {
            "44796579": 3,
            "44279676": 4,
            "43974450": 3,
            "39602571": 4,
            "44733676": 3
        }
    },
    {
        "asset_id": 1,
        "path": "orig_10259212155_540_5s.mp4",
        "content_id": 1,
        "os": {
            "44796579": 3,
            "44279676": 4,
            "43974450": 1,
            "39602571": 3,
            "44733676": 3
        }
    },
    {
        "asset_id": 2,
        "path": "orig_10320058616_540_5s.mp4",
        "content_id": 2,
        "os": {
            "44796579": 3,
            "44279676": 4,
            "43974450": 2,
            "39602571": 4,
            "44733676": 2
        }
    },
    {
        "asset_id": 3,
        "path": "orig_10332507295_540_5s.mp4",
        "content_id": 3,
        "os": {
            "44796579": 3,
            "44279676": 3,
            "43974450": 3,
            "39602571": 4,
            "44733676": 3
        }
    },
    {
        "asset_id": 4,
        "path": "orig_10410965174_540_5s.mp4",
        "content_id": 4,
        "os": {
            "44796579": 3,
            "44279676": 2,
            "43974450": 4,
            "39602571": 4,
            "44733676": 4
        }
    },
    {
        "asset_id": 5,
        "path": "orig_10420562595_540_5s.mp4",
        "content_id": 5,
        "os": {
            "44796579": 3,
            "44279676": 4,
            "43974450": 2,
            "39602571": 1,
            "44733676": 2
        }
    },
    {
        "asset_id": 6,
        "path": "orig_10932852506_540_5s.mp4",
        "content_id": 6,
        "os": {
            "44279676": 2,
            "43974450": 3,
            "39602571": 2,
            "44733676": 3
        }
    }
]

Note that to use DMOS it requires a field called ref_score so that the functions know that 5 is the reference video's quality score.