`TestStringMethods.test_multiExample` fails when run alone

[mohataher]

I have a had a look at your updated code. Interestingly enough, when running the two tests cases in TestStringMethods together, they both pass.

However, when running TestStringMethods.test_multiExample, it fails. I'm not sure why exactly. Do you have an explanation for this?

I ran that using Mac Pro machine, Python 3.6.4 (miniconda). Which machine and Python version did you use?

When test case passes, this is a snippet of the output

`# PASSED`
lam_sparse 0.11
switch_penalty 600
num_cluster 5
num stacked 5
completed getting the data



ITERATION ### 0
OPTIMIZATION for Cluster # 0 DONE!!!
OPTIMIZATION for Cluster # 1 DONE!!!
OPTIMIZATION for Cluster # 2 DONE!!!
OPTIMIZATION for Cluster # 3 DONE!!!
OPTIMIZATION for Cluster # 4 DONE!!!
length of the cluster  0 ------> 2851.       ` # <---- these numbers are different than the failed one.`
length of the cluster  1 ------> 2698
length of the cluster  2 ------> 2094
length of the cluster  3 ------> 7522
length of the cluster  4 ------> 4438
UPDATED THE OLD COVARIANCE
beginning the smoothening ALGORITHM
length of cluster # 0 --------> 2131
length of cluster # 1 --------> 1886
length of cluster # 2 --------> 3072
length of cluster # 3 --------> 7915
length of cluster # 4 --------> 4599
Done writing the figure

And that is a snippet when it fails

 `# FAILED`
lam_sparse 0.11
switch_penalty 600
num_cluster 5
num stacked 5
completed getting the data



ITERATION ### 0
OPTIMIZATION for Cluster # 0 DONE!!!
OPTIMIZATION for Cluster # 1 DONE!!!
OPTIMIZATION for Cluster # 2 DONE!!!
OPTIMIZATION for Cluster # 3 DONE!!!
OPTIMIZATION for Cluster # 4 DONE!!!
length of the cluster  0 ------> 7851.   `# <-----  really higher than when it passed`
length of the cluster  1 ------> 2246
length of the cluster  2 ------> 2261
length of the cluster  3 ------> 2613
length of the cluster  4 ------> 4632
UPDATED THE OLD COVARIANCE
beginning the smoothening ALGORITHM
length of cluster # 0 --------> 7957
length of cluster # 1 --------> 1617
length of cluster # 2 --------> 3102
length of cluster # 3 --------> 3280
length of cluster # 4 --------> 3647
Done writing the figure

The failed test case has successfully convergred, but still fails

CONVERGED!!! BREAKING EARLY!!!



TRAINING F1 score: -1 -1 -1


Failure
Traceback (most recent call last):
  File "/Users/motaher/miniconda2/envs/pyliger/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/Users/motaher/miniconda2/envs/pyliger/lib/python3.6/unittest/case.py", line 605, in run
    testMethod()
  File "/Users/motaher/dev/PycharmProjects/ORIGINAL_TICC/UnitTest.py", line 33, in test_multiExample
    self.assertEqual(sum(val), 0)
  File "/Users/motaher/miniconda2/envs/pyliger/lib/python3.6/unittest/case.py", line 829, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/Users/motaher/miniconda2/envs/pyliger/lib/python3.6/unittest/case.py", line 822, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 45059.0 != 0


Ran 1 test in 53.887s

FAILED (failures=1)

That could be due to the fact that states in stored file and states coming from the algorithms were in different values?

[scoutsaachi]

The reason for this is the unit tests are really simple: they just compare the run of the algorithms when they work to the current result (they're just sanity checks for now, we'll create more robust unit tests at some point).

The result is mildly based off of things such as initialization, which is based off of the random number generator (which we seed at the beginning on package import). That means when the second test is run after the first test, the state of numpy's random generator will be different than if you run the second test on it's own, so the initialization and anything that requires randomization will give you a different value, which means you might fall into a different local optimum. Thus, since the "true result" was based off of the second run happening after the first, it fails on it's own since the random number generator produces different values

[mohataher]

That breaks the F.I.R.S.T principles of unit test cases. They have to be isolated completely. It makes sense to reset the seed at every test case and compare results based on that.

[davidhallac]

Yes, @scoutsaachi is correct that the random initialization step causes this discrepancy.

Thanks for point out the F.I.R.S.T. principles link! As we mentioned, the tests are mostly just sanity checks for now, but we are working on building out the infrastructure around TICC, and we will be sure to fix that issue ASAP as we add full unit test support. (I'll leave this open for now till we fix this inconsistency...)

[davidhallac]

Just pushed a change to make TICC re-seed the random number generator every time it is called. This fixes the Unit Test inconsistency error, so now they should always give reproducible results regardless of the order they are called in - thanks for pointing this out!