Skip to content

Conversation

FrozenGene
Copy link
Member

fix #6309

@jcf94
Copy link
Contributor

jcf94 commented Aug 20, 2020

Should we set the random seed for python random module & numpy random module?

Nerver mind, I noticed that this only used the random engine inside TVM.

@tqchen
Copy link
Member

tqchen commented Aug 20, 2020

We should avoid the usage of seed to resolve the test error wrt to randomness. Since setting seed can hide problems in real usecases.

Instead, we should make sure the testcase pass for most randomness generated, perhaps by increasing the error change range in this case

@tqchen tqchen self-assigned this Aug 20, 2020
@tqchen tqchen added the status: need update need update based on feedbacks label Aug 20, 2020
@FrozenGene
Copy link
Member Author

FrozenGene commented Aug 21, 2020

@tqchen I undenstand your concern. However, I have done some experiments and the generated data is random, we can not make sure completely like np.mean(na) < 0.2 always be true and developers will complaint it again. So to avoid this condition, I have decided to do this change, we have still other assert could help us verify (like np.min / np.max / random_fill test case etc) However, if you still think we should incerease, I could help to do it (like increasing the shape we generated and incerease the error range to make us CI pass more robust)

@tqchen
Copy link
Member

tqchen commented Aug 21, 2020

I do think increase the size and error range should help, and as long as the error happens in low probability(we can calculate the confidence) we should be fine.

@FrozenGene
Copy link
Member Author

FrozenGene commented Aug 21, 2020

Have increased size and error. When to run 100 times, previous code will occur 6 errors, 1000 times will occur 44 errors. New code doesn't have check error. Wish this could let CI be more robust. @tqchen

@tqchen tqchen merged commit 8781d1f into apache:master Aug 21, 2020
trevor-m pushed a commit to trevor-m/tvm that referenced this pull request Aug 26, 2020
* [TEST][FLAKY] fix random fail

* increase size and error check range
trevor-m pushed a commit to trevor-m/tvm that referenced this pull request Aug 26, 2020
* [TEST][FLAKY] fix random fail

* increase size and error check range
trevor-m pushed a commit to trevor-m/tvm that referenced this pull request Aug 26, 2020
* [TEST][FLAKY] fix random fail

* increase size and error check range
electriclilies pushed a commit to electriclilies/tvm that referenced this pull request Aug 26, 2020
* [TEST][FLAKY] fix random fail

* increase size and error check range
trevor-m pushed a commit to trevor-m/tvm that referenced this pull request Sep 2, 2020
* [TEST][FLAKY] fix random fail

* increase size and error check range
trevor-m pushed a commit to neo-ai/tvm that referenced this pull request Sep 3, 2020
* [TEST][FLAKY] fix random fail

* increase size and error check range
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: need update need update based on feedbacks
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[TEST][FLAKY] test_normal fails intermittently
3 participants