Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rf.random for tests #1283

Closed
albertz opened this issue Mar 23, 2023 · 0 comments
Closed

rf.random for tests #1283

albertz opened this issue Mar 23, 2023 · 0 comments
Assignees

Comments

@albertz
Copy link
Member

albertz commented Mar 23, 2023

This is for the new RETURNN frontend (#1120).

In our test_rf_... tests, we want to run the code with multiple backends (e.g. PT and RETURNN net dict (TF)), and then compare the results.

This is straightforward for all deterministic computations. However, for random, we need some special care. Even if the underlying algo is deterministic, it will be different across backends. So the only reasonable thing is that we must make sure that for such tests, random returns the same values in each run.

This is mostly straightforward, except when it comes to control flow (rf.Cond/rf.Loop, see #1282). Ignoring control flow, you could assume that in each framework, the number of calls to random and also the order, would be equal. So that in the first framework, we can cache the results from the N calls, and in the next framework, we can just return them.

With control flow, the number of calls will not be the same, esp not with eager vs graph-based. How to solve this? Maybe it anyway needs to be handled at runtime? So e.g. using some custom EvalLayer in case of RETURNN net dict, which in turn uses some custom tf.py_func or so.

@albertz albertz self-assigned this Mar 27, 2023
albertz added a commit that referenced this issue Apr 3, 2023
This fails currently because we still do not have rf.random (#1283).
@albertz albertz closed this as completed in 6dd279b Apr 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant