Add performance tests for scenarios we care about

[maryamariyan]

From @KrzysztofCwalina on Apr 19, 2018, 2:41 PM PDT

No description provided.

[KrzysztofCwalina]

1. We need to have a set of tests that contributors can run to verify that their PRs have desired impact on perf, i.e. they would run the tests on master, then on the branch with changes, and compare the results.
2. This project is not to do comparative benchmarking between different ML frameworks.

[glebuk]

Sounds good, then we should probably rename the project/file to PerfRegressionTest.
Benchmarks are usually competitive.
Hopefully, in this case the benchmarks can be normalized against hardware, so that if the run is on a different hw platform it will not fail. Alternatively it should track the perf and the hw platform it runs against.

[KrzysztofCwalina]

Hopefully, in this case the benchmarks can be normalized against hardware, so that if the run is on a different hw platform it will not fail. Alternatively it should track the perf and the hw platform it runs against.

The way we use such benchmarks is the following:

1. Run the benchmark on master branch.
2. Switch to PR branch
3. Run the tests again.
4. Post the diff in the PR description.

This "isolates" the hardware.
Having said that, the reports from BDN record information about the hardware, runtime, etc.

[shauheen]

PR #62 merged this.

[KrzysztofCwalina]

The PR just merged an infrastructure project and a "demo" test. I think it would be good to add more real tests, e.g. flight delay prediction. I would be more than happy to add it, if we think this is the right representative test and we add the data set to the repo.