Fix eval in regression test

[ebsmothers]

The config name and the results parsing in our regression test job are incorrect.

This is actually a bit awkward to test now that we (a) don't allow creating PRs from a fork, and (b) don't let forks access the S3 bucket containing regression test artifacts.

So for now I've tested it locally, which I guess is better than nothing?

pytest tests/regression_tests/test_llama2_7b.py -m slow_integration_test
...
====== 1 passed, 1 warning in 267.94s (0:04:27) ======

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1305

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 56ce980 with merge base 00bbd53 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[SalmanMohammadi]

How was this test working before??

[ebsmothers]

It wasn’t, it’s been failing in CI for a while now. But it doesn’t run on PRs or anything, only nightly

[SalmanMohammadi]

This is actually a bit awkward to test now that we (a) don't allow creating PRs from a fork, and (b) don't let forks access the S3 bucket containing regression test artifacts.

I missed this, but we're no longer running integration tests in our CI?

[ebsmothers]

This is actually a bit awkward to test now that we (a) don't allow creating PRs from a fork, and (b) don't let forks access the S3 bucket containing regression test artifacts.

I missed this, but we're no longer running integration tests in our CI?

So we actually have two types of integration tests: recipe tests and regression tests. Recipe tests always run on PRs but only use small checkpoints. Regression tests use the full-size model and run nightly. Currently this is the only regression test we have, but we’ve been wanting to add more and just haven’t had time (e.g. it’d be nice if we could test memory or perf of some of our models too).

[SalmanMohammadi]

I'll raise an issue : )