e2e template tests

[vdice]

Currently, the e2e tests for templated apps use templates installed from the most recent tag.

This presents a problem for PRs that either modify templates or the SDKs that they use, as these changes won't be tested in the e2e framework (as far as I can tell). Additionally, the current tests may fail as they aren't pulling in said changes.

If my understanding is correct, I wonder what we can do so that such changes can be tested via e2e (and how to prevent e2e failing when using potentially-outdated template material)?

[rajatjindal]

this is a (another) good one @vdice. we really need to come up with matrix of scenarios we want to support. I will start something to kickoff the conversation

[vdice]

Thanks @rajatjindal. For this issue I wonder if one approach would be to feed the PR branch name into the spin templates install command here?

For example, I can fetch templates from my branch locally via:

$ spin templates install --git https://github.com/vdice/spin --branch ref/tinygo-0.27.0-updates
Copying remote template source
...

When the branch is created on a fork, fetching and pulling the PR branch as exists on fermyon/spin might be trickier (looking at these docs). So we might need to send in the forked repo in addition to the branch...

[rajatjindal]

I mean it is possible today but we have to change the testcase parameters in tests/testcases/mod.rs

Eg http-js or http-ts tests makes use of it

[vdice]

Also, scratch the --git/--branch approach above. Much simpler to just use the --dir support in the command to reference a relative path.

I'd say for PRs (and maybe main as well), we should be installing from relative directories and then for cron or future support matrix, we should be installing them from default or configured release tags.

[vdice]

@rajatjindal now I've refamiliarized myself and see the template_install_args field on a test case struct -- handy for cases just like this!

What condition could we use to determine whether to send in --dir <path to spin repo root> vs leave empty (install from latest tag via git)? Maybe as straightforward as an env var that we can set (in CI, etc)? eg INSTALL_TEMPLATES_FROM_RELATIVE_DIR or some such? If true, we do the former above, if false, we do the latter.

[rajatjindal]

we can potentially use "USE_LOCAL_TEMPLATES" and based on that add --dir to tests

[lann]

This bit us recently: #3198