Run codec-fixtures tests via Actions

[rvagg]

🤷 giving it a try, what do others think about this approach?

[rvagg]

    --- FAIL: TestCodecs/null (0.00s)

it works at least

[willscott]

I'd split it out into a separate workflow / action, so it shows up as a separate line item in the checks that run against PRs

[warpfork]

Mixed feelings. Compare and contrast with dragging fixture data in as a git submodule:

Submodule:

• 🟢 It's strongly linked. The test success or failure will be reproducible.
• 🟢 It's not dependent on github-specific technology.
• 🟢 It's going to work on localhost development without significant special effort.
• 🟠 It does require a tiny script somewhere that syncs the submodule into place. And git submodule UX is generally considered annoying by developers. (We could minimize this by making these tests skip if the submodule has never been synced?)
• 🟠 We'd probably want a bot that forcibly reminds us to bump the fixture submodule if its main branch advances. Can be done, just another thing to do.

Linking together loosely with github actions like this:

• 🔴 Not strongly linked. A push to the fixture repo may cause tests running in a branch to fail, or re-running tests to fail even without changes in this repo.
• 🟠 It's dependent on github-specific technology.
• 🟠 It does not work on localhost.
• 🟢 It doesn't push any complexity to developers. (But, well, because it doesn't do anything outside of server-side CI, so, ehm.)
• 🟢 It stays "up to date" constantly without intervention or any kind of notification spam. (But relatedly, you can't... stop it from being aggressively "up to date".)

The choice is difficult because one thing that's an outright red in the pros/cons is also directly linked to one of the more valuable greens.

[warpfork]

It's also kinda weird the volume of code that's split into the codec-fixtures repo.

• If we ever have another golang implementation, this'll feel a bit weird to them? (More directories in the codec-fixtures repo, I guess?)
• If we make API changes in this repo, the workflow is... go also make changes in the codec-fixtures repo... then re-run the API changing-branch here? And any other branch here that runs tests in the meanwhile will fail until we merge the API-changer and rebase all other branches to include that? Ugh.

Maybe these are sufficiently low-frequency events we can choose to shrug at it. My inner devops and reproducibility nerd is just 🤕

[mvdan]

My vote goes for a submodule that's periodically updated. Unless we "pin" via a submodule or some other way, CI becomes non-deterministic over time, which can quickly become painful.

See multiformats/go-multicodec#49 for example - we moved go-multicodec to pulling multicodec in via a submodule at a specific commit, rather than always fetching master. Otherwise go generate run in CI would be non-deterministic.

[mvdan]

I should add that I realise that CI is not properly 100% deterministic. But as long as you do things reasonably well, a green check on a specific commit today should remain green in six months or two years. If the commit points to master branches from other repositories, that suddenly becomes quite unlikely :)

[rvagg]

Re:

It's also kinda weird the volume of code that's split into the codec-fixtures repo.

This is a tricky call, but the intention is that it uses codecs to test other codecs, and that's not just codecs that are natively here in ipld-prime, so it broadens out beyond being an ipld-prime concern. As of ipld/codec-fixtures#2 it now does dagpb tests too. I would like to get dag-jose tests in too in the near future (it's possible with JS now, I don't know what the state of Go is). ETH and Bitcoin inclusion would be possible too. Then, any consumer of these tests for third-party implementations can just use the ones for codecs they are interested in testing, but we have a fairly straightforward means of verifying correctness as long as you implement >1 codec (as long as one of them is dag-json or dag-cbor).

So, I don't think it makes sense to absorb any, or much, of the code into this repo. There could even be a case to rename the "go" and "js" directories to be more specific - maybe "js-multiformats" (the moniker that's grown for the new JS stack) and "go-ipld-prime" to make way for alternative implementations of them in those languages.

[rvagg]

I got a cron working in ipld/codec-fixtures that tries out the @master here. I'll let someone else figure out an appropriate strategy here if it's still wanted, this doesn't seem like it.

[warpfork]

I ended up adding a submodule in #231 . It's checked out by CI at .ipld. I don't hate the result yet (although also do suspect we'll need a nag-bot eventually, to yell at us if we let it drift out of sync too far / too long). I also aimed to minimize the irritation a passing-by contributor might feel by making it optional to locally run those tests which depend on it.

... ah, but that's the ipld/ipld, and this is a new repo for just codec fixtures. Right. So we could add another one, anyway, I guess.