[Meta] Test releases for the tests repository #531
Comments
|
Hi everyone, After some more thoughts I also think that I would like to weaken the initially proposed strict bi-weekly release schedule a bit and instead still do regular releases but within a period of 1-4 weeks. In some more dense periods (like atm) there might be the need for more regular releases, e.g. if the last EIP tests are merged, in other periods with low work activity doing a release after 4 weeks might be enough. Due to the nature of this library with more constant updates here and there on tests and less of really feature-centric events/large changes I would generally stick to a regular-release-concept. Here is a draft for some first release notes: Tag: v6.0.0-beta.1 Title: Description: This is the first of a new regular series of test releases, which can be used to reference a static snapshot of the tests within your library. For a starter release notes will give an overall summary of the status of consensus Constantinople Test Summary
Library ChangesBe aware that the format of BlockchainTests recently changed with the introduction of a new field This means that you can faster-execute VersioningLibrary releases will follow semantic versioning:
Here is an example how a follow-up release line could look like:
Put some more thoughts in what would make sense as a concept for the tags. I've now chosen to prefix this with a HF abbreviation ( Once work on the next HF has started one would switch to a new prefix and new starting version number, like As description this now has some overall description of late noteworthy changes + the Constantinople status, in subsequent releases I would give a summary of the stuff changed in between the releases, this is just as a starter. Update: have changed versioning for the draft and added on the release description along with the comment from @carver down the line. |
|
@holgerd77 Thank you very much for jumping on this. 👍 As someone who has implemented the Constantinople EIPs for Trinity I really appreciate your effort to make the process of upgrading the tests more pleasant. Currently, for me, upgrading the tests usually involves lots of digging through other client implementations to find out about the changes they had to implement to make things work (e.g. Your proposed release strategy, versioning scheme and release notes sound good to me. I will add though, that I think the process could be made even easier (for the maintainer) and more transparent (for everyone) if the commits to this repo would follow a commit message convention and then generate release notes automatically. Here's a changelog from Angular that is 100 % generated based on the commits of the project. There's a simple commit message convention to be followed to allow the automatic generation of such changelogs. There are different tools that can generate these changelogs, one of them is clog (DISCLAIMER: I worked on that. Just recommending it because it has binaries, so no JS dependency) I see the main benefit of using such a convention + auto changelog generation on the maintainer side as it removes the tedious job of creating such changelogs by hand. It also eliminates human error. Also, by enforcing clean commits that contain a commit message that goes into such a changelog comes at the benefit of making it much easier for everyone else to look up commits and read about the changes they caused. Adopting such practice comes at the cost of putting a little bit more upfront work into forging meaningful commits and knowing Git well enough to not struggle with that (interactive rebasing etc) so I realize there may be hesitation to buy into such practice. I just wanted to bring it up though, as I think it would greatly improve the whole testing story for a lot of people. Also, if there's buy-in for that, I'm happy to help with the integration of a changelog generator. |
|
Hi @cburgdorf, that's an interesting idea but I wouldn't over-engineer here for now. This requires very much discipline on the commit generation side and I wouldn't want to force everyone into some complete new work structure on a first round, think it's more important to get things off the ground. This also very much requires meaningful PRs, if at the end this gets just a list of "refill tests", "refill tests", "fix typo in docs",... PRs benefit is really limited. So for now I have a strong tendency to not (yet (?)) follow this path and stick to a manual summary. There is also some general overhaul of the complete |
I totally agree. I brought it up as a vision for how things could be as a long term goal. I'm totally on board that improvements need to come step by step.
Yes, basically means either contributors need to learn how to do it or maintainer need to rebase PRs on behalf of those people. Making it harder for external contributors to contribute is basically the main reason why we reject such policy in Trinity for now. That said, I guess the test repo has a more steady circle of contributors and less drive by contributors anyway.
I'm on board. I wasn't expecting this to be adopted any time soon. I do think though, clean semantic commits with description + a generated changelog would make a good long term goal for this project.
Interesting! Paging in @pipermerriam and @carver as I know they have ideas to improve the testing setup as well. E.g. Having a structure for the expected account state (instead of just having a bare |
|
Tests develop branch must always be stable. It is just that this time we created Constantinople tests way before the actuall EIPS were included into the test branch. But I like the idea of meta description of what actually hash changed in the tests and where to expect potential bugs to happen. |
|
Yes, main reason for releases are to have descriptions of what has changed and to be better able to exchange about problems and potential bugs, this is not so much about stability. If develop branch is always stable, even better. |
|
Further, semantic versioning would be helpful here:
Yeah, this totally makes sense. It is a perfect example of how releases can help clients: they can update to the latest version that doesn't include Constantinople tests by upgrading up until just before the next major version number. Also, it would be reasonable to do alpha/beta releases. So if Byzantium was v5, then next releases might be tagged like so:
|
|
Yes, I think the suggestion from @carver for versioning makes more sense than my stuff, will switch to that. |
|
Have updated the draft release notes from above with the changed versioning and notes from @carver. |
|
Have now published a first release v6.0.0-beta.1, have also cross-posted this on Reddit. On a first round I will continue to use this thread on subsequent release note drafts, as I said, will always publish as a draft 24 hours before a release. |
|
(sorry, wrong Reddit link if you got this via mail, use the updated link directly from the post) |
|
@holgerd77 might be good to bake some of this into the project |
|
Thanks @holgerd77 & @winsvega (and others) for all this work, this looks great! |
|
Extcodehash is not implemented yet. There is a test cases list |
|
Hi everyone, My idea for a release path towards a "good enough"
Does this make sense? Comments on the release notes are also very welcome! Cheers Tag: v6.0.0-beta.2 Title: Description: This is the second of a new regular series of test releases, which can be used to reference a static snapshot of the tests within your library. First release has been Constantinople Test Updates
Test Coverage
Library Changes
DocsTest generation docs have been consolidated and integrated in the central ReadTheDocs testing documentation. We also updated outdated parts on this doc section (see PR #539), so it should in principle now be possible to follow the guide and end up with a working test creation setup. There might still be some glitches, please let us know or submit a PR on ethereum/tests to if you stumble over something. Other changes:
|
holgerd77
changed the title
|
Just did the v6.0.0-beta.2 release, here the associated post on Reddit. Since I fuddled around with this myself a bit, here some installation instructions, this should probably also be included in the documentation: cd [SUBMODULE_FOLDER]
git fetch --tags origin develop
git tag -l
git checkout tags/[RELEASE_VERSION]
cd ..
git add [SUBMODULE_FOLDER]
git commit -m "[MESSAGE_ON_TEST_RELEASE_UPDATE]" |
|
What submodule folder? |
Depends on the project: https://github.com/ethereum/tests#clients-using-the-library It's called |
|
would be good to have a text file in test repo called changes.txt But all those records you just add manually while commiting PRs to the test repo. |
|
We've discussed doing something like that in trinity (auto-generating the changelog from the merge commits since the last release). AngularJS does something similar, apparently. It seems like it might be a good idea for trinity to do someday, but the reality is that it is very difficult to get contributors to write good, publicly-readable commit messages. (maybe even tougher to have them make manual entries in a changelog) So we punted on the idea until we can figure out that problem. For now, a manual changelog writeup is the best solution we have. I imagine the |
|
Thats what I say. manually write changes to the file. but automatically mergeit to the change log. |
|
and if I manage to get contributers to write testFiller.json files it should be easy to have them write reasonable change log entries. |
|
Good idea, I will directly create a PR with a collection of the previous release notes within a changelog file. I would then also abandon this release preparation concept of posting release note drafts here but directly open a PR with proposed finalized changelog entry against the changelog file. Then this gets in the normal review process and can be reviewed and approved by @winsvega or someone else and it becomes more transparent for everyone to get noticed and eventually discussed. As one suggestion to simplify things I would suggest to not keep two separate files for this but open a new |
|
It's time for the third release! I've opened #571 which adds a CHANGELOG, and I've also made a draft release, here are the release notes: This is the third release, which can be used to reference a static snapshot of the tests within your library. Potentially breaking changes
EIP 1234
EIP 1052
Misc
|
|
this looks really great! now we could track and refere to a specific pr and tests that covered a specific issue and such much easier. thanks! but please note that this are not all tests we have merged for constantinople. |
|
Thanks!
By this, do you mean that you want me to add a note saying that these are not all the Constantinople tests, just the ones added during this release cycle? Or do you mean that more tests were added during this cycle and I missed them when making these notes? |
|
I mean there will be more before the hardfork |
|
Dropping a link to https://github.com/ethereum/py-evm/issues/1587, there are some good comments in there on ideas for what a changelog should look like. |
|
@winsvega I added a longer description to the release:
and published it here: https://github.com/ethereum/tests/releases/tag/v6.0.0-beta.3 |
|
I also updated #571, please merge it when you get a chance :) |
|
Please notify me if you need specific commit description for auto changelog script. Cause right now I just type a few words in commit title. |
|
Sure thing @winsvega 👍 I don't have strong opinions on commit message titles for now. I made this release by going through all the PRs and reading their descriptions. As long as all commits go through a PR and they include some kind of description of what they're doing (preferably in a comment in the filler) these notes will be pretty easy to generate. |
|
Are there any preferences for github merge type? (Merge commit, rebase commit, ..) |
|
Are this releases still supported? |
|
test releases are now done by github releases |
and others
Hi @winsvega, have you ever thought about making releases on the tests repository, think these could already solve a lot of problems. This can be low-level, e.g. just doing tags on the GitHub release page and adding some CHANGELOG entry, we are doing this over on our proxy ethereumjs-testing library and have made good experiences with it.
This would have the effect that people stay a bit on board about what's going on (test format has changed, Constantinople tests for EIP xxx are ready,...), significantly reduces the risk of people using inconsistent states of the repository and would also help a lot to talk about problems ("What release are you using?" instead of "I am referring to the library in state with latest commit xxx").
What do you think? I could also take on this job this I am doing this anyhow for the JS library mentioned above, but I am also happy with you taking on it if you want to have control over the process.
The text was updated successfully, but these errors were encountered: