Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate plugin version at build time based on git describe #147

Merged

Conversation

torarnv
Copy link
Contributor

@torarnv torarnv commented Apr 7, 2024

The go-gitver package looks at the git describe output and resolves a version based on that.

If the git describe is a plain tag, the resulting plugin version matches the tag.

If there are commits on top of the most recent tag, or the working directory is dirty, the resulting plugin version is the latest plugin version with the patch version bumped, and the "dev" suffix added.

The same logic has been applied to the goreleaser file, so that the releasing logic can be tested with --snapshot

@torarnv
Copy link
Contributor Author

torarnv commented Apr 7, 2024

With this change the version number reported by packer-plugin-tart describe for local builds no longer lag behind the git tag (due to version/version.go being out of date). If pulling the repo without making any changes make install will be equivalent to installing the released version. If there are changes, the version will be the "next" version, which a "dev" suffix.

The goreleaser machinery will also folllow the same logic when using --snapshot to test the releasing infra.

@torarnv torarnv force-pushed the generate-plugin-version-at-build-time branch 2 times, most recently from 4cc9b0b to afcc79e Compare April 7, 2024 13:10
@edigaryev
Copy link
Contributor

Hello Tor Arne 👋

Would you mind explaining the rationale behind this change a bit?

It seems to me that the Packer plugin works just fine without the additional complexity of detecting a version, or I'm missing something?

@torarnv
Copy link
Contributor Author

torarnv commented Apr 8, 2024

Sure 🙂

We currently manage the version of the plugin in two places:

Are you can see, these two are out of sync (1.11.1 vs 1.10.0)

We also have two ways to produce binaries:

Having to maintain the most recent plugin version in two places (both in source and via tags) is cumbersome and easily gets out of sync. As Packer's plugin resolving logic depend on the versions being accurate, this is not ideal.

With the approach in this patch the version is always based on the git tag information, removing the need to manually keep version/version.go up to date.

The second benefit is that for development builds (where there are commits on top of the latest tag, or the working tree is dirty), we now automatically produce a build with a version that bumps the patch version one step (and add a dev suffix). The result is that local development builds are always the "next" release, without having to manually bump a version in version/version.go.

For comparison, the workflow that we have currently relies on always pushing a commit after every release that bumps version/version.go to one patch version after the released tag, to signify that the current HEAD is the "next" version. See hashicorp/packer#12749 (comment). I find that workflow cumbersome, which is why I implemented this automatic approach 😄

The changes to our goreleaser files are not strictly necessary for this local development workflow, but I added them to align the two workflows, so one can use --snapshot to test the goreleaser machinery, and it will automatically use the same logic for versioning as a make dev would.

@edigaryev
Copy link
Contributor

edigaryev commented Apr 8, 2024

How about we just stop updating the version altogether?1

This looks way simpler and solves both of the issues:

  1. No need to bother updating anything version-related in the code.

  2. The make dev plugin will always produce the latest -dev-suffixed version of the Packer plugin Tart on the local machine.

The official, non-dev versions, will keep clocking the version just fine, as you've described.

What do you think?

1: setting it to 0.0.0 is a bit nicer, but would require the developers to manually remove the already installed -dev versions on their machines. Considering there are not too many developers of Packer Plugin Tart at the moment, this might be the way to go.

@torarnv
Copy link
Contributor Author

torarnv commented Apr 8, 2024

How about we just stop updating the version altogether?

I assume you mean we stop updating version/version.go, as we will still have to produce versioned releases to fit in Packer's plugin ecosystem.

That's exactly what this patch does 😄 We stop updating the version, we only push new release git tags.

If by leaving version/version.go at some version and never updating it, without any additional logic, I don't see how that would work?

  1. The make dev plugin will always produce the latest -dev-suffixed version of the Packer plugin Tart on the local machine.

How will it do this? Where will it get the version number from? How will it ensure it's up to date with the latest git tag? How will ensure that it is higher than the latest git tag, which is a requirement for Packer to pick it up with its dev suffix?

@torarnv
Copy link
Contributor Author

torarnv commented Apr 8, 2024

Are you suggesting we hard-code the version.go version to 10000.10000.1000 to ensure it's always and forever higher than whatever the release version is? 🤔 I guess that would work, but doesn't offer many of the benefits of this patch, e.g:

  • If cloning the repo, and not doing any changes, so v1.2.3 is checked out make install will produce 1000.1000.1000-dev instead of v.1.2.3.
  • One can not distinguish which dev version is being run, it's always the same 1000.1000.1000-dev. With this patch, we have more information.
  • The goreleaser machinery does not reflect any of this, and will be out of sync, producing snapshot releases based on the latest git tag, not reflecting 1000.1000.1000-dev

@edigaryev
Copy link
Contributor

If by leaving version/version.go at some version and never updating it, without any additional logic, I don't see how that would work?

For releases, via tags (see https://github.com/hashicorp/packer/blob/1c7930bec11238af88f5c7256baae0abd11be430/packer/plugin-getter/github/getter.go#L198-L200), just as it works right now.

For -dev, it'll just overwrite the previous 0.0.0-dev version installed on the system.

How will it do this? Where will it get the version number from? How will it ensure it's up to date with the latest git tag? How will ensure that it is higher than the latest git tag, which is a requirement for Packer to pick it up with its dev suffix?

You don't need to do any of this as the next invocation of make dev will overwrite the previous 0.0.0-dev version of Packer Plugin Tart.

@torarnv
Copy link
Contributor Author

torarnv commented Apr 8, 2024

If by leaving version/version.go at some version and never updating it, without any additional logic, I don't see how that would work?

For releases, via tags (see https://github.com/hashicorp/packer/blob/1c7930bec11238af88f5c7256baae0abd11be430/packer/plugin-getter/github/getter.go#L198-L200), just as it works right now.

I assume you mean that we will not just git tag a commit to mark a release, but also continue to use the goreleaser machinery to both produce valid release filenames (which include the version in the filename), and also to feed the git tag version to the build process, as we do today via linker flags, so that packer-plugin-tart describe will print a version that matches the tag and file name?

For -dev, it'll just overwrite the previous 0.0.0-dev version installed on the system.

That will not work. The 0.0.0-dev version will not be prioritized by Packer over a 0.1.2 I installed some time ago via e.g. packer init that is still in my packer plugin cache. Going by that solutions means that anyone working on this plugin will have to clean out their packer plugin dir to do a test run of whatever change they are developing, or use a separate sandboxed PACKER_PLUGIN_PATH . I don't find that a particularly smooth workflow 😔

In addition, it doesn't come with any of the other advantages I listed earlier.

  • make install will install git tagv1.11.0 as 0.0.0-dev. With this patch it will install as v1.11.0
  • make install will install v1.11.0-5-gabcde and v2.20.0-4-gabcde as 0.0.0-dev, leaving no information about which version the dev version was based on. With this patch the two versions will be v1.11.1-dev and v2.20.1-dev respectively.
  • The goreleaser --snapshot machinery will not follow the 0.0.0-dev approach

To put it another way, is there a problem with this patch? Why not make things simpler to maintain, and consistent?

@edigaryev
Copy link
Contributor

Are you suggesting we hard-code the version.go version to 10000.10000.1000 to ensure it's always and forever higher than whatever the release version is? 🤔

0.0.0 should be enough.

Due to this code in github.com/hashicorp/go-version, non-prerelease constraints like >= 1.11.1 won't match pre-release versions like 0.0.0-dev.

@edigaryev
Copy link
Contributor

edigaryev commented Apr 8, 2024

That will not work. The 0.0.0-dev version will not be prioritized by Packer over a 0.1.2 I installed some time ago via e.g. packer init that is still in my packer plugin cache.

Neither 1000.1000.1000-dev will be prioritized over 0.1.2 due to this code in https://github.com/hashicorp/go-version.

To put it another way, is there a problem with this patch? Why not make things simpler to maintain, and consistent?

Just want to make sure that we're not solving a non-existent problem here.

P.S. your idea with build times is nice and somewhat resembles the Golang's pseudo-versions, I just don't think that with this state of things the proposed approach will be better and less complex that simply setting the Version variable to 0.0.0 in version/version.go.

@torarnv
Copy link
Contributor Author

torarnv commented Apr 8, 2024

Are you suggesting we hard-code the version.go version to 10000.10000.1000 to ensure it's always and forever higher than whatever the release version is? 🤔

0.0.0 should be enough.

Due to this code in github.com/hashicorp/go-version, non-prerelease constraints like >= 1.11.1 won't match pre-release versions like 0.0.0-dev.

It will when Packer 1.11 is released: https://www.hashicorp.com/blog/predictable-plugin-loading-in-packer-1-11

Which is why I've made this change 🙄

Just want to make sure that we're not solving a non-existent problem here.

I respect that, but I've also highlighter several flaws with the current approach, and I fail to see how the 0.0.0 approach will solve things.

With this patch, and Packer 1.11 I can make install a WIP dev version, and it will be prioritzed and I can test it. With the current state of the repo, it won't, due to version.go being out of sync with the git tag. Nor will I be able to do so with 0.0.0-dev, as it will always be down-priotized.

@edigaryev
Copy link
Contributor

It will when Packer 1.11 is released: https://www.hashicorp.com/blog/predictable-plugin-loading-in-packer-1-11

Interesting, thanks! Perhaps you could've started with that 😅

@torarnv
Copy link
Contributor Author

torarnv commented Apr 8, 2024

It will when Packer 1.11 is released: https://www.hashicorp.com/blog/predictable-plugin-loading-in-packer-1-11

Interesting, thanks! Perhaps you could've started with that 😅

😅 Sorry, yeah, should have done that. It was mentioned in the link to hashicorp/packer#12749 (comment) I posted in the initial reply, but buried quite deep in other stuff 😄

GNUmakefile Show resolved Hide resolved
@torarnv torarnv force-pushed the generate-plugin-version-at-build-time branch from afcc79e to 0464229 Compare April 9, 2024 10:55
@torarnv
Copy link
Contributor Author

torarnv commented Apr 9, 2024

I hope this satisfies your particular taste of regex soup 🍲 😄

You are right that it is simpler than depending on a generator 👍🏻

@fkorotkov
Copy link
Contributor

Picking up on the conversion.

Just couple historical cents. We previously had a case when we need to merge in changes from the packer-plugin-scaffolding which this repository is based of.

This change in the current state will make any similar future cases harder to merge in. Is there an option to get the same result with a smaller diff? Or maybe upstream it to the template?

@torarnv
Copy link
Contributor Author

torarnv commented Apr 11, 2024

The diff for the latest change is quite minimal I would say. Considering the (low) frequency that we do merge in changes from packer-plugin-scaffolding, the (low) chance that these will conflict with this patch, and the (minimal) amount of work needed to resolve those conflicts if they do happen, I don't see it as a big risk. I can volunteer to resolve any such issues if they do come up, if that helps 😄

We've diverged from the scaffolding already, and the scaffolding is anyways meant as a starting point, so I don't see a principal problem with diverging on this specific feature. Especially consider that the scaffolding is set up for a releasing/version model that we do not follow today (bumping version.go immediately after every release).

@edigaryev
Copy link
Contributor

edigaryev commented Apr 11, 2024

The diff for the latest change is quite minimal I would say.

I think we could do better, see the patch attached in #147 (comment).

No need to change anything other than 2 lines in the GNUmakefile file to achieve what you want.

@torarnv
Copy link
Contributor Author

torarnv commented Apr 11, 2024

The diff for the latest change is quite minimal I would say.

I think we could do better, see the patch attached in #147 (comment).

No need to change anything other than 2 lines in the GNUmakefile file to achieve what you want.

As I've tried to explain now, on several occasions, that patch does not solve it fully. Can you please read what I've written, and consider the parts not solved by your suggestion?

I'm fine making changes based on feedback, but this whole process is getting ridiculous.

@torarnv
Copy link
Contributor Author

torarnv commented Apr 11, 2024

@fkorotkov Please read the thread as a whole, including my feedback to @edigaryev's suggestion. Thank you.

@edigaryev
Copy link
Contributor

As I've tried to explain now, on several occasions, that patch does not solve it fully. Can you please read what I've written, and consider the parts not solved by your suggestion?

Are you referring to #147 (comment)?

I think I've missed that indeed. Quoting it here for reference:

The bash version fails to account for v1.11.0, which should not have dev. The command should probably be git describe --tags --dirty --always, replacing anything after - with -dev, and only then bumping the patch version, otherwise leaving it alone. My sed/awk foo is not great enough to formulate that 😉

  1. I thought that the version generated by make dev should always have -dev. Is that not the case?

  2. Could you clarify why do we need --dirty, which adds a -dirty prefix, if we already append -dev?

  3. I agree that we should use --all.

  4. We probably want to use --abbrev=0, though, to avoid adding a commit SHA suffix.

@torarnv
Copy link
Contributor Author

torarnv commented Apr 12, 2024

Good morning ☀️

I see four use-cases here:

  1. GitHub CI building release packages
    • goreleaser release
  2. A user building a released version from source
    • git reset --hard v1.11.1 && make
  3. A developer building a development version
    • echo Hello >> README.md && git commit -a -m "Hack hack hack" && make
  4. A developer testing the goreleaser machinery
    • goreleaser build --clean --snapshot --single-target

The goal for this patch was to ensure we are consistently handling these four cases, by following these principles:

  • Always base version on the git tag, so we don't have to maintain a separate version in a file that gets out of sync
  • Always produce plugin versions that match the git tag if building a clean git tag
  • Always produce -dev suffixed and patch-incremented versions if building a non-tagged commit or dirty working tree

The current state of the repository is as follows:

  1. Works
    • goreleaser is based on git tag, and will only trigger on git tags
  2. Fails
    • Produces 1.10.0-dev, because version.go is out of date.
    • Expect 1.11.1, and without dev as it's a clean tag
  3. Fails
    • Produces 1.10.0-dev, because version.go is out of date.
    • Expect 1.11.2-dev, incrementing the latest release and adding dev
  4. Fails
    • Produces 1.11.1-SNAPSHOT-ebc6ff2 regardless of the state of the tree (clean git tag, or changes on top)
    • This packer version is not compatible with Packer's version parsing (it must be dev if it has a suffix)

With the patch proposed in #147 (comment), we get:

  1. Works
    • goreleaser is based on git tag, and will only trigger on git tags
  2. Fails
    • Produces 1.10.0-dev, because version.go is out of date.
    • Expect 1.11.1, and without dev as it's a clean tag
  3. Works (partially)
    • Produces 1.11.2-dev
    • As long as the developer remembers to use make dev and not just make
      • The latter will still produce 1.10.0-dev, because version.go is out of date.
  4. Fails
    • Produces 1.11.1-SNAPSHOT-ebc6ff2 regardless of the state of the tree (clean git tag, or changes on top)
    • This packer version is not compatible with Packer's version parsing (it must be dev if it has a suffix)

With the patch proposed in this PR, we get:

  1. Works
    • goreleaser is based on git tag, and will only trigger on git tags
  2. Works
    • Produces 1.11.1, because it's a clean git tag without any development changes
  3. Works
    • Produces 1.11.2-dev, because there are local changes (dirty tree or commits on top)
    • Works for both make and make dev
      • The latter being a simple alias for make build now, as it's the what you're building that matters, not the how
  4. Works
    • Produces 1.11.1 for a clean git tag
    • Produces 1.11.2-dev for a dirty tree

Again, my goal has been to be consistent, both in how we pick up the version (from git, rather than git and a file), and how we distinguish release versions (clean git tags) from development versions (dirty tree). I hope that's a goal we generally share. With that goal in mind I think a patch that changes 15 lines seems entirely reasonable.

Claiming "we can do better", when the proposed alternative patch fails on 2 or more of the use-cases seems strange, and I'm left with a feeling that this project doesn't want contributions, or doesn't see the value in contributions that go one step further than fixing the immediate issue. I could have easily hacked around this locally, or in a fork, and gone on with my day. I spent time on and contributed a fix that went beyond that because I thought it would improve the project. I hope that was the right choice.

If there still are concerns with this patch, or the goal itself, then please let me know, but please consider the use-cases above and how the proposed solution will handle all of them (if that's part of the goal).

@edigaryev
Copy link
Contributor

edigaryev commented Apr 12, 2024

Good morning,

Claiming "we can do better", when the proposed alternative patch fails on 2 or more of the use-cases seems strange, [...]

Please take minute to reflect on how could this alternative patch address the use-cases that you've described after the fact.

Now on to the constructive things:

I see four use-cases here:

First of all, thank you for taking your time to list them here.

I only wish if this could be the first message in this PR, and not the 30th one 😅

Just one question:

  1. A user building a released version from source

Why do you need this if we already have a make dev?

Especially in lieu of the upcoming Packer 1.11 release that changes how the prerelease with suffixes like -dev are treated.

Not supporting this use-case would allow to simplify things further and make less deviations from the hashicorp/packer-plugin-scaffolding.

@torarnv
Copy link
Contributor Author

torarnv commented Apr 12, 2024

Good morning,

Claiming "we can do better", when the proposed alternative patch fails on 2 or more of the use-cases seems strange, [...]

Please take minute to reflect on how could this alternative patch address the use-cases that you've described after the fact.

Can you elaborate? What do you mean?

First of all, thank you for taking your time to list them here.

I only wish if this could be the first message in this PR, and not the 30th one 😅

I thought that the intent was fairly clear from the original patch, or at least the discussion that followed. And that there was some level of trust that I had good reasons for making the change as is. That's not the feeling I'm left with after the hoops I've gone through in this review.

If I could make a wish myself, it would be that reviewers assume good intent from the patch author, and consider the possibility that they have missed something if they feel the approach can easily be reduced to a few lines.

Just one question:

  1. A user building a released version from source

Why do you need this if we already have a make dev?

Especially in lieu of the upcoming Packer 1.11 release that changes how the prerelease with suffixes like -dev are treated.

I'm not sure I understand the question. A released version (git tag v1.11.1) is not a "development" version, so why would someone do make dev to install it? make && make install would be the expected command, at least large parts of the software world (coming from autotools e.g.).

I personally don't "need" that. I added it because I saw it as a goal to be consistent in how we handle versioning when building this project. Do we not care about that?

@edigaryev
Copy link
Contributor

I personally don't "need" that. I added it because I saw it as a goal to be consistent in how we handle versioning when building this project. Do we not care about that?

Thanks for explaining your reasoning behind this.

I'm not sure if we care about that, because the plugin is not normally intended to be built by users (unless they're developing it), and packer init will try to pull a new version if it's required in the HCL configuration anyways.

On top of that this might complicate diagnosis of the problems if some user creates an issue and tells us they run version X, but this version X was in fact built on their machine and not in CI, and there's a subtle difference that was introduced when building version X in users environment.

Now that the patchset is much simpler than it was initially we could probably have that too, and easily roll back in case this proves to be a wrong decision.

@edigaryev edigaryev self-requested a review April 12, 2024 12:26
@torarnv
Copy link
Contributor Author

torarnv commented Apr 16, 2024

Thanks @edigaryev! 🎉 @fkorotkov are there any other concerns, or can we land this?

@edigaryev
Copy link
Contributor

@torarnv would you mind updating your PR to use the new version.NewRawVersion() introduced in the latest Packer SDK 0.5.3?

It seems to be more in line with your changes and will also resolve a merge conflict.

@torarnv
Copy link
Contributor Author

torarnv commented Apr 17, 2024

@torarnv would you mind updating your PR to use the new version.NewRawVersion() introduced in the latest Packer SDK 0.5.3?

It seems to be more in line with your changes and will also resolve a merge conflict.

Sure, will have a look later today. It requires updating our QuietUi wrapper as well, as 0.5.3 adds more methods.

@edigaryev
Copy link
Contributor

It requires updating our QuietUi wrapper as well, as 0.5.3 adds more methods.

This is already done as a part of #148, so a rebase should be enough.

If the git describe is a plain tag, the resulting plugin
version matches the tag.

If there are commits on top of the most recent tag, or the
working directory is dirty, the resulting plugin version is
the latest plugin version with the patch version bumped,
and the "dev" suffix added.

The same logic has been applied to the goreleaser file,
so that the releasing logic can be tested with --snapshot
@torarnv torarnv force-pushed the generate-plugin-version-at-build-time branch from 0464229 to 8bc458f Compare April 17, 2024 14:34
@torarnv
Copy link
Contributor Author

torarnv commented Apr 17, 2024

It requires updating our QuietUi wrapper as well, as 0.5.3 adds more methods.

This is already done as a part of #148, so a rebase should be enough.

Ah, excellent. Done

@fkorotkov fkorotkov merged commit 58ce240 into cirruslabs:main Apr 17, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants