pin an agent version with a canary option

[drenalin23]

Pin an ordained version of the te-agent package.

Add an is_canary attribute that if set to true will allow the package version to be empty (and therefore pull the latest from the public repository).

[drenalin23]

Cooper and I are talking on our end. The other option that could be easier for TE customers is to create an additional experimental component in the thousandeyes apt repository. Allow an easy way for this cookbook to have a component override for a canary instance to pull from the experimental group and then have TE delay releases until they see that canary instances are updating correctly at customer endpoints.

[vicgp-te]

Hi @drenalin23 , thanks for your PR.
The problem with this change is that it would break the backwards compatibility, with this change, customers that are expecting the latest version by no passing any parameter, all of a sudden will have the version pinned to 1.66.1-1~xenial unless they are aware of this breaking change and update their chef code setting the new parameter is_canary.
Also, we update the package version every 2 weeks (aprox) and at this moment we are not planning to make a release of this chef cookbook every time it happens.

I'm trying to understand what your use case is. I guess you want to pin the version of your agents, but you maybe want to have part of them using the latest one? Is that the case?

Regarding the experimental apt repository, i think it makes sense but it is not a straightforward change because it involves a significant change in the release/deployment workflow that now is fully automated. Anyway, i will mention this idea to others here and let you know when/if we can do that change.

[drenalin23]

Hi vgarcia - It would be a breaking change.
The reason to start this discussion is that a recent release broke all of our enterprise agents and reduced our monitoring visibility for nearly 24 hours. We need to know that we have some more fine tuned control over how/when agents get updated after that incident.

[vicgp-te]

@drenalin23 thanks for the input. Could you give me more details of when this happened and if you reached out to ThousandEyes about this? I want to make sure we are not releasing agent changes that are not backwards compatible, it shouldn't happen, but if it happens, then this PR would make more sense. First i want to understand what happened to get more context.

Thanks