Add the full version number to the installation directory name

[joshdover]

Dependencies

• Blocked by When upgrading, the elastic-agent watch command should be invoked using the new agent binary #2873
• Blocked by Report the version from the elastic agent package instead of agent binary #2582
• Blocked by ttps://github.com/The division of responsibility between agent versions during an upgrade is incorrect #3639

Issue

We need the full version number of agent to be present in the directory we install into. This is necessary for changing how we release agent.

Suggested solution

Add the full version number to the installation directory name.
Example:

data/
└── elastic-agent-7ac68h-8.8.1
└── elastic-agent-7ac68h-8.8.1.er1

Definition of done

• Installation directory now includes the full version number.
• Add a new integration test here to validate that installation directory is the correct one.

[cmacknz]

Right now the upgrade process temporarily has multiple versions of the agent extracted, so it can rollback. The agent versions are installed under the /data directory with the short Git SHA as the version identifier:

data/
└── elastic-agent-383752
└── elastic-agent-7ac68h

This works fine besides being impossible to tell which version is which by manual inspection. This approach fails if we want the commit hash of the agent itself to be the same for two emergency releases. What you would want instead is something like:

data/
└── elastic-agent-7ac68h-8.8.1
└── elastic-agent-7ac68h-8.8.1.er1

This allows us to keep using the agent Git SHA in the path and also makes it obvious which version is which, which I want for convenience anyway. These elastic-agent-xxxxx paths now show up in our diagnostics and make debugging one step less convenient.

Another approach to this would be to make the unique identifier something other than the agent Git SHA, like computing the hash of the agent package contents at build time. This would also be unique, but the directory name wouldn't be something you could look up in Git. We would need to rely on a manifest file listing the detailed version information of each component of the agent package, but we may want that anyway.

[pchila]

Since we have to change the format of the directories and stamp out new versions, I have 1 question and (maybe) 1 proposal.
The question first: is the precedence relationship below correct? :
8.8.0 < 8.8.1-er.1 < 8.8.1

If the precedence is correct, can we just have the directories to follow the semver version strings like

data/
└── elastic-agent-8.8.1+7ac68h
└── elastic-agent-8.8.1-er.1+7ac68h

@cmacknz , @joshdover what do you think ?

[jlind23]

@pchila this is correct, after discussing this today we will rather rely on changing the patch digit so having only something like:
8.8.1 < 8.8.2 < 8.8.3

[jlind23]

Moving this to a later sprint as the plan changed a bit.

[cmacknz]

I would still like the version numbers in the installation path to make debugging upgrade issues easier, so there's still value in having this regardless of the plan to change how we release agent.

Right now the agent diagnostics will contain these directory version hashes directly and you have to use Git to figure out which version is which. This is an unnecessary barrier to debugging for users who aren't developers.

[joshdover]

Agreed. @jlind23 I'm going to move this back into the current sprint, with a slightly lower priority.

[cmacknz]

We might be able to solve #2780 as part of this as well to let the current version of the agent upgrade to itself. We need a way to break the path collision in this upgrade scenario which we can do by having some special convention for resolving the collision when the agent upgrades to itself.

[pchila]

Just had a quick look at this and there are a couple of elements that may change the sizing and I will report here for posterity:

• the directory name when installing depends on the directory name created in the archive when packaging (since we copy over all the contents during installation).
• during packaging the directory where we place binaries inside /data depends on {{.BeatName}}-{{ commit_short }} and we have multiple occurrences throughout packages.yaml
• agent upgrade:
  • unpacks everything under data as-is retrieving the hash via .elastic-agent.active.commit
  • relies heavily on the convention of data/elastic-agent-<short hash> agent directories to copy runtime data, find and switch links for upgrade/rollback
• throughout agent code the VersionedHome() is hardcoded to the elastic-agent-<short commit>

This means that we cannot have different directory names without some identifier of agent version that lives inside data directory (something very similar to what is needed for #2582) and we need to launch upgrades and watchers from the new version of agent as we now need to support both directory formats as we may be upgrading all the way from back in 7.17.

To sum up:

• We probably need to develop this feature and Report the version from the elastic agent package instead of agent binary #2582 together (not ideal, it only adds to the overall risk) or put Report the version from the elastic agent package instead of agent binary #2582 as prerequisite of this (and not the other way around as suggested by @cmacknz )

• We need to change a very brittle build/packaging system that relies on a lot of ENV vars to communicate between build steps and abuses go templates

  • We need to introduce a separate version for agent which contains exactly the version we set during CI build/release and we need sensible defaults for local builds (what is applicable right now for beats is not good enough for agent)
  • I would suggest to have the exact same agent version (including the build ID) as part of the directory name instead of using the agent commit (we already have that in the package and we can have the full build information in a file inside /data directory)

• We need to support both old and new directories during upgrades

  • install paths of both from and new directories must be included in the update marker and no longer calculated)
  • the path of the new version of the agent has to be deduced from the a marker file created during packaging or from the location of elastic-agent executable inside the archive
  • this will break some of the upgrade paths as old agent versions cannot cope with the new directory structure. To avoid this we can (all the options are in alternative to one another):
    • just add the support for upgrade to a different directory without changing the packaging directory (8.9.0 for example) and then change the packaging in another future release (8.9.1 for example) and mark that upgrades to >8.9.0 must first complete an upgrade to 8.9.0 before proceeding further
    • shift the responsibility of the upgrade to the new version of the agent with the old agent spawning a subprocess after downloading and unpacking the new agent (the same problem with a mandatory specific version upgrade applies as above)
    • try to support both path formats using links, however this defeats the purpose of this issue since clashes are still possible

• We need to check all occurrences in the code of paths.VersionedHome() and paths.Home() to see if we need to duplicate logic between legacy and new directory format

This change is very risky as we don't have good test coverage for packaging, install and upgrade processes, so all of those need to be added to this and I think this makes it quite the big-bang :(

Any thoughts @joshdover , @jlind23 , @cmacknz ?
Do we still want to put this in 8.9.0 ?
Do we introduce a breaking change for the upgrade process ? Should we phase this across multiple versions to reduce risk ?

[jlind23]

Do we still want to put this in 8.9.0 ?
Do we introduce a breaking change for the upgrade process ? Should we phase this across multiple versions to reduce risk ?

If this product is overall not ready then we should rather lean towards postponing the change.
We do NOT want to introduce a breaking change for the upgrade process and we shouldn't force users to upgrade to a specific version before being able to upgrade to a later one. TL;DR: 8.9 shouldn't be a pre-requisites for an upgrade to 8.9+

Happy to make #2582 a pre-requisites for this and I will push this issue to later phase for now until we come up with a proper plan.