chore: remove agent version pinning in the CI

[pkoutsovasilis]

What does this PR do?

This PR removes the pinned AGENT_VERSION, AGENT_PACKAGE_VERSION, and BEAT_VERSION values from the CI environment and updates the .package-version file to 9.1.1.

It also re-enables the upgrade_broken_package_test.go test that was previously skipped due to reliance on pinned versions, since that constraint is now resolved.

Also, I had to bump beats dependency to the commit of 9.1.1 DRA and this required some extra changes in TestGetOtelConfig unit-test

PS: as discussed #8978 (comment) a review from observablt-ci and observablt-ci-contractors is not required (cc @ebeahan @pierrehilbert )

Why is it important?

The version pinning was introduced as a temporary workaround to support the 9.1.0 release. Now that 9.1.1 DRA is released, we are reverting to an unpinned state.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

None. This only impacts internal CI setup and test gating. End users are not affected.

How to test this PR locally

Open a draft PR 🙂

Related issues

• N/A

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 218eba0

History

• 💔 Build #24794 failed 2932c32
• 💔 Build #24762 failed 0f3cb0b
• 💔 Build #24748 failed a309ae8
• 💔 Build #24715 failed a309ae8

cc @pkoutsovasilis

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)