Skip to content
This repository was archived by the owner on Sep 17, 2024. It is now read-only.

Conversation

@mdelapenya
Copy link
Contributor

What is this PR doing?

It changes the compose file for the vanilla box defining a volume to mount the elastic-agent binary on it. We will use environment variables to configure the src and target of the volume.

That way, we will download the agent binary to a temporary dir before the test suite runs (only once), and will remove it from the file system after the tests suite runs.

Why is this important?

It will reduce the number of HTTP connections to download the binary to 1, compared to one for each scenario installing an agent. It usually takes from 5 to 10 seconds, so the gain will be (in average) 7'5 times the number of scenarios (6 at this moment).

It will also bring reliability, because the download of the binary is done from the test runner, so it has the retry-with-backoff capabilities, instead of running it from inside the container.

Related issues

@mdelapenya mdelapenya self-assigned this Jul 17, 2020
@mdelapenya mdelapenya requested review from a team and EricDavisX July 17, 2020 15:37
@mdelapenya mdelapenya marked this pull request as ready for review July 17, 2020 15:38
@mdelapenya mdelapenya assigned mdelapenya and unassigned mdelapenya Jul 17, 2020
// into the Fleet struct, to be used else where
func (fts *FleetTestSuite) downloadAgentBinary() error {
artifact := "elastic-agent"
version := "8.0.0-SNAPSHOT"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This version should be configurable, to support running the test suite against multiple versions of the agent.

I could think of an env var with the URL. If it is present in the execution, simply use it.

@elasticmachine
Copy link
Contributor

elasticmachine commented Jul 17, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #179 opened]

  • Start Time: 2020-07-17T15:37:33.547+0000

  • Duration: 24 min 33 sec

Test stats 🧪

Test Results
Failed 3
Passed 54
Skipped 9
Total 66

Test errors

Expand to view the tests failures

  • Name: Initializing / Tests / Sanity checks / golangcilint – pre_commit.lint

    • Age: 1
    • Duration: 0
    • Error Details: error
  • Name: Initializing / End-To-End Tests / ingest-manager_fleet_mode / Un-enrolling an agent – Fleet Mode Agent

    • Age: 1
    • Duration: 22.02309
    • Error Details: Step the agent is not listed as online in Fleet: The Agent is still online
  • Name: Initializing / End-To-End Tests / ingest-manager_fleet_mode / Re-enrolling an agent – Fleet Mode Agent

    • Age: 1
    • Duration: 16.421597
    • Error Details: Step the agent is listed in Fleet as online: There are 4 online agents. We expected to have exactly one

Steps errors

Expand to view the steps failures

  • Name: Run functional tests for ingest-manager:fleet_mode

    • Description:

    • Duration: 9 min 38 sec

    • Start Time: 2020-07-17T15:47:29.739+0000

    • log

  • Name: Error signal

    • Description:

    • Duration: 0 min 0 sec

    • Start Time: 2020-07-17T15:56:08.019+0000

    • log

  • Name: General Build Step

    • Description: [2020-07-17T15:56:08.501Z] Archiving artifacts
      hudson.AbortException: script returned exit code 1

    • Duration: 0 min 0 sec

    • Start Time: 2020-07-17T15:56:08.491+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-07-17T15:57:03.656Z] metricbeat_elasticsearch_1 is up-to-date
[2020-07-17T15:57:03.656Z] Creating metricbeat_metricbeat_1 ... 
[2020-07-17T15:57:03.656Z] 
Creating metricbeat_metricbeat_1 ... done
time="2020-07-17T15:57:03Z" level=info msg="Metricbeat is running configured for the service" metricbeatVersion=7.8.0 service=mysql serviceVersion=8.0.13 variant=MySQL
[2020-07-17T15:57:25.630Z] time="2020-07-17T15:57:23Z" level=warning msg="Waiting for more hits in the index" currentHits=2 desiredHits=5 elapsedTime=26.379912ms index=metricbeat-7.8.0-mysql-mysql-8.0.13-yrfkepw8 retry=1
[2020-07-17T15:57:25.630Z] time="2020-07-17T15:57:23Z" level=warning msg="Waiting for more hits in the index" currentHits=2 desiredHits=5 elapsedTime=495.994633ms index=metricbeat-7.8.0-mysql-mysql-8.0.13-yrfkepw8 retry=2
[2020-07-17T15:57:25.630Z] time="2020-07-17T15:57:24Z" level=warning msg="Waiting for more hits in the index" currentHits=2 desiredHits=5 elapsedTime=1.534050219s index=metricbeat-7.8.0-mysql-mysql-8.0.13-yrfkepw8 retry=3
[2020-07-17T15:57:26.574Z] time="2020-07-17T15:57:26Z" level=warning msg="Waiting for more hits in the index" currentHits=2 desiredHits=5 elapsedTime=3.048017432s index=metricbeat-7.8.0-mysql-mysql-8.0.13-yrfkepw8 retry=4
[2020-07-17T15:57:29.876Z] time="2020-07-17T15:57:29Z" level=warning msg="Waiting for more hits in the index" currentHits=3 desiredHits=5 elapsedTime=6.18473639s index=metricbeat-7.8.0-mysql-mysql-8.0.13-yrfkepw8 retry=5
[2020-07-17T15:57:36.498Z] time="2020-07-17T15:57:35Z" level=warning msg="Waiting for more hits in the index" currentHits=3 desiredHits=5 elapsedTime=12.634435036s index=metricbeat-7.8.0-mysql-mysql-8.0.13-yrfkepw8 retry=6
[2020-07-17T15:57:40.701Z] time="2020-07-17T15:57:40Z" level=warning msg="Waiting for more hits in the index" currentHits=4 desiredHits=5 elapsedTime=16.950602099s index=metricbeat-7.8.0-mysql-mysql-8.0.13-yrfkepw8 retry=7
[2020-07-17T15:57:47.303Z] time="2020-07-17T15:57:47Z" level=warning msg="Waiting for more hits in the index" currentHits=4 desiredHits=5 elapsedTime=23.860872698s index=metricbeat-7.8.0-mysql-mysql-8.0.13-yrfkepw8 retry=8
[2020-07-17T15:57:51.506Z] time="2020-07-17T15:57:51Z" level=info msg="Hits number satisfied" currentHits=5 desiredHits=5 elapsedTime=27.853914485s retries=9
[2020-07-17T15:57:51.506Z] time="2020-07-17T15:57:51Z" level=info msg="Hits number satisfied" currentHits=5 desiredHits=5 elapsedTime=5.993467ms retries=1
[2020-07-17T15:57:51.774Z] Stopping metricbeat_metricbeat_1 ... 
[2020-07-17T15:57:52.606Z] 
Stopping metricbeat_metricbeat_1 ... done
Removing metricbeat_metricbeat_1 ... 
[2020-07-17T15:57:52.606Z] 
Removing metricbeat_metricbeat_1 ... done
Going to remove metricbeat_metricbeat_1
[2020-07-17T15:57:53.177Z] Stopping metricbeat_mysql_1 ... 
[2020-07-17T15:57:55.874Z] 
Stopping metricbeat_mysql_1 ... done
Removing metricbeat_mysql_1 ... 
[2020-07-17T15:57:55.874Z] 
Removing metricbeat_mysql_1 ... done
Going to remove metricbeat_mysql_1
[2020-07-17T15:57:56.134Z] Pulling mysql (docker.elastic.co/integrations-ci/beats-mysql:percona-5.7.24-1)...
[2020-07-17T15:57:56.704Z] percona-5.7.24-1: Pulling from integrations-ci/beats-mysql
[2020-07-17T15:58:06.733Z] metricbeat_elasticsearch_1 is up-to-date
[2020-07-17T15:58:06.733Z] Creating metricbeat_mysql_1 ... 
[2020-07-17T15:58:31.345Z] 
Creating metricbeat_mysql_1 ... done
Found orphan containers (metricbeat_mysql_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
[2020-07-17T15:58:31.345Z] metricbeat_elasticsearch_1 is up-to-date
[2020-07-17T15:58:31.345Z] Creating metricbeat_metricbeat_1 ... 
[2020-07-17T15:58:31.345Z] 
Creating metricbeat_metricbeat_1 ... done
time="2020-07-17T15:58:30Z" level=info msg="Metricbeat is running configured for the service" metricbeatVersion=7.8.0 service=mysql serviceVersion=5.7.24 variant=Percona
[2020-07-17T15:58:53.324Z] time="2020-07-17T15:58:50Z" level=warning msg="Waiting for more hits in the index" currentHits=2 desiredHits=5 elapsedTime=9.138055ms index=metricbeat-7.8.0-mysql-percona-5.7.24-lqvjr6ll retry=1
[2020-07-17T15:58:53.324Z] time="2020-07-17T15:58:51Z" level=warning msg="Waiting for more hits in the index" currentHits=2 desiredHits=5 elapsedTime=712.023483ms index=metricbeat-7.8.0-mysql-percona-5.7.24-lqvjr6ll retry=2
[2020-07-17T15:58:53.324Z] time="2020-07-17T15:58:51Z" level=warning msg="Waiting for more hits in the index" currentHits=2 desiredHits=5 elapsedTime=1.315355474s index=metricbeat-7.8.0-mysql-percona-5.7.24-lqvjr6ll retry=3
[2020-07-17T15:58:55.243Z] time="2020-07-17T15:58:54Z" level=warning msg="Waiting for more hits in the index" currentHits=3 desiredHits=5 elapsedTime=4.276468346s index=metricbeat-7.8.0-mysql-percona-5.7.24-lqvjr6ll retry=4
[2020-07-17T15:58:57.151Z] time="2020-07-17T15:58:57Z" level=warning msg="Waiting for more hits in the index" currentHits=3 desiredHits=5 elapsedTime=6.580219592s index=metricbeat-7.8.0-mysql-percona-5.7.24-lqvjr6ll retry=5
[2020-07-17T15:59:01.349Z] time="2020-07-17T15:59:00Z" level=warning msg="Waiting for more hits in the index" currentHits=3 desiredHits=5 elapsedTime=10.199307049s index=metricbeat-7.8.0-mysql-percona-5.7.24-lqvjr6ll retry=6
[2020-07-17T15:59:06.633Z] time="2020-07-17T15:59:06Z" level=warning msg="Waiting for more hits in the index" currentHits=4 desiredHits=5 elapsedTime=16.112905996s index=metricbeat-7.8.0-mysql-percona-5.7.24-lqvjr6ll retry=7
[2020-07-17T15:59:10.832Z] time="2020-07-17T15:59:10Z" level=warning msg="Waiting for more hits in the index" currentHits=4 desiredHits=5 elapsedTime=19.827622868s index=metricbeat-7.8.0-mysql-percona-5.7.24-lqvjr6ll retry=8
[2020-07-17T15:59:15.061Z] time="2020-07-17T15:59:14Z" level=warning msg="Waiting for more hits in the index" currentHits=4 desiredHits=5 elapsedTime=23.893120595s index=metricbeat-7.8.0-mysql-percona-5.7.24-lqvjr6ll retry=9
[2020-07-17T15:59:21.648Z] time="2020-07-17T15:59:21Z" level=info msg="Hits number satisfied" currentHits=5 desiredHits=5 elapsedTime=31.06578327s retries=10
[2020-07-17T15:59:21.648Z] time="2020-07-17T15:59:21Z" level=info msg="Hits number satisfied" currentHits=5 desiredHits=5 elapsedTime=5.958045ms retries=1
[2020-07-17T15:59:22.632Z] Stopping metricbeat_metricbeat_1 ... 
[2020-07-17T15:59:22.897Z] 
Stopping metricbeat_metricbeat_1 ... done
Removing metricbeat_metricbeat_1 ... 
[2020-07-17T15:59:22.897Z] 
Removing metricbeat_metricbeat_1 ... done
Going to remove metricbeat_metricbeat_1
[2020-07-17T15:59:23.516Z] Stopping metricbeat_mysql_1 ... 
[2020-07-17T15:59:26.841Z] 
Stopping metricbeat_mysql_1 ... done
Removing metricbeat_mysql_1 ... 
[2020-07-17T15:59:26.841Z] 
Removing metricbeat_mysql_1 ... done
Going to remove metricbeat_mysql_1
[2020-07-17T15:59:27.108Z] Pulling mysql (docker.elastic.co/integrations-ci/beats-mysql:percona-8.0.13-4-1)...
[2020-07-17T15:59:27.733Z] percona-8.0.13-4-1: Pulling from integrations-ci/beats-mysql
[2020-07-17T15:59:37.895Z] metricbeat_elasticsearch_1 is up-to-date
[2020-07-17T15:59:37.895Z] Creating metricbeat_mysql_1 ... 
[2020-07-17T16:00:04.207Z] 
Creating metricbeat_mysql_1 ... done
Found orphan containers (metricbeat_mysql_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
[2020-07-17T16:00:04.207Z] metricbeat_elasticsearch_1 is up-to-date
[2020-07-17T16:00:04.207Z] Creating metricbeat_metricbeat_1 ... 
[2020-07-17T16:00:04.207Z] 
Creating metricbeat_metricbeat_1 ... done
time="2020-07-17T16:00:03Z" level=info msg="Metricbeat is running configured for the service" metricbeatVersion=7.8.0 service=mysql serviceVersion=8.0.13-4 variant=Percona
[2020-07-17T16:00:26.159Z] time="2020-07-17T16:00:23Z" level=warning msg="Waiting for more hits in the index" currentHits=2 desiredHits=5 elapsedTime=8.418267ms index=metricbeat-7.8.0-mysql-percona-8.0.13-4-y8dtj8yj retry=1
[2020-07-17T16:00:26.159Z] time="2020-07-17T16:00:24Z" level=warning msg="Waiting for more hits in the index" currentHits=2 desiredHits=5 elapsedTime=635.380391ms index=metricbeat-7.8.0-mysql-percona-8.0.13-4-y8dtj8yj retry=2
[2020-07-17T16:00:26.159Z] time="2020-07-17T16:00:25Z" level=warning msg="Waiting for more hits in the index" currentHits=2 desiredHits=5 elapsedTime=1.942991632s index=metricbeat-7.8.0-mysql-percona-8.0.13-4-y8dtj8yj retry=3
[2020-07-17T16:00:28.073Z] time="2020-07-17T16:00:27Z" level=warning msg="Waiting for more hits in the index" currentHits=2 desiredHits=5 elapsedTime=4.409186403s index=metricbeat-7.8.0-mysql-percona-8.0.13-4-y8dtj8yj retry=4
[2020-07-17T16:00:30.616Z] time="2020-07-17T16:00:30Z" level=warning msg="Waiting for more hits in the index" currentHits=3 desiredHits=5 elapsedTime=7.147227242s index=metricbeat-7.8.0-mysql-percona-8.0.13-4-y8dtj8yj retry=5
[2020-07-17T16:00:35.898Z] time="2020-07-17T16:00:35Z" level=warning msg="Waiting for more hits in the index" currentHits=3 desiredHits=5 elapsedTime=11.795630416s index=metricbeat-7.8.0-mysql-percona-8.0.13-4-y8dtj8yj retry=6
[2020-07-17T16:00:42.500Z] time="2020-07-17T16:00:42Z" level=warning msg="Waiting for more hits in the index" currentHits=4 desiredHits=5 elapsedTime=18.787165063s index=metricbeat-7.8.0-mysql-percona-8.0.13-4-y8dtj8yj retry=7
[2020-07-17T16:00:49.083Z] time="2020-07-17T16:00:48Z" level=warning msg="Waiting for more hits in the index" currentHits=4 desiredHits=5 elapsedTime=24.707157493s index=metricbeat-7.8.0-mysql-percona-8.0.13-4-y8dtj8yj retry=8
[2020-07-17T16:00:55.663Z] time="2020-07-17T16:00:55Z" level=info msg="Hits number satisfied" currentHits=5 desiredHits=5 elapsedTime=32.10977358s retries=9
[2020-07-17T16:00:55.663Z] time="2020-07-17T16:00:55Z" level=info msg="Hits number satisfied" currentHits=5 desiredHits=5 elapsedTime=5.601852ms retries=1
[2020-07-17T16:00:56.603Z] Stopping metricbeat_metricbeat_1 ... 
[2020-07-17T16:00:56.863Z] 
Stopping metricbeat_metricbeat_1 ... done
Removing metricbeat_metricbeat_1 ... 
[2020-07-17T16:00:56.863Z] 
Removing metricbeat_metricbeat_1 ... done
Going to remove metricbeat_metricbeat_1
[2020-07-17T16:00:57.434Z] Stopping metricbeat_mysql_1 ... 
[2020-07-17T16:01:00.733Z] 
Stopping metricbeat_mysql_1 ... done
Removing metricbeat_mysql_1 ... 
[2020-07-17T16:01:00.733Z] 
Removing metricbeat_mysql_1 ... done
Going to remove metricbeat_mysql_1
[2020-07-17T16:01:00.733Z] Stopping metricbeat_elasticsearch_1 ... 
[2020-07-17T16:01:02.120Z] 
Stopping metricbeat_elasticsearch_1 ... done
Removing metricbeat_elasticsearch_1 ... 
[2020-07-17T16:01:02.120Z] 
Removing metricbeat_elasticsearch_1 ... done
Removing network metricbeat_default
[2020-07-17T16:01:02.121Z] <?xml version="1.0" encoding="UTF-8"?>
[2020-07-17T16:01:02.121Z] <testsuites name="main" tests="7" skipped="0" failures="0" errors="0" time="666.126592536">
[2020-07-17T16:01:02.121Z]   <testsuite name="As a Metricbeat developer I want to check that default configuration works as expected" tests="0" skipped="0" failures="0" errors="0" time="0"></testsuite>
[2020-07-17T16:01:02.121Z]   <testsuite name="As a Metricbeat developer I want to check that the Apache module works as expected" tests="0" skipped="0" failures="0" errors="0" time="0"></testsuite>
[2020-07-17T16:01:02.121Z]   <testsuite name="As a Metricbeat developer I want to check that the MySQL module works as expected" tests="7" skipped="0" failures="0" errors="0" time="619.660812954">
[2020-07-17T16:01:02.121Z]     <testcase name="Check MariaDB-10.2.23 is sending metrics to Elasticsearch without errors" status="passed" time="89.265636866"></testcase>
[2020-07-17T16:01:02.121Z]     <testcase name="Check MariaDB-10.3.14 is sending metrics to Elasticsearch without errors" status="passed" time="83.358153018"></testcase>
[2020-07-17T16:01:02.121Z]     <testcase name="Check MariaDB-10.4.4 is sending metrics to Elasticsearch without errors" status="passed" time="84.250099006"></testcase>
[2020-07-17T16:01:02.121Z]     <testcase name="Check MySQL-5.7.12 is sending metrics to Elasticsearch without errors" status="passed" time="81.559098074"></testcase>
[2020-07-17T16:01:02.121Z]     <testcase name="Check MySQL-8.0.13 is sending metrics to Elasticsearch without errors" status="passed" time="81.959110432"></testcase>
[2020-07-17T16:01:02.121Z]     <testcase name="Check Percona-5.7.24 is sending metrics to Elasticsearch without errors" status="passed" time="86.122142572"></testcase>
[2020-07-17T16:01:02.121Z]     <testcase name="Check Percona-8.0.13-4 is sending metrics to Elasticsearch without errors" status="passed" time="89.182350971"></testcase>
[2020-07-17T16:01:02.121Z]   </testsuite>
[2020-07-17T16:01:02.121Z]   <testsuite name="As a Metricbeat developer I want to check that the Redis module works as expected" tests="0" skipped="0" failures="0" errors="0" time="0"></testsuite>
[2020-07-17T16:01:02.121Z]   <testsuite name="As a Metricbeat developer I want to check that the vSphere module works as expected" tests="0" skipped="0" failures="0" errors="0" time="0"></testsuite>
[2020-07-17T16:01:02.121Z] </testsuites>+ sed -e 's/^[ \t]*//; s#>.*failed$#>#g' outputs/TEST-metricbeat-mysql
[2020-07-17T16:01:02.121Z] + grep -E '^<.*>$'
[2020-07-17T16:01:02.121Z] + exit 0
[2020-07-17T16:01:02.251Z] Recording test results
[2020-07-17T16:01:02.742Z] Archiving artifacts
[2020-07-17T16:01:04.842Z] Stage "Release" skipped due to earlier failure(s)
[2020-07-17T16:01:05.934Z] Running on worker-395930 in /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-179
[2020-07-17T16:01:05.995Z] [INFO] getVaultSecret: Getting secrets
[2020-07-17T16:01:06.063Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-17T16:01:08.147Z] + chmod 755 generate-build-data.sh
[2020-07-17T16:01:08.147Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-179/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-179/runs/1 FAILURE 1413166
[2020-07-17T16:01:08.147Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-179/runs/1/steps/?limit=10000 -o steps-info.json
[2020-07-17T16:01:12.313Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-179/runs/1/tests/?status=FAILED -o tests-errors.json
[2020-07-17T16:01:12.313Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-179/runs/1/log/ -o pipeline-log.txt

@mdelapenya
Copy link
Contributor Author

Failing tests are the ones failing in upstream

@mdelapenya mdelapenya merged commit 15d2db5 into elastic:master Jul 17, 2020
@mdelapenya mdelapenya deleted the 177-download-agent-once branch July 17, 2020 18:18
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Download elastic-agent once

3 participants