Run package tests in vagrant #12646

nik9000 · 2015-08-04T18:34:46Z

This creates a module in distribution called tests that can be run if you
have vagrant and virtualbox installed and will run the packaging tests in
trusty, precise, wheeze, jessie, centos-6.6, and centos-7.0.

See distribution/Vagrantfile for real documentation, but the tl/dr is run
mvn verify -Pvagrant and go get coffee.

Closes #12611

nik9000 · 2015-08-04T18:35:30Z

@electrical, @spinscale, @rmuir probably will have opinions on this.

dadoonet · 2015-08-04T18:55:06Z

That looks awesome! Can we detect if vagrant is installed so we don't add the tests module if not?

Something like in https://github.com/elastic/elasticsearch/blob/master/distribution/pom.xml#L196-L220

nik9000 · 2015-08-04T18:59:54Z

That looks awesome! Can we detect if vagrant is installed so we don't add the tests module if not?

Right now we don't run it at all unless you do -Pvagrant. I fear its just too slow to do it by default if someone has vagrant installed. The first time you do it it downloads approximately 3.5GB of stuff. ~2.5GB of boxes and ~1GB of debs and rpms. I haven't timed the process of running all the tests once the machines are downloaded and provisioned but its tens of minutes. One machine is pretty manageable but 6 is intense.

rmuir · 2015-08-04T19:02:57Z

I only glanced at it, and it looks awesome to me at a glance too. I hope to have some time to try it out locally soon but don't let me hold up the change, this is really needed.

nik9000 · 2015-08-05T17:16:55Z

I've marked this 2.0.0 - if we merge soon then I suspect it'll actually get into beta1.

nik9000 · 2015-08-05T19:06:51Z

@electrical, @spinscale, @rmuir, @dadoonet this could use another reviewer. @dakrone's run through it and found lots of issues I fixed. I'd love someone who's running debian or ubuntu to give it a run through. Baring that someone who can just review the code would be nice.

electrical · 2015-08-05T19:27:41Z

I'll give this a run tomorrow UK time.

nik9000 · 2015-08-05T19:28:35Z

Hurrah! Thanks for that.

nik9000 · 2015-08-06T01:07:00Z

Whoever looks at this next, keep in mind #12682.

spinscale · 2015-08-06T08:47:00Z

distribution/tests/pom.xml

+      <allRpmBoxes>centos-6, centos-7, fedora-22</allRpmBoxes>
+
+      <debBoxes>trusty</debBoxes>
+      <rpmBoxes>centos-7.0</rpmBoxes>


this fails, as the box above has a different name...

dadoonet · 2015-08-06T08:56:29Z

I tried your branch this morning on my laptop. Sadly until now, I did not succeed in making it run.

Here is what I did:

mvn clean install -DskipTests
cd distribution
mvn verify -Pvagrant

It failed. But I saw in logs that the VM was successfully downloaded:

[INFO] Executing tasks

main:
     [echo] Running package tests on trusty, centos-7.0

vagrant-test-all-boxes:

vagrant-up:
     [exec] Bringing machine 'trusty' up with 'virtualbox' provider...
     [exec] ==> trusty: Box 'ubuntu/trusty64' could not be found. Attempting to find and install...
     [exec]     trusty: Box Provider: virtualbox
     [exec]     trusty: Box Version: >= 0
     [exec] ==> trusty: Loading metadata for box 'ubuntu/trusty64'
     [exec]     trusty: URL: https://vagrantcloud.com/ubuntu/trusty64
     [exec] ==> trusty: Adding box 'ubuntu/trusty64' (v20150609.0.10) for provider: virtualbox
     [exec]     trusty: Downloading: https://atlas.hashicorp.com/ubuntu/boxes/trusty64/versions/20150609.0.10/providers/virtualbox.box
     [exec] 
     [exec] There are errors in the configuration of this machine. Please fix
     [exec] the following errors and try again:
     [exec] 
     [exec] vm:
     [exec] * The 'fix-no-tty' provisioner could not be found.
     [exec] * The 'elasticsearch bats dependencies' provisioner could not be found.
     [exec] 
     [exec]     trusty: Progress: 0% (Rate: 0/s, Estimated time remaining: --:--:--)    trusty: Progress: 0% (Rate: 176k/s, Estimated time remaining: 0:39:21)   [...SKIPPED...]
trusty: Successfully added box 'ubuntu/trusty64' (v20150609.0.10) for 'virtualbox'!

Then, I tried to relaunch it again:

cd tests
mvn verify

I get this error:

[INFO] --- maven-antrun-plugin:1.8:run (test-precise) @ elasticsearch-distribution-tests ---
[INFO] Executing tasks

main:
     [echo] Running package tests on trusty, centos-7.0

vagrant-test-all-boxes:

vagrant-up:
     [exec] Bringing machine 'trusty' up with 'virtualbox' provider...
     [exec] There are errors in the configuration of this machine. Please fix
     [exec] the following errors and try again:
     [exec] 
     [exec] vm:
     [exec] * The 'fix-no-tty' provisioner could not be found.
     [exec] * The 'elasticsearch bats dependencies' provisioner could not be found.
     [exec]

Note:

vagrant -v

gives

Vagrant 1.6.5

Do you know what it means? Did I do something wrong?

spinscale · 2015-08-06T09:34:06Z

So when running all those tests, is the output somewhere redirected, so in case a build fails, I can follow what vagrant did without watching the screen?

dadoonet · 2015-08-06T09:38:35Z

So I can confirm it starts with Vagrant 1.7.4 (thanks @spinscale for the help). May be we should try to control the version when starting Ant or at least document it? Not a big deal though.

Tests running ATM... Let's see where it goes...

dadoonet · 2015-08-06T09:41:15Z

I'm wondering if we should not make each test part of the distribution we are testing.

I mean that I'd prefer do something like:

cd distribution/rpm
mvn verify -Pvagrant

cd distribution/deb
mvn verify -Pvagrant

Is this possible?

rmuir · 2015-08-06T09:55:58Z

No, thats not possible or wanted. Sorry, we can't make our integration tests complicated. Please, add this to the qa folder.

dadoonet · 2015-08-06T10:06:30Z

I'm not talking about complicating integration tests. I don't want it too.

I'm saying that if I run:

cd distribution/rpm
mvn verify

It runs what we have today (package the rpm, test it using REST tests).

But it we run

cd distribution/rpm
mvn verify -Pvagrant

It packages the rpm, test it using REST tests and run the Vagrant test for this distribution.

We should may be rename the vagrant profile to qa profile.
So if someone runs in the future (might be run by our release script and/or jenkins):

mvn install -Pqa

It would run for each module we have:

compile
unit test
package
integration test
qa (test coverage, checkstyle, PMD whatever) and for some specific modules Vagrant tests.

So if a module is failing you don't have to run again all the Qa for all the things.

I Maven land, qa tools are supposed to run within their modules. Same for site plugin. If you want to generate a site, you basically generate it by module.

My 2 cents on it

rmuir · 2015-08-06T10:23:26Z

I dont care about maven conventions in this case. Sorry, maven is wrong here and I am -1 to making integration tests complicated in this way.

Please, add it to the qa module. These vagrant tests depend on all kinds of stuff, like plugins. They are not specific to the packaging.

dadoonet · 2015-08-06T10:36:20Z

These vagrant tests depend on all kinds of stuff, like plugins. They are not specific to the packaging.

Fair enough.

rmuir · 2015-08-06T10:49:42Z

and by the way, the current solution (all in distribution/tests) is fine as a quick start. But ultimately we should move it to qa: https://github.com/elastic/elasticsearch/tree/master/qa

This way it can easily depend on e.g. kuromoji plugin or whatever and not require special 'mvn install' or anything like that.

rmuir · 2015-08-06T19:18:03Z

qa/vagrant/pom.xml

+      machines using vagrant and bats.</description>
+    <packaging>pom</packaging>
+
+    <!-- The documentation for how to run this is in ../Vagrantfile -->


But its ../../VagrantFile, is this intended?

I'll fix the note but yeah, I meant to put the file in the root directory. When you run vagrant it walks up your directory tree looking for a Vagrantfile. Whatever it finds it uses. It'd be super weird to put one a / or something but otherwise this is convenient behavior because you can start vagrant from anywhere in your project.

nik9000 · 2015-08-06T19:30:58Z

Maybe it makes sense to move the documentation in Vagrantfile into TESTING.md?

dakrone · 2015-08-06T20:03:47Z

Maybe it makes sense to move the documentation in Vagrantfile into TESTING.md?

+1

nik9000 · 2015-08-06T21:08:21Z

+1

Done.

nik9000 · 2015-08-06T21:09:23Z

I also added a section on testing the packages more manually - involving maven only minimally. Its the kind of tighter loops that I did when working on bugs in the package scripts.

tlrx · 2015-08-07T08:08:02Z

@nik9000 awesome work, thanks! It was really missing.

I noticed a wrong parent version number in qa/vagrant/pom.xml file (.0.0-beta1-SNAPSHOT) and I cannot run mvn -Pvagrant -pl qa/vagrant verifyon Ubuntu 14.04 because rpmbuild is not available in the expected version (error: Only known to work with rpmbuild 5+).

nik9000 · 2015-08-07T11:48:38Z

Ok - fixed the version in the pom.

Above rpmbuild - someone further up told me that the rpms wouldn't work with rpmbuild <5.0. What version of rpmbuild do you have? If it really won't work with that version of rpmbuild then you can work around it with -P-rpm and I can update the error message with that hint.

Soooo - this pull request has grown some extra commits overnight. Did this come from the accidental force push to master?

nik9000 · 2015-08-07T12:56:03Z

on Ubuntu 14.04 because rpmbuild is not available in the expected version (error: Only known to work with rpmbuild 5+).

I just dropped that check.

dakrone · 2015-08-07T13:39:22Z

+1, tested this locally and got it running.

polyfractal · 2015-08-07T22:42:11Z

I'm having some trouble getting this running, which is very likely my own fault :)

[INFO] Reactor Summary:
[INFO] 
[INFO] Elasticsearch Distribution ........................ SUCCESS [5.090s]
[INFO] Elasticsearch with all optional dependencies ...... SUCCESS [1.868s]
[INFO] Elasticsearch Shaded Distribution ................. SUCCESS [18.603s]
[INFO] Elasticsearch TAR Distribution .................... FAILURE [36.933s]
[INFO] Elasticsearch ZIP Distribution .................... SKIPPED
[INFO] Elasticsearch DEB Distribution .................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:03.751s
[INFO] Finished at: Fri Aug 07 18:36:41 EDT 2015
[INFO] Final Memory: 39M/788M
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "vagrant" could not be activated because it does not exist.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (integ-setup) on project elasticsearch: An Ant BuildException has occured: The following error occurred while executing this line:
[ERROR] /Users/tongz/Documents/elasticsearch_master2/distribution/tar/target/dev-tools/ant/integration-tests.xml:262: The following error occurred while executing this line:
[ERROR] /Users/tongz/Documents/elasticsearch_master2/distribution/tar/target/dev-tools/ant/integration-tests.xml:182: ES instance did not start
[ERROR] around Ant part ...<ant antfile="/Users/tongz/Documents/elasticsearch_master2/distribution/tar/target/dev-tools/ant/integration-tests.xml" target="start-external-cluster-tar">... @ 4:159 in /Users/tongz/Documents/elasticsearch_master2/distribution/tar/target/antrun/build-main.xml

Full gist of logs: https://gist.github.com/polyfractal/b49abd33b475a8890efa

Lemme know what else you need :)

dadoonet · 2015-08-08T07:04:15Z

it's because you ran with -Pvagrant in distribution module.
It moved to qa module.

So if your are in distribution module, just run mvn package

In qa module mvn package -Pvagrant

polyfractal · 2015-08-08T13:20:48Z

That did the trick, thanks @dadoonet!

[INFO] BUILD SUCCESS

dadoonet · 2015-08-08T14:38:35Z

I think we should add an "empty" vagrant profile in the parent pom so It should fix that. WDYT @nik9000?

rjernst · 2015-08-08T18:15:29Z

Can we instead not use profiles? Could we use ant logic to run by default if vagrant is installed? And if people really want to turn this off for verify runs, we can have a sys prop to disable?

rjernst · 2015-08-08T18:18:06Z

Btw, I only mean that as a suggestion for the future. This seems to be working, we should get it in.

dadoonet · 2015-08-08T19:16:40Z

Agreed. I'd prefer folks not to use -Pxxxx

Like here: https://github.com/elastic/elasticsearch/blob/master/distribution/pom.xml#L202

rmuir · 2015-08-08T19:56:52Z

or activate the profile with a property. like -Dtests.vagrant or something. let the profile be a impl detail.

nik9000 · 2015-08-10T09:29:03Z

Agreed. I'd prefer folks not to use -Pxxxx

OK - I'll fix that after merging. Looks like I have enough approval to merge this so I'm going to rebase and merge. Unless the rebase is substantively complicated.

This creates a module in qa called vagrant that can be run if you have vagrant and virtualbox installed and will run the packaging tests in trusty and centos-7.0. You can ask it to run tests in other linuxes. This is the full list: * precise aka Ubuntu 12.04 * trusty aka Ubuntu 14.04 * vivid aka Ubuntun 15.04 * wheezy aka Debian 7, the current debian oldstable distribution * jessie aka Debian 8, the current debina stable distribution * centos-6 * centos-7 * fedora-22 * oel-7 There is lots of documentation on how to do this in the TESTING.asciidoc. Closes elastic#12611

nik9000 · 2015-08-10T09:32:58Z

Rebase was unremarkable - just required three way merge and git did it on its own. Merging and then hopping on a plane.

Run package tests in vagrant

nik9000 added >test Issues or PRs that are addressing/adding tests review :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts WIP labels Aug 4, 2015

nik9000 mentioned this pull request Aug 4, 2015

Bats tests should use plugins built in the elasticsearch repository #12651

Closed

nik9000 changed the title ~~[Packaging] run tests in vagrant~~ Run tests in vagrant Aug 5, 2015

nik9000 changed the title ~~Run tests in vagrant~~ Run package tests in vagrant Aug 5, 2015

nik9000 added v2.0.0 and removed WIP labels Aug 5, 2015

nik9000 mentioned this pull request Aug 6, 2015

Bats test for removing the rpm fails every once in a while #12682

Closed

spinscale reviewed Aug 6, 2015
View reviewed changes

nik9000 mentioned this pull request Aug 6, 2015

rpm and deb create scripts directory #12704

Merged

rmuir reviewed Aug 6, 2015
View reviewed changes

areek force-pushed the master branch from f50531c to 0a1c9de Compare August 7, 2015 04:37

nik9000 force-pushed the automate_bats branch from d26dfd2 to 9b5debc Compare August 7, 2015 12:11

clintongormley added v2.0.0-beta1 and removed v2.0.0 labels Aug 9, 2015

nik9000 force-pushed the automate_bats branch from 8cf86ae to f84552d Compare August 10, 2015 09:32

nik9000 added a commit that referenced this pull request Aug 10, 2015

Merge pull request #12646 from nik9000/automate_bats

6d7d975

Run package tests in vagrant

nik9000 merged commit 6d7d975 into elastic:master Aug 10, 2015

nik9000 mentioned this pull request Aug 10, 2015

Move vagrant activation to a parameter #12769

Merged

mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020

Run package tests in vagrant #12646

Run package tests in vagrant #12646

Conversation

nik9000 commented Aug 4, 2015

nik9000 commented Aug 4, 2015

dadoonet commented Aug 4, 2015

nik9000 commented Aug 4, 2015

rmuir commented Aug 4, 2015

nik9000 commented Aug 5, 2015

nik9000 commented Aug 5, 2015

electrical commented Aug 5, 2015

nik9000 commented Aug 5, 2015

nik9000 commented Aug 6, 2015

spinscale Aug 6, 2015

Choose a reason for hiding this comment

dadoonet commented Aug 6, 2015

spinscale commented Aug 6, 2015

dadoonet commented Aug 6, 2015

dadoonet commented Aug 6, 2015

rmuir commented Aug 6, 2015

dadoonet commented Aug 6, 2015

rmuir commented Aug 6, 2015

dadoonet commented Aug 6, 2015

rmuir commented Aug 6, 2015

rmuir Aug 6, 2015

Choose a reason for hiding this comment

nik9000 Aug 6, 2015

Choose a reason for hiding this comment

nik9000 commented Aug 6, 2015

dakrone commented Aug 6, 2015

nik9000 commented Aug 6, 2015

nik9000 commented Aug 6, 2015

tlrx commented Aug 7, 2015

nik9000 commented Aug 7, 2015

nik9000 commented Aug 7, 2015

dakrone commented Aug 7, 2015

polyfractal commented Aug 7, 2015

dadoonet commented Aug 8, 2015

polyfractal commented Aug 8, 2015

dadoonet commented Aug 8, 2015

rjernst commented Aug 8, 2015

rjernst commented Aug 8, 2015

dadoonet commented Aug 8, 2015

rmuir commented Aug 8, 2015

nik9000 commented Aug 10, 2015

nik9000 commented Aug 10, 2015