tests: add external spread backend #1918

fgimenez · 2016-09-15T09:56:34Z

This backend will enable the execution of the spread suite on all-snap running instances, so that the image validation can be automated.

With the current image state, it relies on the previous setup of the default user and network on the instances through console-conf and the execution of tests/lib/external/prepare-ssh.sh script included in this changeset. It also needs the SPREAD_EXTERNAL_ADDRESS environment variable set on the host with the address and port of the running instance, something like localhost:8023.

Currently, the backend includes three systems, ubuntu-core-16-64, ubuntu-core-16-arm-32 and ubuntu-core-16-arm-64.

…grade tests

… about tasks without system assigned

fgimenez · 2016-09-20T09:05:09Z

@niemeyer thanks for the review! :) comments addressed, pls take another look when you have the time

niemeyer

Have added updates to the issues above. Also, tests seem broken (perhaps the old create-key issue), and there are conflicts.

niemeyer

Rather large review, sorry:

niemeyer · 2016-09-26T22:09:51Z

tests/lib/adhoc-ubuntu-core/prepare-instance.sh

@@ -0,0 +1,14 @@
+#!/bin/sh


Can we please move this file to /external/prepare-ssh.sh?

niemeyer · 2016-09-26T22:10:44Z

tests/lib/prepare.sh

-    gadget_name=$(get_gadget_name)
-    for name in $gadget_name ${gadget_name}-kernel ubuntu-core; do
+    . $TESTSLIB/names.sh
+    for name in $gadget_name ${kernel_name} ubuntu-core; do


This can be $kernel_name, same as $gadget_name in the same line.

niemeyer · 2016-09-26T22:11:06Z

tests/lib/reset.sh

    for snap in $(ls /snap); do
-        if [ "$snap" = "bin" ] || [ "$snap" = "$gadget_name" ] ||  [ "$snap" = "${gadget_name}-kernel" ] || [ "$snap" = "ubuntu-core" ]; then
+        if [ "$snap" = "bin" ] || [ "$snap" = "$gadget_name" ] ||  [ "$snap" = "${kernel_name}" ] || [ "$snap" = "ubuntu-core" ]; then


This can be $kernel_name, same as $gadget_name in the same line.

niemeyer · 2016-09-26T22:12:12Z

tests/main/refresh-devmode/task.yaml

@@ -36,6 +35,9 @@ restore: |
    teardown_store $STORE_TYPE $BLOB_DIR

 execute: |
+    # skip the test for nonproduction stores on ubuntu-core systems
+    if [ "$STORE_TYPE" = "fake" -a "$SPREAD_SYSTEM" =~ ubuntu-core-16-* ]; then exit; fi


Would read better in three lines rather than one. Can even drop the comment, since it's just re-stating exactly what the code is doing.

niemeyer · 2016-09-26T22:21:32Z

spread.yaml

+            ADDRESS $SPREAD_ADHOC_UC_ADDRESS
+        systems:
+            # FIXME: remove the ubuntu-core-16.04-64 dummy system once the runner handles tasks without systems assigned
+            - ubuntu-core-16.04-64:


What's the comment above about?

This system is a bit awkward, as there's no such thing as an ubuntu-core-16.04. Happy to fix Spread if that's what the comment implies. Not sure about what's broken, though.

Yes, we need changes in spread, currently if you have a backend with specific systems and there's one task that excludes all of them, then you get an error, for instance, if we remove the dummy system above and try to execute the suite you get:

fgimenez@arkham:~/Desktop/snappy_validate_image$ spread -v -reuse external:ubuntu-core-16-device-amd64 2016/09/27 09:57:44 Found /home/fgimenez/Desktop/snappy_validate_image/spread.yaml. error: no systems specified for tests/main/interfaces-locale-control

The erroring task is different with each execution, probably related to the random ordering of tasks. Adding the dummy systems allow to execute the suite, tests/main/interfaces-locale-control is not executed anyway but doesn't throw any error.

I see, thanks. Spread was fixed in snap and github.

Thanks a lot, works great

niemeyer · 2016-09-26T22:41:52Z

spread.yaml

+            snap remove classic
+        fi
+        mkdir -p /etc/systemd/system/snapd.socket.d
+        cat <<EOF > /etc/systemd/system/snapd.socket.d/local.conf


Why is this needed? And why only here and not on classic?

Sorry, somehow this response got lost.
On amd64 images I was getting the same errors that we got with the service, now with the socket:

+ systemctl start snapd.service snapd.socket A dependency job for snapd.service failed. See 'journalctl -xe' for details. Job for snapd.socket failed. See "systemctl status snapd.socket" and "journalctl -xe" for details.

and in the journal:

Sep 19 16:07:05 localhost.localdomain systemd[1]: snapd.service: Job snapd.service/start failed with result 'dependency'. Sep 19 16:07:05 localhost.localdomain systemd[1]: snapd.socket: Start request repeated too quickly. Sep 19 16:07:05 localhost.localdomain systemd[1]: Failed to listen on Socket activation for snappy daemon. -- Subject: Unit snapd.socket has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit snapd.socket has failed.

Setting the burst limit for the socket solves the issue, not sure if there's a better way.

Okay, sounds good, but this doesn't respond the second part of the question: why only here and not on classic?

I guess that it might be because it tries to restart the socket quicker than in other systems, how can we design an experiment to know for sure the root cause?

Okay, we're already doing that for snapd.service, so should be okay for the socket.

Let's please do both on the same place, instead of special casing this system and backend.

niemeyer · 2016-09-26T22:45:09Z

spread.yaml

@@ -57,6 +57,24 @@ backends:
                username: ubuntu
                password: ubuntu

+    external-ubuntu-core:


Sorry, I know I've suggested this before, but can we please rename this one further to just external? It'll be nicer to see external:ubuntu-core* on the output than external-ubuntu-core:ubuntu-core.

niemeyer · 2016-09-26T22:47:59Z

tests/main/refresh/task.yaml

@@ -36,6 +35,9 @@ restore: |
    teardown_store $STORE_TYPE $BLOB_DIR

 execute: |
+    # skip the test for nonproduction stores on ubuntu-core systems
+    if [ "$STORE_TYPE" = "fake" -a "$SPREAD_SYSTEM" =~ ubuntu-core-16-* ]; then exit; fi


Same as related comment.

niemeyer · 2016-09-26T22:48:17Z

tests/main/revert-devmode/task.yaml

@@ -25,6 +24,9 @@ restore: |
    teardown_store $STORE_TYPE $BLOB_DIR

 execute: |
+    # skip the test for nonproduction stores on ubuntu-core systems
+    if [ "$STORE_TYPE" = "fake" -a "$SPREAD_SYSTEM" =~ ubuntu-core-16-* ]; then exit; fi


Same as related comment.

niemeyer · 2016-09-26T22:48:27Z

tests/main/revert/task.yaml

@@ -25,6 +24,9 @@ restore: |
    teardown_store $STORE_TYPE $BLOB_DIR

 execute: |
+    # skip the test for nonproduction stores on ubuntu-core systems
+    if [ "$STORE_TYPE" = "fake" -a "$SPREAD_SYSTEM" =~ ubuntu-core-16-* ]; then exit; fi


Same as related comment.

…stems

…to spread-validate-image

…ustomized image

Imported using git-ubuntu import. Changelog parent: d92891f New changelog entries: * New upstream release, LP: #1637215: - release: os-release on core has changed - tests: /dev/ptmx does not work on powerpc, skip here - docs: moved to github.com/snapcore/snapd/wiki (canonical#2258) - debian: golang is not installable on powerpc, use golang-any * New upstream release, LP: #1637215: - overlord/ifacestate: add unit tests for undo of setup-snap- security (canonical#2243) - daemon,overlord,snap,tests: download to .partial in final dir (canonical#2237) - overlord/state: marshaling tests for lanes (canonical#2245) - overlord/state: introduce state lanes (canonical#2241) - overlord/snapstate: fix revert+refresh (canonical#2224) - interfaces/sytemd: enable/disable generated service units (canonical#2229) - many: fix incorrect security files generation on undo - overlord/snapstate: add dynamic snapdX.Y assumes (canonical#2227) - interfaces: network-manager: give slot full read-write access to /run/NetworkManager - docs: update the name of the command for the cross-build - overlord/snapstate: fix missing argument to Noticef - snapstate: ensure gadget/core/kernel can not be disabled (canonical#2218) - asserts: limit to 1y only if len(models) == 0 (canonical#2219) - debian: only install share/locale if available (missing on powerpc) - overlrod/snapstate: fix revert followed by refresh to old-current (canonical#2214) - interfaces/builtin: network-manager and bluez can change hostname (canonical#2204) - snap: switch the auto-import dir to /run/snapd/auto-import - docs: less details about cloud.cfg as requested in trello (canonical#2206) - spread.yaml: Ensure ubuntu user has passwordless sudo for autopkgtests (canonical#2201) - interfaces/builtin: add dcdbas-control interface - boot: do not set boot to try mode if the revision is unchanged - interfaces: add shutdown interface (canonical#2162) - interfaces: add system-power-control interface - many: use the new systemd backend for configuring GPIOs - overlord/ifacestate: setup security for slots before plugs - snap: spool assertion candidates if snapd is not up yet - store,daemon,overlord: download things to a partials dir - asserts,daemon: implement system-user-authority header/concept - interfaces/builtin: home base declaration rule using on-classic for its policy - interfaces/builtin: finish decl based checks - asserts: bump snap-declaration to allow signing with new-style plugs and slots - overlord: checks for kernel installation/refresh based on model assertion and previous kernel - tests/lib/fakestore: fix logic to distinguish assertion not found errors - client: add a few explicit error types (around the request cycle) - tests/lib/fakestore/cmd/fakestore: make it log, and fix a typo - overlord/snapstate: two bugs for one - snappy: disable auto-import of assertions on classic (canonical#2122) - overlord/snapstate: move trash cleanup to a cleanup handler (canonical#2173) - daemon: make create-user --known fail on classic without --force- managed (canonical#2123) - asserts,interfaces/policy: implement on-classic plug/slot constraints - overlord: check that the first installed gadget matches the model assertion - tests: use the snapd-control-consumer snap from the store - cmd/snap: make snap run not talk to snapd for finding the revision - snap/squashfs: try to hard link instead of copying. Also, switch to osutil.CopyFile for cp invocation. - store: send supported max-format when retrieving assertions - snapstate, devicestate: do not remove seed - boot,image,overlord,partition: read/write boot variables in single operation - tests: reenable ubuntu-core tests on qemu - asserts,interfaces/policy: allow OR-ing of subrule constraints in plug/slot rules - many: move from flags as ints to flags as structs-of-bools (canonical#2156) - many: add supports for keeping and finding assertions with different format iterations - snap: stop using ubuntu-core-launcher, use snap-confine - many: introduce an assertion format iteration concept, refuse to add unsupported assertion - interfaces: tweak wording and comment - spread.yaml: dump apparmor denials on spread failure - tests: unflake ubuntu-core-reboot (canonical#2150) - cmd/snap: tweak unknown command error message (canonical#2139) - client,daemon,cmd: add payment-declined error kind (canonical#2107) - cmd/snap: update remove command help (canonical#2145) - many: removed frameworks target and fixed service files (canonical#2138) - asserts,snap: validate attributes to a JSON-compatible type subset (canonical#2140) - asserts: remove unused serial-proof type - tests: skip auto-import tests on systems without test keys (canonical#2142) - overlord/devicestate: don't spam the debug log on classic (canonical#2141) - cmd/snap: simplify auto-import mountinfo parsing (canonical#2135) - tests: run ubuntu-core upgrades on isolated machine (canonical#2137) - overlord/devicestate: recover seeding from old external approach (canonical#2134) - overlord: merge overlord/boot pkg into overlord/devicestate (canonical#2118) - daemon: add postCreateUserSuite test suite (canonical#2124) - tests: abort tests if an update process is scheduled (canonical#2119) - snapstate: avoid reboots if nothing in the boot setup has changed (canonical#2117) - cmd/snap: do not auto-import from loop or non-dev devices (canonical#2121) - tests: add spread test for `snap auto-import` (canonical#2126) - tests: add test for auto-mount assertion import (canonical#2127) - osutil: add missing unit tests for IsMounted (canonical#2133) - tests: check for failure creating user on managed ubuntu-core systems (canonical#2096) - snap: ignore /dev/loop addings from udev (canonical#2111) - tests: remove snapd.boot-ok reference (canonical#2109) - tests: enable tests related to the home interface in all-snaps (canonical#2106) - snapstate: only import defaults from gadget on install (canonical#2105) - many: move firstboot code into the snapd daemon (canonical#2033) - store: send correct JSON type of string for expected payment amount (canonical#2103) - cmd/snap: rename is-managed to managed and tune (canonical#2102) - interfaces,overlord/ifacestate: initial cleaning up of no arg AutoConnect related bits (canonical#2090) - client, cmd: prompt for password when buying (canonical#2086) - snapstate: fix hanging `snap remove` if snap is no longer mounted - image: support gadget specific cloud.conf file (canonical#2101) - cmd/snap,ctlcmd: fix behavior of snap(ctl) get (canonical#2093) - store: local users download from the anonymous url (canonical#2100) - docs/hooks.md: fix typos (canonical#2099) - many: check installation of slots and plugs against declarations - docs: fix missing "=" in the systemd-active docs - store: do not set store auth for local users (canonical#2092) - interfaces,overlord/ifacestate: use declaration-based checking for auto-connect (canonical#2071) - overlord, daemon, snap: support gadget config defaults (canonical#2082)The main semantic changes are: - tests: fix snap-disconnect tests after core rename (canonical#2088) - client,daemon,overlord,cmd: add /v2/users and create-user on auto- import (canonical#2074) - many: abbreviated forms of disconnect (canonical#2066) - asserts: require lowercase model until insensitive matching is ready (canonical#2076) - cmd/snap: add version command, same as --version (canonical#2075) - all: use "core" by default but allow "ubuntu-core" still (canonical#2070) - overlord/devicestate, docs/hooks.md: nest prepare-device configuration options - daemon: fix login API to return local macaroons (canonical#2078) - daemon: do not hardcode UID in userLookup (canonical#2080) - client, cmd: connect fixes (canonical#2026) - many: preparations for switching most of autoconnect to use the declarationsfor now: - overlord/auth: update CheckMacaroon to verify local snapd macaroons (canonical#2069) - cmd/snap: trivial auto-import and download tweaks (canonical#2067) - interfaces: add repo.ResolveConnect that handles name resolution - interfaces/policy: introduce InstallCandidate and its checks - interfaces/policy,overlord: check connection requests against the declarations in ifacestate - many: setup snapd macaroon for local users (canonical#2051)Next step: do snapd macaroons verification. - interfaces/policy: implement snap-id/publisher-id checks - many: change Connect to take ConnRef instead of strings (canonical#2060) - snap: auto mount block devices and import assertions (canonical#2047) - daemon: add `snap create-user --force-managed` support (canonical#2041) - docs: remove references to removed buying features (canonical#2057) - interfaces,docs: allow sharing SNAP{,_DATA,_COMMON} via content iface (canonical#2063) - interfaces: add Plug/Slot/Connection reference helpers (canonical#2056) - client,daemon,cmd/snap: improve create-user APIs (canonical#2054) - many: introduce snap refresh --ignore-validation <snap> to override refresh validation (canonical#2052) - daemon: add support for `snap create-user --known` (canonical#2040) - interfaces/policy: start of interface policy checking code based on declarations (canonical#2050) - overlord/configstate: support nested configuration (canonical#2039) - asserts,interfaces/builtin,overlord/assertstate: introduce base- declaration (canonical#2037) - interfaces: builtin: Allow writing DHCP lease files to /run/NetworkManager/dhcp (canonical#2049) - many: remove all traces of the /v2/buy/methods endpoint (canonical#2045) - tests: add external spread backend (canonical#1918) - asserts: parse the slot rules in snap-declarations (canonical#2035) - interfaces: allow read of /etc/ld.so.preload by default for armhf on series 16 (canonical#2048) - store: change purchase to order and store clean up first pass (canonical#2043) - daemon, store: switch to new store APIs in snapd (canonical#2036) - many: add email to UserState (canonical#2038) - asserts: support parsing the plugs stanza i.e. plug rules in snap- declarations (canonical#2027) - store: apply deltas if explicitly enabled (canonical#2031) - tests: fix create-key/snap-sign test isolation (canonical#2032) - snap/implicit: don't restrict the camera iface to clasic (canonical#2025) - client, cmd: change buy command to match UX document (canonical#2011) - coreconfig: nuke it. Also, ignore po/snappy.pot. (canonical#2030) - store: download deltas if explicitly enabled (canonical#2017) - many: allow use of the system user assertion with create-user (canonical#1990) - asserts,overlord,snap: add prepare-device hook for device registration (canonical#2005) - debian: adjust packaging for trusty/deputy systemd (canonical#2003) - asserts: introduce AttributeConstraints (canonical#2015) - interface/builtin: access system bus on screen-inhibit-control - tests: add firewall-control interface test (canonical#2009) - snapstate: pass errors from ListRefresh in updateInfo (canonical#2018) - README: add links to IRC, mailing list and social media (canonical#2022) - docs: add `configure` hook to hooks list (canonical#2024)LP: #1596629 - cmd/snap,configstate: rename apply-config variables to configure. (canonical#2023) - store: retry download on 500 (canonical#2019) - interfaces/builtin: support time and date settings via 'org.freedesktop.timedate1 (canonical#1832)

Federico Gimenez added 30 commits September 14, 2016 10:41

prevent prepare step for adhoc ubuntu-core

0ebb109

more changes

2e3c769

tasks blacklisted for the ubuntu-core-16-device system

1717ec7

renamed device to amd64; test user

9d65377

add prepare-image.sh script

bf6a27e

more tasks adapted to device system

f7f4262

added ubuntu-core-16-device-amd64 system to ubuntu-core specific tasks

c36777f

added ubuntu-core-16-device-arm system

af2a6fe

removed arm system from upgrade suite

4730fc0

added uboot test and removed arm system from grub tests

1c9d3eb

fixed task summary

c96d078

excluded from arm device until we have a snap for them

9e59ea7

fixed find glob

3f69cbc

for now check if grub-editenv is present before using it

7a350e0

reverted grub-editenv changes for arm and removed aarm system from up…

20b0c89

…grade tests

fuse interface test enabled on arm (test snap already uploaded)

1ad3d3b

add dummy system to adhoc-ubuntu-core backend to prevent runner error…

62916e2

… about tasks without system assigned

merged master

ac93668

enable ubuntu-core-upgrade-core for uboot based devices

7a0402b

snap_try_core and snap_mode checks for uboot in upgrade-core

426af6f

enable ubuntu-core-update-rollback-stresstest for uboot based devices

e926560

master merged and deconflicted

0ef8db1

removed ubuntu-core system from revert-devmode-nonproduction

8dd3470

changes to handle pi2-kernel with pi3 gadget

e9decf3

build test binaries on target

317f5d1

merged master

a64280b

removed unneeded variants

0792643

remove /home/fgimenez/workspace/gocode/src on backend restore

de059b4

read change ids from list

bc9a45c

make disabled burst limit available to adhoc-ubuntu-core too

804382e

niemeyer changed the title ~~spread.yaml, tests: add adhoc-ubuntu-core spread backend~~ tests: add adhoc-ubuntu-core spread backend Sep 19, 2016

Federico Gimenez added 2 commits September 20, 2016 09:56

Merge branch 'master' into spread-validate-image

4afba8e

review comments addressed

3ba99b3

fgimenez changed the title ~~tests: add adhoc-ubuntu-core spread backend~~ tests: add external-ubuntu-core spread backend Sep 20, 2016

prevent snapd.socket: Start request repeated too quickly. on amd64

46ca6fa

niemeyer requested changes Sep 25, 2016

View reviewed changes

Federico Gimenez added 6 commits September 26, 2016 10:51

Merge branch 'master' into spread-validate-image

2e06506

merged master and conflicts resolved

1deddf0

remove unneeded restore thanks to new spread reuse capabilities

16cf045

add missing ubuntu-core system restrictions

bdb32f1

skip tests instead of split to filter by system

570efbd

fix system check

898add2

niemeyer requested changes Sep 26, 2016

View reviewed changes

Federico Gimenez added 3 commits September 27, 2016 09:49

master merged and deconflicted

919b2c1

address review feedback

ed17fbf

fixed skip check and added to prepare and restore steps too

0c20edd

fgimenez changed the title ~~tests: add external-ubuntu-core spread backend~~ tests: add external spread backend Sep 27, 2016

Federico Gimenez added 9 commits September 28, 2016 14:07

Merge branch 'master' into spread-validate-image

c95e0c1

Merge branch 'master' into spread-validate-image

fffc48b

addressed review comments

81875ed

addressed review comments; added new ubuntu-core tests to external sy…

3966250

…stems

Merge branch 'spread-validate-image' of github.com:fgimenez/snappy in…

8d2b9e8

…to spread-validate-image

removed external systems from ubuntu-core-device-reg test

7984751

address review comments

83f7696

Merge branch 'master' into spread-validate-image

a716f4e

check for env var instead of backend for skipping tests that need a c…

d87fbdc

…ustomized image

niemeyer approved these changes Sep 30, 2016

View reviewed changes

niemeyer merged commit 0c28c25 into canonical:master Sep 30, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: add external spread backend #1918

tests: add external spread backend #1918

fgimenez commented Sep 15, 2016 •

edited

Loading

fgimenez commented Sep 20, 2016

niemeyer left a comment

niemeyer left a comment

niemeyer Sep 26, 2016 •

edited

Loading

fgimenez Sep 27, 2016

niemeyer Sep 26, 2016

fgimenez Sep 27, 2016

niemeyer Sep 26, 2016 •

edited

Loading

fgimenez Sep 27, 2016

niemeyer Sep 26, 2016 •

edited

Loading

fgimenez Sep 27, 2016

niemeyer Sep 26, 2016 •

edited

Loading

fgimenez Sep 27, 2016

niemeyer Sep 28, 2016

fgimenez Sep 29, 2016

niemeyer Sep 26, 2016

fgimenez Sep 27, 2016

niemeyer Sep 28, 2016

fgimenez Sep 29, 2016

niemeyer Sep 29, 2016

fgimenez Sep 30, 2016

niemeyer Sep 26, 2016

fgimenez Sep 27, 2016

niemeyer Sep 26, 2016 •

edited

Loading

fgimenez Sep 27, 2016

niemeyer Sep 26, 2016

fgimenez Sep 27, 2016

niemeyer Sep 26, 2016

fgimenez Sep 27, 2016

tests: add external spread backend #1918

tests: add external spread backend #1918

Conversation

fgimenez commented Sep 15, 2016 • edited Loading

fgimenez commented Sep 20, 2016

niemeyer left a comment

Choose a reason for hiding this comment

niemeyer left a comment

Choose a reason for hiding this comment

niemeyer Sep 26, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niemeyer Sep 26, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niemeyer Sep 26, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niemeyer Sep 26, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niemeyer Sep 26, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fgimenez commented Sep 15, 2016 •

edited

Loading

niemeyer Sep 26, 2016 •

edited

Loading

niemeyer Sep 26, 2016 •

edited

Loading

niemeyer Sep 26, 2016 •

edited

Loading

niemeyer Sep 26, 2016 •

edited

Loading

niemeyer Sep 26, 2016 •

edited

Loading