Implement group=True for group_manifests plugin by mlangsdorf · Pull Request #778 · containerbuildsystem/atomic-reactor

mlangsdorf · 2017-08-17T19:34:25Z

Add support for creating manifest lists. Includes four commits:

utils: allow get_manifest_digests() to return a digest from a manifest list

get_manifest_digests now supports v1, v2, and list.v2.

and

post_group_manifests: implement creation of the manifest list

when group=True, group_manifest is meant to create a manifest list.
It loops through all the platforms in the worker annotations and
creates a manifest_list_spec, using that platform's worker image
to create the image entry and the goarch information to create
the platform information.

The manifest_list_spec is then repeatedly retagged with the tags
from tag_conf.images, written to a yaml file, and pushed via
manifest-list. After all the images are tagged, the last image
is used to create a manifest digest of the manifest list. The manifest
digest is added to push_conf under the registry's name and the
image's tag, so that delete_from_registry can delete the
manifest list later.

A list of the manifest digest from each registry is returned.

and

group_manifests: add koji_import support for manifest lists

If group_manifests created a manifest list, koji_import should create a dict
in 'extras': 'index' that contains the pullspec and tags for the manifest
list. Have koji_import check the return of group_manifests, and if it
got a list of manifest digests, have it set up the 'extras': 'index'
dictionary.

and

delete_from_registry: delete manifest lists if they exist

If group_manifests creates a manifest list, it stores the registry and tag
in push_conf. If handle_worker_digests works correctly, we need to check to
see if there is a push_conf registry. If there isn't, there's no need to
post a warning.

twaugh · 2017-08-18T09:59:47Z

I just spotted this: estesp/manifest-tool#32

Since manifest-tool-0.6.0 you can add a list of additional tags! So we'd just need a single call to manifest-tool.

twaugh · 2017-08-18T09:49:19Z

tests/plugins/test_group_manifests.py

+        runner = PostBuildPluginsRunner(tasker, workflow, plugins_conf)
+        if valid:
+            result = runner.run()
+            test_results = []


Make this (and expected_results) a set() and use .add() instead of .append(), so that the order of the elements is not significant.

twaugh · 2017-08-18T09:56:25Z

atomic_reactor/util.py

    """Wrapper for digests for a docker manifest."""

-    def __init__(self, v1=None, v2=None):
+    def __init__(self, v1=None, v2=None, list=None):


This shadows the built-in type list. Maybe v2_list or something?

Should the default property prefer the manifest list digest to either the schema 2 or schema 1 image manifest digest? I think it should.

twaugh · 2017-08-18T10:08:14Z

atomic_reactor/plugins/post_group_manifests.py

+            push_conf_registry = self.workflow.push_conf.add_docker_registry(registry,
+                                                                             insecure=insecure)
+            push_conf_registry.config = get_config_from_registry(registry_image, registry, None,
+                                                                 insecure, secret_path, 'v2')


This will fetch the 'config' blob for the architecture=amd64 image. Are we only fetching 'config' because of the way push_conf works?

The only thing atomic-reactor does with 'config' is store an edited version of it in the Koji build metadata, and the worker builds will already have taken care of that.

If we don't need the config, I am happy to not get it. I was just copying the code from post_tag_and_push but looking at the workflow, it doesn't look like anyone requires a config for the docker registries.

twaugh · 2017-08-18T14:15:09Z

atomic_reactor/plugins/post_group_manifests.py

+
+            image_tags = []
+            for image in self.workflow.tag_conf.images:
+                image_tags.append(image.to_str(registry=False))


We want just the suffix part here I think, not the repository name. Also, we only need to set additional tags other than the one used in 'image'.

registry_image = self.workflow.tag_conf.images[0].copy() image_tags = [image.tag for image in self.workflow.tag_conf.images[1:]]

twaugh · 2017-08-18T14:43:24Z

atomic_reactor/util.py



 def get_manifest_media_type(version):
+    if version == 'list':


Maybe use 'v2_list' throughout, instead of 'list'? There's another type in the works as well, 'oci', so at some point we might need to define some constants to hold these names.

stickler-ci · 2017-08-21T14:47:20Z

atomic_reactor/plugins/post_group_manifests.py

+        """
+        # call parent constructor
+        super(GroupManifestsPlugin, self).__init__(tasker, workflow)
+        self.group = True # group


E261 at least two spaces before inline comment

stickler-ci · 2017-08-21T14:47:21Z

atomic_reactor/plugins/post_group_manifests.py

+    def submit_manifest_list(self, registry, registry_conf, manifest_list_spec):
+        docker_secret_path = registry_conf.get('secret', None)
+#        dockercfg = "{0}/.dockercfg".format(registry_conf.get('secret', None))
+


W293 blank line contains whitespace

stickler-ci · 2017-08-21T14:47:23Z

tests/plugins/test_group_manifests.py

+import json
+import responses
+from copy import deepcopy
+from tempfile import mkdtemp


F401 'tempfile.mkdtemp' imported but unused

stickler-ci · 2017-08-21T14:47:23Z

tests/plugins/test_group_manifests.py

+import responses
+from copy import deepcopy
+from tempfile import mkdtemp
+import os


F401 'os' imported but unused

stickler-ci · 2017-08-21T14:47:24Z

tests/plugins/test_group_manifests.py

+
+class TestGroupManifests(object):
+    @pytest.mark.parametrize(('goarch', 'worker_annotations', 'valid'), [
+#        ({}, {}, False),


E122 continuation line missing indentation or outdented

stickler-ci · 2017-08-21T14:47:24Z

tests/plugins/test_group_manifests.py

+#        ({}, {}, False),
+        ({'ppc64le': 'powerpc', 'x86_64': 'amd64'},
+         {'ppc64le': PPC_ANNOTATIONS, 'x86_64': X86_ANNOTATIONS}, True),
+#        ({'ppc64le': 'powerpc', 'x86_64': 'amd64'},


E122 continuation line missing indentation or outdented

stickler-ci · 2017-08-21T14:47:25Z

tests/plugins/test_group_manifests.py

+        ({'ppc64le': 'powerpc', 'x86_64': 'amd64'},
+         {'ppc64le': PPC_ANNOTATIONS, 'x86_64': X86_ANNOTATIONS}, True),
+#        ({'ppc64le': 'powerpc', 'x86_64': 'amd64'},
+#         {'ppc64le': PPC_ANNOTATIONS, 'x86_64': X86_ANNOTATIONS}, False),


E122 continuation line missing indentation or outdented

stickler-ci · 2017-08-21T14:47:26Z

tests/plugins/test_group_manifests.py

+            if valid:
+                repo_and_tag = workflow.tag_conf.images[0].to_str(registry=False).split(':')
+                url = 'http://{0}/v2/{1}/manifests/{2}'.format(registry, repo_and_tag[0],
+                                                                repo_and_tag[1])


E127 continuation line over-indented for visual indent

twaugh

Don't forget to update koji_import to make it add the new image.index metadata when a manifest list was created.

twaugh · 2017-08-21T15:12:13Z

atomic_reactor/plugins/post_group_manifests.py

+
+            cmd = ['manifest-tool', '--docker-cfg=%s' % docker_secret_path,
+                   'push', 'from-spec', fp.name]
+            subprocess.check_call(cmd, env={'HOME': docker_secret_path})


Could add a comment explaining that docker/cli (used by manifest-tool) only reads .dockercfg from $HOME.

twaugh · 2017-08-21T15:13:02Z

atomic_reactor/plugins/post_group_manifests.py

+            fp.flush()
+            self.log.debug("Wrote to file %s with config %s", fp.name, docker_secret_path)
+
+            cmd = ['manifest-tool', '--docker-cfg=%s' % docker_secret_path,


Do we think --docker-cfg is useful here at all, given how docker/cli behaves with .dockercfg?

It seems to me that only the newer-style 'config.json' file is read from the --docker-cfg location.

twaugh · 2017-08-21T15:14:21Z

atomic_reactor/plugins/post_group_manifests.py

+        """
+        # call parent constructor
+        super(GroupManifestsPlugin, self).__init__(tasker, workflow)
+        self.group = True # group


Let's not hard-code it to True. :-)

twaugh · 2017-08-21T15:16:16Z

atomic_reactor/plugins/post_group_manifests.py

+                manifest_list_spec['manifests'].append(arch_entry)
+
+            manifest_list_spec['tags'] = [image.tag for image in self.workflow.tag_conf.images]
+            # use a unique image tag because manifest_list doesn't like digests


It's manifest-tool that doesn't like digests. I still don't understand why.

Ah... we need to use a tag for the output we want manifest-tool to create because the output doesn't exist yet, so we can't very well know its digest.

twaugh · 2017-08-21T15:17:24Z

atomic_reactor/plugins/post_group_manifests.py

+            registry_image = self.workflow.tag_conf.unique_images[0]
+            registry_image.registry = registry
+            manifest_list_spec['image'] = registry_image.to_str()
+            self.log.info("Submitting manifest-list %s", manifest_list_spec)


The manifest-tool documentation calls this input file a 'spec'. Let's call it that in the log message too, to avoid confusion with the actual 'manifest list' that manifest-tool creates as its output.

twaugh · 2017-08-21T16:31:19Z

atomic_reactor/plugins/post_group_manifests.py

+                digest = worker_image['digest']
+                repository = worker_image['repository']
+                arch_entry = {
+                    'image': '{0}/{1}@{2}'.format(registry, repository, digest),


I think we need to use the unique tag instead of the digest here, and move the comment about manifest-tool not liking digests to here.

stickler-ci · 2017-08-22T15:14:20Z

atomic_reactor/plugins/exit_koji_import.py

+            repo = ImageName.parse(repositories[0]).repo
+            # group_manifests added the registry, so this should be valid
+            for registry in self.workflow.push_conf.docker_registries:
+                pullspec = "{0}/{1}@{2}".format(registry.uri, repo, manifest_list_digests[0].v2_list)


E501 line too long (101 > 100 characters)

stickler-ci · 2017-08-22T15:14:21Z

tests/plugins/test_koji_import.py

+    @pytest.mark.parametrize('digests', [
+        [],
+        [{'tag': 'testtag', 'schemaVersion': '2'}],
+        [ManifestDigest(v2_list='sha256:e6593f3e14000439dc62829a50945221c612e47cf5aa85a101677f7b76cf5811')],


E501 line too long (108 > 100 characters)

twaugh · 2017-08-24T07:46:47Z

atomic_reactor/plugins/exit_koji_import.py

+    def set_manifest_list_info(self, extra, worker_metadatas):
+        manifest_list_digests = self.workflow.postbuild_results.get(PLUGIN_GROUP_MANIFESTS_KEY)
+        # group_manifests returns either a list of manifest digests of the manifest lists,
+        # or a list of the image_manifests


By the way, can we fix this? I think it should always return a list of manifest list digests, and that this list should be empty it didn't create any manifest list (i.e. when group=False). That way the test on the following line can be much simpler: if manifest_list_digests:.

Let's remove this comment as it doesn't appear to be relevant anymore.

twaugh · 2017-08-24T07:52:12Z

atomic_reactor/plugins/exit_koji_import.py

+                index['pull'] = [pullspec]
+                for tag in index['tags']:
+                    pullspec = "{0}/{1}:{2}".format(registry.uri, repo, tag)
+                    index['pull'].append(pullspec)


We must not put "floating" tags here. Ideally we should put the {version}-{release} tag here but no other tags, so it looks like:
pull: [ "registry.example.com/foo@sha256:...", "registry.example.com/foo:1.0-1" ]
tags: [ "1.0-1", "latest", ... ]

One way to arrange this is to use the knowledge that only the {version}-{release} tag may contain a dash character:

... index['pull'] = [pullspec] for tag in index['tags']: if '-' in tag: # {version}-{release} index['pull'].append("{0}/{1}:{2}".format(registry.uri, repo, tag)) break

twaugh · 2017-08-24T07:57:19Z

atomic_reactor/plugins/post_group_manifests.py

+            # --docker-cfg may be rendundant here, but it's how the tool should work
+            cmd = ['manifest-tool', '--docker-cfg=%s' % docker_secret_path,
+                   'push', 'from-spec', fp.name]
+            # docker always looks in $HOME for the .docker_cfg, so set $HOME to the path


nit: typo: ".dockercfg"

twaugh · 2017-08-24T07:57:59Z

atomic_reactor/plugins/post_group_manifests.py

+                manifest_list_spec['manifests'].append(arch_entry)
+
+            manifest_list_spec['tags'] = [image.tag for image in self.workflow.tag_conf.images]
+            # use a unique image tag because manifest_list can't accept a digest that


Nit: typo: "... because manifest-tool can't... "

twaugh · 2017-08-24T07:58:25Z

atomic_reactor/plugins/post_group_manifests.py

+            registry_image = self.workflow.tag_conf.unique_images[0]
+            registry_image.registry = registry
+            manifest_list_spec['image'] = registry_image.to_str()
+            self.log.info("Submitting manifest-list %s", manifest_list_spec)


Can we say "spec" in this log message? Submitting manifest-list spec %s

twaugh · 2017-08-24T08:08:55Z

atomic_reactor/plugins/exit_koji_import.py

+            repositories = self.workflow.build_result.annotations['repositories']['unique']
+            repo = ImageName.parse(repositories[0]).repo
+            # group_manifests added the registry, so this should be valid
+            for registry in self.workflow.push_conf.docker_registries:


If Pulp is used, we only want to include the Pulp registry. Otherwise, we want to include all registries we pushed to. So it should be:

registries = self.workflow.push_conf.pulp_registries if not registries: registries = self.workflow.push_conf.all_registries for registry in registry:

twaugh · 2017-08-24T10:42:38Z

atomic_reactor/plugins/post_group_manifests.py

+
+    def get_grouped_manifests(self):
+        grouped_manifests = []
+        for registry, registry_conf in self.registries.items():


Skip registries where registry.get('version') == 'v1'

twaugh · 2017-08-24T10:59:21Z

atomic_reactor/plugins/post_group_manifests.py

+
+            manifest_list_spec['tags'] = [image.tag for image in self.workflow.tag_conf.images]
+            # use a unique image tag because manifest_list can't accept a digest that
+            # isn't in the respoistory yet


Nit: typo: repository

stickler-ci · 2017-08-25T12:23:37Z

tests/plugins/test_group_manifests.py

+                expected_results.add('v2_list-digest:{0}'.format(url))
+            for image in workflow.tag_conf.images:
+                repo_and_tag = image.to_str(registry=False).split(':')
+                path = '/v2/{0}/manifests/{1}'.format(repo_and_tag[0], repo_and_tag[1])


E501 line too long (87 > 79 characters)

stickler-ci · 2017-08-25T12:23:38Z

tests/plugins/test_group_manifests.py

+                responses.add(responses.GET, https_url, body=ConnectionError())
+                url = 'http://' + registry + path
+                if valid:
+                    responses.add_callback(responses.GET, url, callback=request_callback)


E501 line too long (89 > 79 characters)

stickler-ci · 2017-08-25T12:23:38Z

tests/plugins/test_koji_import.py

 from atomic_reactor.source import GitSource, PathSource
 from atomic_reactor.build import BuildResult
-from atomic_reactor.constants import PLUGIN_PULP_PULL_KEY, PLUGIN_PULP_SYNC_KEY
+from atomic_reactor.constants import (PLUGIN_PULP_PULL_KEY, PLUGIN_PULP_SYNC_KEY,


E501 line too long (81 > 79 characters)

stickler-ci · 2017-08-25T12:48:20Z

tests/plugins/test_koji_import.py

+                                            session=session,
+                                            docker_registry=True)
+        workflow.postbuild_results[PLUGIN_GROUP_MANIFESTS_KEY] = digests
+        orchestrate_plugin = workflow.plugin_workspace[OrchestrateBuildPlugin.key]


E501 line too long (82 > 79 characters)

stickler-ci · 2017-08-25T12:48:20Z

tests/plugins/test_koji_import.py

+        if digests:
+            assert 'index' in image.keys()
+            expected_results = {}
+            expected_results['tags'] = [tag.tag for tag in workflow.tag_conf.images]


E501 line too long (84 > 79 characters)

stickler-ci · 2017-08-25T12:48:21Z

tests/plugins/test_koji_import.py

+            assert 'index' in image.keys()
+            expected_results = {}
+            expected_results['tags'] = [tag.tag for tag in workflow.tag_conf.images]
+            pullspec = "docker.example.com/hello-world@{0}".format(digests[0].v2_list)


E501 line too long (86 > 79 characters)

vrutkovs · 2017-08-25T13:09:06Z

atomic_reactor/plugins/post_group_manifests.py

+            try:
+                subprocess.check_call(cmd)
+            except CalledProcessError as exc:
+                self.log.error("manifest-tool failed with %s", exc.output)


check_output(cmd) must be used to have output in exception, check_call won't cut it

stickler-ci · 2017-08-25T13:26:28Z

tests/plugins/test_group_manifests.py

+         {'ppc64le': PPC_ANNOTATIONS, 'x86_64': X86_ANNOTATIONS}, False),
+    ])
+    @responses.activate  # noqa
+    def test_group_manifests_true(self, tmpdir, goarch, worker_annotations, valid):


E501 line too long (83 > 79 characters)

stickler-ci · 2017-08-25T13:26:29Z

tests/plugins/test_group_manifests.py

+            },
+        }]
+        tasker, workflow = mock_environment(tmpdir, primary_images=test_images,
+                                            worker_annotations=worker_annotations)


E501 line too long (82 > 79 characters)

stickler-ci · 2017-08-25T13:26:29Z

tests/plugins/test_group_manifests.py

+            mock_docker()
+
+        goarch = {'ppc64le': 'powerpc', 'x86_64': 'amd64'}
+        worker_annotations = {'ppc64le': PPC_ANNOTATIONS, 'x86_64': X86_ANNOTATIONS}


E501 line too long (84 > 79 characters)

stickler-ci · 2017-08-25T13:33:54Z

atomic_reactor/plugins/post_group_manifests.py

+                   'push', 'from-spec', fp.name]
+            # docker always looks in $HOME for the .dockercfg, so set $HOME to the path
+            try:
+                check_output(cmd, stderr=STDOUT, env={'HOME': docker_secret_path})


E501 line too long (82 > 79 characters)

…t list get_manifest_digests now supports v1, v2, and list.v2 (as v2_list). Signed-off-by: Mark Langsdorf <mlangsdo@redhat.com>

lcarva · 2017-08-25T14:57:10Z

atomic_reactor/plugins/post_group_manifests.py

+                check_output(cmd, stderr=STDOUT, env={'HOME': docker_secret_path})
+            except CalledProcessError as exc:
+                self.log.error("manifest-tool failed with %s", exc.output)
+                raise RuntimeError


How about just re-raising the previous exception? (raise statement with no exception class)
As it is, the exception that gets propagated contains no useful information.

twaugh · 2017-08-25T17:14:02Z

tests/plugins/test_koji_import.py

+            assert 'index' in image.keys()
+            expected_results = {}
+            expected_results['tags'] = [tag.tag for tag in workflow.tag_conf.images]
+            pullspec = "docker.example.com/hello-world@{0}".format(digests[0].v2_list)


This is incorrect! It should be "docker.example.com/myproject/hello-world@{0}". Without the namespace part included, the image is not pullable.

twaugh · 2017-08-25T17:16:36Z

atomic_reactor/plugins/exit_koji_import.py

+            index = {}
+            index['tags'] = [image.tag for image in self.workflow.tag_conf.images]
+            repositories = self.workflow.build_result.annotations['repositories']['unique']
+            repo = ImageName.parse(repositories[0]).repo


This should be: repo = ImageName.parse(repositories[0]).to_str(registry=False, tag=False).

twaugh · 2017-08-25T22:32:32Z

tests/plugins/test_group_manifests.py

-        test_results = [{'tag': 'testtag', 'schemaVersion': '2'}]
+        test_images = ['registry.example.com/namespace/httpd:2.4',
+                       'registry.example.com/namespace/httpd:latest']
+#        test_results = [{'tag': 'testtag', 'schemaVersion': '2'}]


Not needed?

when group=True, group_manifest is meant to create a manifest list. It loops through all the platforms in the worker annotations and creates a manifest_list_spec, using that platform's worker image to create the image entry and the goarch information to create the platform information. The manifest_list_spec is then repeatedly retagged with the tags from tag_conf.images, written to a yaml file, and pushed via manifest-list. After all the images are tagged, the last image is used to create a manifest digest of the manifest list. The manifest digest is added to push_conf under the registry's name and the image's tag, so that delete_from_registry can delete the manifest list later. A list of the manifest digest from each registry is returned. Signed-off-by: Mark Langsdorf <mlangsdo@redhat.com>

If group_manifests created a manifest list, koji_import should create a dict in 'extras': 'index' that contains the pullspec and tags for the manifest list. Have koji_import check the return of group_manifests, and if it got a list of manifest digests, have it set up the 'extras': 'index' dictionary. Signed-off-by: Mark Langsdorf <mlangsdo@redhat.com>

If group_manifests creates a manifest list, it stores the registry and tag in push_conf. If handle_worker_digests works correctly, we need to check to see if there is a push_conf registry. If there isn't, there's no need to post a warning. Signed-off-by: Mark Langsdorf <mlangsdo@redhat.com>

lcarva · 2017-08-28T13:42:54Z

Updated release notes.

twaugh self-requested a review August 18, 2017 09:21

twaugh reviewed Aug 18, 2017

View reviewed changes

mlangsdorf force-pushed the group_manifests_true branch 2 times, most recently from f9fea7f to 1933f1c Compare August 18, 2017 14:15

twaugh reviewed Aug 18, 2017

View reviewed changes

vrutkovs mentioned this pull request Aug 21, 2017

group_manifests: create minimal plugin for group manifests #756

Merged

mlangsdorf force-pushed the group_manifests_true branch from 1933f1c to 423eebe Compare August 21, 2017 14:46

stickler-ci reviewed Aug 21, 2017

View reviewed changes

twaugh reviewed Aug 21, 2017

View reviewed changes

mlangsdorf force-pushed the group_manifests_true branch 3 times, most recently from 89c6e8d to 69830bc Compare August 22, 2017 11:52

stickler-ci reviewed Aug 22, 2017

View reviewed changes

mlangsdorf force-pushed the group_manifests_true branch 2 times, most recently from 77ab82e to 0d9f03e Compare August 22, 2017 20:30

twaugh reviewed Aug 24, 2017

View reviewed changes

twaugh changed the title ~~Group manifests true~~ Implement group=True for group_manifests plugin Aug 24, 2017

twaugh reviewed Aug 24, 2017

View reviewed changes

stickler-ci reviewed Aug 25, 2017

View reviewed changes

mlangsdorf force-pushed the group_manifests_true branch from 6109db3 to 7b7d7f0 Compare August 25, 2017 12:47

stickler-ci reviewed Aug 25, 2017

View reviewed changes

vrutkovs reviewed Aug 25, 2017

View reviewed changes

mlangsdorf force-pushed the group_manifests_true branch from 7b7d7f0 to aff2dae Compare August 25, 2017 13:26

stickler-ci reviewed Aug 25, 2017

View reviewed changes

mlangsdorf force-pushed the group_manifests_true branch from aff2dae to 1ec1d2d Compare August 25, 2017 13:33

stickler-ci reviewed Aug 25, 2017

View reviewed changes

utils: allow get_manifest_digests() to return a digest from a manifes…

61f3fad

…t list get_manifest_digests now supports v1, v2, and list.v2 (as v2_list). Signed-off-by: Mark Langsdorf <mlangsdo@redhat.com>

mlangsdorf force-pushed the group_manifests_true branch from 1ec1d2d to 2862afc Compare August 25, 2017 14:23

lcarva reviewed Aug 25, 2017

View reviewed changes

mlangsdorf force-pushed the group_manifests_true branch from 2862afc to 9bd0795 Compare August 25, 2017 15:09

twaugh reviewed Aug 25, 2017

View reviewed changes

mlangsdorf force-pushed the group_manifests_true branch from 9bd0795 to 3cdd337 Compare August 25, 2017 17:43

lcarva approved these changes Aug 25, 2017

View reviewed changes

twaugh approved these changes Aug 25, 2017

View reviewed changes

mlangsdorf added 3 commits August 28, 2017 07:22

mlangsdorf force-pushed the group_manifests_true branch from 3cdd337 to 9326aac Compare August 28, 2017 12:22

lcarva merged commit 2d7ea16 into containerbuildsystem:master Aug 28, 2017

mlangsdorf deleted the group_manifests_true branch August 31, 2017 19:26

Conversation

mlangsdorf commented Aug 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

twaugh commented Aug 18, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

twaugh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

mlangsdorf commented Aug 17, 2017 •

edited

Loading