Dockerfile for Twitcher with Magpie adapter

[fmigneault]

Use latest Twitcher with its docker image as base to build a new Magpie-flavoured image overriding the adapter.

[dbyrns]

@huard, @tlvu here is our plan:

• Submit a PR to the birdhouse/twitcher so that interfaces required by Magpie are available as well as providing a dockerfile (which is currently missing).
• Add a dockerfile inside the magpie repo that build on top of the stock twitcher image which install the interface implementation of Magpie (using AuthN and AuthZ of Magpie). This way each time Magpie is updated we can build the two modules.
• Finally the web app will use the new component Weaver to fetch registered processes, launch job, monitor them and get the result all via its rest interface (like finch). That will provide a way to list all processes in a single flat list as required in the past.
• Twitcher will still be used as a policy enforcement point but since all WPS providers will be hidden behind Weaver, this is the only WPS component that will require to be protected. Each registered processes considered as weaver resources with individual permissions set.

Tel me if that sound good to you.

[huard]

+1

…

On Tue, May 7, 2019 at 4:21 PM David Byrns ***@***.***> wrote: @huard <https://github.com/huard>, @tlvu <https://github.com/tlvu> here is our plan: - Submit a PR to the birdhouse/twitcher so that interfaces required by Magpie are available as well as providing a dockerfile (which is currently missing). - Add a dockerfile inside the magpie repo that build on top of the stock twitcher image which install the interface implementation of Magpie (using AuthN and AuthZ of Magpie). This way each time Magpie is updated we can build the two modules. - Finally the web app will use the new component Weaver to fetch registered processes, launch job, monitor them and get the result all via its rest interface (like finch). That will provide a way to list all processes in a single flat as required in the past. - Twitcher will still be used as a policy enforcement point but since all WPS providers will be hidden behind Weaver, this is the only WPS component that will require to be protected. Each registered processes considered as weaver resources with individual permissions set. Tel me if that sound good to you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#182 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAT2QZCOMB7FCMYHRNJHRLPUHQEVANCNFSM4HLMDOVQ> .

-- David Huard, PhD Spécialiste Scénarios et services climatiques, Ouranos

[tlvu]

* Add a dockerfile inside the magpie repo that build on top of the stock twitcher image which install the interface implementation of Magpie (using AuthN and AuthZ of Magpie). This way each time Magpie is updated we can build the two modules.

So the Magpie repo will create 2 differents docker images? pavics/magpie (magpie standalone) and pavics/twitcher-magpie (twitcher image with magpie as adapteur pre-installed to avoid it being installed only at container startup time)?

* Finally the web app will use the new component Weaver to fetch registered processes, launch job, monitor them and get the result all via its rest interface (like finch). That will provide a way to list all processes in a single flat list as required in the past.

The "web app" you mean magpie web interface to set user permissions?

Tel me if that sound good to you.

+1 for me as well, thanks !

[dbyrns]

So the Magpie repo will create 2 differents docker images? pavics/magpie (magpie standalone) and pavics/twitcher-magpie (twitcher image with magpie as adapteur pre-installed to avoid it being installed only at container startup time)?

Yes, that's what we want to avoid. Moreover it will be more intuitive to rebuild the twitcher-magpie image when magpie is updated if its docker is at the same place.

The "web app" you mean magpie web interface to set user permissions?

No I mean what we used to call the platform, but the PAVICS web app, the main interface. What is really blocking us right now to update you to the latest version of magpie / twitcher.

+1 for me as well, thanks !

Thank you for your support on this.

[tlvu]

So the Magpie repo will create 2 differents docker images? pavics/magpie (magpie standalone) and pavics/twitcher-magpie (twitcher image with magpie as adapteur pre-installed to avoid it being installed only at container startup time)?

Yes, that's what we want to avoid. Moreover it will be more intuitive to rebuild the twitcher-magpie image when magpie is updated if its docker is at the same place.

More intuitive like you said, faster twitcher container startup time since no install of magpie to do at the beginning, more reproducible since all magpie recursive dependencies are locked down in the docker image twitcher-magpie, more robust since no external dependencies on github and pypi during container startup since no install is required.

Very happy to see this.

[cehbrecht]

I have updated birdhouse/twitcher on master:

• skipped buildout: skip buildout bird-house/twitcher#49
• replaced mongod by sqlalchemy: replace mongodb by sqlalchemy (sqlite, postgres)? bird-house/twitcher#51

There are slight changes to the api (token has no data, service registry needs name):

$ git diff v0.4.0 twitcher/api.py

Hopefully this does not interfere with your changes. There is still the 0.4.x branch.

Currently the master is for development only ... I need to add ansible scripts for production use ... and there is no Docker.

[dbyrns]

Thank you @cehbrecht for your update. @fmigneault is aware of your update and is planning to start from your work to propose a PR that only add an interface to allow us to use a Magpie implementation (extern to you repo) for authentification and permissions. However the PR should also include a Dockerfile to allow us to build from something. Hopefully this is not something you are completely against it.

[cehbrecht]

... no, go ahead. With the current master the Dockerfile would be different (using waitress).

[fmigneault]

@cehbrecht
Is it possible to update/create the docker hub repo for Twitcher to allow working on this issue?
I do not have the permissions to push a new image.

[cehbrecht]

@fmigneault I have overseen this ... I can add you to the docker-cloud birdhouse organization. Please tell me your docker-id.

[fmigneault]

@cehbrecht fmigneault like on github
thanks ;)

[cehbrecht]

@fmigneault done (docker-cloud).

@cehbrecht fmigneault like on github
thanks ;)

[tomLandry]

Nice job, glad this converged. Looks awesome.