docker, PR testing, py36, and regression testing

[satra]

1. dockerhub builds our containers from master and circle currently rebuilds these.

   • could we reuse the dockerhub containers unless:
     1. the dockerfiles have changed?
     2. our requirements have changed.

2. shall we also always merge upstream branch for any PR during the test? this way it also tests if the PR is mergeable without any conflict?

3. we should also have a lightweight nipype docker, with no dependencies installed

4. extend nipype to py36

5. i've asked @mgxd to work towards regression testing (as a separate project). i'm leaning towards looking into https://github.com/bazelbuild/bazel for it. the test matrix is a really high dimensional space:

• python versions
• dependent python library versions
• dependent imaging software versions
• interfaces/workflows
• a few bids datasets

@oesteban, @chrisfilo, @djarecka - any thoughts.

[oesteban]

1. dockerhub builds our containers from master and circle currently rebuilds these.
   • could we reuse the dockerhub containers unless:
     1. the dockerfiles have changed?
     2. our requirements have changed.

We can keep the automated build but changing the label to "master" or so (I'm to do this right now). The "latest" should be pointing to the latest tagged release, which now will be deployed from circle after successful tag push.

I wouldn't use containers from dockerhub, the circle cache should be enough to avoid the overhead of building them every time. Is that what you meant by "reuse the dockerhub containers"?

2. shall we also always merge upstream branch for any PR during the test? this way it also tests if the PR is mergeable without any conflict?

Hopefully we are reaching to a point where the images don't change very often, so I guess we can leave this as a good practice being flexible enough to avoid introducing noise in third parties PRs.

3. we should also have a lightweight nipype docker, with no dependencies installed

Agreed

4. extend nipype to py36

Agreed again :)

5. i've asked @mgxd to work towards regression testing (as a separate project). i'm leaning towards looking into https://github.com/bazelbuild/bazel for it. the test matrix is a really high dimensional space:

• python versions
• dependent python library versions
• dependent imaging software versions
• interfaces/workflows
• a few bids datasets

This is great, but I think the free tier of CircleCI will fall short soon if we do this. What could the infrastructure be for this?