Added support for grabbing and putting data on S3.

[shariqiqbal2810]

• Added S3DataGrabber to nipype/interfaces/io.py
• Added S3DataSink to nipype/interfaces/io.py
• Added tests to nipype/interfaces/tests/test_io.py
  • Currently the S3 tests are using a bucket on our lab's account.
  • We would like to have a conversation about a more long-term testing solution.

[satra]

for testing the grabber you can use the openfmri bucket (which is a public bucket). for the sink, i need to think of a better solution.

[shariqiqbal2810]

I changed the s3datagrabber test to use the openfmri data. Unfortunately, these files are fairly large, so the tests are slowed (even though I'm only downloading 2 files). Additionally, these modules require the use of boto3 (AWS Python API), and I was wondering what your solution would be to keep this optional. Currently I have this with my imports

try:
    import boto3
except:
    pass

[shariqiqbal2810]

FYI the only tests that are failing are due to a lack of boto3 on the testing system. Tests all pass on my system.

[chrisgorgo]

We should skip those tests on systems without boto3 (since not all users
will use it), but also add it to circle.yml config file so it would be
installed on the test server.

On Fri, Sep 4, 2015 at 3:56 AM, Shariq Iqbal notifications@github.com
wrote:

FYI the only tests that are failing are due to a lack of boto3 on the
testing system. Tests all pass on my system.

—
Reply to this email directly or view it on GitHub
#1201 (comment).

[shariqiqbal2810]

Ok done! Any more thoughts on how to implement the S3DataSink test?

[shariqiqbal2810]

So I think the only remaining things to sort out are:

• Figuring out a more permanent testing solution for S3DataSink
• Getting AWS credentials for the testing server

[chrisgorgo]

What about using a fake S3 server? https://github.com/jubos/fake-s3

[shariqiqbal2810]

I modified the code to use a fake s3 server on the S3DataSink tests and use an anonymous request on the S3DataGrabber test (in order to avoid needing credentials). I added a 'gem install fakes3' line to the circle.yml file. Hopefully that will be enough to get everything working (It should as long as 'gem' works on the testing server)