99 import dataset from renku aware repo

[mohammad-alisafaee]

IMPORTANT: Please do not create a Pull Request without creating an issue first.

Description

This PR enables copying metadata from other Renku projects when adding files to datasets. ATM, only creator is copied over from the original dataset's metadata.

Moreover, if fixes some minor issues with renku dataset add and renku dataset update.

Fixes #99
Fixes #763

Type of change

Please select relevant options.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist

Do not create pull request unless you checked all points.

• I have read and understood the CONTRIBUTING document.
• I have performed a self-review of my own code.
• I have signed and sent the [CLA document]
  (https://github.com/SwissDataScienceCenter/documentation/wiki/files/SDSC_Contributor_License_Agreement_v1.0.pdf).
• I have added tests to cover my changes.
• I have NOT removed any existing tests.
• I have updated the documentation accordingly.
• I have described all changes in the CHANGELOG.rst.
• I have run ./run-tests.sh locally.

[rokroskar]

Thanks for this PR! A couple of minor comments in the code and two main issues:

1. when the remote repo is cloned, it is not done with GIT_LFS_SKIP_SMUDGE=1 so all of the data is downloaded
2. I cannot manage to generate any triples for the files I add - I can't tell if this is an issue with the recent json-ld context changes or with this PR in particular - @Panaetius?
3. This PR allows me to import single files, but what if I want to import an entire dataset? We should add a renku provider in addition to Zenodo and Dataverse probably. Perhaps that should be a separate PR.

[rokroskar]

Using #782 this looks pretty good:

There are two things I'm not sure about.

1. git@renkulab.io:rok.roskar/meteorology.git/data/meteorology/MAWS__SMSAWS__20161123.txt is not a valid git URL. I know that it's meant to show the path within the repo but it's not usable by any client. I can't do git clone <URL-from-above and I can't paste it into my browser. Furthermore, the git@ makes it not a valid URL which is also a bit confusing. It could have the git+ssh:// prefix, but it still won't resolve to anything useful. I'm wondering though if specifying the URL here is even needed since we point to the more complete source information which includes the commit and other information.

2. I know we discussed this already in person but I'm still not super comfortable with isBasedOn - I'm wondering if you have considered any alternatives? Maybe prov:wasDerivedFrom would be better (also because it's a provenance issue, i.e. "where did this file come from")? But really, since the file has not been modified using the schema:sameAs might be the most appropriate.

[mohammad-alisafaee]

2. I know we discussed this already in person but I'm still not super comfortable with isBasedOn - I'm wondering if you have considered any alternatives? Maybe prov:wasDerivedFrom would be better (also because it's a provenance issue, i.e. "where did this file come from")? But really, since the file has not been modified using the schema:sameAs might be the most appropriate.

As we discussed, we need to use isBasedOn because we still don't have a single URL/URI that can point to a file to a Git repo. We need to point to a DatasetFile entity but schema:sameAs can only be a URL.

[mohammad-alisafaee]

@rokroskar please just review the last commit about re-raising exceptions.

[rokroskar]

TIL raise...from :)