Feature request: create a harvested dataset via the API

[hakbailey]

Some background: MIT is working on a research data registry using Dataverse as the repository, and as part of this project we want to ingest metadata into Dataverse such that it is indexed and searchable but the search result points to the source repository instead of a local Dataverse record. This is the exact functionality of harvested records, however we are building our own harvesting service because we need to perform actions not possible in the internal harvester, like parsing returned records to identify MIT-affiliated authors and enhancing the metadata with data from our local data warehouse.

In talking to the Dataverse tech team about this, the question rose of whether it would be feasible/useful to add the ability to create "harvested" datasets via the API. They suggested we add it here as an issue to prompt discussion...questions, comments, and suggestions welcome!

[poikilotherm]

This might be interesting for us along the way, as we have a similar idea for Jülich DATA, mostly targeted to being a metadata -based indexer of all scholarly research data dropping on and off campus.

We haven't decided yet if we would try to look into the https://github.com/IQSS/doi2pmh-server project, but this might be a benefit for that project, too. @RightInTwo wasn't your preferred way back then to push rather than pull?

[RightInTwo]

Hi! I'm sorry to say that I forgot to follow up on that project and do not work in the field of research data anymore/for now. My apologies go out to @tcoupin because I did not give you an update about discontinuing my employment.

But maybe there is an interest to pick it up now, again. The general approach of providing a server that allows to make custom OAI-PMH sets from DOIs (which are not necessarily in a set at the source or even at the same source at all) seems to serve multiple use cases, see #5402 and https://github.com/IQSS/doi2pmh-server/blob/master/README.md.

Even though I'm currently not in a position to invest time, I'd be glad to take part in the discussion.

[djbrooke]

Thanks @hakbailey for creating the issue. It'll be good to follow along with the work in #7325 from @qqmyers, as that may serve your use case as well.

[pacian]

UCPH would be very interested in this long standing feature request. I think it would be well worth upstreaming this feature to the larger community if it should ever come to life.

[poikilotherm]

UCPH would be very interested in this long standing feature request. I think it would be well worth upstreaming this feature to the larger community if it should ever come to life.

@pacian You might also be interested in a recent post of mine to the mailing list: https://groups.google.com/g/dataverse-community/c/L4HivEHETUI

[pdurbin]

I'm just adding the word "import" here because I keep having trouble finding this issue. The way I keep thinking of this issue is something like like: I want to import a dataset and have it appear as if it had been harvested.

[DS-INRAE]

@pacian You might also be interested in a recent post of mine to the mailing list: https://groups.google.com/g/dataverse-community/c/L4HivEHETUI

@poikilotherm has your fork been implemented on your installation since ? :)

[poikilotherm]

Unfortunately, not yet.