Native API: What fields are required for dataset creation

[leeper]

I'm looking at this example file for dataset creation via the API: https://github.com/IQSS/dataverse/blob/master/scripts/api/data/dataset-create-new3.json. It's quite extensive. Currently the R client just requires end-users to know what is required and what is not (something I don't actually know) so they can make their own similar file, but I would like to be able facilitate this a bit (issue here: IQSS/dataverse-client-r#3) to make things easier.

Is there documentation somewhere of what fields are mandatory? If not, could this be added to the API docs?

[posixeleni]

At the moment the only place this is documented in our API docs is for the
SWORD API
http://guides.dataverse.org/en/4.3/api/sword.html#create-a-dataset-with-an-atom-entry

But as mentioned above this needs to be explained in any API guide that
involves deposit or updating datasets.

On Tue, Apr 5, 2016 at 11:09 AM, Thomas J. Leeper notifications@github.com
wrote:

I'm looking at this example file for dataset creation via the API:
https://github.com/IQSS/dataverse/blob/master/scripts/api/data/dataset-create-new3.json
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_IQSS_dataverse_blob_master_scripts_api_data_dataset-2Dcreate-2Dnew3.json&d=CwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=OiTcIL_jSSCUW82Gf-OSkgnZLbG2Yt87eT87BdLiP54&m=4z5nsJFaX-7ZYnMX7G0EH3VkoaKUgzKYMoxeTXS6I_U&s=zJ8a-QwjnV-t_L75c3IHpJteRHz9quCEMMSZbjDaUzk&e=.
It's quite extensive. Currently the R client just requires end-users to
know what is required and what is not (something I don't actually know) so
they can make their own similar file, but I would like to be able
facilitate this a bit (issue here: IQSS/dataverse-client-r#3
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_IQSS_dataverse-2Dclient-2Dr_issues_3&d=CwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=OiTcIL_jSSCUW82Gf-OSkgnZLbG2Yt87eT87BdLiP54&m=4z5nsJFaX-7ZYnMX7G0EH3VkoaKUgzKYMoxeTXS6I_U&s=aZ_7F8c9U0nOjOldPsZHHd_FVsmsy_G-jmlRE8Ov9pY&e=)
to make things easier.

Is there documentation somewhere of what fields are mandatory? If not,
could this be added to the API docs?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_IQSS_dataverse_issues_3060&d=CwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=OiTcIL_jSSCUW82Gf-OSkgnZLbG2Yt87eT87BdLiP54&m=4z5nsJFaX-7ZYnMX7G0EH3VkoaKUgzKYMoxeTXS6I_U&s=PTsvfvjoGrD7SFFK6OBnrJi5Zl16wKku2HyqR20tK_A&e=

Eleni Castro
Research Coordinator, Data Curation and Outreach
IQSS, Harvard University
617-496-0703
http://www.iq.harvard.edu/people/eleni-castro
http://orcid.org/0000-0001-9767-8536

Got Data? Check out the Dataverse Project. http://dataverse.org/

[pdurbin]

@leeper I'm glad you opened this issue because @raprasad and I have been discussing this a lot this past week, especially in light of his recent experiments at https://github.com/IQSS/json-schema-test

http://guides.dataverse.org/en/4.3/api/native-api.html says, "Example json can be found at data/dataset-create-new.json" but doesn't provide a link. Since I'm familiar with the code I know that probably what is meant is https://github.com/IQSS/dataverse/blob/v4.3/scripts/api/data/dataset-create-new.json but at minimum we should make this a link so API users don't need to go hunting for this file.

You're right that these JSON files are quite extensive. They were born out of need to serialize datasets in #422 so we could drop a database on a demo site and repopulate it with the JSON files. For this reason, the JSON sort of looks like a backup with lots of internal information about "primitive" and "compound" that API users probably don't care about. I've been saying to @raprasad that I can imagine a (future) world where the JSON used to create datasets is much smaller and focused.

As far as which fields are mandatory, we do a better job of documenting this for the SWORD API, as @posixeleni points out. Look for for "Y" in the "Required" column at http://guides.dataverse.org/en/4.3/api/sword.html#dublin-core-terms-dc-terms-qualified-mapping-dataverse-db-element-crosswalk . Technically, "Subject" is not required because we fill in "N/A" (for backward compatibility) as explained at http://guides.dataverse.org/en/4.3/api/sword.html#new-features-as-of-v1-1

Unfortunately, the native API doesn't document which fields are mandatory. I guess for now you can try making the JSON smaller and smaller and see when it breaks. You could also try creating datasets in the GUI to get a sense of what's required. I agree that we need to document this. Ideally, you'd even be able to query the system and have it tell you which fields are required.

[leeper]

Thanks, @posixeleni and @pdurbin. Yup, doing the same for the native API as is present for the SWORD API would be awesome. I'll start paring down the requests and see what breaks (and post what I find here).

[pdurbin]

@raprasad just opened #3068 which is related: "Compact"/Make complete the native API output

[pdurbin]

@leeper hi, I just posted a long JSON file at #3859 (comment) that contains nearly all the metadata fields available out of the box:

• default_metadata_blocks-4.6.1.json.txt

I say nearly all because @jggautier pointed out that it's missing "Language" and "topic classification" at https://groups.google.com/d/msg/dataverse-community/qsY8swD9Hh0/BVNNyVqaAgAJ

I'm thinking we should clean up the file a bit and add it to the API Guide, saying, "Here's a JSON file that populates the complete set of metadata fields available out of the box." Something like that. Right now, it's hard to figure out what JSON Dataverse wants, leading to issues like #3859.

[markrlondon]

Philip - I just came here, looking for the Native API dataset-create-new.json, which is mentioned on the Native API page, but is lacking a link. I found that you provided the link here. Who should I talk to, in order for that page to be changed, to provide the link? So that other people don't have to search for it also? Thanks! - Mark

https://github.com/IQSS/dataverse/blob/v4.3/scripts/api/data/dataset-create-new.json

[pdurbin]

@markrlondon would you be interested in making a pull request? We document the process here: http://guides.dataverse.org/en/4.7.1/developers/version-control.html#how-to-make-a-pull-request

[pdurbin]

@markrlondon hi! I just made pull request #6107 and tried to explain which field are required to create a dataset. Here's what I wrote:

Is this sufficient to close this issue from your perspective?

@leeper hi! I'd be happy to get your take as well. 😄

[pdurbin]

I'm thinking we should close this issue in favor of this one:

• Query Dataverse for mandatory metadata fields via API #6978