Merge pull request #6107 from IQSS/6086-api-guide

add beginner friendly API documentation #6086

doc/sphinx-guides/source/_static/api/define-metadatablocks.json

@@ -0,0 +1,8 @@
+[
+  "citation",
+  "geospatial",
+  "socialscience",
+  "astrophysics",
+  "biomedical",
+  "journal"
+]

doc/sphinx-guides/source/_static/docsdataverse_org.css

@@ -114,4 +114,8 @@ div.form-group .glyphicon.glyphicon-asterisk {font-size: .5em; vertical-align: t
 
 /* #sidebar.bs-sidenav.affix {
 	position: static;
-} REMOVED STATIC ToC */
+} REMOVED STATIC ToC */
+
+pre {
+    white-space: pre-wrap;
+}

doc/sphinx-guides/source/admin/troubleshooting.rst

@@ -3,11 +3,35 @@
 Troubleshooting
 ===============
 
-This new (as of v.4.6) section of the Admin guide is for tips on how to diagnose and fix system problems. 
+Sometimes Dataverse users get into trouble. Sometimes Dataverse itself gets into trouble. If something has gone wrong, this section is for you.
 
 .. contents:: Contents:
 	:local:
 
+Using Dataverse APIs to Troubleshoot and Fix Problems
+-----------------------------------------------------
+
+See the :doc:`/api/intro` section of the API Guide for a high level overview of Dataverse APIs. Below are listed problems that support teams might encounter that can be handled via API (sometimes only via API).
+
+A Dataset Is Locked And Cannot Be Edited or Published
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It's normal for the ingest process described in the :doc:`/user/tabulardataingest/ingestprocess` section of the User Guide to take some time but if hours or days have passed and the dataset is still locked, you might want to inspect the locks and consider deleting some or all of them.
+
+See :doc:`dataverses-datasets`.
+
+Someone Created Spam Datasets and I Need to Delete Them
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Depending on how open your installation of Dataverse is to the general public creating datasets, you may sometimes need to deal with spam datasets.
+
+Look for "destroy" in the :doc:`/api/native-api` section of the API Guide.
+
+A User Needs Their Account to Be Converted From Institutional (Shibboleth), ORCID, Google, or GitHub to Something Else
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+See :ref:`converting-shibboleth-users-to-local` and :ref:`converting-oauth-users-to-local`.
+
 Glassfish
 ---------
 

doc/sphinx-guides/source/api/apps.rst

@@ -1,40 +1,69 @@
 Apps
 ====
 
-The introduction of Dataverse APIs has fostered the development of apps that are listed at http://dataverse.org/integrations and the :doc:`/admin/integrations` section of the Admin Guide.
+The introduction of Dataverse APIs has fostered the development of a variety of software applications that are listed in the :doc:`/admin/integrations` and :doc:`/admin/reporting-tools` sections of the Admin Guide and the :doc:`/installation/external-tools` section of the Installation Guide.
 
-The apps below are open source, demonstrating how to use Dataverse APIs. Some of these apps (and others) are built on :doc:`/api/client-libraries` that are available for Dataverse APIs.
+The apps below are open source and demonstrate how to use Dataverse APIs. Some of these apps are built on :doc:`/api/client-libraries` that are available for Dataverse APIs in Python, R, and Java.
 
 .. contents:: |toctitle|
 	:local:
 
 Javascript
 ----------
 
+Data Explorer
+~~~~~~~~~~~~~
+
+Data Explorer is a GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis.
+
+https://github.com/scholarsportal/Dataverse-Data-Explorer
+
+Data Curation Tool
+~~~~~~~~~~~~~~~~~~
+
+Data Curation Tool is  a GUI for curating data by adding labels, groups, weights and other details to assist with informed reuse.
+
+https://github.com/scholarsportal/Dataverse-Data-Curation-Tool
+
+File Previewers
+~~~~~~~~~~~~~~~
+
+File Previewers are tools that display the content of files - including audio, html, Hypothes.is annotations, images, PDF, text, video - allowing them to be viewed without downloading.
+
+https://github.com/QualitativeDataRepository/dataverse-previewers
+
 TwoRavens
 ~~~~~~~~~
 
 TwoRavens is a system of interlocking statistical tools for data exploration, analysis, and meta-analysis.
 
 https://github.com/IQSS/TwoRavens
 
-PHP
----
+Python
+------
 
-OJS
-~~~
+Please note that there are multiple Python modules for Dataverse APIs listed in the :doc:`client-libraries` section.
 
-The Open Journal Systems (OJS) Dataverse Plugin adds data sharing and preservation to the OJS publication process.
+dataverse-sample-data
+~~~~~~~~~~~~~~~~~~~~~
 
-https://github.com/pkp/ojs/tree/ojs-stable-2_4_8/plugins/generic/dataverse
+dataverse-sample-data allows you to populate your Dataverse installation with sample data. It makes uses of pyDataverse, which is listed in the :doc:`client-libraries` section.
 
-Python
-------
+https://github.com/IQSS/dataverse-sample-data
+
+Texas Digital Library dataverse-reports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Dataverse Reports for Texas Digital Library generates and emails statistical reports for an installation of Dataverse using the native API and database queries.
+
+https://github.com/TexasDigitalLibrary/dataverse-reports
 
 OSF
 ~~~
 
-Allows you to view, download, and upload files to and from a Dataverse dataset from an Open Science Framework (OSF) project: https://github.com/CenterForOpenScience/osf.io/tree/develop/addons/dataverse
+OSF allows you to view, download, and upload files to and from a Dataverse dataset from an Open Science Framework (OSF) project.
+
+https://github.com/CenterForOpenScience/osf.io/tree/develop/addons/dataverse
 
 GeoConnect
 ~~~~~~~~~~
@@ -46,22 +75,49 @@ https://github.com/IQSS/geoconnect
 dataverse-metrics
 ~~~~~~~~~~~~~~~~~
 
-dataverse-metrics aggregates and visualizes metrics across multiple Dataverse installations but can also be used with a single installation: https://github.com/IQSS/dataverse-metrics
+dataverse-metrics aggregates and visualizes metrics across multiple Dataverse installations but can also be used with a single installation
+
+https://github.com/IQSS/dataverse-metrics
+
+Whole Tale
+~~~~~~~~~~
+
+Whole Tale enables researchers to analyze data using popular tools including Jupyter and RStudio with the ultimate goal of supporting publishing of reproducible research packages.
+
+https://github.com/whole-tale/girder_wholetale/tree/v0.7/server/lib/dataverse
+
+Archivematica
+~~~~~~~~~~~~~
+
+Archivematica is an integrated suite of open-source tools for processing digital objects for long-term preservation.
+
+https://github.com/artefactual/archivematica/tree/v1.9.2/src/MCPClient/lib/clientScripts
 
 Java
 ----
 
+Please note that there is a Java library for Dataverse APIs listed in the :doc:`client-libraries` section.
+
 DVUploader
-~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~
 
 The open-source DVUploader tool is a stand-alone command-line Java application that uses the Dataverse API to upload files to a specified Dataset. Files can be specified by name, or the DVUploader can upload all files in a directory or recursively from a directory tree. The DVUploader can also verify that uploaded files match their local sources by comparing the local and remote fixity checksums. Source code, release 1.0.0- jar file, and documentation are available on GitHub. DVUploader's creation was supported by the Texas Digital Library.
 
 https://github.com/IQSS/dataverse-uploader
 
-
 Dataverse for Android
 ~~~~~~~~~~~~~~~~~~~~~
 
-For now this is only a proof of concept.
+Dataverse for Android makes use of Dataverse's Search API.
 
 https://github.com/IQSS/dataverse-android
+
+PHP
+---
+
+OJS
+~~~
+
+The Open Journal Systems (OJS) Dataverse Plugin adds data sharing and preservation to the OJS publication process.
+
+https://github.com/pkp/ojs/tree/ojs-stable-2_4_8/plugins/generic/dataverse

doc/sphinx-guides/source/api/auth.rst

@@ -0,0 +1,63 @@
+API Tokens and Authentication 
+=============================
+
+An API token is similar to a password and allows you to authenticate to Dataverse APIs to perform actions as you. Many Dataverse APIs require the use of an API token.
+
+.. contents:: |toctitle|
+    :local:
+
+How to Get an API Token
+-----------------------
+
+Your API token is unique to the server you are using. You cannot take your API token from one server and use it on another server.
+
+Instructions for getting a token are described in the :doc:`/user/account` section of the User Guide.
+
+How Your API Token Is Like a Password
+-------------------------------------
+
+Anyone who has your API Token can add and delete data as you so you should treat it with the same care as a password.
+
+Passing Your API Token as an HTTP Header (Preferred) or a Query Parameter
+-------------------------------------------------------------------------
+
+See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of ``export`` below.
+
+There are two ways to pass your API token to Dataverse APIs. The preferred method is to send the token in the ``X-Dataverse-key`` HTTP header, as in the following curl example.
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export ALIAS=root
+
+  curl -H X-Dataverse-key:$API_TOKEN $SERVER_URL/api/dataverses/$ALIAS/contents
+
+Here's how it looks without the environment variables:
+
+.. code-block:: bash
+
+  curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx https://demo.dataverse.org/api/dataverses/root/contents
+
+The second way to pass your API token is via a query parameter called ``key`` in the URL like below.
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export ALIAS=root
+
+  curl $SERVER_URL/api/dataverses/$ALIAS/contents?key=$API_TOKEN
+
+Here's how it looks without the environment variables:
+
+.. code-block:: bash
+
+  curl https://demo.dataverse.org/api/dataverses/root/contents?key=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+
+Use of the ``X-Dataverse-key`` HTTP header form is preferred to passing ``key`` in the URL because query parameters like ``key`` appear in URLs and might accidentally get shared, exposing your API token. (Again it's like a password.) Additionally, URLs are often logged on servers while it's less common to log HTTP headers.
+
+Resetting Your API Token
+------------------------
+
+You can reset your API Token from your account page in Dataverse as described in the :doc:`/user/account` section of the User Guide.

doc/sphinx-guides/source/api/faq.rst

@@ -0,0 +1,95 @@
+Frequently Asked Questions
+==========================
+
+APIs are less intuitive than graphical user interfaces (GUIs) so questions are expected!
+
+.. contents:: |toctitle|
+    :local:
+
+What is an API?
+---------------
+
+See "What is an API?" in the :doc:`intro` section.
+
+What Are Common Use Cases for Dataverse APIs?
+---------------------------------------------
+
+See the :doc:`getting-started` section for common use cases for researchers and curators. Other types of API users should find starting points at :ref:`types-of-api-users`.
+
+Where Can I Find Examples of Using Dataverse APIs?
+--------------------------------------------------
+
+See the :doc:`getting-started` section links to examples using curl.
+
+For examples in Javascript, Python, R, and Java, and PHP, see the :doc:`apps` and :doc:`client-libraries` sections.
+
+When Should I Use the Native API vs. the SWORD API?
+---------------------------------------------------
+
+The :doc:`sword` is based on a standard, works fine, and is fully supported, but much more development effort has been going into the :doc:`native-api`, which is not based on a standard. It is specific to Dataverse.
+
+SWORD uses XML. The Native API uses JSON.
+
+SWORD only supports a dozen or so operations. The Native API supports many more.
+
+To Operate on a Dataset Should I Use Its DOI (or Handle) or Its Database ID?
+----------------------------------------------------------------------------
+
+It's fine to target a datasets using either its Persistent ID (PID such as DOI or Handle) or its database id.
+
+Here's an example from :ref:`publish-dataset-api` of targeting a dataset using its DOI:
+
+.. code-block:: bash
+
+  curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST "https://demo.dataverse.org/api/datasets/:persistentId/actions/:publish?persistentId=doi:10.5072/FK2/J8SJZB&type=major"
+
+You can target the same dataset with its database ID ("42" in the example below), like this:
+
+.. code-block:: bash
+
+  curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST "https://demo.dataverse.org/api/datasets/42/actions/:publish?type=major"
+
+Note that when multiple query parameters are used (such as ``persistentId`` and ``type`` above) there is a question mark (``?``) before the first query parameter and ampersands (``&``) before each of the subsequent query parameters. Also, ``&`` has special meaning in Unix shells such as Bash so you must put quotes around the entire URL.
+
+Where is the Comprehensive List of All API Functionality?
+---------------------------------------------------------
+
+There are so many Dataverse APIs that a single page in this guide would probably be overwhelming. See :ref:`list-of-dataverse-apis` for links to various pages.
+
+It's possible to get a complete list of API functionality in Swagger/OpenAPI format if you deploy Dataverse to Payara 5+. For details, see https://github.com/IQSS/dataverse/issues/5794
+
+Is There a Changelog of API Functionality That Has Been Added Over Time?
+------------------------------------------------------------------------
+
+No, but there probably should be. If you have suggestions for how it should look, please create an issue at https://github.com/IQSS/dataverse/issues
+
+.. _no-api:
+
+What Functionality is GUI Only and Not Available Via API
+-------------------------------------------------------
+
+The following tasks cannot currently be automated via API because no API exists for them. The web interface should be used instead for these GUI-only features:
+
+- Setting a logo image, URL, and tagline when creating a dataverse.
+- Editing properties of an existing dataverse.
+- Set "Enable Access Request" for Terms of Use: https://groups.google.com/d/msg/dataverse-community/oKdesT9rFGc/qM6wrsnnBAAJ
+- Downloading a guestbook.
+- Set guestbook_id for a dataset: https://groups.google.com/d/msg/dataverse-community/oKdesT9rFGc/qM6wrsnnBAAJ
+- Filling out a guestbook. See also https://groups.google.com/d/msg/dataverse-dev/G9FNGP_bT0w/dgE2Fk4iBQAJ
+- Seeing why a file failed ingest.
+- Dataset templates.
+- Deaccessioning datasets.
+
+If you would like APIs for any of the features above, please open a GitHub issue at https://github.com/IQSS/dataverse/issues
+
+You are also welcome to open an issue to add to the list above. Or you are welcome to make a pull request. Please see the :doc:`/developers/documentation` section of the Developer Guide for instructions.
+
+Why Aren't the Return Values (HTTP Status Codes) Documented?
+------------------------------------------------------------
+
+They should be. Please consider making a pull request to help. The :doc:`/developers/documentation` section of the Developer Guide should help you get started. :ref:`create-dataverse-api` has an example you can follow or you can come up with a better way.
+
+What If My Question Isn't Answered Here?
+----------------------------------------
+
+Please ask! For information on where to ask, please see :ref:`getting-help-with-apis`.