Support partially unzipped archival bags

doc/release-notes/12063-ORE-and-Bag-updates.md

@@ -0,0 +1,15 @@
+This release contains multiple updates to the OAI-ORE metadata export and archival Bag output:
+
+OAI-ORE
+- now uses URI for checksum algorithms
+- a bug causing failures with deaccessioned versions when the deaccession note ("Deaccession Reason" in the UI) was null (which has been allowed via the API). 
+- the "https://schema.org/additionalType" is updated to "Dataverse OREMap Format v1.0.2" to indicate that the out has changed 
+
+Archival Bag 
+- for dataset versions with no files, the (empty) manifest-<alg>.txt file created will now use the default algorithm defined by the "FileFixityChecksumAlgorithm" setting rather than always defaulting to "md5"
+- a bug causing the bag-info.txt to not have information on contacts when the dataset version has more than one contact has been fixed
+- values used in the bag-info.txt file that may be multi-line (with embedded CR or LF characters) are now properly indented/formatted per the BagIt specification (i.e. Internal-Sender-Identifier, External-Description, Source-Organization, Organization-Address).
+- the name of the dataset is no longer used as a subdirectory under the data directory (dataset names can be long enough to cause failures when unzipping)
+- a new key, "Dataverse-Bag-Version" has been added to bag-info.txt with a value "1.0", allowing tracking of changes to Dataverse's arhival bag generation
+- improvements to file retrieval w.r.t. retries on errors or throttling
+- retrieval of files for inclusion in the bag is no longer counted as a download by Dataverse

doc/release-notes/12144-un-holey-bags.md

@@ -0,0 +1,21 @@
+This release contains multiple updates to the OAI-ORE metadata export and archival Bag output:
+
+OAI-ORE
+- now uses URI for checksum algorithms
+- a bug causing failures with deaccessioned versions when the deaccession note ("Deaccession Reason" in the UI) was null (which has been allowed via the API). 
+- the "https://schema.org/additionalType" is updated to "Dataverse OREMap Format v1.0.2" to indicate that the out has changed 
+
+Archival Bag 
+- for dataset versions with no files, the (empty) manifest-<alg>.txt file created will now use the default algorithm defined by the "FileFixityChecksumAlgorithm" setting rather than always defaulting to "md5"
+- a bug causing the bag-info.txt to not have information on contacts when the dataset version has more than one contact has been fixed
+- values used in the bag-info.txt file that may be multi-line (with embedded CR or LF characters) are now properly indented/formatted per the BagIt specification (i.e. Internal-Sender-Identifier, External-Description, Source-Organization, Organization-Address).
+- the name of the dataset is no longer used as a subdirectory under the data directory (dataset names can be long enough to cause failures when unzipping)
+- a new key, "Dataverse-Bag-Version" has been added to bag-info.txt with a value "1.0", allowing tracking of changes to Dataverse's arhival bag generation
+- improvements to file retrieval w.r.t. retries on errors or throttling
+- retrieval of files for inclusion in the bag is no longer counted as a download by Dataverse
+- the size of data files and total dataset size that will be included in an archival bag can now be limited. Admins can choose whether files above these limits are transferred along with the zipped bag (creating a complete archival copy) or are just referenced (using the concept of a "holey" bag and just listing the oversized files and the Dataverse urls from which they can be retrieved. In the holey bag case, an active service on the archiving platform must retrieve the oversized files (using appropriate credentials as needed) to make a complete copy
+
+### New JVM Options (MicroProfile Config Settings)
+dataverse.bagit.zip.holey
+dataverse.bagit.zip.max-data-size
+dataverse.bagit.zip.max-file-size 

doc/sphinx-guides/source/admin/big-data-administration.rst

@@ -302,6 +302,7 @@ There are a broad range of options (that are not turned on by default) for impro
 - :ref:`:DisableSolrFacetsWithoutJsession` - disables facets for users who have disabled cookies (e.g. for bots)
 - :ref:`:DisableUncheckedTypesFacet` - only disables the facet showing the number of collections, datasets, files matching the query (this facet is potentially less useful than others)
 - :ref:`:StoreIngestedTabularFilesWithVarHeaders` - by default, Dataverse stores ingested files without headers and dynamically adds them back at download time. Once this setting is enabled, Dataverse will leave the headers in place (for newly ingested files), reducing the cost of downloads
+- :ref:`dataverse.bagit.zip.max-file-size`, :ref:`dataverse.bagit.zip.max-data-size`, and :ref:`dataverse.bagit.zip.holey` - options to control the size and temporary storage requirements when generating archival Bags - see :ref:`BagIt Export`
 
 
 Scaling Infrastructure

doc/sphinx-guides/source/installation/config.rst

@@ -2259,6 +2259,8 @@ These archival Bags include all of the files and metadata in a given dataset ver
 
 The Dataverse Software offers an internal archive workflow which may be configured as a PostPublication workflow via an admin API call to manually submit previously published Datasets and prior versions to a configured archive such as Chronopolis. The workflow creates a `JSON-LD <http://www.openarchives.org/ore/0.9/jsonld>`_ serialized `OAI-ORE <https://www.openarchives.org/ore/>`_ map file, which is also available as a metadata export format in the Dataverse Software web interface.
 
+The size of the zipped archival Bag can be limited, and files that don't fit within that limit can either be transferred separately (placed so that they are correctly positioned according to the BagIt specification when the zipped bag in unzipped in place) or just referenced for later download (using the BagIt concept of a 'holey' bag with a list of files in a ``fetch.txt`` file) can now be configured for all archivers. These settings allow for managing large datasets by excluding files over a certain size or total data size, which can be useful for archivers with size limitations or to reduce transfer times. See the :ref:`dataverse.bagit.zip.max-file-size`, :ref:`dataverse.bagit.zip.max-data-size`, and :ref:`dataverse.bagit.zip.holey` JVM options for more details.  
+
 At present, archiving classes include the DuraCloudSubmitToArchiveCommand, LocalSubmitToArchiveCommand, GoogleCloudSubmitToArchive, and S3SubmitToArchiveCommand , which all extend the AbstractSubmitToArchiveCommand and use the configurable mechanisms discussed below. (A DRSSubmitToArchiveCommand, which works with Harvard's DRS also exists and, while specific to DRS, is a useful example of how Archivers can support single-version-only semantics and support archiving only from specified collections (with collection specific parameters)). 
 
 All current options support the :ref:`Archival Status API` calls and the same status is available in the dataset page version table (for contributors/those who could view the unpublished dataset, with more detail available to superusers).
@@ -3868,6 +3870,21 @@ This can instead be restricted to only superusers who can publish the dataset us
 
 Example: ``dataverse.coar-notify.relationship-announcement.notify-superusers-only=true``
 
+.. _dataverse.bagit.zip.holey:
+
+``dataverse.bagit.zip.holey``
+  A boolean that, if true, will cause the BagIt archiver to create a "holey" bag. In a holey bag, files that are not included in the bag are listed in the ``fetch.txt`` file with a URL from which they can be downloaded. This is used in conjunction with ``dataverse.bagit.zip.max-file-size`` and/or ``dataverse.bagit.zip.max-data-size``. Default: false.
+
+.. _dataverse.bagit.zip.max-data-size:
+
+``dataverse.bagit.zip.max-data-size``
+  The maximum total (uncompressed) size of data files (in bytes) to include in a BagIt zip archive. If the total size of the dataset files exceeds this limit, files will be excluded from the zipped bag (starting from the largest) until the total size is under the limit. Excluded files will be handled as defined by ``dataverse.bagit.zip.holey`` - just listed if that setting is true or being transferred separately and placed next to the zipped bag. When not set, there is no limit.
+
+.. _dataverse.bagit.zip.max-file-size:
+
+``dataverse.bagit.zip.max-file-size``
+  The maximum (uncompressed) size of a single file (in bytes) to include in a BagIt zip archive. Any file larger than this will be excluded. Excluded files will be handled as defined by ``dataverse.bagit.zip.holey`` - just listed if that setting is true or being transferred separately and placed next to the zipped bag. When not set, there is no limit.
+
 .. _feature-flags:
 
 Feature Flags

src/main/java/edu/harvard/iq/dataverse/DataFile.java

@@ -109,18 +109,22 @@ public class DataFile extends DvObject implements Comparable {
      * The list of types should be limited to the list above in the technote
      * because the string gets passed into MessageDigest.getInstance() and you
      * can't just pass in any old string.
+     * 
+     * The URIs are used in the OAI_ORE export. They are taken from the associated XML Digital Signature standards.
      */
     public enum ChecksumType {
 
-        MD5("MD5"),
-        SHA1("SHA-1"),
-        SHA256("SHA-256"),
-        SHA512("SHA-512");
+        MD5("MD5", "http://www.w3.org/2001/04/xmldsig-more#md5"),
+        SHA1("SHA-1", "http://www.w3.org/2000/09/xmldsig#sha1"),
+        SHA256("SHA-256", "http://www.w3.org/2001/04/xmlenc#sha256"),
+        SHA512("SHA-512", "http://www.w3.org/2001/04/xmlenc#sha512");
 
         private final String text;
+        private final String uri;
 
-        private ChecksumType(final String text) {
+        private ChecksumType(final String text, final String uri) {
             this.text = text;
+            this.uri = uri;
         }
 
         public static ChecksumType fromString(String text) {
@@ -131,13 +135,30 @@ public static ChecksumType fromString(String text) {
                     }
                 }
             }
-            throw new IllegalArgumentException("ChecksumType must be one of these values: " + Arrays.asList(ChecksumType.values()) + ".");
+            throw new IllegalArgumentException(
+                    "ChecksumType must be one of these values: " + Arrays.asList(ChecksumType.values()) + ".");
+        }
+
+        public static ChecksumType fromUri(String uri) {
+            if (uri != null) {
+                for (ChecksumType checksumType : ChecksumType.values()) {
+                    if (uri.equals(checksumType.uri)) {
+                        return checksumType;
+                    }
+                }
+            }
+            throw new IllegalArgumentException(
+                    "ChecksumType must be one of these values: " + Arrays.asList(ChecksumType.values()) + ".");
         }
 
         @Override
         public String toString() {
             return text;
         }
+
+        public String toUri() {
+            return uri;
+        }
     }
 
     //@Expose

src/main/java/edu/harvard/iq/dataverse/engine/command/impl/DuraCloudSubmitToArchiveCommand.java

@@ -7,16 +7,24 @@
 import edu.harvard.iq.dataverse.authorization.users.ApiToken;
 import edu.harvard.iq.dataverse.engine.command.DataverseRequest;
 import edu.harvard.iq.dataverse.engine.command.RequiredPermissions;
+import edu.harvard.iq.dataverse.util.bagit.BagGenerator;
+import edu.harvard.iq.dataverse.util.bagit.OREMap;
+
 import static edu.harvard.iq.dataverse.settings.SettingsServiceBean.Key.DuraCloudContext;
 import static edu.harvard.iq.dataverse.settings.SettingsServiceBean.Key.DuraCloudHost;
 import static edu.harvard.iq.dataverse.settings.SettingsServiceBean.Key.DuraCloudPort;
 import edu.harvard.iq.dataverse.workflow.step.Failure;
 import edu.harvard.iq.dataverse.workflow.step.WorkflowStepResult;
 
+import java.io.File;
+import java.io.FileOutputStream;
 import java.io.IOException;
+import java.io.InputStream;
 import java.io.PipedInputStream;
 import java.io.PipedOutputStream;
 import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
 import java.security.DigestInputStream;
 import java.security.MessageDigest;
 import java.security.NoSuchAlgorithmException;
@@ -96,6 +104,8 @@ public WorkflowStepResult performArchiveSubmission(DatasetVersion dv, ApiToken t
                 statusObject.add(DatasetVersion.ARCHIVAL_STATUS, DatasetVersion.ARCHIVAL_STATUS_FAILURE);
                 statusObject.add(DatasetVersion.ARCHIVAL_STATUS_MESSAGE, "Bag not transferred");
 
+                Path tempBagFile = null;
+
                 try {
                     /*
                      * If there is a failure in creating a space, it is likely that a prior version
@@ -161,20 +171,38 @@ public void run() {
                         // Add BagIt ZIP file
                         // Although DuraCloud uses SHA-256 internally, it's API uses MD5 to verify the
                         // transfer
+                        Path bagFile = null;
+
 
-                        messageDigest = MessageDigest.getInstance("MD5");
-                        try (PipedInputStream in = new PipedInputStream(100000);
-                                DigestInputStream digestInputStream2 = new DigestInputStream(in, messageDigest)) {
-                            Thread bagThread = startBagThread(dv, in, digestInputStream2, dataciteXml, token);
-                            checksum = store.addContent(spaceName, fileName, digestInputStream2, -1l, null, null, null);
-                            bagThread.join();
-                            if (success) {
-                                logger.fine("Content: " + fileName + " added with checksum: " + checksum);
-                                localchecksum = Hex.encodeHexString(digestInputStream2.getMessageDigest().digest());
+                        tempBagFile = Files.createTempFile("dataverse-bag-", ".zip");
+                        logger.fine("Creating bag in temporary file: " + tempBagFile.toString());
+
+                        BagGenerator bagger = new BagGenerator(new OREMap(dv, false), dataciteXml);
+                        bagger.setAuthenticationKey(token.getTokenString());
+                        // Generate bag to temporary file using the provided ore JsonObject
+                        try (FileOutputStream fos = new FileOutputStream(tempBagFile.toFile())) {
+                            if (!bagger.generateBag(fos)) {
+                                throw new IOException("Bag generation failed");
                             }
-                            if (!success || !checksum.equals(localchecksum)) {
+                        }
+
+                        // Store BagIt file
+                        long bagSize = Files.size(tempBagFile);
+                        logger.fine("Bag created successfully, size: " + bagSize + " bytes");
+
+                            // Now upload the bag file
+                           messageDigest = MessageDigest.getInstance("MD5");
+                            try (InputStream is = Files.newInputStream(bagFile);
+                                    DigestInputStream bagDigestInputStream = new DigestInputStream(is, messageDigest)) {
+                                checksum = store.addContent(spaceName, fileName, bagDigestInputStream, bagFile.toFile().length(), "application/zip", null, null);
+                                localchecksum = Hex.encodeHexString(bagDigestInputStream.getMessageDigest().digest());
+
+                            if (checksum != null && checksum.equals(localchecksum)) {
+                                logger.fine("Content: " + fileName + " added with checksum: " + checksum);
+                                success = true;
+                            } else {
                                 logger.severe("Failure on " + fileName);
-                                logger.severe(success ? checksum + " not equal to " + localchecksum : "failed to transfer to DuraCloud");
+                                logger.severe(checksum + " not equal to " + localchecksum);
                                 try {
                                     store.deleteContent(spaceName, fileName);
                                     store.deleteContent(spaceName, baseFileName + "_datacite.xml");
@@ -185,9 +213,6 @@ public void run() {
                                         "DuraCloud Submission Failure: incomplete archive transfer");
                             }
                         }
-
-                        logger.fine("DuraCloud Submission step: Content Transferred");
-
                         // Document the location of dataset archival copy location (actually the URL
                         // where you can
                         // view it as an admin)
@@ -223,8 +248,20 @@ public void run() {
                     return new Failure("Unable to create DuraCloud space with name: " + baseFileName, mesg);
                 } catch (NoSuchAlgorithmException e) {
                     logger.severe("MD5 MessageDigest not available!");
+                } catch (Exception e) {
+                    logger.warning(e.getLocalizedMessage());
+                    e.printStackTrace();
+                    return new Failure("Error in transferring file to DuraCloud",
+                            "DuraCloud Submission Failure: internal error");
                 }
                 finally {
+                    if (tempBagFile != null) {
+                        try {
+                            Files.deleteIfExists(tempBagFile);
+                        } catch (IOException e) {
+                            logger.warning("Failed to delete temporary bag file: " + tempBagFile + " : " + e.getMessage());
+                        }
+                    }
                     dv.setArchivalCopyLocation(statusObject.build().toString());
                 }
             } else {