Merge branch 'DANS-KNAW:develop' into UseSupportEmailInClosingText

.github/ISSUE_TEMPLATE/bug_report.md

@@ -14,7 +14,7 @@ Thank you for contributing to the Dataverse Project through the creation of a bu
 WARNING: If this is a security issue it should be reported privately to security@dataverse.org
 
 More information on bug issues and contributions can be found in the "Contributing to Dataverse" page:
-https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md#bug-reportsissues
+https://guides.dataverse.org/en/latest/contributor/index.html
 
 Please fill out as much of the template as you can.
 Start below this comment section.
@@ -44,7 +44,6 @@ Start below this comment section.
 **Any related open or closed issues to this bug report?**
 
 
-
 **Screenshots:**
 
 No matter the issue, screenshots are always welcome.
@@ -53,3 +52,7 @@ To add a screenshot, please use one of the following formats and/or methods desc
 
 * https://help.github.com/en/articles/file-attachments-on-issues-and-pull-requests
 *
+
+
+**Are you thinking about creating a pull request for this issue?**
+Help is always welcome, is this bug something you or your organization plan to fix?

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,7 +1,7 @@
 ---
 name: Feature request
 about: Suggest an idea or new feature for the Dataverse software!
-title: 'Feature Request/Idea:'
+title: 'Feature Request:'
 labels: 'Type: Feature'
 assignees: ''
 
@@ -11,7 +11,7 @@ assignees: ''
 Thank you for contributing to the Dataverse Project through the creation of a feature request!
 
 More information on ideas/feature requests and contributions can be found in the "Contributing to Dataverse" page:
-https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md#ideasfeature-requests
+https://guides.dataverse.org/en/latest/contributor/index.html
 
 Please fill out as much of the template as you can.
 Start below this comment section.
@@ -34,3 +34,6 @@ Start below this comment section.
 
 
 **Any open or closed issues related to this feature request?**
+
+**Are you thinking about creating a pull request for this feature?**  
+Help is always welcome, is this feature something you or your organization plan to implement?

.github/ISSUE_TEMPLATE/idea_proposal.md

@@ -0,0 +1,40 @@
+---
+name: Idea proposal
+about: Propose a new idea for discussion to improve the Dataverse software!
+title: 'Suggestion:'
+labels: 'Type: Suggestion'
+assignees: ''
+
+---
+
+<!--
+Thank you for contributing to the Dataverse Project through the creation of a feature request!
+
+More information on ideas/feature requests and contributions can be found in the "Contributing to Dataverse" page:
+https://guides.dataverse.org/en/latest/contributor/index.html
+
+Please fill out as much of the template as you can.
+Start below this comment section.
+-->
+
+**Overview of the Suggestion**
+
+
+**What kind of user is the suggestion intended for?**
+(Example users roles: API User, Curator, Depositor, Guest, Superuser, Sysadmin)
+
+
+**What inspired this idea?**
+
+
+**What existing behavior do you want changed?**
+
+
+**Any brand new behavior do you want to add to Dataverse?**
+
+
+**Any open or closed issues related to this suggestion?**
+
+
+**Are you thinking about creating a pull request for this issue?**
+Help is always welcome, is this idea something you or your organization plan to implement?

.github/workflows/check_property_files.yml

@@ -0,0 +1,32 @@
+name: "Properties Check"
+on:
+    pull_request:
+        paths:
+            - "src/**/*.properties"
+            - "scripts/api/data/metadatablocks/*"
+jobs:
+    duplicate_keys:
+        name: Duplicate Keys
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - name: Run duplicates detection script
+              shell: bash
+              run: tests/check_duplicate_properties.sh
+
+    metadata_blocks_properties:
+        name: Metadata Blocks Properties
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - name: Setup GraalVM + Native Image
+              uses: graalvm/setup-graalvm@v1
+              with:
+                github-token: ${{ secrets.GITHUB_TOKEN }}
+                java-version: '21'
+                distribution: 'graalvm-community'
+            - name: Setup JBang
+              uses: jbangdev/setup-jbang@main
+            - name: Run metadata block properties verification script
+              shell: bash
+              run: tests/verify_mdb_properties.sh

.github/workflows/shellspec.yml

@@ -24,28 +24,11 @@ jobs:
               run: |
                   cd tests/shell
                   shellspec
-    shellspec-centos7:
-        name: "CentOS 7"
+    shellspec-rocky9:
+        name: "RockyLinux 9"
         runs-on: ubuntu-latest
         container:
-            image: centos:7
-        steps:
-            - uses: actions/checkout@v2
-            - name: Install shellspec
-              run: |
-                  curl -fsSL https://github.com/shellspec/shellspec/releases/download/${{ env.SHELLSPEC_VERSION }}/shellspec-dist.tar.gz | tar -xz -C /usr/share
-                  ln -s /usr/share/shellspec/shellspec /usr/bin/shellspec
-            - name: Install dependencies
-              run: yum install -y ed
-            - name: Run shellspec
-              run: |
-                  cd tests/shell
-                  shellspec
-    shellspec-rocky8:
-        name: "RockyLinux 8"
-        runs-on: ubuntu-latest
-        container:
-            image: rockylinux/rockylinux:8
+            image: rockylinux/rockylinux:9
         steps:
             - uses: actions/checkout@v2
             - name: Install shellspec

.gitignore

@@ -34,6 +34,7 @@ oauth-credentials.md
 /src/main/webapp/oauth2/newAccount.html
 scripts/api/setup-all.sh*
 scripts/api/setup-all.*.log
+src/main/resources/edu/harvard/iq/dataverse/openapi/
 
 # ctags generated tag file
 tags

CONTRIBUTING.md

@@ -1,67 +1,7 @@
 # Contributing to Dataverse
 
-Thank you for your interest in contributing to Dataverse!  We are open to contributions from everyone. You don't need permission to participate. Just jump in. If you have questions, please reach out using one or more of the channels described below.
+Thank you for your interest in contributing to Dataverse! We are open to contributions from everyone.
 
-We aren't just looking for developers. There are many ways to contribute to Dataverse.  We welcome contributions of ideas, bug reports, usability research/feedback, documentation, code, and more!
+Please see our [Contributor Guide][] for how you can help!
 
-## Ideas/Feature Requests
-
-Your idea or feature request might already be captured in the Dataverse [issue tracker] on GitHub but if not, the best way to bring it to the community's attention is by posting on the [dataverse-community Google Group][] or bringing it up on a [Community Call][]. You're also welcome to make some noise in [chat.dataverse.org][] or cram your idea into 280 characters and mention [@dataverseorg][] on Twitter. To discuss your idea privately, please email it to support@dataverse.org
-
-There's a chance your idea is already on our roadmap, which is available at https://www.iq.harvard.edu/roadmap-dataverse-project
-
-[chat.dataverse.org]: http://chat.dataverse.org
-[issue tracker]: https://github.com/IQSS/dataverse/issues
-[@dataverseorg]: https://twitter.com/dataverseorg
-
-## Usability testing
-
-Please email us at support@dataverse.org if you are interested in participating in usability testing.
-
-## Bug Reports/Issues
-
-An issue is a bug (a feature is no longer behaving the way it should) or a feature (something new to Dataverse that helps users complete tasks). You can browse the Dataverse [issue tracker] on GitHub by open or closed issues or by milestones.
-
-Before submitting an issue, please search the existing issues by using the search bar at the top of the page. If there is an existing open issue that matches the issue you want to report, please add a comment to it.
-
-If there is no pre-existing issue or it has been closed, please click on the "New Issue" button, log in, and write in what the issue is (unless it is a security issue which should be reported privately to security@dataverse.org).
-
-If you do not receive a reply to your new issue or comment in a timely manner, please email support@dataverse.org with a link to the issue.
-
-### Writing an Issue
-
-For the subject of an issue, please start it by writing the feature or functionality it relates to, i.e. "Create Account:..." or "Dataset Page:...". In the body of the issue, please outline the issue you are reporting with as much detail as possible. In order for the Dataverse development team to best respond to the issue, we need as much information about the issue as you can provide. Include steps to reproduce bugs. Indicate which version you're using, which is shown at the bottom of the page. We love screenshots!
-
-### Issue Attachments
-
-You can attach certain files (images, screenshots, logs, etc.) by dragging and dropping, selecting them, or pasting from the clipboard. Files must be one of GitHub's [supported attachment formats] such as png, gif, jpg, txt, pdf, zip, etc. (Pro tip: A file ending in .log can be renamed to .txt so you can upload it.) If there's no easy way to attach your file, please include a URL that points to the file in question.
-
-[supported attachment formats]: https://help.github.com/articles/file-attachments-on-issues-and-pull-requests/
-
-## Documentation
-
-The source for the documentation at http://guides.dataverse.org/en/latest/ is in the GitHub repo under the "[doc][]" folder. If you find a typo or inaccuracy or something to clarify, please send us a pull request! For more on the tools used to write docs, please see the [documentation][] section of the Developer Guide.
-
-[doc]: https://github.com/IQSS/dataverse/tree/develop/doc/sphinx-guides/source
-[documentation]: http://guides.dataverse.org/en/latest/developers/documentation.html
-
-## Code/Pull Requests
-
-We love code contributions. Developers are not limited to the main Dataverse code in this git repo. You can help with API client libraries in your favorite language that are mentioned in the [API Guide][] or create a new library. You can help work on configuration management code that's mentioned in the [Installation Guide][]. The Installation Guide also covers a relatively new concept called "external tools" that allows developers to create their own tools that are available from within an installation of Dataverse.
-
-[API Guide]: http://guides.dataverse.org/en/latest/api
-[Installation Guide]: http://guides.dataverse.org/en/latest/installation
-
-If you are interested in working on the main Dataverse code, great! Before you start coding, please reach out to us either on the [dataverse-community Google Group][], the [dataverse-dev Google Group][], [chat.dataverse.org][], or via support@dataverse.org to make sure the effort is well coordinated and we avoid merge conflicts. We maintain a list of [community contributors][] and [dev efforts][] the community is working on so please let us know if you'd like to be added or removed from either list.
-
-Please read http://guides.dataverse.org/en/latest/developers/version-control.html to understand how we use the "git flow" model of development and how we will encourage you to create a GitHub issue (if it doesn't exist already) to associate with your pull request. That page also includes tips on making a pull request.
-
-After making your pull request, your goal should be to help it advance through our kanban board at https://github.com/orgs/IQSS/projects/34 . If no one has moved your pull request to the code review column in a timely manner, please reach out. Note that once a pull request is created for an issue, we'll remove the issue from the board so that we only track one card (the pull request).
-
-Thanks for your contribution!
-
-[dataverse-community Google Group]: https://groups.google.com/group/dataverse-community
-[Community Call]: https://dataverse.org/community-calls
-[dataverse-dev Google Group]: https://groups.google.com/group/dataverse-dev
-[community contributors]: https://docs.google.com/spreadsheets/d/1o9DD-MQ0WkrYaEFTD5rF_NtyL8aUISgURsAXSL7Budk/edit?usp=sharing
-[dev efforts]: https://github.com/orgs/IQSS/projects/34/views/6
+[Contributor Guide]: https://guides.dataverse.org/en/latest/contributor/index.html

README.md

@@ -7,7 +7,7 @@ Dataverse is an [open source][] software platform for sharing, finding, citing,
 
 We maintain a demo site at [demo.dataverse.org][] which you are welcome to use for testing and evaluating Dataverse.
 
-To install Dataverse, please see our [Installation Guide][] which will prompt you to download our [latest release][].
+To install Dataverse, please see our [Installation Guide][] which will prompt you to download our [latest release][]. Docker users should consult the [Container Guide][].
 
 To discuss Dataverse with the community, please join our [mailing list][], participate in a [community call][], chat with us at [chat.dataverse.org][], or attend our annual [Dataverse Community Meeting][].
 
@@ -28,6 +28,7 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
 [Dataverse community]: https://dataverse.org/developers
 [Installation Guide]: https://guides.dataverse.org/en/latest/installation/index.html
 [latest release]: https://github.com/IQSS/dataverse/releases
+[Container Guide]: https://guides.dataverse.org/en/latest/container/index.html
 [features]: https://dataverse.org/software-features
 [project board]: https://github.com/orgs/IQSS/projects/34
 [roadmap]: https://www.iq.harvard.edu/roadmap-dataverse-project

conf/solr/9.3.0/schema.xml → conf/solr/schema.xml

@@ -142,6 +142,7 @@
 
     <field name="dvName" type="text_en" stored="true" indexed="true" multiValued="false"/>
     <field name="dvAlias" type="text_en" stored="true" indexed="true" multiValued="false"/>
+    <field name="dvParentAlias" type="text_en" stored="true" indexed="true" multiValued="false"/>
     <field name="dvAffiliation" type="text_en" stored="true" indexed="true" multiValued="false"/>
     <field name="dvDescription" type="text_en" stored="true" indexed="true" multiValued="false"/>
 
@@ -157,7 +158,8 @@
     <field name="publicationStatus" type="string" stored="true" indexed="true" multiValued="true"/>
     <field name="externalStatus" type="string" stored="true" indexed="true" multiValued="false"/>
     <field name="embargoEndDate" type="plong" stored="true" indexed="true" multiValued="false"/>
-
+    <field name="retentionEndDate" type="plong" stored="true" indexed="true" multiValued="false"/>
+
     <field name="subtreePaths" type="string" stored="true" indexed="true" multiValued="true"/>
 
     <field name="fileName" type="text_en" stored="true" indexed="true" multiValued="true"/>
@@ -204,6 +206,7 @@
     <field name="entityId" type="plong" stored="true" indexed="true" multiValued="false"/>
 
     <field name="datasetVersionId" type="plong" stored="true" indexed="true" multiValued="false"/>
+    <field name="datasetType" type="string" stored="true" indexed="true" multiValued="false"/>
 
     <!-- Added for Dataverse 4.0 alpha 1 to sort by name  -->
     <!-- https://redmine.hmdc.harvard.edu/issues/3482 -->
@@ -325,6 +328,7 @@
     <field name="journalVolumeIssue" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="keyword" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="keywordValue" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="keywordTermURI" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="keywordVocabulary" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="keywordVocabularyURI" type="text_en" multiValued="true" stored="true" indexed="true"/>
     <field name="kindOfData" type="text_en" multiValued="true" stored="true" indexed="true"/>
@@ -424,6 +428,7 @@
     <copyField source="dvAlias" dest="_text_" maxChars="3000"/>
     <copyField source="dvAffiliation" dest="_text_" maxChars="3000"/>
     <copyField source="dsPersistentId" dest="_text_" maxChars="3000"/>
+    <copyField source="datasetType" dest="_text_" maxChars="3000"/>
     <!-- copyField commands copy one field to another at the time a document
         is added to the index.  It's used either to index the same field differently,
         or to add multiple fields to the same field for easier/faster searching.  -->
@@ -564,6 +569,7 @@
     <copyField source="journalVolumeIssue" dest="_text_" maxChars="3000"/>
     <copyField source="keyword" dest="_text_" maxChars="3000"/>
     <copyField source="keywordValue" dest="_text_" maxChars="3000"/>
+    <copyField source="keywordTermURI" dest="_text_" maxChars="3000"/>
     <copyField source="keywordVocabulary" dest="_text_" maxChars="3000"/>
     <copyField source="keywordVocabularyURI" dest="_text_" maxChars="3000"/>
     <copyField source="kindOfData" dest="_text_" maxChars="3000"/>

conf/solr/9.3.0/solrconfig.xml → conf/solr/solrconfig.xml

@@ -290,7 +290,7 @@
          have some sort of hard autoCommit to limit the log size.
       -->
     <autoCommit>
-      <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
+      <maxTime>${solr.autoCommit.maxTime:30000}</maxTime>
       <openSearcher>false</openSearcher>
     </autoCommit>
 
@@ -301,7 +301,7 @@
       -->
 
     <autoSoftCommit>
-      <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
+      <maxTime>${solr.autoSoftCommit.maxTime:1000}</maxTime>
     </autoSoftCommit>
 
     <!-- Update Related Event Listeners

doc/release-notes/10169-JSON-schema-validation.md

@@ -0,0 +1,3 @@
+### Improved JSON Schema validation for datasets
+
+Enhanced JSON schema validation with checks for required and allowed child objects, type checking for field types including `primitive`, `compound` and `controlledVocabulary`. More user-friendly error messages to help pinpoint the issues in the dataset JSON. See [Retrieve a Dataset JSON Schema for a Collection](https://guides.dataverse.org/en/6.3/api/native-api.html#retrieve-a-dataset-json-schema-for-a-collection) in the API Guide and PR #10543.

doc/release-notes/10341-croissant.md

@@ -0,0 +1,9 @@
+A new metadata export format called Croissant is now available as an external metadata exporter. It is oriented toward making datasets consumable by machine learning.
+
+When enabled, Croissant replaces the Schema.org JSON-LD format in the `<head>` of dataset landing pages. For details, see the [Schema.org JSON-LD/Croissant Metadata](https://dataverse-guide--10533.org.readthedocs.build/en/10533/admin/discoverability.html#schema-org-head) under the discoverability section of the Admin Guide.
+
+For more about the Croissant exporter, see https://github.com/gdcc/exporter-croissant
+
+For installation instructions, see [Enabling External Exporters](https://dataverse-guide--10533.org.readthedocs.build/en/10533/installation/advanced.html#enabling-external-exporters) in the Installation Guide.
+
+See also Issue #10341 and PR #10533.