Merge branch 'develop' into 4396-support-multiple-file-storage-locati…

…ons #4396

Conflicts (new methods added in same place on both branches):
src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java

conf/docker-aio/c7.dockerfile

@@ -1,17 +1,23 @@
 FROM centos:7
 # OS dependencies
-RUN yum install -y java-1.8.0-openjdk-headless postgresql-server sudo epel-release unzip perl curl
+RUN yum install -y java-1.8.0-openjdk-headless postgresql-server sudo epel-release unzip perl curl httpd
 RUN yum install -y jq
 
 # copy and unpack dependencies (solr, glassfish)
 COPY dv /tmp/dv
 COPY testdata/schema.xml /tmp/dv
 COPY testdata/solrconfig.xml /tmp/dv
+# IPv6 and localhost appears to be related to some of the intermittant connection issues
+COPY disableipv6.conf /etc/sysctl.d/
+RUN rm /etc/httpd/conf/*
+COPY testdata/httpd.conf /etc/httpd/conf 
 RUN cd /opt ; tar zxf /tmp/dv/deps/solr-7.2.1dv.tgz 
 RUN cd /opt ; tar zxf /tmp/dv/deps/glassfish4dv.tgz
 
+# this copy of domain.xml is the result of running `asadmin set server.monitoring-service.module-monitoring-levels.jvm=LOW` on a default glassfish installation (aka - enable the glassfish REST monitir endpoint for the jvm`
+COPY domain-restmonitor.xml /opt/glassfish4/glassfish/domains/domain1/config/domain.xml
+
 RUN sudo -u postgres /usr/bin/initdb -D /var/lib/pgsql/data
-#RUN sudo -u postgres createuser dvnapp
 
 # copy configuration related files
 RUN cp /tmp/dv/pg_hba.conf /var/lib/pgsql/data/
@@ -30,6 +36,9 @@ EXPOSE 5432
 # glassfish port
 EXPOSE 8080
 
+# apache port, http
+EXPOSE 80
+
 RUN mkdir /opt/dv
 
 # yeah - still not happy if glassfish isn't in /usr/local :<
@@ -41,4 +50,8 @@ COPY testdata /opt/dv/testdata
 COPY testscripts/* /opt/dv/testdata/
 COPY setupIT.bash /opt/dv
 WORKDIR /opt/dv
+
+# healthcheck for glassfish only (assumes modified domain.xml); 
+#  does not check dataverse application status.
+HEALTHCHECK CMD curl --fail http://localhost:4848/monitoring/domain/server.json || exit 1
 CMD ["/opt/dv/entrypoint.bash"]

conf/docker-aio/disableipv6.conf

@@ -0,0 +1 @@
+net.ipv6.conf.all.disable_ipv6 = 1

conf/docker-aio/entrypoint.bash

@@ -6,6 +6,9 @@ cd /opt/solr-7.2.1/
 bin/solr start -force
 bin/solr create_core -c collection1 -d server/solr/collection1/conf -force
 
+# start apache, in both foreground and background...
+apachectl -DFOREGROUND &
+
 # TODO: Run Glassfish as non-root.
 cd /opt/glassfish4
 bin/asadmin start-domain

conf/docker-aio/readme.txt

@@ -9,21 +9,23 @@ Per-build:
 - `cd conf/docker-aio`, and run `./1prep.sh` to copy files for integration test data into docker build context; `1prep.sh` will also build the war file and installation zip file
 - build the docker image: `docker build -t dv0 -f c7.dockerfile .`
 
-- Run image: `docker run -d -p 8083:8080 --name dv dv0` (aka - forward port 8083 locally to 8080 in the container)
+- Run image: `docker run -d -p 8083:8080 -p 8084:80 --name dv dv0` (aka - forward port 8083 locally to 8080 in the container for glassfish, and 8084 to 80 for apache)
 
 Note: If you see an error like this... `docker: Error response from daemon: Conflict. The container name "/dv" is already in use by container "5f72a45b68c86c7b0f4305b83ce7d663020329ea4e30fa2a3ce9ddb05223533d". You have to remove (or rename) that container to be able to reuse that name.` ... run something like `docker ps -a | grep dv` to see the container left over from the last run and something like `docker rm 5f72a45b68c8` to remove it. Then try the `docker run` command above again.
 
+Another troubleshooting note: `empty reply from server` or `Failed to connect to ::1: Cannot assign requested address` tend to indicate either that you haven't given glassfish enough time to start, or your docker setup is in an inconsistent state and should probably be restarted.
+
 - Installation (integration test): `docker exec -it dv /opt/dv/setupIT.bash`
 
 (Note that it's possible to customize the installation by editing `conf/docker-aio/default.config` and running `docker exec -it dv /opt/dv/install.bash` but for the purposes of integration testing, the `setupIT.bash` script above works fine.)
 
-- update `dataverse.siteUrl` (appears only necessary for `DatasetsIT.testPrivateUrl`): `docker exec -it dv /usr/local/glassfish4/bin/asadmin create-jvm-options "-Ddataverse.siteUrl=http\://localhost\:8083"`
+- update `dataverse.siteUrl` (appears only necessary for `DatasetsIT.testPrivateUrl`): `docker exec -it dv /usr/local/glassfish4/bin/asadmin create-jvm-options "-Ddataverse.siteUrl=http\://localhost\:8084"`
 
 Run integration tests: 
 
 First, cd back to the root of the repo where the `pom.xml` file is (`cd ../..` assuming you're still in the `conf/docker-aio` directory). Then run the test suite with script below:
 
 `conf/docker-aio/run-test-suite.sh`
 
-There isn't any strict requirement on the local port (8083 in this doc), the name of the image (dv0) or container (dv), these can be changed as desired as long as they are consistent.
+There isn't any strict requirement on the local port (8083, 8084 in this doc), the name of the image (dv0) or container (dv), these can be changed as desired as long as they are consistent.
 

conf/docker-aio/run-test-suite.sh

@@ -1,4 +1,10 @@
 #!/bin/sh
 # This is the canonical list of which "IT" tests are expected to pass.
+
+dvurl=$1
+if [ -z "$dvurl" ]; then
+	dvurl="http://localhost:8084"
+fi
+
 # Please note the "dataverse.test.baseurl" is set to run for "all-in-one" Docker environment.
-mvn test -Dtest=DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT -Ddataverse.test.baseurl='http://localhost:8083'
+mvn test -Dtest=DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT -Ddataverse.test.baseurl=$dvurl

conf/docker-aio/seturl.bash

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+
+docker exec -it dv /usr/local/glassfish4/bin/asadmin create-jvm-options "\"-Ddataverse.siteUrl=http\://localhost\:8084\""