Update tools through the API, automate updates.

[natefoo]

We shouldn't be doing this through the UI on Test or Main. Updates of most tools should happen automatically when there are new versions in the Tool Shed. This would probably be in the form of a script run from cron or Jenkins.

Tool installations and manual updates

I'd like to see this done by hooking CI on this repo and then writing tools.yaml files. We can validate them on PR and run them on merge. The process to run them will be basically the same as the update process below but with an existing tools.yaml instead of one generated from /api/tools, so this would be something we do after automated updates.

Automated updates

A rough outline of what would need to happen (see also: installing tools on Test and Main:

1. Get current list of tools from /api/tools
2. Create a tools.yaml for these tools based on a whitelist of repo owners (e.g. iuc, devteam) and individual repositories
3. Do something similar to the usegalaxy_cvmfs role's update procedure:
   1. cvmfs_server transaction {test,main}.galaxyproject.org, do not continue if this fails (there is already an open transaction). Failure on any subsequent steps should be sure to call cvmfs_server abort -f {test,main}.galaxyproject.org
   2. Run the installer application. Failure on any subsequent steps should be sure to call docker kill <container>:
      • docker run --rm -d -v /cvmfs:/cvmfs -p 127.0.0.1:8808:8888 galaxy/test_installer
      • docker run --rm -d -v /cvmfs:/cvmfs -v /galaxy-repl:/galaxy-repl:ro -p 127.0.0.1:8818:8888 galaxy/main_installer for main
   3. Wait until the installer is responding (e.g. with galaxy-wait
      • https://test-installer.galaxyproject.org/ for Test
      • https://main-installer.galaxyproject.org/ for Main
   4. Install tools with shed-tools
4. docker kill <container_id> to stop the installer
5. cvmfs_server publish -a <some_tag_id> -m '<Some description>' {test,main}.galaxyproject.org

We have a sandbox CVMFS repository that can be used for testing by logging in as sandbox@cvmfs0-psu0.galaxyproject.org and using the repo name sandbox.galaxyproject.org.

@erasche has some existing tooling to do this for usegalaxy.eu (but without CVMFS) in usegalaxy-eu/usegalaxy-eu-tools.

I also have this script that attempts to run the UCSC builds update automatically every week for the data.galaxyproject.org repository, but it fails (note it does not do any of the "failure on cleanup" stuff described above):

#!/bin/bash 
#
# Script to update UCSC shared data tables.  The idea is to update, but if
# the update fails, not replace current data/tables with error
# messages.
set -e

# setup directories
BINDIR=$(dirname $0)
BROWSER="http://genome-test.cse.ucsc.edu/cgi-bin/das/dsn"
LENGTHS="/cvmfs/data.galaxyproject.org/managed/len/ucsc"
LOCATION="/cvmfs/data.galaxyproject.org/managed/location"
WORKDIR=$(mktemp -t -d ucsc_builds_XXXXXXXX)
mkdir ${WORKDIR}/chrom ${WORKDIR}/build

# the only dependency is six
. ${HOME}/ucsc_update/bin/activate

echo "Working directory: $WORKDIR"

echo "Updating builds.txt"
python ${BINDIR}/parse_builds.py ${BROWSER} > ${WORKDIR}/build/builds.txt
echo "Updating ucsc_build_sites.txt"
python ${BINDIR}/parse_builds_3_sites.py > ${WORKDIR}/build/ucsc_build_sites.txt
echo "Building chromInfo tables"
python ${BINDIR}/build_chrom_db.py ${WORKDIR}/chrom/ ${WORKDIR}/build/builds.txt
echo "Adding Manual Builds"
curl https://raw.githubusercontent.com/galaxyproject/galaxy/dev/tool-data/shared/ucsc/manual_builds.txt.sample \
    | python ${BINDIR}/add_manual_builds.py /dev/stdin ${WORKDIR}/build/builds.txt ${WORKDIR}/chrom/

cvmfs_server transaction data.galaxyproject.org

for f in builds.txt ucsc_build_sites.txt; do
    echo "Comparing ${WORKDIR}/build/$f with ${LOCATION}/$f"
    diff -u ${WORKDIR}/build/$f ${LOCATION}/$f || { echo "Installing new $f"; cp ${WORKDIR}/build/$f ${LOCATION}/$f; }
done

echo "Comparing length files"
for f in ${WORKDIR}/chrom/*.len; do
    f=$(basename $f)
    diff -q ${WORKDIR}/chrom/$f ${LENGTHS}/$f || { echo "Installing new $f"; cp ${WORKDIR}/chrom/$f ${LENGTHS}/$f; }
done

cvmfs_server publish -a "ucsc-builds-$(date +%s)" -m "Update UCSC builds $(date +%Y-%m-%d)" data.galaxyproject.org

rm -rf ${WORKDIR}
echo "Update complete"

Note that installing updates where there's not a new version of the tool (only new revisions of the current version) appear to not be working right with ephemeris for me as described in this comment (also includes an example tools.yaml and shed-tools run).