############################################################
Author
############################################################
Authors: Yifan Ning, Wen Zhang, Richard Boyce

Copyright 2016-2017 University of Pittsburgh

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

######################################################################
Deploy DBMI-Annotator on production mode
######################################################################

(1) Install docker(v1.12.3), docker-compose (v1.8.1) and apache (v2.4) server

    Apache2 server configuration:

    RewriteEngine on

    ## DBMI-Annotator Docker configuration ###############################
    # proxy for docker exposed service at port 8085
    ProxyPass /dbmiannotator http://localhost:8085/dbmiannotator
    RewriteRule /dbmiannotator(.*) http://localhost:8085/dbmiannotator$1 [P]

    # proxy for docker exposed service at port 8085 for annotator store
    ProxyPass /annotatorstore http://localhost:8085/annotatorstore
    RewriteRule /annotatorstore(.*) http://localhost:8085/annotatorstore$1 [P]

    # stylesheet folder relocate at Document root(dailymed, PMC)
    RewriteRule /dbmiannotator/dailymed(.*) http://localhost/DDI-labels/dailymed$1 [P]
    RewriteRule /dbmiannotator/PMC(.*)_files/(.*) http://localhost/PMC/PMC$1_files/$2 [P]

    # RDF redirects
    RewriteRule /DDI-pdfs/(.*) http://localhost:8085/DDI-pdfs/$1 [P]

(2) Create local volumes for postgres and elasticsearch data mount point

    $ bash scripts/create-docker-volumns.sh

(3) Run dbmi-annotator with all dependencies in docker container, takes 20 secs for setup all services

    $ bash scripts/clean-docker-ps.sh
    $ cd dbmi-annotator
    $ docker-compose up -d

(4) Optional: add extra articles for annotating. Currently version is supporting Pubmed centre article (PMC), Dailymed product labeling (SPL) and PDFs.
    
    4.1 For PMC article:
    $ docker cp /path/to/html apache2:/var/www/html/PMC/

    Example: 
    <1> we saved PMC article with pmcid 4536363 as PMC4536363
    <2> copy the html resources to apache home dir in docker container apache2
    $ docker cp PMC4536363* apache2:/var/www/html/PMC/
    
    Access the article by copy and paste the link to input box on main page
    http://localhost/PMC/PMC4536363.html 

    4.2 For PDF:
    $ docker cp /path/to/PDF apache2:/var/www/html/DDI-pdfs

    Example:
    $ docker cp /home/websites/html/DDI-pdfs/Martin_2002_12445025.pdf apache2:/var/www/html/DDI-pdfs/

    Access the article by copy and paste the link to input box on main page
    http://localhost/DDI-pdfs/Martin_2002_12445025.pdf

    4.3 Recursively copy contents of directory into container apache2
    $ docker cp -L /home/websites/dbmi-dt-139.dbmi.pitt.edu/DDI-pdfs apache2:/var/www/html/

    4.4 Configure source list on AnnotationPress main page. In docker container dbmi-annotator, add entry to "dbmi-annotator/article-list/", supporting publishers according to exists source list csv:

    PMC: pmc-list.csv
    PDF document: test-pdf-list.csv
    Dailymed SPL: dailymed-list.csv    
    FutureMedicine: future-list.csv
    Sage: sage-list.csv
    Taylor & Francis: taylor-list.csv
    Wolters Kluwer: wolters-list.csv
    Elsevier: elsevier-list.csv
    Springer: springer-list.csv
    Wiley: wiley-list.csv

    convention for new entry in source list csv: <name of document>,<URL>
    Refresh main page will update list of articles automatically, allowing configure at runtime


######################################################################
Docker Administration
######################################################################

----------------------------------------------------------------------
Docker clean and restart
----------------------------------------------------------------------

Stop and clean all processes (required for rerun docker-compose when made modifications)

$ bash scripts/clean-docker-ps.sh

----------------------------------------------------------------------
Work in container
----------------------------------------------------------------------

Get bash shell within container (ex. apache2):
$ docker exec -it apache2 bash

----------------------------------------------------------------------
Delete annotations from host:
----------------------------------------------------------------------

Truncate all data in docker containers
$ bash scripts/recreate-docker-volumns.sh

Delete annotation in docker by condition
$ docker exec dbmi-annotator curl -XDELETE 'http://elasticsearch:9200/annotator/annotation/_query?q=email:test@gmail.com'

----------------------------------------------------------------------
Backup elasticsearch store
----------------------------------------------------------------------

(1) create elasticsearch snapshot 

$ docker exec -it dbmiannotator bash

root@dbmiannotator# cd backup-program
root@dbmiannotator# bash elastic-snapshot.sh elasticsearch 9200

moint point (Ubuntu) is at 
/var/lib/docker/volumes/elastic-snapshot-volume/_data

Run from host (add to crontab for daily backups):
$ docker exec dbmi-annotator bash backup-program/elastic-snapshot.sh elasticsearch 9200


----------------------------------------------------------------------
Docker related Issues fixing
----------------------------------------------------------------------
If run into error when build:
docker not find endpoint count for network bridge: key not found in store

Run following commands and rebuild image:
$ rm -rf /var/lib/docker/network/files
$ sudo service docker restart

----------------------------------------------------------------------
Docker commands
----------------------------------------------------------------------
Clean docker containers:

$ docker stop $(docker ps -a -q)
$ docker rm $(docker ps -a -q)

Volume inspect:

$ docker volume inspect postgres-volume

Clean all orphaned volumes

$ docker volume rm $(docker volume ls -qf dangling=true)

Check dependent child processes of specific ps
$ docker inspect --format='{{.Id}} {{.Parent}}' $(docker images --filter since=<parent ps Id> -q)


######################################################################
Evidence base translation tool - In development (Optional)
######################################################################

$ docker exec -it dbmi-annotator bash
$ cd translation/mp-evidence-base-ETL

Optional: Annotation translation (Elasticsearch to PostgreSQL)

    python loadAnnotatorAnnsToRDB.py <elastic host> <elastic port> <pg host> <pg port> <pg username> <pg password> <OPTIONS (1: clean all tables, 2 drop and recreate all tables, 0: keep existing data)>

    Example:
    $ python loadAnnotatorAnnsToRDB.py elasticsearch 9200 postgres 5432 dbmiannotator <password> 0


Optional: Dailymed SPLs Annotation translation (CSV to PostgreSQL)

    python loadDomeoAnnsToRDB.py <pg hostname> <pg port> <pg username> <pg password> <OPTIONS (1: clean all tables, 2 drop and recreate all tables, 0: keep existing data)>

    Example:
    python loadDomeoAnnsToRDB.py postgres 5432 dbmiannotator <password> 0


Optional: pre-load annotations (postgreSQL to elasticsearch)

    python load-rdb-annotations.py <pg hostname> <pg username> <pg password> <es hostname> <annotation author>

    Example:
    $ python translateRDB2Elastic.py postgres dbmiannotator <password> elasticsearch test@gmail.com

######################################################################
Build dependencies in docker image for DBMI-Annotator
######################################################################

Description: This repository holds all Dockerfile as dependencies of dbmi-annotator to built 

Author: Yifan Ning, University of Pittsburgh

Environment: Ubuntu 14.04/15.10, docker > 1.9 

----------------------------------------------------------------------
Install docker and docker-compose
----------------------------------------------------------------------

(1) Install docker 
sudo apt-get install docker

(2) Install specific version of docker-compose (preferred):
$ cd
$ curl -L https://github.com/docker/compose/releases/download/1.8.1/docker-compose-`uname -s`-`uname -m` > ~/docker-compose
$ sudo mv docker-compose /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
Reference: http://www.dynomitedb.com/docs/dynomite/v0.5.7/install-docker-ubuntu-14-04/ 

Optional: Install default version of docker-compose based on Ubuntu kernel version
Recommand use virtualenv in order to avoiding conflict of docker-compose python lib dependencies and local python denpendencies
  
sudo pip install docker-compose

----------------------------------------------------------------------
Build Postgres
----------------------------------------------------------------------

Build:

(1) Install postgres 9.3
(2) Create user 'dbmiannotator' and database 'dbmiannotator'
(3) Execute script rdb-postgres-schema.sql to create table schema
(4) Execute script rdb-postgres-initial.sql to initialize plugins information
(5) build docker image <dockerhub>/postgres-9.3:latest

$ cd postgres-9.3
$ docker build -t <dockerhub>/postgres-9.3:latest .

Optional: run independently on localhost and port 5432 after built:
$ docker run -p 0.0.0.0:5432:5432 -d --name postgres <dockerhub>/postgres-9.3

Reference: 
https://docs.docker.com/engine/examples/postgresql_service/

----------------------------------------------------------------------
Build nodejs server DBMI-Annotator
----------------------------------------------------------------------

Dependency:
Postgres db 9.3 running on port 5432

Built:
(1) Download dbmi-annotator

$ git clone https://github.com/dbmi-pitt/dbmi-annotator.git

(2) Install node modules

$ npm install

(3) Create config.js and modify

$ cp config/config.sample.js config/config.js
$ cp config/production.sample.conf config/production.conf

configure domain name and postgres DB connection information in and production.conf for production mode

(4) Build docker image dbmi-annotator

$ docker build -t <dockerhub>/dbmi-annotator:latest .

(5) Push images to docker hub

$ docker push <dockerhub>/dbmi-annotator:latest

Optional: run dbmi-annotator on localhost:3000

$ docker run -p 0.0.0.0:3000:3000 -d --name dbmi-annotator <dockerhub>/dbmi-annotator node server.js  --link postgres --link elastic -e DATABASE_URL=postgre://postgres:<pin>@0.0.0.0:5432/dbmiannotator 

----------------------------------------------------------------------
Build Elasticsearch
----------------------------------------------------------------------

Build:

(1) Install Java JDK 8 (Oracle) and set JAVA_HOME environment variable
(2) Install elasticsearch 1.7.1
(3) Install plugin head and build docker images <dockerhub>/elasticsearch-1.7:1.7.1
(4) Build docker image <dockerhub>/elasticsearch-1.7:1.7.1

$ cd elasticsearch-1.7
$ docker build -t <dockerhub>/elasticsearch-1.7:1.7.1 .

Optional: run independently on localhost and port 9200 after built:
$ docker run -p 9200:9200 -d --name elastic <dockerhub>/elasticsearch-1.7:1.7.1

----------------------------------------------------------------------
Build Annotator-store
----------------------------------------------------------------------
Dependency:
Elasticsearch 1.7.1 running on port 9200

Build:

(1) Download annotator store repository

$ git clone https://github.com/ningyifan/annotator-store.git

(2) Setup python virtual environment and build docker image 

$ cd annotator-store
$ docker build --no-cache -t <dockerhub>/annotator-store:latest .

Optional: run annotator store independently on localhost and port 5000:
 
$ docker run -p 0.0.0.0:5000:5000 -d --name annotator-store <dockerhub>/annotator-store python /<path to repo>/annotator-store/run.py -link <dockerhub>/elasticsearch-1.7 -e ELASTICSEARCH_HOST=elasticsearch

----------------------------------------------------------------------
Build Apache server 2.4
----------------------------------------------------------------------

Build:

(1) Download and install apache2 2.4
(2) Replace default server configuration at /etc/apache2/sites-enabled/ 
(3) Copy and configure 
(4) Start apache2 server
(5) build docker image

$ docker build -t <dockerhub>/apache2:2.4 .

Optional: run independently on localhost and port 80

$ docker run -d -p 80:80  --name=apache2 <dockerhub>/apache2:2.4