pga-create docker image

[mcarmonaa]

This PR add two things:

• a new step in .travis.yml to do a push of the pga-create docker image
• a Dockerfile for pga-create

How pga-create image works

The pga-create image will generate the list of repositories to be included in the PGA and the index in two separated steps through two different commands: select-repos and index-repos. The download of the repositories is done in other step using borges.

Step 1: select-repos

select repos needs a volume mounted on /pga/data. This step will leave the processed data necessary in that volume.

$ docker run --rm --name pga-create_1 \
-v /home/manu/pga-docker/data:/pga/data \
-e "STARS=1000" \
pga-create:test select-repos

The environment variable STARS configures the number of stars to filter repositories that will be included in the PGA list.

Under the mounted volume, the data needed for the next steps is placed in:

$ ls /home/manu/pga-docker/data
languages.gz  pga.list  repositories.gz  stars.gz

Step 2: borges

Using borges and the generated list in the previous step /home/manu/pga-docker/data/pga.listdownload the repositories. It's important that the bucket-size here and in the next step are the same.

Step 3: index-repos

To generate the pga-index:

$ docker run --rm --name pga-create_1 \
-v /home/manu/pga-docker/data:/pga/data \
-v /home/manu/pga-docker/root-repositories:/pga/root-repositories \
--network host \
-e "PGA_VERSION=v2-test" \
-e "BUCKET_SIZE=2" \
pga-create:test index-repos

It's important to note two things: "BUCKET_SIZE=2" should be the same used by borges and -v /home/manu/pga-docker/root-repositories:/pga/root-repositories it's where we have the repositories downloaded by borges.

After this step we will have two sets of data under the mounted volumes:

# DATA generated by pga-create
$ ls /home/manu/pga-docker/data 
index.csv  index_v2-test.csv  languages.gz  pga.list  repositories.gz  stars.gz

# repositories (siva files) and the `index_XXX.tar.gz`
$ ls root-repositories 
00  06  0c  12  18  1e  24  2a  30  36  3c  42  48  4e  54  5a  60  66  6c  72  78  7e  84  8a  90  96  9c  a2  a8  ae  b4  ba  c0  c6  cc  d2  d8  de  e4  ea  f0  f6  fc
01  07  0d  13  19  1f  25  2b  31  37  3d  43  49  4f  55  5b  61  67  6d  73  79  7f  85  8b  91  97  9d  a3  a9  af  b5  bb  c1  c7  cd  d3  d9  df  e5  eb  f1  f7  fd
02  08  0e  14  1a  20  26  2c  32  38  3e  44  4a  50  56  5c  62  68  6e  74  7a  80  86  8c  92  98  9e  a4  aa  b0  b6  bc  c2  c8  ce  d4  da  e0  e6  ec  f2  f8  fe
03  09  0f  15  1b  21  27  2d  33  39  3f  45  4b  51  57  5d  63  69  6f  75  7b  81  87  8d  93  99  9f  a5  ab  b1  b7  bd  c3  c9  cf  d5  db  e1  e7  ed  f3  f9  ff
04  0a  10  16  1c  22  28  2e  34  3a  40  46  4c  52  58  5e  64  6a  70  76  7c  82  88  8e  94  9a  a0  a6  ac  b2  b8  be  c4  ca  d0  d6  dc  e2  e8  ee  f4  fa  index_v2-test.tar.gz
05  0b  11  17  1d  23  29  2f  35  3b  41  47  4d  53  59  5f  65  6b  71  77  7d  83  89  8f  95  9b  a1  a7  ad  b3  b9  bf  c5  cb  d1  d7  dd  e3  e9  ef  f5  fb

$ ls -lah /home/manu/pga-docker/root-repositories/index_v2-test.tar.gz
-rw-r--r-- 1 root root 52K Mar 20 07:51 root-repositories/index_v2-test.tar.gz

So under XXX/root-repositories is left all the data (siva-files and the index.tar.gz) needed by the web server.

The addition of the docker-hub credentials and the creation of the srcd/pga-create on docker-hub is still pending until this is merged, I'll do it then.

[jfontan]

leftover?

[mcarmonaa]

indeed

[jfontan]

Maybe we can extract these two scripts to files in the repository to make them easier to read ad modify.

Also, instead of using && to continue only when the previous command is successful you can enable exiting the shell on error with:

set -e

https://linux.die.net/man/1/ash