CD: Package Index

[ax3l]

Build the docker image for the new package index REST API on each merge/push to develop and master.

Follow-up to: #11652

Manual Build Instructions

Here is how to build the image for the service manually/locally.

spack list --format version_json > packages.json
docker build . -t spack/packages.spack.io:latest

docker run -p 8080:80 spack/packages.spack.io:latest

Now the server is started at http://localhost:8080

[tgamblin]

minor nitpick and one major request: can we not check packages.json into the Spack repo? The CI should just run it.

[ax3l]

removed it already, accidentally pushed

[tgamblin]

ok can we remove initial spaces from shebangs?

[ax3l]

huh! clumsy f1ngers

[opadron]

I'm guessing this docker image provides a listing of packages... available for installation from the built-in repo? Is that right?

In any case, this all looks OK to me. I might make a pass on some of this after further simplifying #11621, just to make sure all of our docker stuff is consistent.

[ax3l]

Wonderful! Yes, this provides a nginx image with static files for all packages as shown in #11652

[ax3l]

@opadron I already pushed a first image manually to spack/packages.spack.io. If you spin up packages.spack.io with it we can provide version badges to our users! :)
badges/shields#3536

[opadron]

@ax3l Done!

But, it looks like something isn't working quite right. Here's some log output that I see after hitting packages.spack.io a few times:

2019/06/08 00:30:44 [error] 7#7: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 192.168.118.41, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "packages.spack.io"
192.168.118.41 - - [08/Jun/2019:00:30:44 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0" "75.183.5.177"
192.168.118.41 - - [08/Jun/2019:00:31:16 +0000] "GET /.git/config HTTP/1.1" 404 153 "http://packages.spack.io/.git/config" "Go-http-client/1.1" "195.176.3.20"
2019/06/08 00:31:16 [error] 7#7: *2 open() "/usr/share/nginx/html/.git/config" failed (2: No such file or directory), client: 192.168.118.41, server: localhost, request: "GET /.git/config HTTP/1.1", host: "packages.spack.io", referrer: "http://packages.spack.io/.git/config"
192.168.118.41 - - [08/Jun/2019:00:31:40 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0" "75.183.5.177"
2019/06/08 00:31:40 [error] 7#7: *3 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 192.168.118.41, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "packages.spack.io"
2019/06/08 00:32:44 [error] 7#7: *4 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 192.168.118.41, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "packages.spack.io"
192.168.118.41 - - [08/Jun/2019:00:32:44 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0" "75.183.5.177"

[opadron]

Taking a closer look, it seems like there isn't supposed to be a landing page, and the default page was just left unmodified? I'm not familiar with the API this server is supposed to serve, but e.g.: packages.spack.io/z/zlib.json seems to work just fine.

[ax3l]

Yes, we didn't add one yet and consider adding a little search frontend later on, such as https://github.com/Homebrew/formulae.brew.sh

For the API itself a landing page is not needed, but will look more professional once we add one.
Note: Update to add an api/ prefix, so we can add a static page later on without potential collisions.

The package index API is as simple as:
https://packages.spack.io/api/:firstLetter/:packageName.json
which will 200 a JSON file for all valid packages (content from spack list in #11652) and 404 for all other package names.
e.g. https://packages.spack.io/api/a/adios2.json and https://packages.spack.io/api/p/py-pandas.json
There is also the full index available under https://packages.spack.io/api/packages.json

The corresponding shields.io API (badges/shields#3536) for badges/shields will then be as simple as:
https://img.shields.io/spack/v/:packageName.svg,
e.g. adios2: and py-pandas:

[ax3l]

@opadron I pushed the endpoint change to include api/ now but the service does not seam to automatically re-deploy with the updated image on dockerhub.

[ax3l]

@opadron @tgamblin shall we merge this already?

[tgamblin]

@ax3l: it’s approved and I think you have merge rights :)

[ax3l]

Okay, I just wasn't sure if self-merge is fine :)