Migrate to docker

[HarelM]

Feature/Infra

Current implementation of services and site is based on windows specific technology and due to this fact it's hard to deploy and create a staging environment (IIS for the site and windows services for routing and DB).

Presumably, Docker should be able to support all this kind of requirements including keep-alive mechanism, and loading of Elasticsearch and GraphHopper as required containers.

This requires a lot of learning effort though...

[HarelM]

Initial commit via #639

[HarelM]

Containers that will be probable needed.
Gpsbabel
https://hub.docker.com/r/jamesmstone/gpsbabel/
Graphhopper:
https://hub.docker.com/r/graphhopper/graphhopper/
ElasticSearch:
https://www.docker.elastic.co/
OsmCTools:
https://hub.docker.com/r/michelesr/osmctools/

[HarelM]

Using some of the containers above is an issue since they might be platform specific and to run them you'll need to know that they are running using Docker which kind of breaks the concept of decoupling and abstraction.
An interesting solution to this issue might be to make everything a REST server (mainly GPSBabel and OsmCTools) so that it will be cross-platform and cross language.
I started to look at C++ libraries that might do it like C++ REST SDK but it looks too complex and I hate C++ to be honest.
A possible simple and elegant solution might be to use a NodeJs docker image and create a very small HTTP server to host these executable.

[HarelM]

Node JS docker is still not supported on windows platform unfortunately.
See nodejs/docker-node#695
A possible solution to this problem, which also might make the docker file simpler is to wrap the HTTP using .Net Core and use a NuGet Package (actually updating the current package) with the compiled binaries (for all OSs).
The first step though is to move every executable usage to be an HTTP micro service.
This probably means that every micros service will have its own repository here at IHM.
Microservices that will turn to docker containers:

1. GPSBabel
2. OSM update
3. Elevation?
4. GraphHopper (already has docker, I think)
5. Elastic search (already has docker, I think)

[HarelM]

For future reference the following commands will build GPSBabel on Debian with docker:

FROM debian:latest

RUN apt-get update  
RUN apt-get install -y git 
RUN git clone https://github.com/gpsbabel/gpsbabel.git gpsbabel
RUN apt-get install -y qtbase5-dev 
RUN apt-get install -y qttools5-dev 
RUN apt-get install -y build-essential
RUN apt-get install -y qt5-default
WORKDIR gpsbabel
RUN ./configure
RUN make
RUN ./gpsbabel
ENTRYPOINT	["/gpsbabel/gpsbabel"]

The following will do it on alpine:

FROM alpine:latest

RUN apk add --no-cache --update alpine-sdk
RUN apk add libusb-dev
RUN apk add sed
RUN apk add qt5-qtbase
RUN apk add qt5-qtmultimedia-dev
RUN apk add git
RUN git clone https://github.com/gpsbabel/gpsbabel.git gpsbabel \
    && cd gpsbabel \
    && ./configure \
    && make

ENTRYPOINT	["/gpsbabel"]

I have managed to extract the relevant so files from the docker container to create a NuGet package that will work on debian.

[HarelM]

Related articles that we might need to install on the server:
https://docs.docker.com/machine/drivers/hyper-v/
https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization
https://docs.docker.com/toolbox/toolbox_install_windows/
https://blog.docker.com/2016/09/dockerforws2016/
https://docs.docker.com/machine/get-started/

[HarelM]

Used the following command to enable hyper-v on the server:
Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V –All -NoRestart
Based on this video:
https://www.youtube.com/watch?v=eMBtnmxbz0M
Replaced docker-machine according to comment here:
docker/machine#4424 (comment)
Ran:
docker-machine --debug create -d hyperv --hyperv-virtual-switch "Primary Virtual Switch" main
Got error:
The Virtual Machine Management Service failed to start the virtual machine 'main' because one of the Hyper-components is not running (Virtual machine ID 0603AE2F-CAAC-4DDD-8A37-E32146D83FF7).

Might be related to the following article:
https://www.yoyoclouds.com/2012/07/create-and-setup-vm-in-hyper-v.html

[HarelM]

Hopefully visual studio support for docker in now better:
https://developercommunity.visualstudio.com/content/problem/238120/docker-compose-doesnt-show-in-selection.html

[HarelM]

As an initial step, instead of trying to fully migrate everything to docker, the following might prove worthy of trying.

1. Use docker compose for elasticsearch and graphhopper only.
2. Migrate other services later on.

Elastic search is probably the most interesting one to migrate and should be easy.
Graphopper is probably the most valuable one since it will allow creating two instances side by side and allow switching instead of using a proprietary version I wrote which was not updated for a long time.
Also it might help with: #1124 and macOS development.
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html
Disable X-Pack:
https://discuss.elastic.co/t/how-do-i-disable-x-pack-security-on-the-elasticsearch-5-2-2-docker-image/78183/2
https://gist.github.com/shreyu86/90377c42a78f1ff2d0e17a9ad502ac31

[HarelM]

The current state seems that using docker compose the following will happen:

1. Load Elasticsearch
2. Make Graphhopper create a routing for Israel
3. Compile and load the site

What's left is the following:

1. Create a script that uses docker to create a Graphhopper cache, replace the existing and restarting Graphhopper instead of using two instances (more down time, less complexity).
2. Move GPSBabel to docker using GPSBabelWebApi
3. Find a solution to OSM-C-Tools

[HarelM]

Docker update script was created and tested.
Last two things:

1. GPSBabel
2. OSM-C tools.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days

[HarelM]

GPSBabel was implemented and pushed.
Elasticsearch was updated to latest version and can now in theory be moved to docker but it requires some changes in the server. nevertheless debug env is using docker now.
So the only missing micro-service that needs to be moved to docker is the code around OSM-C-Tools.
There's not a lot of logic there, just an up-to-date pbf file and some changes files.
The following repository is the target for that micro service:
https://github.com/IsraelHikingMap/OSMPbfWebAPI