docker based build #4415

clement-igonet · 2024-07-17T13:06:28Z

Setup to buils maplibre-gl-js with docker compose build command.
For development purpose: docker compose up
related issue: /link Suggest docker solution to help on setting up build/test/run environment #57

Fixes #57

codecov-commenter · 2024-07-17T13:09:12Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.58%. Comparing base (4b98e75) to head (d11a438).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4415   +/-   ##
=======================================
  Coverage   87.57%   87.58%           
=======================================
  Files         246      246           
  Lines       33396    33396           
  Branches     2223     2198   -25     
=======================================
+ Hits        29247    29249    +2     
- Misses       3130     3138    +8     
+ Partials     1019     1009   -10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

louwers · 2024-07-17T13:17:38Z

Can this run the browser tests? I had some trouble with that on an M1 Mac.

louwers · 2024-07-17T13:18:20Z

docker-compose.yml

@@ -0,0 +1,15 @@
+services:


Does a docker-compose really make sense with only one service?

I would reccomnd to add an entrypoint to the docker file by using npm run start and export 9966 port.
Also add some explanation to the CONTRIBUTING.md file on how to use this image.

Does a docker-compose really make sense with only one service?

I already imagine at least 1 or 2 other services (so I start by a first):

a headless web browser for testing purposes

a web server to mak run a release with some examples

I still don't see a huge value here...
You can simply mount the entire cloned repo instead of a selective number of folders.
You can switch the entry point command to run different scripts from the package.json file.
Bottom line, this can be dramatically simplified.

louwers · 2024-07-17T13:20:15Z

Dockerfile

+
+RUN echo "hello world"
+
+RUN apt-get update && apt-get install -y \


The .devcontainer should probably use the Dockerfile, otherwise this is duplicated.

I don't think this docker container is published, so I'm not sure .devcontainer can use this docker.
I would reccoment trying to find a container that already has most of these installed and simply use it instead of installing everything.

I would reccoment trying to find a container that already has most of these installed and simply use it instead of installing everything.

I'm sure to get this specific point. Are you talking about .devcontainer stuff ?

You can use a Dockerfile https://docs.github.com/en/codespaces/setting-up-your-project-for-codespaces/adding-a-dev-container-configuration/introduction-to-dev-containers#dockerfile

OK, good point.

I did not consider to integrate docker facilities so far away, I just though about not having to change my environment (installing node and dependencies).

So, I do not aim to manage .devcontainer in that specific P.R.

Is it something mandatory here ?

Dockerfile

HarelM · 2024-07-17T14:47:01Z

Dockerfile

+  xvfb \
+  && rm -rf /var/lib/apt/lists/*
+
+RUN apt-get update && apt-get install -y \


Why is this split into two lines? Why not installing everything at one?

Why is this split into two lines? Why not installing everything at one?

Well, I'm just following documentation from CONTRIBUTING.md : https://github.com/maplibre/maplibre-gl-js/blob/main/CONTRIBUTING.md#linux-and-by-extension-github-codespaces

Probably a mistake there too, or done with readability in mind.
We should aim at reducing docker layers.

OK, it will be done today.

clement-igonet · 2024-07-19T17:59:45Z

Can this run the browser tests? I had some trouble with that on an M1 Mac.

Good question.
What is the npm command to launch for that ?
I'm ready to have a try on my side, and to look for a fix if needed.

HarelM · 2024-07-19T18:05:51Z

Checkout the CI code, it has most of what the CI is running, which is stuff in most cases you should run locally to make sure the PR checks pass.
npm run test-unit, npm run test-integration, npm run test-render

clement-igonet · 2024-07-19T22:04:09Z

npm run test-unit, npm run test-integration, npm run test-render

Only npm run test-unit runs well in docker way for the moment.
I suggest to focus on npm run test-integration and npm run test-render in another step

HarelM · 2024-07-20T12:20:58Z

Not sure what you mean by another step, but if the docker image is not good at the moment, I suggest to fix it before merging it, otherwise it misses the point a bit...
Also the part about devcontainer is worth looking into as part of this PR.

clement-igonet · 2024-07-20T13:39:00Z

Not sure what you mean by another step, but if the docker image is not good at the moment, I suggest to fix it before merging it, otherwise it misses the point a bit... Also the part about devcontainer is worth looking into as part of this PR.

I"ll do my best. github CI is not a part I know well. It seems to be github specific, as I though to suggest something more independent, not linked to github CI.

HarelM · 2024-07-20T14:13:48Z

It might be interesting to publish this dev container to ghcr for every push to main so that people won't need to build it locally...

clement-igonet · 2024-07-20T14:54:55Z

It might be interesting to publish this dev container to ghcr for every push to main so that people won't need to build it locally...

OK.
It could be this docker tag: ghcr.io/maplibre/maplibre-gl-js:dev
In that case, do we already have a github docker repository for maplibre ?
secrets.GITHUB_TOKEN would be also required

HarelM · 2024-07-20T18:38:25Z

.devcontainer/Dockerfile

+
+RUN \
+    # Configure global npm install location, use group to adapt to UID/GID changes
+    if ! cat /etc/group | grep -e "^npm:" > /dev/null 2>&1; then groupadd -r npm; fi \


Why not use a node docker image as a base image instead of all this complexity?

HarelM · 2024-07-20T18:39:14Z

.devcontainer/Dockerfile

+    && chmod g+s ${NPM_GLOBAL} \
+    && npm config -g set prefix ${NPM_GLOBAL} \
+    && su ${USERNAME} -c "npm config -g set prefix ${NPM_GLOBAL}" \
+    # Install eslint


Not sure why you would need to install eslint globally, it is installed locally as part of npm ci command.

HarelM · 2024-07-20T18:41:24Z

.devcontainer/devcontainer.json

@@ -1,4 +1,5 @@
 {
    "name": "MapLibre container using node.js",
+    "build": { "dockerfile": "Dockerfile" },


I would assume that using a docker image would avoid the need for all the below installations.
Also, if we publish an image as part of the commits to main I would expect to be able to use this image for codespaces...?

HarelM · 2024-07-20T18:42:12Z

.github/workflows/release.yml

@@ -6,6 +6,26 @@ on:
  workflow_dispatch:

 jobs:
+  push-store-image:


Suggested change

push-store-image:

push-dev-image:

OK, applied.

HarelM · 2024-07-20T18:42:46Z

.github/workflows/release.yml

+        working-directory: './devcontainer'
+    steps:
+      - name: 'Checkout GitHub Action'
+        uses: actions/checkout@main


Better to use a version, like in other places in the CI.

Better to use a version, like in other places in the CI.

OK. So I used this:

ghcr.io/maplibre/maplibre-gl-js:${{ steps.package-version.outputs.current-version }}

I meant v4 or so for the checkout action...

HarelM · 2024-07-20T18:47:10Z

.devcontainer/Dockerfile

+    libgif-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+CMD npm ci && npm run start


Add npm run build-dist here.
npm run start should be the entry point to allow replacing it with other commands.
This will allow for better ergonomics and avoid the need for docker compose.

clement-igonet · 2024-07-20T21:47:00Z

A clarification point: what is expected for CI ?
What released docker image should do ?

Just contains environment to run any node comment for maplibre ?
Contains also source code ?
Contain only distribution folder content ?

HarelM · 2024-07-21T04:53:59Z

CI should stay the same, besides pushing an image for every commit to main.
Docker image should contain everything to allow running any maplibre script written in package.json (besides docker within a docker I guess).
It should contain the src and dist, and should allow a developer to replace them with their code.

clement-igonet · 2024-07-21T12:16:59Z

CI should stay the same, besides pushing an image for every commit to main. Docker image should contain everything to allow running any maplibre script written in package.json (besides docker within a docker I guess). It should contain the src and dist, and should allow a developer to replace them with their code.

I'll work on that way.

However, in term of use cases, here is what I have in mind:

I'd like to rely on an environment that is easy to reproduce (in dev environment or production environment)
- So a Dockerfile without sources is welcome, for dev and CI purposes
As a developer, I'd like to make run maplibre from my own code
- So a Dockerfile without sources is enough, as docker or docker-compose can bind the current local code inside the docker container
As a user, I'd like to have a try with any version of maplibre-gl-js library
- In that way, I do not need docker as I can manage maplibre-gl-js version in html source code.

=> So I would suggest not to include source code inside provided docker image

It does not really cover a clearly defined use case
It would provide a big docker image for no value

What do you think ?

HarelM · 2024-07-21T12:24:54Z

I don't think it will change the size dramatically, so the argument is weak.
Since you can override the code I don't think it will matter much.
In order to run a debug version (npm run start) you'll need the code, otherwise it won't work.
And I think that for completeness and simplicity it's better to include it.

We can always remove it later on, I suggest to include it, see how it feels and then see if we truly need to optimize it.

clement-igonet · 2024-07-21T12:48:45Z

F.Y.I.: For testing purpose, I realize that google-chrome must be installed.

HarelM · 2024-07-21T17:44:03Z

.devcontainer/Dockerfile

@@ -0,0 +1,38 @@
+ARG VARIANT=20-bookworm-slim


Since we are using puppeteer for e2e tests I would recommend a base image that already has chrome and node like:
https://pptr.dev/guides/docker
And avoid all these complexity of installing chrome etc.

HarelM · 2024-07-27T18:57:03Z

I would consider starting from the following docker image and see if there are missing stuff that needs to be added instead of bootstrapping an image:
cypress/browsers.

clement-igonet force-pushed the main branch from 0aa06dc to 3a9a226 Compare July 17, 2024 13:14

louwers reviewed Jul 17, 2024

View reviewed changes

HarelM reviewed Jul 17, 2024

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

HarelM reviewed Jul 17, 2024

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

HarelM reviewed Jul 17, 2024

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

HarelM reviewed Jul 17, 2024

View reviewed changes

clement-igonet force-pushed the main branch 3 times, most recently from 71edd03 to 5915500 Compare July 19, 2024 22:01

clement-igonet added 5 commits July 20, 2024 17:17

docker based build

44b15d5

fix after review 01

ffe06d1

reducing docker layers

2b112e0

merge with .devcontainer

11183c8

dev container to ghcr

385c298

clement-igonet force-pushed the main branch from baf1559 to 385c298 Compare July 20, 2024 15:17

HarelM reviewed Jul 20, 2024

View reviewed changes

fix after review 02

0990cc4

fix after review 02

74aa2a8

include source code and build dist

d11a438

HarelM reviewed Jul 21, 2024

View reviewed changes

HarelM added the waiting for code review fixes label Aug 1, 2024

HarelM mentioned this pull request Nov 4, 2024

Add a shell.nix or prefarably a flake.nix #4973

Open


		RUN echo "hello world"

		RUN apt-get update && apt-get install -y \

docker based build #4415

Are you sure you want to change the base?

docker based build #4415

Conversation

clement-igonet commented Jul 17, 2024 • edited by HarelM Loading

codecov-commenter commented Jul 17, 2024 • edited Loading

Codecov Report

louwers commented Jul 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

louwers Jul 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clement-igonet commented Jul 19, 2024

HarelM commented Jul 19, 2024

clement-igonet commented Jul 19, 2024 • edited Loading

HarelM commented Jul 20, 2024

clement-igonet commented Jul 20, 2024 • edited Loading

HarelM commented Jul 20, 2024

clement-igonet commented Jul 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clement-igonet commented Jul 20, 2024

HarelM commented Jul 21, 2024

clement-igonet commented Jul 21, 2024 • edited Loading

HarelM commented Jul 21, 2024

clement-igonet commented Jul 21, 2024

Choose a reason for hiding this comment

HarelM commented Jul 27, 2024

clement-igonet commented Jul 17, 2024 •

edited by HarelM

Loading

codecov-commenter commented Jul 17, 2024 •

edited

Loading

louwers Jul 17, 2024 •

edited

Loading

clement-igonet commented Jul 19, 2024 •

edited

Loading

clement-igonet commented Jul 20, 2024 •

edited

Loading

clement-igonet commented Jul 20, 2024 •

edited

Loading

clement-igonet commented Jul 21, 2024 •

edited

Loading