Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs(docker): add troubleshooting section for python #9548

Merged
merged 5 commits into from
Nov 19, 2023
Merged

docs(docker): add troubleshooting section for python #9548

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f45e907
feat: add troubleshooting section
jtoar Nov 18, 2023
227371a
fix headings
jtoar Nov 18, 2023
9caf814
add make and gcc, and comment template
jtoar Nov 19, 2023
4897d13
general improvements
jtoar Nov 19, 2023
97a7193
general improvements
jtoar Nov 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
151 changes: 115 additions & 36 deletions docs/docs/docker.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ description: Redwood's Dockerfile

:::note The Dockerfile is experimental

We've worked hard to optimize the Dockerfile and make the whole experience from setup to deploy smooth, but the Dockerfile still may change slightly as we make more optimizations and collaborate with more deploy providers.
Redwood's Dockerfile is the collective effort of several hard-working community members.
We've worked hard to optimize the it, but expect changes as we we collaborate with users and deploy providers.

:::

Redwood's Dockerfile is the accumulation of the hard work of several community members.
If you're not familiar with Docker, we recommend going through their [getting started](https://docs.docker.com/get-started/) documentation.

## Set up
Expand Down Expand Up @@ -41,13 +41,13 @@ And the prod compose file with:
docker compose -f ./docker-compose.prod.yml up
```

:::info make sure to specify env vars
:::info make sure to specify build args

If your api side or web side depend on env vars at build time, you may need to supply them as `--build-args`, or in the compose files.

:::

The first time you do this, you'll have to use the console stage to go in and migrate the database—just like you would with a Redwood app on your machine:
The first time you do this, you'll have to use the `console` stage to go in and migrate the database—just like you would with a Redwood app on your machine:

```
docker compose -f ./docker-compose.dev.yml run --rm -it console /bin/bash
Expand All @@ -57,10 +57,9 @@ root@...:/home/node/app# yarn rw prisma migrate dev
## The Dockerfile in detail

The documentation here goes through and explains every line of Redwood's Dockerfile.
If you'd like to see the whole Dockerfile for reference, you can find it [here](https://github.com/redwoodjs/redwood/tree/main/packages/cli/src/commands/experimental/templates/docker/Dockerfile).
Or by setting it up in your project: `yarn rw experimental setup-docker`.
If you'd like to see the whole Dockerfile for reference, you can find it [here](https://github.com/redwoodjs/redwood/tree/main/packages/cli/src/commands/experimental/templates/docker/Dockerfile) or by setting it up in your project: `yarn rw experimental setup-docker`.

Redwood takes advantage of [Docker's multi-stage build support](https://docs.docker.com/build/building/multi-stage/) to keep the final production image lean.
Redwood takes advantage of [Docker's multi-stage build support](https://docs.docker.com/build/building/multi-stage/) to keep the final production images lean.

### The `base` stage

Expand All @@ -71,41 +70,45 @@ It's used as the base image for the build stages and the `console` stage.
FROM node:18-bookworm-slim as base
```

We use a Node.js 18 image as the base image because that's what Redwood targets.
We use a Node.js 18 image as the base image because that's the version Redwood targets.
"bookworm" is the codename for the current stable distribution of Debian (version 12).
We think it's important to pin the version of the OS just like we pin the version of Node.js.
Lastly, the "slim" variant of the `node:18-bookworm` image only includes what Node.js needs which reduces the image's size while making it more secure.

:::tip Why not alpine?

While alpine may be smaller, it uses an unofficial C runtime.
While alpine may be smaller, it uses musl, a different C standard library.
In developing this Dockerfile, we prioritized security over size.

If you know what you're doing feel free to change this—it's your Dockerfile now! But you'll also probably have to change the apt-get instructions below.
If you know what you're doing feel free to change this—it's your Dockerfile now!
Just remember to change the `apt-get` instructions below too if needed.

:::


```Dockerfile
RUN apt-get update && apt-get install -y \
openssl \
# python3 make gcc \
&& rm -rf /var/lib/apt/lists/*
```

The `node:18-bookworm-slim` image doesn't have [OpenSSL](https://www.openssl.org/), which [seems to be a bug](https://github.com/nodejs/docker-node/issues/1919).
(It was included in the bullseye image, the codename for Debian 11.)
When running on Linux distributions, [Prisma needs OpenSSL](https://www.prisma.io/docs/reference/system-requirements#linux-runtime-dependencies).
(It was included in the "bullseye" image, the codename for Debian 11.)
On Linux, [Prisma needs OpenSSL](https://www.prisma.io/docs/reference/system-requirements#linux-runtime-dependencies).
After installing it, we clean up the apt cache, adhering to [Docker best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#apt-get).

[It's recommended](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#apt-get) to combine `apt-get update` and `apt-get install -y` in the same `RUN` statement for cache busting.

Lastly, Python and its dependencies are there ready to be uncommented if you need them. See the [Troubleshooting](#python) section for more.

```Dockerfile
USER node
```

This and subsequent `chown` options in `COPY` instructions are for security.
[Services that can run without privileges should](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#user).
The Node.js image includes a user, created with an explicit `uid` and `gid`, node; we reuse it.
The Node.js image includes a user, `node`, created with an explicit `uid` and `gid`.
We reuse it.

```Dockerfile
WORKDIR /home/node/app
Expand All @@ -122,7 +125,7 @@ COPY --chown=node:node yarn.lock .
Here we copy the minimum set of files that the `yarn install` step needs.
The order isn't completely arbitrary—it tries to maximize [Docker's layer caching](https://docs.docker.com/build/cache/).
We expect `yarn.lock` to change more than the package.json files, the package.json files to change more than `.yarnrc.yml` , and `.yarnrc.yml` to change more than the binary, etc.
That said, it's hard to argue that these files couldn't be arranged differently, or tht the COPY instructions couldn't be combined.
That said, it's hard to argue that these files couldn't be arranged differently, or tht the `COPY` instructions couldn't be combined.
The important thing is that they're all here, before the `yarn install` step:

```Dockerfile
Expand All @@ -131,7 +134,6 @@ RUN --mount=type=cache,target=/home/node/.yarn/berry/cache,uid=1000 \
CI=1 yarn install
```

The `yarn install` step.
This step installs all your project's dependencies—production and dev.
Since we use multi-stage builds, your production images won't pay for the dev dependencies installed in this step.
The build stages need the dev dependnecies.
Expand All @@ -148,11 +150,7 @@ The last thing to note is that we designate the node user.
[The node user's `uid` is `1000`](https://github.com/nodejs/docker-node/blob/57d57436d1cb175e5f7c8d501df5893556c886c2/18/bookworm-slim/Dockerfile#L3-L4).

One more thing to note: without setting `CI=1`, depending on the deploy provider, yarn may think it's in a TTY, making the logs difficult to read. With this set, yarn adapts accordingly.
Enabling CI enables `--immutable --inline-builds`, both of which are highly recommended. For more information on those settings:

- https://yarnpkg.com/configuration/yarnrc#enableInlineBuilds
- https://yarnpkg.com/configuration/yarnrc#enableProgressBars
- https://yarnpkg.com/configuration/yarnrc#enableTelemetry
Enabling CI enables [immutable installs](https://yarnpkg.com/configuration/yarnrc#enableImmutableInstalls) and [inline builds](https://yarnpkg.com/configuration/yarnrc#enableInlineBuilds), both of which are highly recommended. For more information on those settings:

```Dockerfile
COPY --chown=node:node redwood.toml .
Expand All @@ -162,16 +160,17 @@ COPY --chown=node:node .env.defaults .env.defaults

We'll need these config files for the build and production stages.
The `redwood.toml` file is Redwood's de-facto config file.
Both the build and serve stages pull from it to enable and configure functionality.
Both the build and serve stages read it to enable and configure functionality.

.env.defaults is ok to include. Because this file is committed to git.
But .env is not. If you add a secret to the Dockerfile, it can be excavated.
`.env.defaults` is ok to include.
This file is committed to git, but `.env` is not.
If you add a secret to the Dockerfile, it can be excavated.
While it's technically true that multi stage builds add a sort of security layer, it's not a best practice.
Leave them out and figure it out with your deploy provider.
Leave them out and look to your deploy provider for further configuration.

### The `api_build` stage

The `api_build` builds the api side.
The `api_build` stage builds the api side:

```Dockerfile
FROM base as api_build
Expand All @@ -185,23 +184,22 @@ COPY --chown=node:node api api
RUN yarn redwood build api
```

After the work we did in the base stage, building the api side amounts to copying in the api directory and running yarn redwood build api.

Remember not to put secrets here.
After the work we did in the base stage, building the api side amounts to copying in the api directory and running `yarn redwood build api`.

### The `api_serve` stage

The `api_serve` stage serves your GraphQL api and functions.
The `api_serve` stage serves your GraphQL api and functions:

```Dockerfile
FROM node:18-bookworm-slim as api_serve

RUN apt-get update && apt-get install -y \
openssl \
# python3 make gcc \
&& rm -rf /var/lib/apt/lists/*
```

We don't start from the base stage, but begin anew with the `node:18-bookworm-slim` image.
We don't start from the `base` stage, but begin anew with the `node:18-bookworm-slim` image.
Since this is a production stage, it's important for it to be as small as possible.
Docker's [multi-stage builds](https://docs.docker.com/build/building/multi-stage/) enables this.

Expand Down Expand Up @@ -231,7 +229,7 @@ This is a critical step for image size.
We don't use the regular `yarn install` command.
Using the [official workspaces plugin](https://github.com/yarnpkg/berry/tree/master/packages/plugin-workspace-tools)—which will be included by default in yarn v4—we "focus" on the api workspace, only installing its production dependencies.

The cache mount will be populated at this point from the install in the `base` stage, so the fetch step in the yarn install should fly by.
The cache mount will be populated at this point from the install in the `base` stage, so the fetch step should fly by.

```Dockerfile
COPY --chown=node:node redwood.toml .
Expand Down Expand Up @@ -259,12 +257,12 @@ Lastly, the default command is to start the api server using the bin from the `@
You can override this command if you have more specific needs.

Note that the Redwood CLI isn't available anymore.
To access the server bin, we have to find it's path in node_modules.
Though this is somewhat discouraged in modern yarn, since we're using the `node_modules` linker, it's in `node_modules/.bin`.
To access the server bin, we have to find it's path in `node_modules`.
Though this is somewhat discouraged in modern yarn, since we're using the `node-modules` node linker, it's in `node_modules/.bin`.

### The `web_build` stage

This `web_build` builds the web side.
This `web_build` builds the web side:

```Dockerfile
FROM base as web_build
Expand Down Expand Up @@ -338,7 +336,7 @@ Lastly, note that we use the shell form of `CMD` here for its variable expansion

### The `console` stage

The `console` stage is an optional stage for debugging.
The `console` stage is an optional stage for debugging:

```Dockerfile
FROM base as console
Expand Down Expand Up @@ -372,3 +370,84 @@ docker run --rm -it console /bin/bash

As the comment says, feel free to add more packages.
We intentionally kept them to a minimum in the base stage, but you shouldn't worry about the size of the image here.

## Troubleshooting

### Python

We tried to make the Dockerfile as lean as possible.
In some cases, that means we excluded a dependency your project needs.
And by far the most common is Python.

During a stage's `yarn install` step (`RUN ... yarn install`), if you see an error like the following:

```
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python Python is not set from command line or npm configuration
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python Python is not set from environment variable PYTHON
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python checking if "python3" can be used
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python - executable path is ""
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python - "" could not be run
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python checking if "python" can be used
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python - executable path is ""
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python - "" could not be run
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python **********************************************************
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python You need to install the latest version of Python.
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python you can try one of the following options:
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python - Use the switch --python="/path/to/pythonexecutable"
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python (accepted by both node-gyp and npm)
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python - Set the environment variable PYTHON
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python - Set the npm configuration variable python:
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python npm config set python "/path/to/pythonexecutable"
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python For more information consult the documentation at:
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python **********************************************************
➤ YN0000: │ bufferutil@npm:4.0.8 STDERR gyp ERR! find Python
```

It's because your project depends on Python and the image doesn't provide it.

It's easy to fix: just add `python3` and its dependencies (usually `make` and `gcc`):

```diff
FROM node:18-bookworm-slim as base

RUN apt-get update && apt-get install -y \
openssl \
+ python3 make gcc \
&& rm -rf /var/lib/apt/lists/*
```

Not sure why your project depends on Python? `yarn why` is your friend.
From the error message, we know `bufferutil` couldn't build.
But why do we have `bufferutil`?

```
yarn why bufferutil
└─ websocket@npm:1.0.34
└─ bufferutil@npm:4.0.8 (via npm:^4.0.1)
```

`websocket` needs `bufferutil`. But why do we have `websocket`?
Keep pulling the thread till you get to a top-level dependency:

```
yarn why websocket
└─ @supabase/realtime-js@npm:2.8.4
└─ websocket@npm:1.0.34 (via npm:^1.0.34)

yarn why @supabase/realtime-js
└─ @supabase/supabase-js@npm:2.38.4
└─ @supabase/realtime-js@npm:2.8.4 (via npm:^2.8.4)

yarn why @supabase/supabase-js
├─ api@workspace:api
│ └─ @supabase/supabase-js@npm:2.38.4 (via npm:^2.21.0)
│
└─ web@workspace:web
└─ @supabase/supabase-js@npm:2.38.4 (via npm:^2.21.0)
```

In this case, it looks like it's ultimately because of our auth provider, `@supabase/supabase-js`.
2 changes: 2 additions & 0 deletions packages/cli/src/commands/experimental/templates/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ FROM node:18-bookworm-slim as base

RUN apt-get update && apt-get install -y \
openssl \
# python3 make gcc \
&& rm -rf /var/lib/apt/lists/*

USER node
Expand Down Expand Up @@ -57,6 +58,7 @@ FROM node:18-bookworm-slim as api_serve

RUN apt-get update && apt-get install -y \
openssl \
# python3 make gcc \
&& rm -rf /var/lib/apt/lists/*

USER node
Expand Down
Loading