feat(docker): Create non-root `clp-user` and move env setup after flattening (fixes #1378, fixes #1379). #1413

junhaoliao · 2025-10-13T02:44:21Z

- Move USER and ENV directives after image flattening in clp-package Dockerfile (fixes #1379).
- Reduce layers using multi-line ENV (fixes #1378).

Description

This PR:

Moves the USER and ENV directives to after the image flattening step in the clp-package Dockerfile. This ensures that the environment variables and user settings are correctly applied to the final image, rather than being overridden or lost during the flattening process. (fixes Docker image flattening loses USER and ENV directives in clp-package Dockerfile #1379)
- Creates the clp-user user with a home directory and shell in the final image, ensuring that the user environment is properly set up.
- Sets the working directory to /opt/clp for the clp-user, ensuring that the user starts in the correct location when running a container from this image.
- Ensures that the PATH includes both /opt/clp/bin and /opt/clp/sbin, allowing access to all installed binaries.
Reduces the number of layers in the Docker image by combining multiple ENV directives into a single multi-line ENV directive. This helps to optimize the image size and improve build performance. (fixes Optimize ENV declarations in clp-package Dockerfile to reduce layers #1378)

Checklist

The PR satisfies the contribution guidelines.
This is a breaking change and that has been indicated in the PR title, OR this isn't a
breaking change.
Necessary docs have been updated, OR no docs need to be updated.

Validation performed

before

without the changes in the PR

junhao@ASUS-X870E:~/workspace/2-clp/tools/docker-images/clp-package$ ./build.sh
[+] Building 1.1s (12/12) FINISHED                                                                                                                                                                                  docker:default
...
junhao@ASUS-X870E:~/workspace/2-clp/tools/docker-images/clp-package$ docker run -it --rm "$(docker images --format '{{.Repository}}:{{.Tag}}' \
  | grep 'clp-package' | head -n1)" /bin/bash
root@563207a4425c:/# echo $SHELL
/bin/bash
root@563207a4425c:/# env
HOSTNAME=563207a4425c
PWD=/
HOME=/root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
TERM=xterm
SHLVL=1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env
root@563207a4425c:/# id
uid=0(root) gid=0(root) groups=0(root)

after

with the changes in the PR

junhao@ASUS-X870E:~/workspace/2-clp/tools/docker-images/clp-package$ ./build.sh 
[+] Building 5.3s (14/14) FINISHED                                                                                                                                                                                   docker:default
...
junhao@ASUS-X870E:~/workspace/2-clp/tools/docker-images/clp-package$ docker run -it --rm "$(docker images --format '{{.Repository}}:{{.Tag}}' \
  | grep 'clp-package' | head -n1)" /bin/bash
clp-user@6984500ebb62:~$ echo $SHELL
/bin/bash
clp-user@6984500ebb62:~$ env
HOSTNAME=6984500ebb62
PWD=/opt/clp
CLP_HOME=/opt/clp
HOME=/opt/clp
PYTHONPATH=/opt/clp/lib/python3/site-packages
TERM=xterm
USER=clp-user
SHLVL=1
PATH=/opt/clp/sbin:/opt/clp/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env
clp-user@6984500ebb62:~$ id
uid=1000(clp-user) gid=1000(clp-user) groups=1000(clp-user)

Observed that:

The image size remains the same: 827MB -> 827MB. (Not shown from above output.)
Number of layers changed: [+] Building 1.1s (12/12) -> Building 5.3s (14/14).
UID / GID were set as expected: uid=0(root) gid=0(root) groups=0(root) -> uid=1000(clp-user) gid=1000(clp-user) groups=1000(clp-user).
The PATH was set as expected: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -> PATH=/opt/clp/sbin:/opt/clp/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.
Env var LS_COLORS was lost, which is even better.
CLP_HOME is set as expected: CLP_HOME missing -> CLP_HOME=/opt/clp.
HOME is set as expected: HOME=/root -> HOME=/opt/clp.
PYTHONPATH is set as expected: PYTHONPATH missing -> PYTHONPATH=/opt/clp/lib/python3/site-packages.

Summary by CodeRabbit

Chores
- Moved runtime environment and user setup into the final image stage for a cleaner runtime.
- Consolidated environment variable declarations to simplify configuration.
- Ensured PATH includes system sbin alongside application binaries.
- Set a dedicated runtime user and working directory to standardize execution.
- Reduced image layering for a leaner final image and potentially faster pulls.

…lp-package Dockerfile (fixes y-scope#1379); Reduce layers using multi-line ENV (fixes y-scope#1378).

coderabbitai · 2025-10-13T02:44:47Z

Walkthrough

Moved CLP packaging runtime metadata and user creation from the build/base stage into the final scratch stage of tools/docker-images/clp-package/Dockerfile, copying only built artifacts into the final stage and defining CLP_HOME, PATH, PYTHONPATH, and USER there; consolidated ENV declarations into a multi-line block.

Changes

Cohort / File(s)	Summary of changes
Dockerfile stage/env adjustments `tools/docker-images/clp-package/Dockerfile`	Moved `ENV` and `USER` directives from the base stage to the final `FROM scratch` stage (after `COPY --from=base / /`), added user creation and `WORKDIR` in final stage, copied built package into `CLP_HOME`, updated `PATH` entries (bin and sbin) and `PYTHONPATH`, and consolidated multiple `ENV` lines into a single multi-line `ENV`. Removed the previous COPY from base to `/opt/clp` in the base stage.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    actor Builder as Docker Builder
    participant Base as Base Stage
    participant Final as Final Stage (scratch)
    participant Image as Resulting Image

    Builder->>Base: Build files & artifacts
    Note over Base: No final ENV/USER set here
    Builder->>Final: FROM scratch\nCOPY --from=base / /
    Final->>Final: create clp-user\ncopy package into CLP_HOME\nset WORKDIR
    Final->>Image: ENV CLP_HOME, PATH (bin,sbin), PYTHONPATH\nUSER clp-user
    Note over Image: ENV and USER persist in final image metadata

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related issues

Optimize clp-package Docker build to avoid intermediate copying #1358 — Related: both modify clp-package Dockerfile to change where built artifacts are copied and remove intermediate copy steps; this PR addresses that objective.

Possibly linkable linked issues

Docker image flattening loses USER and ENV directives in clp-package Dockerfile #1379 — Addressed: ENV and USER were moved to the final stage so their metadata persists after flattening.
Optimize ENV declarations in clp-package Dockerfile to reduce layers #1378 — Addressed: Multiple ENV statements were consolidated into a single multi-line ENV to reduce layers.

Pre-merge checks and finishing touches

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title succinctly summarizes the creation of a non-root `clp-user` and the relocation of environment setup to the final stage, clearly reflecting the primary changes in the Dockerfile without introducing unnecessary details.
Linked Issues Check	✅ Passed	The changes correctly move all ENV and USER directives into the final scratch stage and consolidate separate ENV statements into a single multi-line declaration, fulfilling the requirements of issue #1379 and reducing layer count per issue #1378.
Out of Scope Changes Check	✅ Passed	All modifications are confined to the clp-package Dockerfile to address the linked issues without any unrelated code or file changes, so no out-of-scope alterations are present.
Docstring Coverage	✅ Passed	No functions found in the changes. Docstring coverage check skipped.

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

junhaoliao · 2025-10-13T02:48:50Z

tools/docker-images/clp-package/Dockerfile

To fix #1410, we need to "Move USER and ENV directives after image flattening in clp-package Dockerfile"

junhaoliao · 2025-10-13T02:53:10Z

tools/docker-images/clp-package/Dockerfile

+ENV CLP_HOME="/opt/clp"
+ENV PATH="${CLP_HOME}/bin:${PATH}" \
+    PYTHONPATH="${CLP_HOME}/lib/python3/site-packages"
+ENV PATH="${CLP_HOME}/sbin:${PATH}"


We still have to keep three ENV directives because the later ENV settings depend on the former ones.

Technically, we could combine the PATH settings into a single line like PATH="${CLP_HOME}/sbin:${CLP_HOME}/bin:${PATH}". For readability, we previously agreed in feat(docker): Add container image containing the CLP package, for Docker Compose integration (resolves #1164). #1166 that we want to split them into separate lines. Although the aim is to cut down on number of layers, we still should keep the lines separated given the benefits of readability should outweigh the layer count reduction

sitaowang1998 · 2025-10-13T03:54:14Z

tools/docker-images/clp-package/Dockerfile

+    PYTHONPATH="${CLP_HOME}/lib/python3/site-packages"
+ENV PATH="${CLP_HOME}/sbin:${PATH}"
+
+USER 1000:1000


The messages of groups: cannot find name for group ID 1000 and I have no name! are a little scary. Could we find a way to make them disappear?

…t working directory.

…ckerfile.

coderabbitai

Actionable comments posted: 1

♻️ Duplicate comments (1)

tools/docker-images/clp-package/Dockerfile (1)
18-21: Restore one-variable-per-ENV instruction

We previously agreed (PR #1166) to keep each ENV directive focused on a single variable for readability and to avoid order-dependent surprises. Please split the combined ENV back into separate statements for PATH, PYTHONPATH, and USER, even if it costs an extra layer. Example:
-ENV PATH="${CLP_HOME}/bin:${PATH}"
-ENV PATH="${CLP_HOME}/sbin:${PATH}" \
-    PYTHONPATH="${CLP_HOME}/lib/python3/site-packages" \
-    USER="clp-user"
+ENV PATH="${CLP_HOME}/bin:${PATH}"
+ENV PATH="${CLP_HOME}/sbin:${PATH}"
+ENV PYTHONPATH="${CLP_HOME}/lib/python3/site-packages"
+ENV USER="clp-user"
Based on learnings.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7d18129 and 3ef8714.

📒 Files selected for processing (1)

tools/docker-images/clp-package/Dockerfile (1 hunks)

🧰 Additional context used

🧠 Learnings (2)

📓 Common learnings

Learnt from: junhaoliao
PR: y-scope/clp#1414
File: tools/docker-images/clp-package/Dockerfile:20-24
Timestamp: 2025-10-13T03:32:19.293Z
Learning: In the clp repository's Dockerfiles (e.g., tools/docker-images/clp-package/Dockerfile), ENV directives should be split into separate lines for readability rather than consolidated to reduce layer count. This is especially true for PATH modifications, as agreed upon in PR #1166. Later ENV settings may depend on earlier ones (e.g., referencing CLP_HOME).

📚 Learning: 2025-10-13T03:32:19.293Z

Learnt from: junhaoliao
PR: y-scope/clp#1414
File: tools/docker-images/clp-package/Dockerfile:20-24
Timestamp: 2025-10-13T03:32:19.293Z
Learning: In the clp repository's Dockerfiles (e.g., tools/docker-images/clp-package/Dockerfile), ENV directives should be split into separate lines for readability rather than consolidated to reduce layer count. This is especially true for PATH modifications, as agreed upon in PR #1166. Later ENV settings may depend on earlier ones (e.g., referencing CLP_HOME).

Applied to files:

tools/docker-images/clp-package/Dockerfile

🪛 Checkov (3.2.334)

tools/docker-images/clp-package/Dockerfile

[low] 1-25: Ensure that HEALTHCHECK instructions have been added to container images

(CKV_DOCKER_2)

coderabbitai · 2025-10-14T06:51:38Z

tools/docker-images/clp-package/Dockerfile

+COPY ./build/clp-package ${CLP_HOME}
+
+ENV PATH="${CLP_HOME}/bin:${PATH}"
+ENV PATH="${CLP_HOME}/sbin:${PATH}" \
+    PYTHONPATH="${CLP_HOME}/lib/python3/site-packages" \
+    USER="clp-user"
+
+RUN useradd --uid 1000 --shell /bin/bash --home-dir ${CLP_HOME} ${USER}
+USER ${USER}
+WORKDIR ${CLP_HOME}


⚠️ Potential issue | 🟠 Major

Home directory ownership remains root-owned

COPY ./build/clp-package ${CLP_HOME} leaves /opt/clp owned by root, and the subsequent useradd … --home-dir ${CLP_HOME} does not adjust ownership. When the container starts as clp-user, the user cannot write to their home (e.g., creating dotfiles or logs). Please ensure the home directory is owned by UID/GID 1000 before switching users—for example, by adding chown -R ${USER}:${USER} ${CLP_HOME} (or using COPY --chown=1000:1000 …) prior to the USER instruction.

🧰 Tools

🪛 Checkov (3.2.334)

[low] 1-25: Ensure that HEALTHCHECK instructions have been added to container images

(CKV_DOCKER_2)

🤖 Prompt for AI Agents

In tools/docker-images/clp-package/Dockerfile around lines 16 to 25, the COPY leaves ${CLP_HOME} owned by root so clp-user (UID 1000) cannot write to its home; before the USER instruction change ownership of the directory to UID/GID 1000 (either by using COPY --chown=1000:1000 ./build/clp-package ${CLP_HOME} or running chown -R ${USER}:${USER} ${CLP_HOME} in a RUN step) so the home directory and its contents are writable by clp-user.

fix(docker): Move USER and ENV directives after image flattening in c…

7d18129

…lp-package Dockerfile (fixes y-scope#1379); Reduce layers using multi-line ENV (fixes y-scope#1378).

junhaoliao requested a review from a team as a code owner October 13, 2025 02:44

junhaoliao requested review from kirkrodrigues and sitaowang1998 October 13, 2025 02:46

junhaoliao commented Oct 13, 2025

View reviewed changes

junhaoliao mentioned this pull request Oct 13, 2025

fix(docker): Include libmariadbcpp.so (required by spider_scheduler and spider_worker) in the clp-package image; Set mariadb-connector-cpp build type to Release (fixes #1410). #1414

Draft

3 tasks

Merge branch 'main' into fix-clp-package-docker

b193c2f

sitaowang1998 requested changes Oct 13, 2025

View reviewed changes

junhaoliao marked this pull request as draft October 14, 2025 05:23

junhaoliao added 3 commits October 14, 2025 01:51

feat(docker): Update clp-package Dockerfile; add non-root user and se…

3195ef0

…t working directory.

fix(docker): Reorder and restructure ENV directives in clp-package Do…

18a1b18

…ckerfile.

add \

3ef8714

junhaoliao changed the title ~~fix(docker): Create non-root clp-user and move env setup after flattening (fixes #1378, fixes #1379).~~ feat(docker): Create non-root clp-user and move env setup after flattening (fixes #1378, fixes #1379). Oct 14, 2025

junhaoliao requested a review from sitaowang1998 October 14, 2025 06:48

junhaoliao marked this pull request as ready for review October 14, 2025 06:48

coderabbitai bot reviewed Oct 14, 2025

View reviewed changes

Merge branch 'main' into fix-clp-package-docker

1b2d4cb

junhaoliao mentioned this pull request Oct 16, 2025

feat(deployment)!: Migrate package orchestration to Docker Compose (resolves #1177). #1178

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(docker): Create non-root `clp-user` and move env setup after flattening (fixes #1378, fixes #1379). #1413

feat(docker): Create non-root `clp-user` and move env setup after flattening (fixes #1378, fixes #1379). #1413

Uh oh!

junhaoliao commented Oct 13, 2025 •

edited by coderabbitai bot

Loading

Uh oh!

coderabbitai bot commented Oct 13, 2025 •

edited

Loading

Uh oh!

junhaoliao Oct 13, 2025

Uh oh!

junhaoliao Oct 13, 2025

Uh oh!

sitaowang1998 Oct 13, 2025

Uh oh!

coderabbitai bot left a comment

Uh oh!

coderabbitai bot Oct 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(docker): Create non-root clp-user and move env setup after flattening (fixes #1378, fixes #1379). #1413

Are you sure you want to change the base?

feat(docker): Create non-root clp-user and move env setup after flattening (fixes #1378, fixes #1379). #1413

Uh oh!

Conversation

junhaoliao commented Oct 13, 2025 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Validation performed

before

after

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related issues

Possibly linkable linked issues

Pre-merge checks and finishing touches

Uh oh!

junhaoliao Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

junhaoliao Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

sitaowang1998 Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(docker): Create non-root `clp-user` and move env setup after flattening (fixes #1378, fixes #1379). #1413

feat(docker): Create non-root `clp-user` and move env setup after flattening (fixes #1378, fixes #1379). #1413

junhaoliao commented Oct 13, 2025 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Oct 13, 2025 •

edited

Loading