[Bug Report] Docker container always rebuilds upon "start"

[vassil-atn]

Describe the bug

Whenever the docker container is started (with ./container.py start) it gets rebuilt despite no changes happening to the Dockerfile which takes about two minutes each time. Some parts do get loaded from cache but not all of them, which still takes a significant amount of time to build. Is this intended or a bug?

Steps to reproduce

$ ./docker/container.py start
[INFO] Using container profile: base
[INFO] X11 Forwarding is configured as '1' in '.container.cfg'.
    To disable X11 forwarding, set 'X11_FORWARDING_ENABLED=0' in '.container.cfg'.
[INFO] Building the docker image and starting the container 'isaac-lab-base' in the background...

[+] Building 94.9s (19/19) FINISHED                                                                                      docker:default
 => [isaac-lab-base internal] load build definition from Dockerfile.base                                                           0.0s
 => => transferring dockerfile: 4.53kB                                                                                             0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG nvcr.io/nvidia/isaac-sim:${ISAACSIM_VERSION_ARG} results in empty or inv  0.0s
 => [isaac-lab-base internal] load metadata for nvcr.io/nvidia/isaac-sim:4.2.0                                                     0.7s
 => [isaac-lab-base internal] load .dockerignore                                                                                   0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => [isaac-lab-base base  1/13] FROM nvcr.io/nvidia/isaac-sim:4.2.0@sha256:X  0.0s
 => [isaac-lab-base internal] load build context                                                                                   0.3s
 => => transferring context: 1.08MB                                                                                                0.3s
 => CACHED [isaac-lab-base base  2/13] RUN --mount=type=cache,target=/var/cache/apt     apt-get update && apt-get install -y --no  0.0s
 => [isaac-lab-base base  3/13] COPY ../ /workspace/isaaclab                                                                      12.8s
 => [isaac-lab-base base  4/13] RUN ln -sf /isaac-sim /workspace/isaaclab/_isaac_sim                                               0.2s
 => [isaac-lab-base base  5/13] RUN --mount=type=cache,target=/var/cache/apt     /workspace/isaaclab/isaaclab.sh -p /workspace/is  0.4s
 => [isaac-lab-base base  6/13] RUN mkdir -p /isaac-sim/kit/cache &&     mkdir -p /root/.cache/ov &&     mkdir -p /root/.cache/pi  0.3s
 => [isaac-lab-base base  7/13] RUN touch /bin/nvidia-smi &&     touch /bin/nvidia-debugdump &&     touch /bin/nvidia-persistence  0.3s 
 => [isaac-lab-base base  8/13] RUN --mount=type=cache,target=/root/.cache/pip     /workspace/isaaclab/isaaclab.sh --install      53.7s 
 => [isaac-lab-base base  9/13] RUN cd /workspace/isaaclab/rsl_rl-algorithms && /workspace/isaaclab/_isaac_sim/python.sh -m pip i  5.0s 
 => [isaac-lab-base base 10/13] RUN cd /workspace/isaaclab/PMTG && /workspace/isaaclab/_isaac_sim/python.sh -m pip install -e .    3.5s 
 => [isaac-lab-base base 11/13] RUN echo "export ISAACLAB_PATH=/workspace/isaaclab" >> ${HOME}/.bashrc &&     echo "alias isaacla  0.2s 
 => [isaac-lab-base base 12/13] RUN echo "machine api.wandb.ai login user password X" >> $  0.2s 
 => [isaac-lab-base base 13/13] WORKDIR /workspace/isaaclab                                                                        0.1s 
 => [isaac-lab-base] exporting to image                                                                                           17.0s 
 => => exporting layers                                                                                                           16.9s 
 => => writing image sha256:X                                       0.0s 
 => => naming to docker.io/library/isaac-lab-base                                                                                  0.0s
 => [isaac-lab-base] resolving provenance for metadata file                                                                        0.0s
[+] Building 0.8s (19/19) FINISHED                                                                                       docker:default
 => [isaac-lab-base internal] load build definition from Dockerfile.base                                                           0.0s
 => => transferring dockerfile: 4.53kB                                                                                             0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG nvcr.io/nvidia/isaac-sim:${ISAACSIM_VERSION_ARG} results in empty or inv  0.0s
 => [isaac-lab-base internal] load metadata for nvcr.io/nvidia/isaac-sim:4.2.0                                                     0.3s
 => [isaac-lab-base internal] load .dockerignore                                                                                   0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => [isaac-lab-base base  1/13] FROM nvcr.io/nvidia/isaac-sim:4.2.0@sha256:X  0.0s
 => [isaac-lab-base internal] load build context                                                                                   0.3s
 => => transferring context: 1.04MB                                                                                                0.3s
 => CACHED [isaac-lab-base base  2/13] RUN --mount=type=cache,target=/var/cache/apt     apt-get update && apt-get install -y --no  0.0s
 => CACHED [isaac-lab-base base  3/13] COPY ../ /workspace/isaaclab                                                                0.0s
 => CACHED [isaac-lab-base base  4/13] RUN ln -sf /isaac-sim /workspace/isaaclab/_isaac_sim                                        0.0s
 => CACHED [isaac-lab-base base  5/13] RUN --mount=type=cache,target=/var/cache/apt     /workspace/isaaclab/isaaclab.sh -p /works  0.0s
 => CACHED [isaac-lab-base base  6/13] RUN mkdir -p /isaac-sim/kit/cache &&     mkdir -p /root/.cache/ov &&     mkdir -p /root/.c  0.0s
 => CACHED [isaac-lab-base base  7/13] RUN touch /bin/nvidia-smi &&     touch /bin/nvidia-debugdump &&     touch /bin/nvidia-pers  0.0s
 => CACHED [isaac-lab-base base  8/13] RUN --mount=type=cache,target=/root/.cache/pip     /workspace/isaaclab/isaaclab.sh --insta  0.0s
 => CACHED [isaac-lab-base base  9/13] RUN cd /workspace/isaaclab/rsl_rl-algorithms && /workspace/isaaclab/_isaac_sim/python.sh -  0.0s
 => CACHED [isaac-lab-base base 10/13] RUN cd /workspace/isaaclab/PMTG && /workspace/isaaclab/_isaac_sim/python.sh -m pip install  0.0s
 => CACHED [isaac-lab-base base 11/13] RUN echo "export ISAACLAB_PATH=/workspace/isaaclab" >> ${HOME}/.bashrc &&     echo "alias   0.0s
 => CACHED [isaac-lab-base base 12/13] RUN echo "machine api.wandb.ai login user password X  0.0s
 => CACHED [isaac-lab-base base 13/13] WORKDIR /workspace/isaaclab                                                                 0.0s
 => [isaac-lab-base] exporting to image                                                                                            0.0s
 => => exporting layers                                                                                                            0.0s
 => => writing image sha256:X                                       0.0s
 => => naming to docker.io/library/isaac-lab-base                                                                                  0.0s
 => [isaac-lab-base] resolving provenance for metadata file                                                                        0.0s
[+] Running 1/1

System Info

Describe the characteristic of your environment:

• Commit:
• Isaac Sim Version: 4.2.0
• OS: Ubuntu 20.04
• GPU: RTX 4090
• CUDA: 12,4
• GPU Driver: 550.120

Additional context

Add any other context about the problem here.

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

• Fixed if bug.

[RandomOakForest]

This is due to a relatively early COPY in the Dockerfile which invalidates the build cache and forces a new rebuild of the layers that follow. We will review. Thanks.

[scottrbrtsn]

I tried to reproduce but could not.
Below is output from running with changes triggering a build, stopping, then rerunning. On rerun, the cache was used for the COPY.

COPY should hash all copied files. if this hash is different for any file it will rerun.

Are you sure no files are changed?

---

Running with a change in a .py which triggers the build:

9:34:26  isaac  ~  projects  …  .venv  IsaacLab  docker   main  ✎ 
python container.py start
[INFO] Using container profile: base
[INFO] X11 Forwarding is configured as '1' in '.container.cfg'.
        To disable X11 forwarding, set 'X11_FORWARDING_ENABLED=0' in '.container.cfg'.
[INFO] Building the docker image and starting the container 'isaac-lab-base' in the background...

[+] Building 69.0s (16/16) FINISHED                                                                            docker:default
 => [isaac-lab-base internal] load build definition from Dockerfile.base                                                 0.0s
 => => transferring dockerfile: 4.02kB                                                                                   0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG ${ISAACSIM_BASE_IMAGE_ARG}:${ISAACSIM_VERSION_ARG} results in   0.0s
 => [isaac-lab-base internal] load metadata for nvcr.io/nvidia/isaac-sim:4.2.0                                           0.1s
 => [isaac-lab-base internal] load .dockerignore                                                                         0.0s
 => => transferring context: 378B                                                                                        0.0s
 => [isaac-lab-base base  1/10] FROM nvcr.io/nvidia/isaac-sim:4.2.0@sha256:b606646df3aab3f38ba655ce96cdf8f45a13bb167f01  0.0s
 => [isaac-lab-base internal] load build context                                                                         0.1s
 => => transferring context: 97.39kB                                                                                     0.1s
 => CACHED [isaac-lab-base base  2/10] RUN --mount=type=cache,target=/var/cache/apt     apt-get update && apt-get insta  0.0s
 => [isaac-lab-base base  3/10] COPY ../ /workspace/isaaclab                                                             0.1s
 => [isaac-lab-base base  4/10] RUN ln -sf /isaac-sim /workspace/isaaclab/_isaac_sim                                     0.3s
 => [isaac-lab-base base  5/10] RUN --mount=type=cache,target=/var/cache/apt     /workspace/isaaclab/isaaclab.sh -p /wo  0.6s
 => [isaac-lab-base base  6/10] RUN mkdir -p /isaac-sim/kit/cache &&     mkdir -p /root/.cache/ov &&     mkdir -p /root  0.4s
 => [isaac-lab-base base  7/10] RUN touch /bin/nvidia-smi &&     touch /bin/nvidia-debugdump &&     touch /bin/nvidia-p  0.3s
 => [isaac-lab-base base  8/10] RUN --mount=type=cache,target=/root/.cache/pip     /workspace/isaaclab/isaaclab.sh --i  60.7s
 => [isaac-lab-base base  9/10] RUN echo "export ISAACLAB_PATH=/workspace/isaaclab" >> ${HOME}/.bashrc &&     echo "ali  0.5s
 => [isaac-lab-base base 10/10] WORKDIR /workspace/isaaclab                                                              0.1s
 => [isaac-lab-base] exporting to image                                                                                  5.7s
 => => exporting layers                                                                                                  5.6s
 => => writing image sha256:aea5287f7dbfec301767c393090998c45857867ee089e5c99d55c9c660d84c1a                             0.0s
 => => naming to docker.io/library/isaac-lab-base                                                                        0.0s
 => [isaac-lab-base] resolving provenance for metadata file                                                              0.0s
[+] Building 0.3s (16/16) FINISHED                                                                             docker:default
 => [isaac-lab-base internal] load build definition from Dockerfile.base                                                 0.0s
 => => transferring dockerfile: 4.02kB                                                                                   0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG ${ISAACSIM_BASE_IMAGE_ARG}:${ISAACSIM_VERSION_ARG} results in   0.0s
 => [isaac-lab-base internal] load metadata for nvcr.io/nvidia/isaac-sim:4.2.0                                           0.2s
 => [isaac-lab-base internal] load .dockerignore                                                                         0.0s
 => => transferring context: 378B                                                                                        0.0s
 => [isaac-lab-base base  1/10] FROM nvcr.io/nvidia/isaac-sim:4.2.0@sha256:b606646df3aab3f38ba655ce96cdf8f45a13bb167f01  0.0s
 => [isaac-lab-base internal] load build context                                                                         0.1s
 => => transferring context: 95.97kB                                                                                     0.0s
 => CACHED [isaac-lab-base base  2/10] RUN --mount=type=cache,target=/var/cache/apt     apt-get update && apt-get insta  0.0s
 => CACHED [isaac-lab-base base  3/10] COPY ../ /workspace/isaaclab                                                      0.0s
 => CACHED [isaac-lab-base base  4/10] RUN ln -sf /isaac-sim /workspace/isaaclab/_isaac_sim                              0.0s
 => CACHED [isaac-lab-base base  5/10] RUN --mount=type=cache,target=/var/cache/apt     /workspace/isaaclab/isaaclab.sh  0.0s
 => CACHED [isaac-lab-base base  6/10] RUN mkdir -p /isaac-sim/kit/cache &&     mkdir -p /root/.cache/ov &&     mkdir -  0.0s
 => CACHED [isaac-lab-base base  7/10] RUN touch /bin/nvidia-smi &&     touch /bin/nvidia-debugdump &&     touch /bin/n  0.0s
 => CACHED [isaac-lab-base base  8/10] RUN --mount=type=cache,target=/root/.cache/pip     /workspace/isaaclab/isaaclab.  0.0s
 => CACHED [isaac-lab-base base  9/10] RUN echo "export ISAACLAB_PATH=/workspace/isaaclab" >> ${HOME}/.bashrc &&     ec  0.0s
 => CACHED [isaac-lab-base base 10/10] WORKDIR /workspace/isaaclab                                                       0.0s
 => [isaac-lab-base] exporting to image                                                                                  0.0s
 => => exporting layers                                                                                                  0.0s
 => => writing image sha256:aea5287f7dbfec301767c393090998c45857867ee089e5c99d55c9c660d84c1a                             0.0s
 => => naming to docker.io/library/isaac-lab-base                                                                        0.0s
 => [isaac-lab-base] resolving provenance for metadata file                                                              0.0s
[+] Running 1/1
 ✔ Container isaac-lab-base  Started                                                                                     2.0s

stop the container:

 9:35:43  isaac  ~  projects  …  .venv  IsaacLab  docker   main  ✎ 
 %  python container.py stop
[INFO] Using container profile: base
[INFO] Stopping the launched docker container 'isaac-lab-base'...

[+] Running 1/1
 ✔ Container isaac-lab-base  Removed                                                                                    10.7s
[INFO] Removing temporary Isaac Lab '.xauth' file: /tmp/tmp.eHKXkstkVK/tmp.H8nblZP05q.xauth.

no changes, rerun the container:

9:35:56   isaac  ~  projects  …  .venv  IsaacLab  docker   main  ✎ 
 %  python container.py start
[INFO] Using container profile: base
[INFO] X11 Forwarding is configured as '1' in '.container.cfg'.
        To disable X11 forwarding, set 'X11_FORWARDING_ENABLED=0' in '.container.cfg'.
[INFO] Building the docker image and starting the container 'isaac-lab-base' in the background...

[+] Building 0.3s (16/16) FINISHED                                                                             docker:default
 => [isaac-lab-base internal] load build definition from Dockerfile.base                                                 0.0s
 => => transferring dockerfile: 4.02kB                                                                                   0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG ${ISAACSIM_BASE_IMAGE_ARG}:${ISAACSIM_VERSION_ARG} results in   0.0s
 => [isaac-lab-base internal] load metadata for nvcr.io/nvidia/isaac-sim:4.2.0                                           0.1s
 => [isaac-lab-base internal] load .dockerignore                                                                         0.0s
 => => transferring context: 378B                                                                                        0.0s
 => [isaac-lab-base base  1/10] FROM nvcr.io/nvidia/isaac-sim:4.2.0@sha256:b606646df3aab3f38ba655ce96cdf8f45a13bb167f01  0.0s
 => [isaac-lab-base internal] load build context                                                                         0.1s
 => => transferring context: 95.97kB                                                                                     0.1s
 => CACHED [isaac-lab-base base  2/10] RUN --mount=type=cache,target=/var/cache/apt     apt-get update && apt-get insta  0.0s
 => CACHED [isaac-lab-base base  3/10] COPY ../ /workspace/isaaclab                                                      0.0s
 => CACHED [isaac-lab-base base  4/10] RUN ln -sf /isaac-sim /workspace/isaaclab/_isaac_sim                              0.0s
 => CACHED [isaac-lab-base base  5/10] RUN --mount=type=cache,target=/var/cache/apt     /workspace/isaaclab/isaaclab.sh  0.0s
 => CACHED [isaac-lab-base base  6/10] RUN mkdir -p /isaac-sim/kit/cache &&     mkdir -p /root/.cache/ov &&     mkdir -  0.0s
 => CACHED [isaac-lab-base base  7/10] RUN touch /bin/nvidia-smi &&     touch /bin/nvidia-debugdump &&     touch /bin/n  0.0s
 => CACHED [isaac-lab-base base  8/10] RUN --mount=type=cache,target=/root/.cache/pip     /workspace/isaaclab/isaaclab.  0.0s
 => CACHED [isaac-lab-base base  9/10] RUN echo "export ISAACLAB_PATH=/workspace/isaaclab" >> ${HOME}/.bashrc &&     ec  0.0s
 => CACHED [isaac-lab-base base 10/10] WORKDIR /workspace/isaaclab                                                       0.0s
 => [isaac-lab-base] exporting to image                                                                                  0.0s
 => => exporting layers                                                                                                  0.0s
 => => writing image sha256:aea5287f7dbfec301767c393090998c45857867ee089e5c99d55c9c660d84c1a                             0.0s
 => => naming to docker.io/library/isaac-lab-base                                                                        0.0s
 => [isaac-lab-base] resolving provenance for metadata file                                                              0.0s
[+] Building 0.3s (16/16) FINISHED                                                                             docker:default
 => [isaac-lab-base internal] load build definition from Dockerfile.base                                                 0.0s
 => => transferring dockerfile: 4.02kB                                                                                   0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG ${ISAACSIM_BASE_IMAGE_ARG}:${ISAACSIM_VERSION_ARG} results in   0.0s
 => [isaac-lab-base internal] load metadata for nvcr.io/nvidia/isaac-sim:4.2.0                                           0.1s
 => [isaac-lab-base internal] load .dockerignore                                                                         0.0s
 => => transferring context: 378B                                                                                        0.0s
 => [isaac-lab-base base  1/10] FROM nvcr.io/nvidia/isaac-sim:4.2.0@sha256:b606646df3aab3f38ba655ce96cdf8f45a13bb167f01  0.0s
 => [isaac-lab-base internal] load build context                                                                         0.1s
 => => transferring context: 95.97kB                                                                                     0.1s
 => CACHED [isaac-lab-base base  2/10] RUN --mount=type=cache,target=/var/cache/apt     apt-get update && apt-get insta  0.0s
 => CACHED [isaac-lab-base base  3/10] COPY ../ /workspace/isaaclab                                                      0.0s
 => CACHED [isaac-lab-base base  4/10] RUN ln -sf /isaac-sim /workspace/isaaclab/_isaac_sim                              0.0s
 => CACHED [isaac-lab-base base  5/10] RUN --mount=type=cache,target=/var/cache/apt     /workspace/isaaclab/isaaclab.sh  0.0s
 => CACHED [isaac-lab-base base  6/10] RUN mkdir -p /isaac-sim/kit/cache &&     mkdir -p /root/.cache/ov &&     mkdir -  0.0s
 => CACHED [isaac-lab-base base  7/10] RUN touch /bin/nvidia-smi &&     touch /bin/nvidia-debugdump &&     touch /bin/n  0.0s
 => CACHED [isaac-lab-base base  8/10] RUN --mount=type=cache,target=/root/.cache/pip     /workspace/isaaclab/isaaclab.  0.0s
 => CACHED [isaac-lab-base base  9/10] RUN echo "export ISAACLAB_PATH=/workspace/isaaclab" >> ${HOME}/.bashrc &&     ec  0.0s
 => CACHED [isaac-lab-base base 10/10] WORKDIR /workspace/isaaclab                                                       0.0s
 => [isaac-lab-base] exporting to image                                                                                  0.0s
 => => exporting layers                                                                                                  0.0s
 => => writing image sha256:aea5287f7dbfec301767c393090998c45857867ee089e5c99d55c9c660d84c1a                             0.0s
 => => naming to docker.io/library/isaac-lab-base                                                                        0.0s
 => [isaac-lab-base] resolving provenance for metadata file                                                              0.0s
[+] Running 1/1
 ✔ Container isaac-lab-base  Started                                                                                     0.5s

[vassil-atn]

Yeah I think the COPY is the culprit, since my development environments were under extensions as part of the main Isaac Lab repository.

I was actually not aware of the Isaac Lab Extensions Template (https://github.com/isaac-sim/IsaacLabExtensionTemplate/tree/main) which seems a much better way of developing additional environments, etc. in Isaac Lab. Switching to the Extension template basically avoids this issue all-together, and seems to be the recommended approach (that I somehow missed) so I'd be happy for this issue to be closed.