Add CPU only to compose script #3365

jbkyang-nvi · 2021-09-14T00:34:58Z

Enable compose script to compose cpu-only containers. Fixed upstream-version confusion to ensure compose script works for all branches

…ctured evn variable checking logic

…stream version to use actual upstream version in build.py

deadeyegoodwin · 2021-09-14T15:57:59Z

build.py

@@ -760,6 +761,7 @@ def dockerfile_prepare_container_linux(argmap, backends, enable_gpu):
 ENV TF_ADJUST_SATURATION_FUSED  1
 ENV TF_ENABLE_WINOGRAD_NONFUSED 1
 ENV TF_AUTOTUNE_THRESHOLD       2
+ENV TRITON_GPU_ENABLED_BUILD    {gpu_enabled}        


TRITON_SERVER_GPU_ENABLED

deadeyegoodwin · 2021-09-14T16:07:16Z

compose.py

@@ -150,41 +151,79 @@ def create_argmap(images):
    # Extract information from upstream build and create map other functions can
    # use
    upstreamDockerImage = images["full"]
+    minDockerImage = images["min"]


use lower-case underscores style for variables
"upstreamDockerImage" confuses upstream meaning. Use full_docker_image

deadeyegoodwin · 2021-09-14T16:07:41Z

compose.py

+    minDockerImage = images["min"]
+    enable_gpu = FLAGS.enable_gpu
+    # Docker inspect enviroment variables
+    baseRunArgs = ['docker', 'inspect', '-f']


variable naming style

deadeyegoodwin · 2021-09-14T16:10:35Z

docs/compose.md

@@ -54,7 +63,44 @@ will provide a container `tritonserver` locally. You can access the container wi
 $ docker run -it tritonserver:latest
 ```

-Note: If `compose.py` is run on release versions `r21.08` and older, the resulting container will have DCGM version 2.2.3 installed. This may result in different GPU statistic reporting behavior.
+Note: If `compose.py` is run on release versions `r21.08` and older, 


"and earlier"

deadeyegoodwin · 2021-09-14T16:10:58Z

docs/compose.md

+### Compose a specific version of Triton
+
+`compose.py` requires two containers to build, a `min` container which is the 
+base the compose container is build from and a `full` container where the script


build -> built

CoderHam · 2021-09-14T17:33:56Z

compose.py

+        gpu_enabled = False if e == None else True
+        fail_if(
+            not gpu_enabled,
+            '\'enable-gpu\' flag added but min container provided does not have CUDA installed'


Instead of added say specified.

Please address changes here and in other places

were you looking at a old version? I think I addressed all your comments

Yes, I was. Looks like you address all the changes

CoderHam · 2021-09-14T17:37:02Z

compose.py

+                    "nvcr.io/nvidia/tritonserver:{}-cpu-only-py3".format(
+                        FLAGS.container_version),
+                "min":
+                    "ubuntu:20.04".format(FLAGS.container_version)


Why are you adding .format(FLAGS.container_version)?

CoderHam · 2021-09-14T17:37:39Z

docs/compose.md

-`compose.py` provides `--backend`, `--repoagent` options that allow you to specify which backends and repository agents to include in the custom image. The `--enable-gpu` flag indicates that you want to create an image that supports NVIDIA GPUs. For example, the following creates a new docker image that contains only the TensorFlow 1 and TensorFlow 2 backends and the checksum repository agent.
+The `compose.py` script can be found in the [server repository](https://github.com/triton-inference-server/server).
+Simply clone the repository and run `compose.py` to create a custom container. 
+Note created container version will depend on the branch that was cloned. 


Note: or Note that the

CoderHam · 2021-09-14T17:38:09Z

docs/compose.md

+`compose.py` provides `--backend`, `--repoagent` options that allow you to 
+specify which backends and repository agents to include in the custom image. 
+The `--enable-gpu` flag indicates that you want to create an image that supports
+ NVIDIA GPUs. For example, the following creates a new docker image that 


extra space at start

CoderHam · 2021-09-14T17:39:38Z

docs/compose.md

+### Compose a specific version of Triton
+
+`compose.py` requires two containers to build, a `min` container which is the 
+base the compose container is build from and a `full` container where the script


from which the script will extract components.

CoderHam · 2021-09-14T17:40:12Z

docs/compose.md

+python3 compose.py --backend tensorflow1 --repoagent checksum --enable-gpu
+```
+on branch [r21.08](https://github.com/triton-inference-server/server/tree/r21.08) pulls:
+-  `min` container `nvcr.io/nvidia/tritonserver:21.08-py3-min` 


extra space at start

CoderHam · 2021-09-14T17:40:43Z

docs/compose.md

+
+### CPU only container composition
+
+To compose a container that is build for only cpu usage, simply remove the 


CoderHam · 2021-09-14T17:41:15Z

docs/compose.md

+
+To compose a container that is build for only cpu usage, simply remove the 
+`--enable-gpu` flag when running `compose.py`. 
+This will build a container from `ubuntu:20.04` docker and use 


using ubuntu:20.04 as base

CoderHam · 2021-09-14T17:41:25Z

docs/compose.md

+This will build a container from `ubuntu:20.04` docker and use 
+backends/repoagents from `nvcr.io/nvidia/tritonserver:<upstream-container-version>-cpu-only-py3`.
+Note: 
+1. If composing a CPU only container, both `min` and `full` containers should be built for CPU only and not have CUDA installed.


When composing

CoderHam · 2021-09-14T21:52:09Z

compose.py

+        gpu_enabled = False if e == None else True
+        fail_if(
+            not gpu_enabled,
+            '\'enable-gpu\' flag added but min container provided does not have CUDA installed'


Please address changes here and in other places

jbkyang-nvi added 5 commits September 9, 2021 15:49

create container composer for cpu-only build

f369199

fix true/false mismatch, added check on min container and also restru…

9a18265

…ctured evn variable checking logic

added documentation

6722697

added documentation

0d2f761

added more checking for gpu-enabled compose capability and updated up…

506ea14

…stream version to use actual upstream version in build.py

jbkyang-nvi requested review from CoderHam, deadeyegoodwin and GuanLuo and removed request for CoderHam September 14, 2021 00:35

deadeyegoodwin suggested changes Sep 14, 2021

View reviewed changes

CoderHam suggested changes Sep 14, 2021

View reviewed changes

jbkyang-nvi force-pushed the kyang-cpu-only-composer branch from 48d5990 to 2921195 Compare September 14, 2021 18:36

fixed naming according to conventions and updated wording for readme

8b6a9e5

jbkyang-nvi force-pushed the kyang-cpu-only-composer branch from 2921195 to 8b6a9e5 Compare September 14, 2021 19:03

jbkyang-nvi requested review from deadeyegoodwin and CoderHam September 14, 2021 19:03

deadeyegoodwin approved these changes Sep 14, 2021

View reviewed changes

CoderHam suggested changes Sep 14, 2021

View reviewed changes

jbkyang-nvi requested a review from CoderHam September 14, 2021 22:08

CoderHam approved these changes Sep 14, 2021

View reviewed changes

jbkyang-nvi merged commit 0801098 into main Sep 15, 2021

jbkyang-nvi deleted the kyang-cpu-only-composer branch September 15, 2021 17:05

tanmayv25 mentioned this pull request Apr 7, 2023

Updating container versions for Triton 23.04 release #5611

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CPU only to compose script #3365

Add CPU only to compose script #3365

jbkyang-nvi commented Sep 14, 2021

deadeyegoodwin Sep 14, 2021

deadeyegoodwin Sep 14, 2021

deadeyegoodwin Sep 14, 2021

deadeyegoodwin Sep 14, 2021

deadeyegoodwin Sep 14, 2021

CoderHam Sep 14, 2021

CoderHam Sep 14, 2021

jbkyang-nvi Sep 14, 2021

CoderHam Sep 14, 2021

CoderHam Sep 14, 2021

CoderHam Sep 14, 2021

CoderHam Sep 14, 2021

CoderHam Sep 14, 2021

CoderHam Sep 14, 2021

CoderHam Sep 14, 2021

CoderHam Sep 14, 2021

CoderHam Sep 14, 2021

CoderHam Sep 14, 2021


		### CPU only container composition

		To compose a container that is build for only cpu usage, simply remove the

Add CPU only to compose script #3365

Add CPU only to compose script #3365

Conversation

jbkyang-nvi commented Sep 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment