Custom container composition script #3039

jbkyang-nvi · 2021-06-23T17:02:57Z

Make compose.py so users can compose custom containers with different backends themselves.
Gitlab test PR: https://gitlab-master.nvidia.com/dl/dgx/tritonserver/-/merge_requests/327

compose.py

qa/L0_infer/infer_test.py

deadeyegoodwin · 2021-06-23T18:45:12Z

compose.py

@@ -0,0 +1,305 @@
+#!/usr/bin/env python3
+# Copyright (c) 2020-2021, NVIDIA CORPORATION & AFFILIATES. All rights reserved.


Probably need to remove the "(c)" so the formatter doesn't wrap line?

It doesn't wrap line for # started comments. I've removed the (c) for consistency

deadeyegoodwin · 2021-06-23T18:52:43Z

compose.py

+                          required=False,
+                          help='Enable verbose output.')
+    parser.add_argument(
+        '--build-name',


--output-name the name of the generated container, default is 'tritonserver'

Note that there is no "building" happening so we should avoid using that terminology as it can be confusing with a source build performed by build.py.

deadeyegoodwin · 2021-06-23T18:54:03Z

compose.py

+        'Build name. Default is "tritonserver_build".'
+    )
+    parser.add_argument(
+        '--build-dir',


--work-dir we don't do any clones or builds so the description should say something about "generated dockerfiles are placed here". Should default to the current directory.

deadeyegoodwin · 2021-06-23T18:54:58Z

compose.py

+        FLAGS.upstream_container_version = TRITON_VERSION_MAP[
+            FLAGS.version][1]
+
+    dockerfile_name = 'Dockerfile.buildbase'


Dockerfile.compose

deadeyegoodwin · 2021-06-23T19:03:15Z

compose.py

+    with open(os.path.join(ddir, dockerfile_name), "a") as dfile:
+        dfile.write(df)
+
+### create build container


remove build terminology

deadeyegoodwin · 2021-06-23T19:50:03Z

docs/compose.md

+nvcr.io/nvidia/tritonserver:<xx.yy>-py3-min
+nvcr.io/nvidia/tritonserver:<xx.yy>-py3
+```
+to build the new custom container. The resulting container version will be `xx.yy` as it inherits from the pre-mentioned builds.


"new custom Triton Docker image".

I would drop "The resulting ..." sentence

deadeyegoodwin · 2021-06-23T19:51:00Z

docs/compose.md

+```
+to build the new custom container. The resulting container version will be `xx.yy` as it inherits from the pre-mentioned builds.
+
+### Customizing backends and repo-agents


I would not create a new section here but just continue the previous paragraph...
"compose.py provides --backend and --repoagent options that allow you to specify which backends and repository agents you want to include in the custom image. The --enable-gpu flag indicates that you want to create an image that supports NVIDIA GPUs. For example, the following creates a new docker image that contains only the TensorFlow 1 and TensorFlow 2 backends and the checksum repository agent.

deadeyegoodwin · 2021-06-23T19:51:19Z

docs/compose.md

+
+Example:
+```
+python3 compose.py --build-dir . --backend tensorflow1 --backend tensorflow2 --repoagent checksum


Add --enable-gpu and remove --build-dir

docs/compose.md

qa/L0_infer/infer_test.py

compose.py

build.py

compose.py

build.py

deadeyegoodwin · 2021-06-29T15:23:29Z

build.py

 ENV NVIDIA_BUILD_ID {}
 LABEL com.nvidia.build.id={}
 LABEL com.nvidia.build.ref={}
 '''.format(argmap['NVIDIA_BUILD_ID'], argmap['NVIDIA_BUILD_ID'],
           argmap['NVIDIA_BUILD_REF'])

-    # Add feature labels for SageMaker endpoint
+# Add feature labels for SageMaker endpoint


deadeyegoodwin · 2021-06-29T15:25:45Z

build.py

+
+
+def dockerfile_add_installation_linux(argmap, backends, endpoints):
+    df='''


Add description:
"""Common steps for production docker image, shared by build.py and compose.py"""

deadeyegoodwin · 2021-06-29T15:26:25Z

compose.py

+import docker
+
+### Global variables
+TRITON_VERSION_MAP = {


Remove, you should use this from build.py when needed

compose.py

deadeyegoodwin · 2021-06-29T15:33:13Z

docs/compose.md

+```
+to create the new custom Triton Docker image.
+
+`compose.py` provides `--container-version`, `--backend`, `--repoagent` and `--endpoint` options that allow you to specify which upstream container to use and which backends, repository agents and endpoints to include in the custom image. The `--enable-gpu` flag indicates that you want to create an image that supports NVIDIA GPUs. For example, the following creates a new docker image that contains only the TensorFlow 1 and TensorFlow 2 backends and the checksum repository agent.


remove --container-version

deadeyegoodwin · 2021-06-29T15:34:13Z

docs/compose.md

+nvcr.io/nvidia/tritonserver:<xx.yy>-py3-min
+nvcr.io/nvidia/tritonserver:<xx.yy>-py3
+```
+to create the new custom Triton Docker image.


describe here how you need to "git clone" the release branch of the server repo that corresponds to the docker container release that you want to use to compose a new image

Would it not be easier to ask the user specify the upstream container version and if they don't specify, then fallback to branch specific version? This way, people don't have to clone the entire repo just for this script

We can make --container-version optional. The potential problem is that the branch they run from (e.g. "main") may have changes in compose.py that are not valid for the release container they target. So the default should be to work using TRITON_VERSION but it is ok to have --container-version flag to override if the user knows what they are doing.

build.py

compose.py

deadeyegoodwin · 2021-06-29T20:07:53Z

build.py

+        dfile.write(df)
+
+
+"""Common steps for production docker image, shared by build.py and compose.py"""


In python function comments go after the def, so this should be after line 689.

deadeyegoodwin · 2021-06-29T20:16:43Z

build.py

@@ -742,9 +766,7 @@ def create_dockerfile_linux(ddir, dockerfile_name, argmap, backends, repoagents,
 COPY --chown=1000:1000 --from=tritonserver_build /workspace/build/sagemaker/serve /usr/bin/.


There is no "tritonserver_build" when running from compose.py. So you likely need to move the sagemaker out of the common function and modify it as needed in compose.py.

compose.py

deadeyegoodwin · 2021-06-29T20:20:14Z

compose.py

+        help=
+        'Generated dockerfiles are placed here. Default to current directory.')
+    parser.add_argument(
+        '--upstream-container-version',


just --container-version not upstream-container-version

deadeyegoodwin · 2021-06-29T20:22:02Z

docs/compose.md

-following commands.
+## Use the compose.py script
+
+The `compose.py` script can be found in the [server repository](https://github.com/triton-inference-server/server). Simply clone the repository and run `compose.py` to create a custom container. 


Need to help them know which branch to clone. They need to clone the release branch corresponding to the NGC release that they want to base their custom image on. For example they need to use r21.06 branch if they want to create an image based on the NGC 21.06 Triton release.

deadeyegoodwin · 2021-06-30T20:38:06Z

build.py

-       '2021.2.200', # ORT OpenVINO
-       '2021.2')     # Standalone OpenVINO
+    '2.12.0dev': (
+        '21.06',  # triton container


deadeyegoodwin · 2021-06-30T20:53:40Z

compose.py

+        'NVIDIA_BUILD_ID': p_build.stdout.rstrip(),
+        'TRITON_VERSION': version,
+        'TRITON_CONTAINER_VERSION': container_version,
+        'ENDPOINTS': f is not None,


Can you name this "SAGEMAKER_ENDPOINT" to be more descriptive

deadeyegoodwin · 2021-06-30T20:54:52Z

compose.py

+        type=str,
+        required=False,
+        help=
+        'The version to use for the generated Docker image. If not specified the container version will be chosen automatically based on --version value.'


There is no --version flag, so change desc to be "... will be chosen automatically based on the repository branch."

deadeyegoodwin · 2021-06-30T20:55:42Z

compose.py

+        '--dry-run',
+        action="store_true",
+        required=False,
+        help='Only create Dockerfile, does not create a container.')


"Only create Dockerfile.compose, do not build the Docker image"

deadeyegoodwin · 2021-06-30T20:57:34Z

docs/compose.md

-cuDNN, etc. dependencies that are required to run Triton. The
-\<xx.yy\>-py3 image contains the complete Triton with all options and
-backends.
+The `compose.py` script can be found in the [server repository](https://github.com/triton-inference-server/server). Simply clone the repository and run `compose.py` to create a custom container. Note created container version will depend on the branch that was cloned. For example branch [r21.06](https://github.com/triton-inference-server/server/tree/r21.06) is needed to create a image based on the NGC 21.06 Triton release. 


"is needed to create" -> "should be used to create"

deadeyegoodwin · 2021-06-30T20:57:49Z

docs/compose.md

-customization currently available is limited. As a result the minimum
-Triton still contains both HTTP/REST and GRPC endpoints; S3, GCS and
-Azure Storage filesystem support; and the TensorRT backend.
+`compose.py` provides `--backend`, `--repoagent` and `--endpoint` options that allow you to specify which backends, repository agents and endpoints to include in the custom image. The `--enable-gpu` flag indicates that you want to create an image that supports NVIDIA GPUs. For example, the following creates a new docker image that contains only the TensorFlow 1 and TensorFlow 2 backends and the checksum repository agent.


Remove --endpoint

build.py

docs/compose.md

qa/L0_infer/install_and_test.sh

CoderHam · 2021-07-22T20:54:16Z

docs/compose.md

@@ -41,7 +41,7 @@ from source to get more exact customization.

 ## Use the compose.py script

-The `compose.py` script can be found in the [server repository](https://github.com/triton-inference-server/server). Simply clone the repository and run `compose.py` to create a custom container. Note created container version will depend on the branch that was cloned. For example branch [r21.06](https://github.com/triton-inference-server/server/tree/r21.06) should be used to create a image based on the NGC 21.06 Triton release. 
+The `compose.py` script can be found in the [server repository](https://github.com/triton-inference-server/server). Simply clone the repository and run `compose.py` to create a custom container. Note created container version will depend on the branch that was cloned. For example branch [r21.08](https://github.com/triton-inference-server/server/tree/r21.06) should be used to create a image based on the NGC 21.08 Triton release. 


Arent the doc updates to reflect 21.08 does during the 21.08 codefreeze? These should be 21.07 for now

This is not going to be merged in before the 21.07 release.. lets change it back to 21.06.

CoderHam · 2021-07-22T20:55:57Z

docs/compose.md

@@ -54,6 +54,7 @@ will provide a container `tritonserver` locally. You can access the container wi
 $ docker run -it tritonserver:latest
 ```

+Note: If `compose.py` is run on release versions older than r21.08, the resulting container will have DCGM version 2.2.8 installed. This may result in different GPU statistic reporting behavior.


Same here. This should be added during codefreeze. Additionally won't 21.07 have 2.2.3 with your latest changes?

CoderHam · 2021-07-22T22:42:10Z

compose.py

+    dcgm_ver = re.search("DCGM_VERSION=([\S]{4,}) ", vars)
+    dcgm_version = ""
+    if dcgm_ver == None:
+        dcgm_version = "2.2.3"


Why was this changed back to 2.2.3?

Since the two versions that matter -- 21.06 and 21.07 will have 2.2.3, we should just use that for the older versions. 21.08 will have 2.2.8,

Note the buildbase is the only time we will need the patch anyway, so this should work (and additionally make the composed container have the same version of dcgm as 21.06 and 21.07).

jbkyang-nvi requested review from deadeyegoodwin, CoderHam and GuanLuo June 23, 2021 17:02

deadeyegoodwin changed the title ~~Kyang make compose.py~~ Custom container composition script Jun 23, 2021

GuanLuo requested changes Jun 23, 2021

View reviewed changes

compose.py Outdated Show resolved Hide resolved

compose.py Show resolved Hide resolved

qa/L0_infer/infer_test.py Outdated Show resolved Hide resolved

deadeyegoodwin suggested changes Jun 23, 2021

View reviewed changes

CoderHam suggested changes Jun 24, 2021

View reviewed changes

compose.py Show resolved Hide resolved

jbkyang-nvi requested review from deadeyegoodwin, CoderHam and GuanLuo June 24, 2021 20:01

deadeyegoodwin suggested changes Jun 24, 2021

View reviewed changes

jbkyang-nvi force-pushed the kyang-make-container branch from db0eb02 to 830816b Compare June 25, 2021 22:10

jbkyang-nvi commented Jun 25, 2021

View reviewed changes

build.py Show resolved Hide resolved

jbkyang-nvi requested a review from deadeyegoodwin June 28, 2021 17:53

jbkyang-nvi force-pushed the kyang-make-container branch from 478f43e to 5a73b4f Compare June 28, 2021 18:29

deadeyegoodwin suggested changes Jun 28, 2021

View reviewed changes

build.py Show resolved Hide resolved

compose.py Outdated Show resolved Hide resolved

compose.py Outdated Show resolved Hide resolved

compose.py Outdated Show resolved Hide resolved

compose.py Outdated Show resolved Hide resolved

compose.py Outdated Show resolved Hide resolved

jbkyang-nvi requested a review from deadeyegoodwin June 29, 2021 02:23

deadeyegoodwin suggested changes Jun 29, 2021

View reviewed changes

GuanLuo reviewed Jun 29, 2021

View reviewed changes

build.py Show resolved Hide resolved

compose.py Outdated Show resolved Hide resolved

jbkyang-nvi requested review from GuanLuo and deadeyegoodwin June 29, 2021 19:58

deadeyegoodwin suggested changes Jun 29, 2021

View reviewed changes

jbkyang-nvi requested a review from deadeyegoodwin June 30, 2021 00:07

jbkyang-nvi mentioned this pull request Jun 30, 2021

Add documentation for making custom min container #3059

Closed

deadeyegoodwin suggested changes Jun 30, 2021

View reviewed changes

jbkyang-nvi requested a review from deadeyegoodwin June 30, 2021 21:07

deadeyegoodwin previously approved these changes Jun 30, 2021

View reviewed changes

GuanLuo previously approved these changes Jul 1, 2021

View reviewed changes

CoderHam suggested changes Jul 1, 2021

View reviewed changes

build.py Outdated Show resolved Hide resolved

docs/compose.md Outdated Show resolved Hide resolved

docs/compose.md Show resolved Hide resolved

qa/L0_infer/install_and_test.sh Outdated Show resolved Hide resolved

qa/L0_infer/install_and_test.sh Outdated Show resolved Hide resolved

jbkyang-nvi dismissed GuanLuo’s stale review via e417e04 July 1, 2021 22:08

jbkyang-nvi added 9 commits July 21, 2021 16:23

remove extra env in dockerfile

dbea483

adjusted build so more similar to old build.py

da3b716

removed log message

344fb50

add images flag so users can specify the image

adf7e9c

remove wrong variable name

32da9db

addressed comments

3144fba

addressed comments

279f9d4

changed function name and add comments

8d72b6d

rebased and reformatted

e1eced4

jbkyang-nvi dismissed CoderHam’s stale review via e1eced4 July 21, 2021 23:25

jbkyang-nvi force-pushed the kyang-make-container branch from 2b73bef to e1eced4 Compare July 21, 2021 23:25

jbkyang-nvi added 2 commits July 21, 2021 16:56

fix build.py problem

46b3be4

fix again

12b78b9

CoderHam previously approved these changes Jul 22, 2021

View reviewed changes

add default DCGM for older container versions

da3c92e

jbkyang-nvi dismissed CoderHam’s stale review via da3c92e July 22, 2021 19:40

change default version to 2.2.8

f3a25d2

deadeyegoodwin previously approved these changes Jul 22, 2021

View reviewed changes

CoderHam previously approved these changes Jul 22, 2021

View reviewed changes

updated documentation

b0748af

jbkyang-nvi dismissed stale reviews from CoderHam and deadeyegoodwin via b0748af July 22, 2021 20:38

CoderHam suggested changes Jul 22, 2021

View reviewed changes

fixed documentation

ad8b523

jbkyang-nvi requested a review from CoderHam July 22, 2021 22:37

CoderHam suggested changes Jul 22, 2021

View reviewed changes

jbkyang-nvi requested a review from CoderHam July 22, 2021 22:51

CoderHam approved these changes Jul 22, 2021

View reviewed changes

jbkyang-nvi merged commit a3faff7 into main Jul 23, 2021

jbkyang-nvi deleted the kyang-make-container branch July 23, 2021 17:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom container composition script #3039

Custom container composition script #3039

jbkyang-nvi commented Jun 23, 2021 •

edited

Loading

deadeyegoodwin Jun 23, 2021

jbkyang-nvi Jun 24, 2021

deadeyegoodwin Jun 23, 2021

deadeyegoodwin Jun 23, 2021

deadeyegoodwin Jun 23, 2021

deadeyegoodwin Jun 23, 2021

deadeyegoodwin Jun 23, 2021

deadeyegoodwin Jun 23, 2021

deadeyegoodwin Jun 23, 2021

deadeyegoodwin Jun 29, 2021

deadeyegoodwin Jun 29, 2021

deadeyegoodwin Jun 29, 2021

deadeyegoodwin Jun 29, 2021

deadeyegoodwin Jun 29, 2021

jbkyang-nvi Jun 29, 2021

deadeyegoodwin Jun 29, 2021

deadeyegoodwin Jun 29, 2021

deadeyegoodwin Jun 29, 2021

deadeyegoodwin Jun 29, 2021

deadeyegoodwin Jun 29, 2021

deadeyegoodwin Jun 30, 2021

deadeyegoodwin Jun 30, 2021

deadeyegoodwin Jun 30, 2021

deadeyegoodwin Jun 30, 2021

deadeyegoodwin Jun 30, 2021

deadeyegoodwin Jun 30, 2021

CoderHam Jul 22, 2021

jbkyang-nvi Jul 22, 2021

CoderHam Jul 22, 2021

CoderHam Jul 22, 2021

jbkyang-nvi Jul 22, 2021

jbkyang-nvi Jul 22, 2021 •

edited

Loading

		@@ -0,0 +1,305 @@
		#!/usr/bin/env python3
		# Copyright (c) 2020-2021, NVIDIA CORPORATION & AFFILIATES. All rights reserved.



		def dockerfile_add_installation_linux(argmap, backends, endpoints):
		df='''

		dfile.write(df)


		"""Common steps for production docker image, shared by build.py and compose.py"""

		@@ -742,9 +766,7 @@ def create_dockerfile_linux(ddir, dockerfile_name, argmap, backends, repoagents,
		COPY --chown=1000:1000 --from=tritonserver_build /workspace/build/sagemaker/serve /usr/bin/.

Custom container composition script #3039

Custom container composition script #3039

Conversation

jbkyang-nvi commented Jun 23, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbkyang-nvi Jul 22, 2021 • edited Loading

Choose a reason for hiding this comment

jbkyang-nvi commented Jun 23, 2021 •

edited

Loading

jbkyang-nvi Jul 22, 2021 •

edited

Loading