Adds notes and example annotation for pytorch job

examples/mnist/README.md

@@ -7,6 +7,8 @@ to switch the distributed backends. The manifests to launch the distributed trai
 file using the pytorch operator are under the respective version folders: [v1](./v1).
 Each folder contains manifests with example usage of the different backends.
 
+**Note**: PyTorch job doesn’t work in a user namespace by default because of Istio [automatic sidecar injection](https://istio.io/v1.3/docs/setup/additional-setup/sidecar-injection/#automatic-sidecar-injection). In order to get it running, it needs annotation sidecar.istio.io/inject: "false" to disable it for either PyTorch pods or namespace.
+
 **Build Image**
 
 The default image name and tag is `kubeflow/pytorch-dist-mnist-test:1.0`.

examples/mnist/v1/pytorch_job_mnist_gloo.yaml

@@ -8,6 +8,9 @@ spec:
       replicas: 1
       restartPolicy: OnFailure
       template:
+        metadata:
+          annotations:
+            sidecar.istio.io/inject: "false"
         spec:
           containers:
             - name: pytorch
@@ -21,6 +24,9 @@ spec:
       replicas: 1
       restartPolicy: OnFailure
       template:
+        metadata:
+          annotations:
+            sidecar.istio.io/inject: "false"
         spec:
           containers: 
             - name: pytorch

examples/mnist/v1/pytorch_job_mnist_mpi.yaml

@@ -8,6 +8,9 @@ spec:
       replicas: 1
       restartPolicy: OnFailure
       template:
+        metadata:
+          annotations:
+            sidecar.istio.io/inject: "false"
         spec:
           containers:
             - name: pytorch
@@ -21,6 +24,9 @@ spec:
       replicas: 1
       restartPolicy: OnFailure 
       template:
+        metadata:
+          annotations:
+            sidecar.istio.io/inject: "false"
         spec:
           containers: 
             - name: pytorch

examples/mnist/v1/pytorch_job_mnist_nccl.yaml

@@ -8,6 +8,9 @@ spec:
       replicas: 1
       restartPolicy: OnFailure
       template:
+        metadata:
+          annotations:
+            sidecar.istio.io/inject: "false"
         spec:
           containers:
             - name: pytorch
@@ -20,6 +23,9 @@ spec:
       replicas: 1
       restartPolicy: OnFailure
       template:
+        metadata:
+          annotations:
+            sidecar.istio.io/inject: "false"
         spec:
           containers: 
             - name: pytorch