Docker Multi-GPU DDP training hang on `destroy_process_group()` with `wandb` option 3

[Yoon5]

Hello, when I try to training using multi gpu based on docker file images. I got the below error. I use Ubuntu 18.04, python 3.8.
<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

root@5a70a5f2d489:/usr/src/app# python -m torch.distributed.run --nproc_per_node 2 train.py --batch 64 --data data.yaml --weights yolov5s.pt --device 0,1
WARNING:__main__:*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
Traceback (most recent call last):
  File "train.py", line 620, in <module>
    main(opt)
  File "train.py", line 497, in main
    check_file(opt.data), check_yaml(opt.cfg), check_yaml(opt.hyp), str(opt.weights), str(opt.project)  # checks
  File "/usr/src/app/utils/general.py", line 326, in check_file
    assert len(files), f'File not found: {file}'  # assert file was found
AssertionError: File not found: data.yaml
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: (30 second timeout) ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 1 (pid: 405) of binary: /opt/conda/bin/python
/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py:367: UserWarning: 

**********************************************************************
               CHILD PROCESS FAILED WITH NO ERROR_FILE                
**********************************************************************
CHILD PROCESS FAILED WITH NO ERROR_FILE
Child process 405 (local_rank 1) FAILED (exitcode 1)
Error msg: Process failed with exitcode 1
Without writing an error file to <N/A>.
While this DOES NOT affect the correctness of your application,
no trace information about the error will be available for inspection.
Consider decorating your top level entrypoint function with
torch.distributed.elastic.multiprocessing.errors.record. Example:

  from torch.distributed.elastic.multiprocessing.errors import record

  @record
  def trainer_main(args):
     # do train
**********************************************************************
  warnings.warn(_no_error_file_warning_msg(rank, failure))
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 702, in <module>
    main()
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 361, in wrapper
    return f(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 698, in main
    run(args)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 689, in run
    elastic_launch(
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 116, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 244, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
***************************************
            train.py FAILED            
=======================================
Root Cause:
[0]:
  time: 2021-10-13_04:30:25
  rank: 1 (local_rank: 1)
  exitcode: 1 (pid: 405)
  error_file: <N/A>
  msg: "Process failed with exitcode 1"
=======================================
Other Failures:
  <NO_OTHER_FAILURES>
***************************************

root@5a70a5f2d489:/usr/src/app#

[glenn-jocher]

@Yoon5 thanks for the bug report! If you uninstall wandb before training (pip uninstall wandb), or login to wandb before training (wandb login API_KEY), does this resolve the issue for you?

@AyushExel I did some testing and it seems like wandb may be causing issues with DDP. I train all of my DDP models already logged in, but if not logged in and presented with 1,2,3 options query training may crash as above, or if training completes process group is not destroyed and system hangs. The steps I used to reproduce are on a 2-GPU training are here. Can you try to reproduce on your end?

# Pull image
t=ultralytics/yolov5:latest && sudo docker pull $t && sudo docker run -it --ipc=host --gpus all $t

# Train 3 epochs COCO128 with DDP
python -m torch.distributed.launch --nproc_per_node 1 --master_port 2 train.py --data coco128.yaml --epochs 3

Hang looks like this, seems to occur with wandb installed and enabled:

EDIT1: Summary is here:

• Docker DDP default: user selects wandb option 3, hang on finish bug
• Docker DDP with pip uninstall wandb: training runs correctly
• Docker DDP with wandb login API_KEY: training runs correctly

[Yoon5]

Thank you I will try)))))

[Yoon5]

I tried above lines (# Pull image
t=ultralytics/yolov5:latest && sudo docker pull $t && sudo docker run -it --ipc=host --gpus all $t

Train 3 epochs COCO128 with DDP

python -m torch.distributed.launch --nproc_per_node 1 --master_port 2 train.py --data coco128.yaml --epochs 3). And I do not have wandb. I am did not installed it to my env. And I got this after

~/Desktop/yolov5-master$ t=ultralytics/yolov5:latest && sudo docker pull $t && sudo docker run -it --ipc=host --gpus all $t
latest: Pulling from ultralytics/yolov5
Digest: sha256:eee5c66aa087376ab6b70b737b6825dcc59bc1059407a6875c8ef627e2e11f9c
Status: Image is up to date for ultralytics/yolov5:latest
docker.io/ultralytics/yolov5:latest

=============
== PyTorch ==

NVIDIA Release 21.05 (build 22595835)
PyTorch Version 1.9.0a0+2ecb2c7

Container image Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.

Copyright (c) 2014-2021 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006 Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015 Google Inc.
Copyright (c) 2015 Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

NVIDIA Deep Learning Profiler (dlprof) Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

ERROR: This container was built for NVIDIA Driver Release 465.19 or later, but
version 460.91.03 was detected and compatibility mode is UNAVAILABLE.

   [[Forward compatibility was attempted on non supported HW (CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE) cuInit()=804]]

NOTE: MOFED driver for multi-node communication was not detected.
Multi-node communication performance may be reduced.

root@1709fa266811:/usr/src/app# python -m torch.distributed.launch --nproc_per_node 1 --master_port 2 train.py --data coco128.yaml --epochs 3
/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torch.distributed.run.
Note that --use_env is set by default in torch.distributed.run.
If your script expects --local_rank argument to be set, please
change it to read from os.environ['LOCAL_RANK'] instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions

warnings.warn(
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: (30 second timeout) wandb login API_KEY
wandb: WARNING Invalid choice
wandb: Enter your choice: (30 second timeout)
wandb: W&B disabled due to login timeout.
train: weights=yolov5s.pt, cfg=, data=coco128.yaml, hyp=data/hyps/hyp.scratch.yaml, epochs=3, batch_size=16, imgsz=640, rect=False, resume=False, nosave=False, noval=False, noautoanchor=False, evolve=None, bucket=, cache=None, image_weights=False, device=, multi_scale=False, single_cls=False, adam=False, sync_bn=False, workers=8, project=runs/train, name=exp, exist_ok=False, quad=False, linear_lr=False, label_smoothing=0.0, patience=100, freeze=0, save_period=-1, local_rank=0, entity=None, upload_dataset=False, bbox_interval=-1, artifact_alias=latest
github: skipping check (Docker image), for updates see https://github.com/ultralytics/yolov5
/opt/conda/lib/python3.8/site-packages/torch/cuda/init.py:106: UserWarning:
GeForce RTX 3080 Ti with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the GeForce RTX 3080 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
YOLOv5 🚀 v6.0-3-g20a809d torch 1.9.1+cu102 CUDA:0 (GeForce RTX 3080 Ti, 12053.8125MB)

Added key: store_based_barrier_key:1 to store for rank: 0
Rank 0: Completed store-based barrier for 1 nodes.
hyperparameters: lr0=0.01, lrf=0.1, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=0.05, cls=0.5, cls_pw=1.0, obj=1.0, obj_pw=1.0, iou_t=0.2, anchor_t=4.0, fl_gamma=0.0, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0
TensorBoard: Start with 'tensorboard --logdir runs/train', view at http://localhost:6006/

WARNING: Dataset not found, nonexistent paths: ['/usr/src/datasets/coco128/images/train2017']
Downloading https://github.com/ultralytics/yolov5/releases/download/v1.0/coco128.zip to coco128.zip...
100%|██████████████████████████████████████| 6.66M/6.66M [00:00<00:00, 8.76MB/s]
Dataset autodownload success, saved to ../datasets

Downloading https://github.com/ultralytics/yolov5/releases/download/v6.0/yolov5s.pt to yolov5s.pt...
100%|██████████████████████████████████████| 14.0M/14.0M [00:04<00:00, 3.64MB/s]

             from  n    params  module                                  arguments                     

0 -1 1 3520 models.common.Conv [3, 32, 6, 2, 2]
1 -1 1 18560 models.common.Conv [32, 64, 3, 2]
2 -1 1 18816 models.common.C3 [64, 64, 1]
3 -1 1 73984 models.common.Conv [64, 128, 3, 2]
4 -1 2 115712 models.common.C3 [128, 128, 2]
5 -1 1 295424 models.common.Conv [128, 256, 3, 2]
6 -1 3 625152 models.common.C3 [256, 256, 3]
7 -1 1 1180672 models.common.Conv [256, 512, 3, 2]
8 -1 1 1182720 models.common.C3 [512, 512, 1]
9 -1 1 656896 models.common.SPPF [512, 512, 5]
10 -1 1 131584 models.common.Conv [512, 256, 1, 1]
11 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
12 [-1, 6] 1 0 models.common.Concat [1]
13 -1 1 361984 models.common.C3 [512, 256, 1, False]
14 -1 1 33024 models.common.Conv [256, 128, 1, 1]
15 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
16 [-1, 4] 1 0 models.common.Concat [1]
17 -1 1 90880 models.common.C3 [256, 128, 1, False]
18 -1 1 147712 models.common.Conv [128, 128, 3, 2]
19 [-1, 14] 1 0 models.common.Concat [1]
20 -1 1 296448 models.common.C3 [256, 256, 1, False]
21 -1 1 590336 models.common.Conv [256, 256, 3, 2]
22 [-1, 10] 1 0 models.common.Concat [1]
23 -1 1 1182720 models.common.C3 [512, 512, 1, False]
24 [17, 20, 23] 1 229245 models.yolo.Detect [80, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]], [128, 256, 512]]
Model Summary: 270 layers, 7235389 parameters, 7235389 gradients, 16.5 GFLOPs

Traceback (most recent call last):
File "train.py", line 620, in
main(opt)
File "train.py", line 517, in main
train(opt.hyp, opt, device, callbacks)
File "train.py", line 119, in train
csd = ckpt['model'].float().state_dict() # checkpoint state_dict as FP32
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 692, in float
return self._apply(lambda t: t.float() if t.is_floating_point() else t)
File "/usr/src/app/models/yolo.py", line 239, in _apply
self = super()._apply(fn)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 530, in _apply
module._apply(fn)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 552, in _apply
param_applied = fn(param)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 692, in
return self._apply(lambda t: t.float() if t.is_floating_point() else t)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 362) of binary: /opt/conda/bin/python
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/opt/conda/lib/python3.8/site-packages/torch/distributed/run.py", line 689, in run
elastic_launch(
File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 116, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 244, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

---

        train.py FAILED            

=======================================
Root Cause:
[0]:
time: 2021-10-13_05:37:52
rank: 0 (local_rank: 0)
exitcode: 1 (pid: 362)
error_file: <N/A>
msg: "Process failed with exitcode 1"

Other Failures:
<NO_OTHER_FAILURES>

---

---

[glenn-jocher]

@Yoon5 before you do anything you need to update your nvidia drivers as your error message states:

ERROR: This container was built for NVIDIA Driver Release 465.19 or later, but
version 460.91.03 was detected and compatibility mode is UNAVAILABLE.

@AyushExel I'm manually pushing a new ultralytics/yolov5:latest image without wandb for now until we can sort this out, so if you pull the image after seeing this message you'll have to pip install wandb in the image to get started testing.

[Yoon5]

Thank you

[AyushExel]

@glenn-jocher I'm testing this now

[AyushExel]

@glenn-jocher The problem doesn't occur for me. I'm running on 2 T4 GPUs and the program exited fine. I've tried this 2 times.

Full trace:

/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torch.distributed.run.
Note that --use_env is set by default in torch.distributed.run.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: (30 second timeout) 
wandb: W&B disabled due to login timeout.
train: weights=yolov5s.pt, cfg=, data=coco128.yaml, hyp=data/hyps/hyp.scratch.yaml, epochs=3, batch_size=16, imgsz=640, rect=False, resume=False, nosave=False, noval=False, noautoanchor=False, evolve=None, bucket=, cache=None, image_weights=False, device=, multi_scale=False, single_cls=False, adam=False, sync_bn=False, workers=8, project=runs/train, name=exp, exist_ok=False, quad=False, linear_lr=False, label_smoothing=0.0, patience=100, freeze=0, save_period=-1, local_rank=0, entity=None, upload_dataset=False, bbox_interval=-1, artifact_alias=latest
github: skipping check (Docker image), for updates see https://github.com/ultralytics/yolov5
YOLOv5 🚀 v6.0-4-gb754525 torch 1.9.1+cu102 CUDA:0 (Tesla T4, 15109.75MB)

Added key: store_based_barrier_key:1 to store for rank: 0
Rank 0: Completed store-based barrier for 1 nodes.
hyperparameters: lr0=0.01, lrf=0.1, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=0.05, cls=0.5, cls_pw=1.0, obj=1.0, obj_pw=1.0, iou_t=0.2, anchor_t=4.0, fl_gamma=0.0, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0
TensorBoard: Start with 'tensorboard --logdir runs/train', view at http://localhost:6006/

WARNING: Dataset not found, nonexistent paths: ['/usr/src/datasets/coco128/images/train2017']
Downloading https://github.com/ultralytics/yolov5/releases/download/v1.0/coco128.zip to coco128.zip...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 6.66M/6.66M [00:00<00:00, 123MB/s]
Dataset autodownload success, saved to ../datasets

Downloading https://github.com/ultralytics/yolov5/releases/download/v6.0/yolov5s.pt to yolov5s.pt...
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 14.0M/14.0M [00:00<00:00, 31.5MB/s]


                 from  n    params  module                                  arguments                     
  0                -1  1      3520  models.common.Conv                      [3, 32, 6, 2, 2]              
  1                -1  1     18560  models.common.Conv                      [32, 64, 3, 2]                
  2                -1  1     18816  models.common.C3                        [64, 64, 1]                   
  3                -1  1     73984  models.common.Conv                      [64, 128, 3, 2]               
  4                -1  2    115712  models.common.C3                        [128, 128, 2]                 
  5                -1  1    295424  models.common.Conv                      [128, 256, 3, 2]              
  6                -1  3    625152  models.common.C3                        [256, 256, 3]                 
  7                -1  1   1180672  models.common.Conv                      [256, 512, 3, 2]              
  8                -1  1   1182720  models.common.C3                        [512, 512, 1]                 
  9                -1  1    656896  models.common.SPPF                      [512, 512, 5]                 
 10                -1  1    131584  models.common.Conv                      [512, 256, 1, 1]              
 11                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 12           [-1, 6]  1         0  models.common.Concat                    [1]                           
 13                -1  1    361984  models.common.C3                        [512, 256, 1, False]          
 14                -1  1     33024  models.common.Conv                      [256, 128, 1, 1]              
 15                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 16           [-1, 4]  1         0  models.common.Concat                    [1]                           
 17                -1  1     90880  models.common.C3                        [256, 128, 1, False]          
 18                -1  1    147712  models.common.Conv                      [128, 128, 3, 2]              
 19          [-1, 14]  1         0  models.common.Concat                    [1]                           
 20                -1  1    296448  models.common.C3                        [256, 256, 1, False]          
 21                -1  1    590336  models.common.Conv                      [256, 256, 3, 2]              
 22          [-1, 10]  1         0  models.common.Concat                    [1]                           
 23                -1  1   1182720  models.common.C3                        [512, 512, 1, False]          
 24      [17, 20, 23]  1    229245  models.yolo.Detect                      [80, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]], [128, 256, 512]]
Model Summary: 270 layers, 7235389 parameters, 7235389 gradients, 16.5 GFLOPs

Transferred 349/349 items from yolov5s.pt
Scaled weight_decay = 0.0005
optimizer: SGD with parameter groups 57 weight, 60 weight (no decay), 60 bias
train: Scanning '../datasets/coco128/labels/train2017' images and labels...128 found, 0 missing, 2 empty, 0 corrupted: 100%|█| 128/128 [00
train: New cache created: ../datasets/coco128/labels/train2017.cache
val: Scanning '../datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|█| 128/12
Plotting labels... 
val: Scanning '../datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|█| 128/12
val: Scanning '../datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|█| 128/12
val: Scanning '../datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|█| 128/12
val: Scanning '../datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|█| 128/12
val: Scanning '../datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|█| 128/12
val: Scanning '../datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|█| 128/12
val: Scanning '../datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|█| 128/12
val: Scanning '../datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|█| 128/12

autoanchor: Analyzing anchors... anchors/target = 4.26, Best Possible Recall (BPR) = 0.9946
Image sizes 640 train, 640 val
Using 8 dataloader workers
Logging results to runs/train/exp
Starting training for 3 epochs...

     Epoch   gpu_mem       box       obj       cls    labels  img_size
       0/2     2.53G   0.04048   0.07431   0.02063       210       640:  12%|███▉                           | 1/8 [00:06<00:44,  6.40s/it]Reducer buckets have been rebuilt in this iteration.
       0/2     6.65G   0.04252   0.06144   0.02085       232       640: 100%|███████████████████████████████| 8/8 [00:08<00:00,  1.02s/it]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|███████████████| 4/4 [00:09<00:00,  2.44s/it]
                 all        128        929      0.671      0.533      0.621      0.407

     Epoch   gpu_mem       box       obj       cls    labels  img_size
       1/2     7.05G   0.04501   0.06472    0.0191       180       640: 100%|███████████████████████████████| 8/8 [00:02<00:00,  4.00it/s]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|███████████████| 4/4 [00:03<00:00,  1.22it/s]
                 all        128        929      0.697      0.536      0.631      0.416

     Epoch   gpu_mem       box       obj       cls    labels  img_size
       2/2     7.05G   0.04566   0.06474   0.02024       305       640: 100%|███████████████████████████████| 8/8 [00:01<00:00,  4.31it/s]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|███████████████| 4/4 [00:03<00:00,  1.20it/s]
                 all        128        929      0.704      0.547      0.633      0.418

3 epochs completed in 0.009 hours.
Optimizer stripped from runs/train/exp/weights/last.pt, 14.9MB
Optimizer stripped from runs/train/exp/weights/best.pt, 14.9MB

Validating runs/train/exp/weights/best.pt...
Fusing layers... 
Model Summary: 213 layers, 7225885 parameters, 0 gradients, 16.5 GFLOPs
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|███████████████| 4/4 [00:05<00:00,  1.28s/it]
                 all        128        929      0.704      0.547      0.634      0.419
              person        128        254      0.807      0.681      0.774      0.508
             bicycle        128          6      0.748      0.496      0.545      0.318
                 car        128         46      0.801      0.351      0.484      0.212
          motorcycle        128          5      0.672        0.6        0.8      0.633
            airplane        128          6          1      0.831      0.995      0.721
                 bus        128          7      0.648      0.714      0.694       0.59
               train        128          3      0.809          1      0.995      0.632
               truck        128         12      0.605       0.25      0.446      0.251
                boat        128          6      0.735      0.333      0.489      0.141
       traffic light        128         14      0.587      0.143      0.238      0.149
           stop sign        128          2      0.627        0.5      0.828      0.663
               bench        128          9      0.908      0.444       0.57      0.243
                bird        128         16      0.888      0.991      0.988      0.652
                 cat        128          4          1      0.729      0.836      0.691
                 dog        128          9      0.916      0.667      0.887      0.547
               horse        128          2      0.705          1      0.995      0.697
            elephant        128         17          1      0.927      0.946      0.696
                bear        128          1       0.52          1      0.995      0.995
               zebra        128          4      0.849          1      0.995      0.952
             giraffe        128          9      0.813      0.778      0.869      0.576
            backpack        128          6          1      0.307      0.452      0.201
            umbrella        128         18      0.724      0.556      0.722      0.394
             handbag        128         19      0.662      0.104      0.167       0.11
                 tie        128          7      0.895      0.571      0.693      0.466
            suitcase        128          4          1      0.992      0.995      0.621
             frisbee        128          5      0.652        0.8      0.798      0.694
                skis        128          1      0.616          1      0.995      0.497
           snowboard        128          7          1      0.705      0.766      0.558
         sports ball        128          6      0.659        0.5      0.622      0.341
                kite        128         10      0.557        0.5      0.557      0.204
        baseball bat        128          4      0.391        0.5      0.275      0.136
      baseball glove        128          7      0.474      0.391      0.327      0.197
          skateboard        128          5      0.754      0.614      0.792      0.557
       tennis racket        128          7      0.536      0.571      0.538      0.299
              bottle        128         18      0.649      0.389      0.484      0.286
          wine glass        128         16      0.771      0.875      0.853      0.397
                 cup        128         36      0.852      0.361      0.493      0.294
                fork        128          6      0.378      0.167      0.252      0.194
               knife        128         16      0.888      0.625      0.667      0.449
               spoon        128         22      0.811      0.391      0.531      0.257
                bowl        128         28       0.75      0.571      0.617      0.461
              banana        128          1          0          0      0.142     0.0142
            sandwich        128          2          0          0     0.0957     0.0743
              orange        128          4          1          0      0.578      0.199
            broccoli        128         11      0.418      0.182      0.314      0.273
              carrot        128         24      0.716      0.542      0.636      0.383
             hot dog        128          2        0.4      0.699      0.497      0.465
               pizza        128          5      0.629          1      0.831      0.603
               donut        128         14      0.692          1      0.952      0.823
                cake        128          4       0.73          1      0.895      0.713
               chair        128         35      0.458      0.486      0.476      0.232
               couch        128          6      0.723      0.333      0.801      0.453
        potted plant        128         14      0.791      0.714      0.806      0.447
                 bed        128          3          1          0      0.746      0.275
        dining table        128         13       0.83      0.462      0.476      0.299
              toilet        128          2      0.456        0.5      0.566      0.496
                  tv        128          2      0.752          1      0.995      0.846
              laptop        128          3          1          0      0.426      0.185
               mouse        128          2          1          0     0.0268     0.0215
              remote        128          8       0.71      0.625      0.635      0.506
          cell phone        128          8      0.599      0.199      0.429      0.202
           microwave        128          3      0.402          1      0.995      0.743
                oven        128          5      0.364        0.4      0.427      0.248
                sink        128          6      0.344      0.167      0.265      0.161
        refrigerator        128          5      0.704        0.8      0.814      0.456
                book        128         29      0.601      0.138      0.294      0.133
               clock        128          9       0.87      0.778       0.91      0.599
                vase        128          2      0.286          1      0.663      0.597
            scissors        128          1          1          0     0.0302    0.00603
          teddy bear        128         21       0.85      0.381      0.613      0.344
          toothbrush        128          5          1      0.477      0.708      0.438
Results saved to runs/train/exp
root@903423287a25:/usr/src/app# 

[glenn-jocher]

@AyushExel oh interesting. Can you try again and enter wandb: (3) Don't visualize my results