Skip to content

Latest commit

 

History

History
213 lines (192 loc) · 29.3 KB

MODEL_ZOO.md

File metadata and controls

213 lines (192 loc) · 29.3 KB
Raw

Welcome to PytorchAutoDrive model zoo

Lane detection performance

Data Augmentation levels:

  • level 0: only small rotation and resize
  • level 1a: the LSTR augmentations
  • level 1b: the BézierLaneNet augmentations
  • level 1c: the LaneATT augmentations
method backbone data
augmentation		 resolution mixed precision? dataset metric average best training time
(2080 Ti)
Baseline VGG16 level 0 360 x 640 yes TuSimple Accuracy 93.79% 93.94% 1.5h
Baseline ResNet18 level 0 360 x 640 yes TuSimple Accuracy 94.18% 94.25% 0.7h
Baseline ResNet34 level 0 360 x 640 yes TuSimple Accuracy 95.23% 95.31% 1.1h
Baseline ResNet34 level 1a 360 x 640 no TuSimple Accuracy 92.14% 92.68% 1.2h*
Baseline ResNet50 level 0 360 x 640 yes TuSimple Accuracy 95.07% 95.12% 1.5h
Baseline ResNet101 level 0 360 x 640 yes TuSimple Accuracy 95.15% 95.19% 2.6h
Baseline ERFNet level 0 360 x 640 yes TuSimple Accuracy 96.02% 96.04% 0.8h
Baseline ERFNet level 1a 360 x 640 no TuSimple Accuracy 94.21% 94.37% 0.9h*
Baseline ENet# level 0 360 x 640 yes TuSimple Accuracy 95.55% 95.61% 1h+
Baseline MobileNetV2 level 0 360 x 640 yes TuSimple Accuracy 93.98% 94.07% 0.5h
Baseline MobileNetV3-Large level 0 360 x 640 yes TuSimple Accuracy 92.09% 92.18% 0.5h
SCNN VGG16 level 0 360 x 640 yes TuSimple Accuracy 95.01% 95.17% 2h
SCNN ResNet18 level 0 360 x 640 yes TuSimple Accuracy 94.69% 94.77% 1.2h
SCNN ResNet34 level 0 360 x 640 yes TuSimple Accuracy 95.19% 95.25% 1.6h
SCNN ResNet34 level 1a 360 x 640 no TuSimple Accuracy 92.62% 93.42% 1.7h*
SCNN ResNet50 level 0 360 x 640 yes TuSimple Accuracy 95.43% 95.56% 2.4h
SCNN ResNet101 level 0 360 x 640 yes TuSimple Accuracy 95.56% 95.69% 3.5h
SCNN ERFNet level 0 360 x 640 yes TuSimple Accuracy 96.18% 96.29% 1.6h
SCNN ERFNet level 1a 360 x 640 no TuSimple Accuracy 95.00% 95.26% 1.7h*
RESA ResNet18 level 0 360 x 640 no TuSimple Accuracy 94.64% 95.24% 1.2h*
RESA ResNet34 level 0 360 x 640 no TuSimple Accuracy 94.84% 95.15% 1.6h*
RESA ResNet50 level 0 360 x 640 no TuSimple Accuracy 95.34% 95.50% 2.4h*
RESA ResNet101 level 0 360 x 640 no TuSimple Accuracy 95.24% 95.56% 3.5h*
RESA ERFNet level 0 360 x 640 no TuSimple Accuracy 95.73% 95.76% 1.7h
RESA MobileNetV2 level 0 360 x 640 yes TuSimple Accuracy 94.61% 95.21% 0.7h
RESA MobileNetV3-Large level 0 360 x 640 yes TuSimple Accuracy 94.56% 94.99% 0.7h
LSTR ResNet18s# level 0 360 x 640 no TuSimple Accuracy 91.91% 92.40% 14.2h
LSTR ResNet18s# level 1a 360 x 640 no TuSimple Accuracy 94.91% 95.06% 15.5h
BézierLaneNet ResNet18 level 1b 360 x 640 no TuSimple Accuracy 95.01% 95.41% 5.5h
BézierLaneNet ResNet34 level 1b 360 x 640 no TuSimple Accuracy 95.17% 95.65% 6.5h
Baseline VGG16 level 0 288 x 800 yes CULane F1 65.93 66.09 9.3h
Baseline ResNet18 level 0 288 x 800 yes CULane F1 65.19 65.30 5.3h
Baseline ResNet34 level 0 288 x 800 yes CULane F1 69.82 69.92 7.3h
Baseline ResNet50 level 0 288 x 800 yes CULane F1 68.31 68.48 12.4h
Baseline ResNet101 level 0 288 x 800 yes CULane F1 71.29 71.37 20.0h
Baseline ERFNet level 0 288 x 800 yes CULane F1 73.40 73.49 6h
Baseline ENet# level 0 288 x 800 yes CULane F1 69.39 69.90 6.4h+
Baseline MobileNetV2 level 0 288 x 800 yes CULane F1 67.34 67.41 3.0h
Baseline MobileNetV3-Large level 0 288 x 800 yes CULane F1 68.27 68.42 3.0h
Baseline RepVGG-A0 level 0 288 x 800 yes CULane F1 70.22 70.56 3.3h**
Baseline RepVGG-A1 level 0 288 x 800 yes CULane F1 70.73 70.85 4.1h**
Baseline RepVGG-B0 level 0 288 x 800 yes CULane F1 71.77 71.81 6.2h**
Baseline RepVGG-B1g2 level 0 288 x 800 yes CULane F1 72.08 72.20 10.0h**
Baseline RepVGG-B2 level 0 288 x 800 yes CULane F1 72.24 72.33 13.2h**
Baseline Swin-Tiny level 0 288 x 800 yes CULane F1 69.75 69.90 12.1h**
SCNN VGG16 level 0 288 x 800 yes CULane F1 74.02 74.29 12.8h
SCNN ResNet18 level 0 288 x 800 yes CULane F1 71.94 72.19 8.0h
SCNN ResNet34 level 0 288 x 800 yes CULane F1 72.44 72.70 10.7h
SCNN ResNet50 level 0 288 x 800 yes CULane F1 72.95 73.03 17.9h
SCNN ResNet101 level 0 288 x 800 yes CULane F1 73.29 73.58 25.7h
SCNN ERFNet level 0 288 x 800 yes CULane F1 73.85 74.03 11.3h
SCNN RepVGG-A1 level 0 288 x 800 yes CULane F1 72.88 72.89 5.7h**
RESA ResNet18 level 0 288 x 800 no CULane F1 72.76 72.90 8.0h*
RESA ResNet34 level 0 288 x 800 no CULane F1 73.29 73.66 10.7h*
RESA ResNet50 level 0 288 x 800 no CULane F1 73.99 74.19 17.9h*
RESA ResNet101 level 0 288 x 800 no CULane F1 73.96 74.04 25.7h*
RESA ERFNet level 0 288 x 800 no CULane F1 73.28 73.32 9.1h
RESA MobileNetV2 level 0 288 x 800 yes CULane F1 72.28 72.36 4.6h
RESA MobileNetV3-Large level 0 288 x 800 yes CULane F1 70.23 70.61 4.6h
LSTR ResNet18s-2X# level 0 288 x 800 no CULane F1 36.27 39.77 28.5h*
LSTR ResNet18s-2X# level 1a 288 x 800 no CULane F1 68.35 68.72 31.5h*
LSTR ResNet34 level 1a 288 x 800 no CULane F1 72.17 72.48 45.0h*
LaneATT ResNet18 level 1c 360 x 640 no CULane F1 74.71 74.87 3.6h**
LaneATT ResNet34 level 1c 360 x 640 no CULane F1 75.76 75.82 4.0h**
BézierLaneNet ResNet18 level 1b 288 x 800 yes CULane F1 73.36 73.67 9.9h
BézierLaneNet ResNet34 level 1b 288 x 800 yes CULane F1 75.30 75.57 11.0h
Baseline ERFNet level 0 360 x 640 yes LLAMAS F1 95.94 96.13 10.9h+
Baseline VGG16 level 0 360 x 640 yes LLAMAS F1 95.05 95.11 9.3h
Baseline ResNet34 level 0 360 x 640 yes LLAMAS F1 95.90 95.91 7.0h
SCNN ERFNet level 0 360 x 640 yes LLAMAS F1 95.89 95.94 14.2h+
SCNN VGG16 level 0 360 x 640 yes LLAMAS F1 96.39 96.42 12.5h
SCNN ResNet34 level 0 360 x 640 yes LLAMAS F1 96.17 96.19 10.1h
BézierLaneNet ResNet18 level 1b 360 x 640 yes LLAMAS F1 95.42 95.52 5.5h
BézierLaneNet ResNet34 level 1b 360 x 640 yes LLAMAS F1 96.04 96.11 6.1h

All performance is measured with ImageNet pre-training and reported as 3 times average/best on test set.

The test set annotations of LLAMAS are not public, so we provide validation set result in this table.

+ Measured on a single GTX 1080Ti.

# No pre-training.

* Trained on a 1080 Ti cluster, with CUDA 9.0 PyTorch 1.3, training time is estimated as: single 2080 Ti, mixed precision.

** Trained on two 2080ti.

TuSimple detailed performance (best):

method backbone data
augmentation		 accuracy FP FN
Baseline VGG16 level 0 93.94% 0.0998 0.1021 model | shell
Baseline ResNet18 level 0 94.25% 0.0881 0.0894 model | shell
Baseline ResNet34 level 0 95.31% 0.0640 0.0622 model | shell
Baseline ResNet34 level 1a 92.68% 0.1073 0.1221 model | shell
Baseline ResNet50 level 0 95.12% 0.0649 0.0653 model | shell
Baseline ResNet101 level 0 95.19% 0.0619 0.0620 model | shell
Baseline ERFNet level 0 96.04% 0.0591 0.0365 model | shell
Baseline ERFNet level 1a 94.37% 0.0846 0.0770 model | shell
Baseline ENet level 0 95.61% 0.0655 0.0503 model | shell
Baseline MobileNetV2 level 0 94.07% 0.0792 0.0866 model | shell
Baseline MobileNetV3-Large level 0 92.18% 0.1149 0.1322 model | shell
SCNN VGG16 level 0 95.17% 0.0637 0.0622 model | shell
SCNN ResNet18 level 0 94.77% 0.0753 0.0737 model | shell
SCNN ResNet34 level 0 95.25% 0.0627 0.0634 model | shell
SCNN ResNet34 level 1a 93.42% 0.0868 0.0998 model | shell
SCNN ResNet50 level 0 95.56% 0.0561 0.0556 model | shell
SCNN ResNet101 level 0 95.69% 0.0519 0.0504 model | shell
SCNN ERFNet level 0 96.29% 0.0470 0.0318 model | shell
SCNN ERFNet level 1a 95.26% 0.0625 0.0512 model | shell
RESA ResNet18 level 0 95.24% 0.0685 0.0571 model | shell
RESA ResNet34 level 0 95.15% 0.0690 0.0592 model | shell
RESA ResNet50 level 0 95.50% 0.0550 0.0507 model | shell
RESA ResNet101 level 0 95.56% 0.0580 0.0513 model | shell
RESA ERFNet level 0 95.76% 0.0648 0.0439 model | shell
RESA MobileNetV2 level 0 95.21% 0.0642 0.0552 model | shell
RESA MobileNetV3-Large level 0 94.99% 0.0841 0.0597 model | shell
LSTR ResNet18s level 1a 95.06% 0.0486 0.0418 model | shell
LSTR ResNet18s level 0 92.40% 0.1289 0.1127 model | shell
BézierLaneNet ResNet18 level 1b 95.41% 0.0531 0.0458 model | shell
BézierLaneNet ResNet34 level 1b 95.65% 0.0513 0.0386 model | shell

CULane detailed performance (best):

method backbone data
augmentation		 normal crowded night no line shadow arrow dazzle
light		 curve crossroad total
Baseline VGG16 level 0 85.51 64.05 61.14 35.96 59.76 78.43 53.25 62.16 2224 66.09 model | shell
Baseline ResNet18 level 0 85.45 62.63 61.04 33.88 51.72 78.15 53.05 59.70 1915 65.30 model | shell
Baseline ResNet34 level 0 89.46 66.66 65.38 40.43 62.17 83.18 58.51 63.00 1713 69.92 model | shell
Baseline ResNet50 level 0 88.15 65.73 63.74 37.96 62.59 81.68 59.47 64.01 2046 68.48 model | shell
Baseline ResNet101 level 0 90.11 67.89 67.01 43.10 70.56 85.09 61.77 65.47 1883 71.37 model | shell
Baseline ERFNet level 0 91.48 71.27 68.09 46.76 74.47 86.09 64.18 66.89 2102 73.49 model | shell
Baseline ENet level 0 89.26 68.15 62.99 42.43 68.59 83.10 58.49 63.23 2464 69.90 model | shell
Baseline MobileNetV2 level 0 87.82 65.09 61.46 38.15 57.34 79.29 55.89 60.29 2114 67.41 model | shell
Baseline MobileNetV3-Large level 0 88.20 66.33 63.08 40.41 56.15 79.81 59.15 61.96 2304 68.42 model | shell
Baseline RepVGG-A0 level 0 89.74 67.68 65.21 42.51 67.85 83.13 60.86 63.63 2011 70.56 model | shell
Baseline RepVGG-A1 level 0 89.92 68.60 65.43 41.99 66.64 84.78 61.38 64.85 2127 70.85 model | shell
Baseline RepVGG-B0 level 0 90.86 69.32 66.68 43.53 67.83 85.43 59.80 66.47 2189 71.81 model | shell
Baseline RepVGG-B1g2 level 0 90.85 69.31 67.94 43.81 68.45 85.85 60.64 67.69 2092 72.20 model | shell
Baseline RepVGG-B2 level 0 90.82 69.84 67.65 43.02 72.08 85.76 61.75 67.67 2000 72.33 model | shell
Baseline Swin-Tiny level 0 89.55 68.36 63.56 42.53 61.96 82.64 60.81 65.21 2813 69.90 model | shell
SCNN VGG16 level 0 92.02 72.31 69.13 46.01 76.37 87.71 64.68 68.96 1924 74.29 model | shell
SCNN ResNet18 level 0 90.98 70.17 66.54 43.12 66.31 85.62 62.20 65.58 1808 72.19 model | shell
SCNN ResNet34 level 0 91.06 70.41 67.75 44.64 68.98 86.50 61.57 65.75 2017 72.70 model | shell
SCNN ResNet50 level 0 91.38 70.60 67.62 45.02 71.24 86.90 66.03 66.17 1958 73.03 model | shell
SCNN ResNet101 level 0 91.10 71.43 68.53 46.39 72.61 86.87 61.95 67.01 1720 73.58 model | shell
SCNN ERFNet level 0 91.82 72.13 69.49 46.68 70.59 87.40 64.18 68.30 2236 74.03 model | shell
SCNN RepVGG-A0 level 0 91.06 71.30 67.23 44.75 70.51 87.11 61.73 66.61 1963 72.89 model | shell
RESA ResNet18 level 0 91.23 70.57 67.16 45.24 68.01 86.56 64.32 66.19 1679 72.90 model | shell
RESA ResNet34 level 0 91.31 71.80 67.54 46.57 72.74 86.94 64.46 67.31 1701 73.66 model | shell
RESA ResNet50 level 0 91.52 72.49 68.44 47.02 72.56 87.34 63.11 68.21 1493 74.19 model | shell
RESA ResNet101 level 0 91.45 71.51 69.01 46.54 75.52 87.75 63.90 68.24 1522 74.04 model | shell
RESA ERFNet level 0 91.18 71.07 68.50 45.49 69.53 87.68 64.52 65.56 1777 73.32 model | shell
RESA MobileNetV2 level 0 90.58 70.42 67.19 45.29 62.80 85.52 66.00 65.19 1945 72.36 model | shell
RESA MobileNetV3-Large level 0 89.53 67.63 65.74 43.08 66.07 84.61 60.10 63.14 2218 70.61 model | shell
LSTR ResNet18s-2X level 0 56.17 39.10 22.90 25.62 25.49 52.09 40.21 30.33 1690 39.77 model | shell
LSTR ResNet18s-2X level 1a 86.78 67.34 59.92 40.10 59.82 78.66 56.63 56.64 1166 68.72 model | shell
LSTR ResNet34 level 1a 89.73 69.77 66.72 45.32 68.16 85.03 64.34 64.13 1247 72.48 model | shell
LaneATT ResNet18 level 1c 90.74 72.63 69.53 47.71 70.38 86.55 65.02 65.73 1036 74.87 model | shell
LaneATT ResNet34 level 1c 91.36 73.72 70.71 48.40 73.69 86.86 68.95 66.00 965 75.82 model | shell
BézierLaneNet ResNet18 level 1b 90.22 71.55 68.70 45.30 70.91 84.09 62.49 58.98 996 73.67 model | shell
BézierLaneNet ResNet34 level 1b 91.59 73.20 69.90 48.05 76.74 87.16 69.20 62.45 888 75.57 model | shell

LLAMAS detailed performance (best):

method backbone data
augmentation		 F1 TP FP FN Precision Recall val / test
Baseline VGG16 level 0 95.11 70263 3460 3772 95.31 94.91 val model | shell
Baseline ResNet34 level 0 95.91 70841 2847 3194 96.14 95.69 val model | shell
Baseline ERFNet level 0 96.13 71136 2830 2899 96.17 96.08 val model | shell
SCNN VGG16 level 0 96.42 71274 2526 2761 96.27 96.42 val model | shell
SCNN ERFNet level 0 95.94 71036 3019 2999 95.92 95.95 val model | shell
SCNN ResNet34 level 0 96.19 71109 2705 2926 96.34 96.05 val model | shell
BézierLaneNet ResNet18 level 1b 95.52 70515 3102 3520 95.79 95.25 val model | shell
BézierLaneNet ResNet34 level 1b 96.11 70959 2667 3076 96.38 95.85 val model | shell

Their test performance can be found at the LLAMAS leaderboard.

Semantic segmentation performance

model resolution mixed precision? dataset average best training time
(2080 Ti)		 best model link
FCN 321 x 321 yes PASCAL VOC 2012 70.72 70.83 3.3h model | shell
FCN 321 x 321 no PASCAL VOC 2012 70.91 71.55 6.3h model | shell
DeeplabV2 321 x 321 yes PASCAL VOC 2012 74.59 74.74 3.3h model | shell
DeeplabV3 321 x 321 yes PASCAL VOC 2012 78.11 78.17 7h model | shell
FCN 256 x 512 yes Cityscapes 68.05 68.20 2.2h model | shell
DeeplabV2 256 x 512 yes Cityscapes 68.65 68.90 2.2h model | shell
DeeplabV3 256 x 512 yes Cityscapes 69.87 70.37 4.5h model | shell
DeeplabV2 256 x 512 no Cityscapes 68.45 68.89 4h model | shell
ERFNet 512 x 1024 yes Cityscapes 71.99 72.47 5h model | shell
ENet 512 x 1024 yes Cityscapes 65.54 65.74 10.6h model | shell
DeeplabV2 512 x 1024 yes Cityscapes 71.78 72.12 9h model | shell
DeeplabV3 512 x 1024 yes Cityscapes 74.64 74.67 20.1h model | shell
DeeplabV2 512 x 1024 yes GTAV 32.90 33.88 13.8h model | shell
DeeplabV2 512 x 1024 yes SYNTHIA* 33.89 34.86 10.4h model | shell

All performance is measured with ImageNet pre-training and reported as 3 times average/best mIoU (%) on val set.

* mIoU-16.