Trouble importing labels for object detection

[natedailey7]

I'm training a model for object detection, and I'm having trouble importing my labels with their bounding boxes. I'm wondering... does creating a ObjectDetectionSlidingWindowGeoDataset with the convenience method .from_uris() automatically create bounding boxes for the label vectors? Or should I pivot toward creating a ObjectDetectionSlidingWindowGeoDataset with the default constructor?

Currently, I'm not able to achieve a mAP score great than 0.0. I'm working with this code:

class_config = ClassConfig(
    names=['background', '1'],
    colors=['lightgray', 'darkred'],
    null_class='background')

val_image_uri   = "path"
val_label_uri   = "path"
train_image_uri = "path"
train_label_uri = "path"

window_width = 100
window_height = 100

training_epochs = 10

train_ds = ObjectDetectionSlidingWindowGeoDataset.from_uris(
    class_config=class_config,
    image_uri=train_image_uri,
    label_vector_uri=train_label_uri,
    label_vector_default_class_id=class_config.get_class_id('1'),
    size=window_width,
    stride=window_width,
    transform=A.Resize(window_width, window_height),
)

val_ds = ObjectDetectionSlidingWindowGeoDataset.from_uris(
    class_config=class_config,
    image_uri=train_image_uri,
    label_vector_uri=val_label_uri,
    label_vector_default_class_id=class_config.get_class_id('1'),
    size=window_width,
    stride=window_width,
    transform=A.Resize(window_width, window_height),
)

data_cfg = ObjectDetectionGeoDataConfig(
    class_names=class_config.names,
    class_colors=class_config.colors,
	img_channels=4,
    num_workers=0,
)

solver_cfg = SolverConfig(
    batch_sz=8,
    lr=0.005,
)

model_cfg = ObjectDetectionModelConfig(backbone='resnet50', pretrained=True)

learner_cfg = ObjectDetectionLearnerConfig(
    data=data_cfg, 
    solver=solver_cfg,
    model=model_cfg, 
)

learner = ObjectDetectionLearner(
    cfg=learner_cfg,
    output_dir='path',
    train_ds=train_ds,
    valid_ds=val_ds,
)
learner.log_data_stats()

learner.train(epochs=training_epochs)

When I run .plot_predictions(), I see that my input does not contain any bounding boxes / vectors, so I suspect that I am inputting my labels incorrectly.

learner.plot_predictions(split='valid', show=True)

Let me know if you have any advice, thanks!

[AdeelH]

does creating a ObjectDetectionSlidingWindowGeoDataset with the convenience method .from_uris() automatically create bounding boxes for the label vectors

It should, yes.

You can also see samples of what the training chips look like by doing learner.plot_dataloaders().

If you don't see any labels there, I would suggest

• Checking if the labels and the imagery are properly aligned when you open them in QGIS
• Ensuring that the labels GeoJSON is ESPG:4326 and not some other CRS.
• Checking if you see any log messages like "RasterioSource bbox does not intersect with ObjectDetectionLabelSource bbox" from Raster Vision when you create your dataset.
• Inspecting the coords of the bboxes and comparing them to the corresponding raster source's extent.

  extent = train_ds.scene.extent
  npboxes, _, _ = train_ds.scene.label_source[extent]
  print(npboxes)
  print(extent)

[natedailey7]

I changed my learning rate to 1e-4 and it seems to have helped! Here is what I'm seeing now. My mAP value is non-zero, but still pretty low... do you have any recommendations from here? Should I just continue fiddling with my learning rate and test iteratively? Are there any other parameters I should try tweaking? Here are my epochs 2 through 5:

2023-11-27 15:45:14:rastervision.pytorch_learner.learner: INFO - epoch: 2
Training: 100%|██████████| 38/38 [00:17<00:00,  2.19it/s]
Validating: 100%|██████████| 38/38 [00:16<00:00,  2.29it/s]
2023-11-27 15:45:48:rastervision.pytorch_learner.learner: INFO - metrics:
{'epoch': 2,
 'loss_classifier': 0.17477168142795563,
 'loss_box_reg': 0.24905365705490112,
 'loss_objectness': 0.45493578910827637,
 'loss_rpn_box_reg': 1.5294535160064697,
 'train_loss': 2.4082143306732178,
 'train_time': '0:00:17.451940',
 'mAP': 0.0,
 'mAP50': 0.0,
 'valid_time': '0:00:16.596459'}
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.08s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
2023-11-27 15:45:49:rastervision.pytorch_learner.learner: INFO - epoch: 3
Training: 100%|██████████| 38/38 [00:17<00:00,  2.19it/s]
Validating: 100%|██████████| 38/38 [00:16<00:00,  2.27it/s]
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.01s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
2023-11-27 15:46:27:rastervision.pytorch_learner.learner: INFO - metrics:
{'epoch': 3,
 'loss_classifier': 0.18088671565055847,
 'loss_box_reg': 0.2787415385246277,
 'loss_objectness': 0.4575013518333435,
 'loss_rpn_box_reg': 1.4427378177642822,
 'train_loss': 2.359867572784424,
 'train_time': '0:00:17.496760',
 'mAP': 1.2532898859506203e-05,
 'mAP50': 0.00012532898859506203,
 'valid_time': '0:00:16.776175'}
DONE (t=3.47s).
Accumulating evaluation results...
DONE (t=0.06s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
2023-11-27 15:46:27:rastervision.pytorch_learner.learner: INFO - epoch: 4
Training: 100%|██████████| 38/38 [00:17<00:00,  2.16it/s]
Validating: 100%|██████████| 38/38 [00:17<00:00,  2.22it/s]
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
2023-11-27 15:47:03:rastervision.pytorch_learner.learner: INFO - metrics:
{'epoch': 4,
 'loss_classifier': 0.1250934898853302,
 'loss_box_reg': 0.27645573019981384,
 'loss_objectness': 0.4403263330459595,
 'loss_rpn_box_reg': 1.3784470558166504,
 'train_loss': 2.220322608947754,
 'train_time': '0:00:17.700590',
 'mAP': 1.544551115875564e-05,
 'mAP50': 4.9258657209004484e-05,
 'valid_time': '0:00:17.118714'}
DONE (t=0.75s).
Accumulating evaluation results...
DONE (t=0.03s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
2023-11-27 15:47:04:rastervision.pytorch_learner.learner: INFO - epoch: 5
Training: 100%|██████████| 38/38 [00:17<00:00,  2.16it/s]
Validating: 100%|██████████| 38/38 [00:16<00:00,  2.24it/s]
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
2023-11-27 15:47:39:rastervision.pytorch_learner.learner: INFO - metrics:
{'epoch': 5,
 'loss_classifier': 0.12306996434926987,
 'loss_box_reg': 0.18609346449375153,
 'loss_objectness': 0.22219280898571014,
 'loss_rpn_box_reg': 0.6633540987968445,
 'train_loss': 1.194710373878479,
 'train_time': '0:00:17.675159',
 'mAP': 0.0003086843630925069,
 'mAP50': 0.001651274211019902,
 'valid_time': '0:00:16.968000'}
DONE (t=0.93s).
Accumulating evaluation results...
DONE (t=0.02s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.002
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.001
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.003
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.005
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.005
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.013
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000

[AdeelH]

Some things that might be worth trying:

• Make sure you can overfit the training set i.e. use the same image for training and validation and see if you can get a very high accuracy.
• Try to increase the amount of training data. You can create multiple GeoDatasets (each corresponding to a different scene) and then combine them into one using torch.utils.data.ConcatDataset.
• Try to use more data augmentation. See here for an example.

[natedailey7]

Thanks so much for those suggestions!

• I tried using identical training and validation datasets, but I'm still seeing the same behavior (mAP score of zero). Do you think something might be incorrect with my framework?
• I researched Albumentations a bit and it seems like it will be very helpful!

Here's what I'm currently working with:

class_config = ClassConfig(
    names=['0'],
    colors=['darkred'],
    )

val_image_uri   = "path"
val_label_uri   = "path"
train_image_uri = "path"
train_label_uri = "path"

data_augmentation_transform = A.Compose([
    A.OneOf([
        A.ChannelShuffle(),
        A.ChannelDropout(),
        A.Blur(),
    ]),
])

train_ds = ObjectDetectionRandomWindowGeoDataset.from_uris(
    class_config=class_config,
    image_uri=train_image_uri,
    label_vector_uri=train_label_uri,
    label_vector_default_class_id=class_config.get_class_id('0'),
    transform=data_augmentation_transform,
    neg_ratio=2,
    out_size=256,
    size_lims=(150, 200),
    max_windows=100,
)

data_cfg = ObjectDetectionGeoDataConfig(
    class_names=class_config.names,
    class_colors=class_config.colors,
    img_channels=4,
    num_workers=0, 
)

solver_cfg = SolverConfig(
    batch_sz=8,
    lr=1e-4,
)

model_cfg = ObjectDetectionModelConfig(backbone='resnet50', pretrained=True)

learner_cfg = ObjectDetectionLearnerConfig(
    data=data_cfg, 
    solver=solver_cfg,
    model=model_cfg, 
)

learner = ObjectDetectionLearner(
    cfg=learner_cfg,
    output_dir="path"
    train_ds=train_ds,
    valid_ds=train_ds, # <--------------- identical train and val datasets
)
learner.log_data_stats()

learner.train(epochs=4)

Over 4 epochs, my loss_classifier value goes down from 7.7 to 2.09.

[AdeelH]

• Ideally, we want the validation set to be deterministic so that the metrics are comparable across epochs. So you should use a sliding window dataset and no augmentations for the validation set. Just use the same image URI as the training set.
• Again, make sure that the labels look right when you do plot_dataloaders() try doing that a couple of times to see different random batches.
• Train for longer, as long as the loss keeps going down.

[natedailey7]

@AdeelH, after some extensive testing, I was able to achieve a more accurate result. Thank you for your tips!