Number of classes assertion error

[lcoandrade]

Hi there again,
I'm working on a building footprint extraction Kaggle notebook. The dataset I'm using has images with these characteristics:

Images: torch.Size([3, 256, 256])
Labels: torch.Size([256, 256])

I'm using SemanticSegmentationRandomWindowGeoDataset. For each image in the folders (img and gt), I create one SemanticSegmentationRandomWindowGeoDataset that is inserted in a list that is used to create a concat dataset with PyTorch:

def make_random_window_dataset_from_dict(data_dict):
    datasets = []
    for file in data_dict.keys():    
        dataset = SemanticSegmentationRandomWindowGeoDataset.from_uris(
            class_config=class_config,
            image_uri=data_dict[file].image,
            label_raster_uri=data_dict[file].label,
            size_lims=(IMG_SIZE, 2*IMG_SIZE),
            out_size=IMG_SIZE,
            max_windows=MAX_WINDOWS,
        )
        datasets.append(dataset)
    return ConcatDataset(datasets)

After, I split this concat dataset in training, validation and testing:

train_set, val_set, test_set = torch.utils.data.random_split(concat_dataset, [0.6, 0.2, 0.2])

This is my class config:

names=[
    'Background',
    'Buildings',
]

cmap = [
    'black',
    'white',
]

class_config = ClassConfig(
    names=names,
    colors = cmap,
    null_class='Background',
)

And this is my data config:

#DataConfig – Configure data-related hyper-parameters
data_cfg = SemanticSegmentationGeoDataConfig(
    class_names=class_config.names,
    class_colors=class_config.colors,
    num_workers=4, # increase to use multi-processing
    img_channels=3,
)

This is my learner:

learner = SemanticSegmentationLearner(
    cfg=learner_cfg,
    output_dir=OUTPUT_DIR,
    train_ds=train_set,
    valid_ds=val_set,
    test_ds=test_set,
    training=True,
)

When I start to train, I get:

../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [32,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [33,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [6,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [262,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [263,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [264,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [265,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [517,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [518,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [519,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [520,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [521,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [522,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [523,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [773,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [774,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [775,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [776,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [777,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [778,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [779,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [780,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [781,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [782,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:103: nll_loss2d_forward_kernel: block: [2,0,0], thread: [783,0,0] Assertion `t >= 0 && t < n_classes` failed.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[20], line 3
      1 start = timeit.default_timer() # Measuring the time
----> 3 learner.train(epochs=EPOCHS)
      5 print("The time taken to train was: ", timeit.default_timer() - start)

File /opt/conda/lib/python3.10/site-packages/rastervision/pytorch_learner/learner.py:391, in Learner.train(self, epochs)
    389 for epoch in range(start_epoch, end_epoch):
    390     log.info(f'epoch: {epoch}')
--> 391     train_metrics = self.train_epoch(
    392         optimizer=self.opt, step_scheduler=self.step_scheduler)
    393     if self.epoch_scheduler:
    394         self.epoch_scheduler.step()

File /opt/conda/lib/python3.10/site-packages/rastervision/pytorch_learner/learner.py:416, in Learner.train_epoch(self, optimizer, step_scheduler)
    414 optimizer.zero_grad()
    415 output = self.train_step(batch, batch_ind)
--> 416 output['train_loss'].backward()
    417 optimizer.step()
    418 # detach tensors in the output, if any, to avoid memory leaks

File /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:487, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)
    477 if has_torch_function_unary(self):
    478     return handle_torch_function(
    479         Tensor.backward,
    480         (self,),
   (...)
    485         inputs=inputs,
    486     )
--> 487 torch.autograd.backward(
    488     self, gradient, retain_graph, create_graph, inputs=inputs
    489 )

File /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:200, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    195     retain_graph = create_graph
    197 # The reason we repeat same the comment below is that
    198 # some Python versions print out the first line of a multi-line function
    199 # calls in the traceback and some print out the last line
--> 200 Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    201     tensors, grad_tensors_, retain_graph, create_graph, inputs,
    202     allow_unreachable=True, accumulate_grad=True)

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I've faced this error before when I forgot to divide my labels by 255. But in that case, I was subclassing 'SemanticSegmentationDataReader' and I've have forgotten to divide the labels by 255. This case is diferente. I'm not subclassing. What can be the problem?

[AdeelH]

Check the values in your labels to see what the problem is.

x, y = train_ds[0]
print(y.unique())

[lcoandrade]

I get:

tensor([  0, 255])

It should be:

tensor([  0, 1])

But I'm not subclassing anything. What should I do to solve this?

[AdeelH]

The 255 you were seeing before wasn't because you were subclassing, but rather because your label files were pre-chipped RGB images with pixel values in the range 0-255.

In the present case, the labels are not pre-chipped and are single-band but still 0-255. Since we are reading them via a RasterSource, we can take advantage of the ReclassTransformer:

from rastervision.core.data import ReclassTransformer
...
dataset = SemanticSegmentationRandomWindowGeoDataset.from_uris(
    ...,
    label_raster_source_kw=dict(raster_transformers=[ReclassTransformer({255: 1})])
)

[lcoandrade]

Thank you, very much!
In my notebook from 3 months ago, I didn't need to use ReclassTransformer. Was this some change in the API?

[AdeelH]

Hmm, I don't know why that is. There's been no API change that would cause this.