GPU memory increased by 4GiB when running rasterizer

[CHMimilanlan]

It's a wired problem, which bothered me for a whole day.
when i running code below:

self.raster_settings = GaussianRasterizationSettings(
    image_height=self.image_size,
    image_width=self.image_size,
    tanfovx=self.tanfov,
    tanfovy=self.tanfov,
    bg=self.bg,
    scale_modifier=1.0,
    viewmatrix=w2c,
    projmatrix=full_proj,
    sh_degree=0,
    campos=cam_center,
    prefiltered=False,
    debug=False,
    antialiasing=True
)
self.rasterizer = GaussianRasterizer(raster_settings=self.raster_settings)
rendered_image, _ ,_ = self.rasterizer( 
    means3D = means3D,
    means2D = screenspace_points,
    colors_precomp = colors_precomp,
    opacities = opacities,
    scales = scales,
    rotations = rotations,
    cov3D_precomp = None)

CUDA memory increased by 4GiB, which is unbearable. That code is supposed to run several times. Each time that code is run, CUDA memory grows by 4GB until the CUDA out of memory.
I check the shape of each variable, it seems that it wasn't the problem of the size of variables, every variables are in expected shape.

means3D.shape=torch.Size([50200, 3])
screenspace_points.shape=torch.Size([50200, 3])
colors_precomp.shape=torch.Size([50200, 3])
opacities.shape=torch.Size([50200, 1])
scales.shape=torch.Size([50200, 1])
rotations.shape=torch.Size([50200, 4])

And the output size is also in expected shape, rendered_image.shape=torch.Size([3, 1024, 1024])
I have tried reinstalled diff_gaussian_rasterization and it doesn't work.

Could you tell me some potential possible reasons which cause this problem and it's corresponding solutions?
I will be grateful for your timely response !

[Junhaocai27]

same problem

[ZhaungbiaoXiu]

It's a wired problem, which bothered me for a whole day. when i running code below:

self.raster_settings = GaussianRasterizationSettings(
    image_height=self.image_size,
    image_width=self.image_size,
    tanfovx=self.tanfov,
    tanfovy=self.tanfov,
    bg=self.bg,
    scale_modifier=1.0,
    viewmatrix=w2c,
    projmatrix=full_proj,
    sh_degree=0,
    campos=cam_center,
    prefiltered=False,
    debug=False,
    antialiasing=True
)
self.rasterizer = GaussianRasterizer(raster_settings=self.raster_settings)
rendered_image, _ ,_ = self.rasterizer( 
    means3D = means3D,
    means2D = screenspace_points,
    colors_precomp = colors_precomp,
    opacities = opacities,
    scales = scales,
    rotations = rotations,
    cov3D_precomp = None)

CUDA memory increased by 4GiB, which is unbearable. That code is supposed to run several times. Each time that code is run, CUDA memory grows by 4GB until the CUDA out of memory. I check the shape of each variable, it seems that it wasn't the problem of the size of variables, every variables are in expected shape.

means3D.shape=torch.Size([50200, 3])
screenspace_points.shape=torch.Size([50200, 3])
colors_precomp.shape=torch.Size([50200, 3])
opacities.shape=torch.Size([50200, 1])
scales.shape=torch.Size([50200, 1])
rotations.shape=torch.Size([50200, 4])

And the output size is also in expected shape, rendered_image.shape=torch.Size([3, 1024, 1024]) I have tried reinstalled diff_gaussian_rasterization and it doesn't work.

Could you tell me some potential possible reasons which cause this problem and it's corresponding solutions? I will be grateful for your timely response !

Have you solved this problem？

[ZhaungbiaoXiu]

same problem

Have you solved this problem？

[fgz-ai]

I faced the same issue of out of GPU memory while the size of input tensors are small (in scale of 10K points). Any pointer to resolve this issue is appreciated.

[Langwenchong]

When using DDP for multi-GPU training (where each GPU independently calls the render function without using the backward process — this is a feedforward GS reconstruction project), I noticed that the underlying CUDA implementation might involve a large amount of fragmented memory allocation. As a result, whether I run it on a single Tesla V100 32GB card or multiple cards, it always reports an OOM error with an absurdly large allocation request (it tries to allocate 130 TB >> 32 GB). Meanwhile, when I run the same test on an RTX 3090, it seems that thanks to its more advanced virtualization technology, it can keep running for 4–5 hours before the GPU eventually crashes (most likely also due to OOM). I suspect this is because the GS rasterizer and NCCL modules may be making many fragmented memory allocations, and even the 3090’s virtualization can’t sustain it, leading to GPU crashes. Therefore, I’m wondering if the underlying CUDA rasterizer implementation might cause extremely high memory overhead?

[shashanknsridhara]

I am facing a similar issue. If anyone knows how to fix this, please post the solution.