Skip to content

hwdec_cuda/vulkan: fix partial failure edge cases in error paths#18017

Open
philipl wants to merge 2 commits into
mpv-player:masterfrom
philipl:unslopped-hwdec-fixes
Open

hwdec_cuda/vulkan: fix partial failure edge cases in error paths#18017
philipl wants to merge 2 commits into
mpv-player:masterfrom
philipl:unslopped-hwdec-fixes

Conversation

@philipl
Copy link
Copy Markdown
Member

@philipl philipl commented May 23, 2026

These changes were inspired by the slopbot, but I went through and manually verified them, identifying additional problems, and better solutions along the way.


hwdec_cuda: fix partial failure edge cases in error paths

The slopbot spam did find valid issues in the error paths in this code, and as
part of verifying them, I have additional fixes.

  • ensure that the return value -1 from cuda_init() if ext_init() fails
  • zero alloc the ext_vk and ext_gl structs so that we have a known initial
    state
  • explicitly initialise the sem_handle.fd to -1 to ensure we don't
    accidentally close stdin in the error path. But also don't do this on
    windows as an invalid Handle is 0.
  • Don't do a double cuCtxPopCurrent() in the ext_gl init failure path

Fixes #17976, #17988, #17989


hwdec_vulkan: fix partial failure edge cases in error paths

This is another file where the slopbot identified a problem, but I was
surprised to see it fix it in a poor way and also miss an adjacement
problem.

mapper_map calls mapper_unmap with the intention of using it to clean
up partial failure after some planes are wrapped and released while others
are not. But the current code would actually segfault in that situation.

There are two fixes required:

  • Ensure p->vkf is set by mapper_map early so that its available to
    mapper_unmap in the error path
  • Don't call vkfc->unlock_frame in the mapper_map error path, as
    this will be called by mapper_unmap

Fixes #17977


philipl added 2 commits May 23, 2026 11:00
The slopbot spam did find valid issues in the error paths in this code, and as
part of verifying them, I have additional fixes.

* ensure that the return value -1 from `cuda_init()` if `ext_init()` fails
* zero alloc the `ext_vk` and `ext_gl` structs so that we have a known initial
  state
* explicitly initialise the `sem_handle.fd` to -1 to ensure we don't
  accidentally close stdin in the error path. But also don't do this on
  windows as an invalid Handle is 0.
* Don't do a double `cuCtxPopCurrent()` in the `ext_gl` init failure path

Fixes mpv-player#17976, mpv-player#17988, mpv-player#17989
This is another file where the slopbot identified a problem, but I was
surprised to see it fix it in a poor way and also miss an adjacement
problem.

`mapper_map` calls `mapper_unmap` with the intention of using it to clean
up partial failure after some planes are wrapped and released while others
are not. But the current code would actually segfault in that situation.

There are two fixes required:

* Ensure `p->vkf` is set by `mapper_map` early so that its available to
  `mapper_unmap` in the error path
* Don't call `vkfc->unlock_frame` in the `mapper_map` error path, as
  this will be called by `mapper_unmap`

Fixes mpv-player#17977
@philipl philipl force-pushed the unslopped-hwdec-fixes branch from 826d640 to 7858bf8 Compare May 23, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] hwdec_vulkan: NULL pointer dereference in mapper_unmap when p->vkf is NULL [Bug] hwdec_cuda: mapper_init returns 0 on ext_init failure

1 participant