Skip to content

Vulkan: Don't default to CPU device (like llvmpipe) #14099

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 10, 2025
Merged

Vulkan: Don't default to CPU device (like llvmpipe) #14099

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions ggml/src/ggml-vulkan/ggml-vulkan.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3595,11 +3595,11 @@ static void ggml_vk_instance_init() {

vk_perf_logger_enabled = getenv("GGML_VK_PERF_LOGGER") != nullptr;

size_t num_available_devices = vk_instance.instance.enumeratePhysicalDevices().size();

// Emulate behavior of CUDA_VISIBLE_DEVICES for Vulkan
char * devices_env = getenv("GGML_VK_VISIBLE_DEVICES");
if (devices_env != nullptr) {
size_t num_available_devices = vk_instance.instance.enumeratePhysicalDevices().size();

std::string devices(devices_env);
std::replace(devices.begin(), devices.end(), ',', ' ');

Expand All @@ -3615,9 +3615,9 @@ static void ggml_vk_instance_init() {
} else {
std::vector<vk::PhysicalDevice> devices = vk_instance.instance.enumeratePhysicalDevices();

// Make sure at least one device exists
// If no vulkan devices are found, return early
if (devices.empty()) {
std::cerr << "ggml_vulkan: Error: No devices found." << std::endl;
GGML_LOG_INFO("ggml_vulkan: No devices found.\n");
return;
}

Expand Down Expand Up @@ -3700,9 +3700,20 @@ static void ggml_vk_instance_init() {
}
}

// If no dedicated GPUs found, fall back to GPU 0
// If no dedicated GPUs found, fall back to the first non-CPU device.
// If only CPU devices are available, return without devices.
if (vk_instance.device_indices.empty()) {
for (size_t i = 0; i < devices.size(); i++) {
if (devices[i].getProperties().deviceType != vk::PhysicalDeviceType::eCpu) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's possible we want to consider other device types here too like:

if (devices[i].getProperties().deviceType != vk::PhysicalDeviceType::eCpu && devices[i].getProperties().deviceType != vk::PhysicalDeviceType::eIntegratedGpu)

Most integrated GPUs are slower than CPU inferencing, especially if an Integrated GPU has < 1GB VRAM it gets very questionable.

But could be discussion for another PR...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually looking at the various types it can be, I'd flip it:

if (devices[i].getProperties().deviceType == vk::PhysicalDeviceType::eDiscreteGpu)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is true, but there are a lot of iGPUs that run better than CPU with Vulkan, too. It is not as straightforward to decide here, we might need a black- or whitelist.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considering we can always override with GGML_VK_VISIBLE_DEVICES, only eDiscreteGpu would get my vote

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious what's your take @a-ghorbani from the Android perspective

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most integrated GPUs are slower than CPU inferencing, especially if an Integrated GPU has < 1GB VRAM it gets very questionable.

Considering we can always override with GGML_VK_VISIBLE_DEVICES, only eDiscreteGpu would get my vote

In the vast majority of cases an integrated GPU with Vulkan and -ngl 0 is going to perform better than CPU in prompt processing. Also pretty much all computer iGPUs that support Vulkan (so anything newer than Intel Skylake or the AMD GCN2 APUs) should be able to access several GBs of memory no problem. I'll admit I'm not sure about phones though.

If you look at the chart a lot of the newer integrated chips are running very well even with the model fully offloaded. Also Intel, AMD, and Nvidia are beginning to follow Apple by making fast iGPUs with more memory bandwidth.

Now there's a case where the CPU might win for prompt processing though and that's when you have one of those new 16 core AMD Zen 5 CPUs with the little 2 CU iGPU.

vk_instance.device_indices.push_back(i);
break;
}
}
}

if (vk_instance.device_indices.empty()) {
vk_instance.device_indices.push_back(0);
GGML_LOG_INFO("ggml_vulkan: No devices found.\n");
return;
}
}
GGML_LOG_DEBUG("ggml_vulkan: Found %zu Vulkan devices:\n", vk_instance.device_indices.size());
Expand Down
Loading