Doesn't work on the latest RTX 50 Series cards due to Python Torch version

[digitalassassins]

The Python torch version is outdated and doesn't support the latest RTX 50 series cards.

If you want to use it on Docker with the latest RTX 50 series cards, you need to

1. Download the git repository.
2. edit the pyproject.toml

Change project-optional.dependencies to:

[project.optional-dependencies]
gpu = [
    "torch==2.7.0+cu128",
]

That's lines 43 - 46

Change tool.uv.index

[[tool.uv.index]]
name = "pytorch-cuda"
url = "https://download.pytorch.org/whl/cu128"
explicit = true

That's line 78 - 81

Build Docker package as normal, and the package will now work with RTX50 series GPUs

[RBEmerson970]

Um, and testing for 50 series before making the changes, I trust?

[digitalassassins]

yes, I had to do it, as it no longer works since upgrading to a 50 series, so I had to make the changes to make it work. Just put the update on here in case others are having the same issue.

All users of 50 series cards are having problems with PyTorch version being outdated in all software:

comfyanonymous/ComfyUI#7127

InvokeAI have just bumped the pytorch version to 2.7.0 to support 50 series cards on desktop version, but not in the latest Docker image yet. So that has to be done manually as well.

[RBEmerson970]

Check the postings in here, and you'll see PyTorch versions are very much an issue across the spectrum of GPU's. "Latest and Greatest" breaks things for some folks.

[digitalassassins]

Yeah, I understand that, thats why I didn't add a pull request. checked the same build image using Docker on an RTX2060 and works fine, but I understand it's not guaranteed to work on all GPUs and can cause issues. I can't test every GPU, so I just put this here for others in case they are struggling to get it to work.

[FlippingBinary]

I was using the Kokoro-FastAPI Docker image until I upgraded my hardware and was disappointed to discover this total blocker of an issue. The legacy version of PyTorch is not compatible with the current generation of NVidia GPUs at all.

Even if PyTorch's current release causes issues for some people, Docker containers don't automatically replace themselves with the latest image unless something has been configured to do so. That means current users won't have any problems out of the blue, even if this project updates to the current PyTorch release. Similarly, users who try the "Latest and Greatest" version of this project can always choose to use an older version if the newer version introduces breaks something for them. Users running it natively can just pull a tagged commit from GitHub, and Docker users can simply replace latest with v0.2.4, or whichever version they want to use. A well-designed Docker image will always work if it ever worked.

People with current generation hardware have a workaround that involves building the Docker image locally, but that can take some tinkering and may result in build errors that are time consuming to resolve and completely duplicates the efforts of multiple people. It also increases the difficulty of updating to new versions of this project because it can cause merge conflicts.

For me, it made more sense to switch to Speaches, which is a project that provides speech recognition with Whisper in addition to speech synthesis with Kokoro, and it even lets you pull any compatible model you want from Hugging Face. It uses ONNX instead of PyTorch, so there is a potential tradeoff in terms of performance, but the Kokoro-FastAPI project looks like it's also moving towards ONNX anyway and doesn't seem to be updating PyTorch versions any time soon.

[digitalassassins]

I have added many custom modifications to Kokoro-FastAPI filtering to make it perfect for my needs. So updating the PyTorch version was a no-brainer. It's also lightning fast on the 50 series cards 😁

[FlippingBinary]

It's a tradeoff for sure, but I'm too quick to get sucked into side projects at the detriment of whatever I need to be doing. I was doing that with an attempt at adding the current version of PyTorch as an optional feature of this project with the intention of submitting a pull request, but I had to set it aside after running into build errors and running out of time for it. I just went without it. More recently, I got Home Assistant up and running again and wanted to add both STT and TTS, so switching to Speaches just made more sense. It generates speech just as fast, if not faster, than any other voice assistant, so it works well for me. I've seen benchmarks comparing PyTorch with ONNX showing some differences that swing the needle either way depending on the length of the spoken text and whether the model is in memory and things like that, but I don't think I can notice a perceptible difference for my use case. If anything, Speaches with ONNX seems a bit faster, but it's not a fair comparison because I haven't tested it head-to-head.