Issue running on RTX 3090 + A100

[ayunami2000]

Hi,

I used the demo code, and Python 3.13, and it always seems to freeze when i run the demo code. On my local machine with a 3090 and 32GB ram, it eventually SIGKILLs; in a Kubernetes cluster with an A100 and 16GB ram (I can increase this if needed for testing, let me know), it eventually terminates the entire container.

Here is what it said when I ran it locally:

This happened right after my music I also have playing on the local machine stuttered/lagged a bit, indicating there are potentially RAM requirements not stated in your docs.

Here is my contents of test.py:

from steerling import SteerlingGenerator, GenerationConfig

generator = SteerlingGenerator.from_pretrained("guidelabs/steerling-8b")

text = generator.generate(
    "The key to understanding neural networks is",
    GenerationConfig(max_new_tokens=100, seed=42),
)
print(text)

[adebayoj]

Hi @ayunami2000, Thanks for the update. We will try to replicate this and get back to you. It was tested on A100 and H100 setups, so perhaps there is indeed an issue.