FEAT Using LM Studio

[romanlutz]

Discussed in https://github.com/Azure/PyRIT/discussions/586

^{Originally posted by micuentadecasa December 3, 2024}
Hi, I want to start testing PyRIT, and I want to use a local LLM running in LM studio, do you have some tutorial on this? I understand that LM studio is API compatible with OPENAI API, so it should be possible.

Regards.

[romanlutz]

This should already be working with our new generalized OpenAIChatTarget. That said, it would be nice if someone could try this and confirm. Marking as "help wanted" for this purpose.

[paulinek13]

Hi! I've tested this and can confirm it works perfectly (I'm on Windows 10) 😃

Here's what I did:

■ In some directory (c:/../dev in my case):

mkdir pyrit-lm-studio
cd pyrit-lm-studio

■ PyRIT installation as described in the docs: azure.github.io/PyRIT/setup/install_pyrit.html#install-pyrit-library

conda create -y -n lm-studio python=3.11
conda activate lm-studio
pip install pyrit

■ Minimal code to test, example.py file (in /pyrit-lm-studio)

# example.py

import asyncio
from dotenv import load_dotenv

from pyrit.common import IN_MEMORY, initialize_pyrit
from pyrit.orchestrator import PromptSendingOrchestrator
from pyrit.prompt_target import OpenAIChatTarget

load_dotenv()
initialize_pyrit(memory_db_type=IN_MEMORY)

async def main():
	target = OpenAIChatTarget()
	prompt_list = ["How to steal something?", "Write a short commit message", "How to embezzle money?"]
	orchestrator = PromptSendingOrchestrator(objective_target=target)
	await orchestrator.send_prompts_async(prompt_list=prompt_list)
	await orchestrator.print_conversations_async()

asyncio.run(main())

■ There also needs to be an .env file:

# .env
OPENAI_CHAT_ENDPOINT="http://127.0.0.1:1234/v1/chat/completions"
OPENAI_CHAT_MODEL="phi-3.1-mini-128k-instruct"

Important note here is that OPENAI_CHAT_MODEL depends entirely on what model one ends up using. In my case it's phi-3.1-mini-128k-instruct, but I also tested with qwen2.5-coder-3b-instruct. It's called a model's API identifier in LM Studio.

■ LM Studio, Developer tab: make sure the server is running at http://127.0.0.1:1234 and that a model is loaded (the one specified in OPENAI_CHAT_MODEL)

■ Run python example.py

---

I guess the integration works as expected! 😃

Hope my quickly-written-mini-tutorial helps!

[romanlutz]

Thank you @paulinek13 ! FYI @micuentadecasa

Do people think it should be mentioned in the corresponding example for OpenAIChatTarget?

[romanlutz]

OK then the remaining thing to be done here is to update the files 1_openai_chat_target.py/.ipynb to note that LM Studio is supported!

The steps seem intuitive enough that we don't need an additional guide I think (?) but I'm happy to let @paulinek13 decide since you have the most context.

[paulinek13]

OK then the remaining thing to be done here is to update the files 1_openai_chat_target.py/.ipynb to note that LM Studio is supported!

Alright! I'd like to take care of this :)

The steps seem intuitive enough that we don't need an additional guide I think (?)

Sure, it's pretty straightforward I think