SceneProgLLM

SceneProgLLM is a powerful and versatile Python package that wraps around LangChain's LLM interface to provide enhanced functionality, including support for text, code, JSON, list and pydantic response formats along with image input/output, caching, and multiple endpoints. This project is built to support SceneProg projects.

---

Features

1. Flexible Response Formats:
   • Supports text, code, list, JSON, pydantic and image outputs.
2. Image Input and Output:
   • Accepts image inputs and enables image generation through Stable Diffusion (SD) or OpenAI's image generation API.
3. Caching:
   • Integrated caching system to store and retrieve previous query responses for faster execution.
4. System Template:
   • Allows users to set a system description template containing placeholders which can be later filled with values.

---

Installation

To install the package and its dependencies, use the following command:

pip install sceneprogllm

For proper usage, create a .env file in the package root with following fields:

TEXT2IMGSD=<endpoint for text to image generation>
OPENAI_API_KEY=<Your OpenAI key>
OLLAMA_HOST=Ollama host IP address
OLLAMA_PORT=Ollama host Port

Getting Started

Importing the Package

from sceneprogllm import LLM

Usage Examples

1. Generating Text Responses

llm = LLM(name="text_bot", response_format="text")
response = llm("What is the capital of France?")
print(response)

>> The capital of France is Paris.

2. Generating JSON Responses

llm = LLM(
    name="json_bot",
    response_format="json",
    json_keys=["capital:str", "currency:str"]
)
query = "What is capital and currency of India?"
response = llm(query)
print(response)

>> {'capital': 'New Delhi', 'currency': 'Indian Rupee'}

3. Generating List Responses

llm = LLM(
   name="list_bot",
   response_format="list",
)
query = "List G7 countries"
response = llm(query)
print(response)

>> ['Canada', 'France', 'Germany', 'Italy', 'Japan', 'United Kingdom', 'United States']

4. Generating Pydantic Responses

from pydantic import BaseModel, Field
class mypydantic(BaseModel):
    country: str = Field(description="Name of the country")
    capital: str = Field(description="Capital city of the country")

llm = LLM(
    name="pydantic_bot",
    response_format="pydantic",
)
response = llm("What is the capital of France?", pydantic_object=mypydantic)
print(response)

>> country='France' capital='Paris'

5. Generating Python Code

llm = LLM(name="code_bot", response_format="code")
query = "Write a Python function to calculate factorial of a number."
response = llm(query)
print(response)

>>
def factorial(n):
    if n < 0:
        raise ValueError("Factorial is not defined for negative numbers")
    elif n == 0 or n == 1:
        return 1
    else:
        result = 1
        for i in range(2, n + 1):
            result *= i
        return result

6. Generating images from text

llm = LLM(name="image_bot", response_format="image")
response = llm("Generate an image of a futuristic cityscape.")
response.save("futuristic_city.jpg")

>> 

7. Query using Images

llm = LLM(name="image_bot", response_format="json", json_keys=["count:int"])
image_paths = ["assets/lions.png"]
response = llm("How many lions are there in the image?", image_paths=image_paths)
print(response)

>> {'count': 6}

8. Clear LLM cache

from sceneprogllm import clear_llm_cache
clear_llm_cache()

9. Set seed and temperature

llm = LLM(
         name="seed_bot",
         seed=0,
         temperature=1.0
         )

10. Control behavious via system description

llm = LLM(
   name="system_bot",
   system_desc="You are a funny AI assistant",
)
response = llm("What is the capital of France")
print(response)

>> 

Ah, the capital of France! That's Paris, the city of romance, lights, and baguettes longer than your arm! Just imagine the Eiffel Tower wearing a beret and saying, "Bonjour!"

11. Using Template

from sceneprogllm import LLM
llm = LLM(
    name="template_bot",
    system_desc="You are a helpful assistant. {description}",
)

response = llm("What is the capital of France?", system_desc_keys={"description": "You are a funny AI assistant"})
print(response)

Using Ollama

Additional models like Deekseek-R1 and Llama3.2-vision are available via Ollama with the LLM class, make sure Ollama is installed. Then, when initializing the LLM object, specify the Ollama Model via model_name. For example, "llama3.2-vision".

See Ollama model site for the available options. Note that different model will support different modes (text, image, etc.).

Here is an example:

from sceneprogllm import LLM

# Example for generating text responses using Ollama
llm = LLM(name="text_bot", response_format="text", model_name="llama3.2-vision:90b")

Queries

Please send your questions to k5gupta@ucsd.edu