How to track input and output tokens to AzureChatOpenaI

[AsmaaMHadir]

I need to track costs across my LangGraph graph whenever I make a call to the Azure deployed openai model. I saw in one of the documentation guide that from langchain_community.callbacks import get_openai_callback is used to get input tokens and cost and other similar metadata. Is this the correct module to use for tracking tokens/cost? I already use Langsmith but I want a programming based implementation to track the tokens in and out for each model call and return It in my API.

[dosubot[bot]]

To track input and output tokens and costs for each model call to an Azure deployed OpenAI model, you can indeed use the get_openai_callback from the langchain_community.callbacks module. This callback allows you to monitor token usage and calculate costs for your model invocations. Here is an example of how to use this callback:

from langchain_community.callbacks import get_openai_callback
from langchain_openai import OpenAI

llm = OpenAI(model_name="gpt-3.5-turbo-instruct")

with get_openai_callback() as cb:
    result = llm.invoke("Tell me a joke")
    print(result)
    print("---")

print(f"Total Tokens: {cb.total_tokens}")
print(f"Prompt Tokens: {cb.prompt_tokens}")
print(f"Completion Tokens: {cb.completion_tokens}")
print(f"Total Cost (USD): ${cb.total_cost}")

This code will track the input and output tokens and calculate the cost based on the usage. The get_openai_callback is indeed the correct module to use for this purpose, and it can be used to track multiple calls as well ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[dosubot[bot]]

The costs associated with retrieval from Azure AI Search typically include:

1. Embeddings Calculation: This involves generating embeddings for the input query, which can incur costs depending on the model and service used for embedding generation.

2. Querying the Index: This includes the cost of searching the index for relevant documents. The cost can vary based on the complexity of the query, the number of documents retrieved, and any additional processing like filtering or ranking.

In your code snippet, the retrieve_documents function uses LangGraphConfig.get_azure_index_retriever to set up the retriever with specific parameters like retrieval_k and retrieval_search_type. These parameters can influence the retrieval cost by affecting the number of documents processed and the type of search performed ^[1].

The line kwargs = {"filters": filter_query} if filter_query else {} is used to apply any document filters specified in the state. This can also impact the cost if the filtering process requires additional computation or affects the number of documents retrieved.

Overall, the main costs are from embeddings calculation and querying the index, but additional factors like filtering and search type can also influence the total cost.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[AsmaaMHadir]

@dosu

How do I track and calculate AzureSearch retrieval costs including embeddings calculation?

[dosubot[bot]]

To track and calculate Azure AI Search retrieval costs, including embeddings calculation, you can follow these steps:

1. Embeddings Calculation: Track the number of embeddings generated for your queries. This can be done by monitoring the input queries and using a callback or logging mechanism to record the number of embeddings processed. If you're using a specific model for embeddings, check if it provides any built-in utilities for tracking usage.

2. Querying the Index: Monitor the number of queries made to the Azure AI Search index and the number of documents retrieved. This can be done by logging each query and its results. The LangGraphConfig.get_azure_index_retriever function allows you to specify parameters like top_k and search_type, which can affect the cost by determining the number of documents retrieved and the complexity of the search ^[1].

3. Cost Calculation: Combine the data from embeddings and index querying to calculate the total cost. Azure AI Search typically charges based on the number of operations and the amount of data processed, so you'll need to refer to Azure's pricing details to convert your usage metrics into costs.

By implementing logging or using callbacks to track these metrics, you can effectively monitor and calculate the costs associated with Azure AI Search operations.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[AsmaaMHadir]

@dosu

I use

AzureSearch(
            azure_search_endpoint=LangGraphConfig.AZURE_SEARCH_ENDPOINT,
            azure_search_key=LangGraphConfig.AZURE_SEARCH_KEY,
            index_name=LangGraphConfig.AZURE_AI_SEARCH_INDEX_NAME,
            embedding_function=embeddings.embed_query,
            additional_search_client_options={"retry_total": 4},
            semantic_configuration_name="semanticConfig",
            vector_search_profile_name="vector-profile-latest" # switch from "vector-profile-1739219669397"
        )

with embeddings being:

AzureOpenAIEmbeddings(
            azure_endpoint=LangGraphConfig.AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT,
            api_version=LangGraphConfig.AZURE_EMBEDDINGS_API_VERSION,
            api_key=LangGraphConfig.AZURE_API_KEY
        )

How do I calculate the cost of invoking the AzureSearch instance for retrieval of k documents given a query?

I need to know how to get:

• Cost for calculating embeddings of the input query
• Cost for retrieving k documents from the azure index

[dosubot[bot]]

To calculate the cost of invoking the AzureSearch instance for retrieval of k documents given a query, you need to consider two main components: the cost of calculating embeddings for the input query and the cost of retrieving documents from the Azure index.

1. Cost for Calculating Embeddings:

   • The cost of generating embeddings depends on the pricing model of the Azure OpenAI service you are using. Typically, this cost is based on the number of tokens processed. You can track the number of tokens used for generating embeddings by using a callback or logging mechanism, similar to how you track tokens for model invocations.
   • You will need to refer to Azure's pricing details for the specific model and API version you are using to convert the number of tokens into a cost.

2. Cost for Retrieving Documents:

   • The cost of querying the Azure Search index is generally based on the number of queries and the complexity of each query. This includes factors like the number of documents retrieved (k), the type of search (e.g., similarity, hybrid), and any additional processing such as filtering or semantic search.
   • Azure Search pricing typically involves charges for the number of operations and the volume of data processed. You can monitor the number of queries and the parameters used (e.g., k, search type) to estimate the cost.
   • Again, refer to Azure's pricing details for Azure Cognitive Search to determine the cost per query and any additional charges based on your configuration.

By tracking these metrics and referring to Azure's pricing documentation, you can calculate the total cost of using the AzureSearch instance for document retrieval.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}