Cost calculation is always zero with fine-tuned OpenAI models

[robatwilliams]

System Info

LangChain 0.0.240

but code on latest master has same issue

Who can help?

@agola11

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

Use a fine-tuned OpenAI model, named like "ft:gpt-2.5-turbo-0613:orgname@12abCDEF"

with get_openai_callback() as openai_callback:

openai_callback.total_cost

The cost is zero.

Expected behavior

The cost is returned.

The issue is in https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/callbacks/openai_info.py#L67

which expects fine-tuned models' names' to start with "ft-" (dash, should be colon)

[dosubot]

🤖

Hello,

Thank you for bringing this issue to our attention. It seems like the problem lies in the standardize_model_name method in the OpenAICallbackHandler class. This method is used to standardize the model name to a format that can be used in the OpenAI API.

For fine-tuned models, the method splits the model name on the colon and only keeps the part before the colon:

if "ft-" in model_name:
    return model_name.split(":")[0] + "-finetuned"

However, as you pointed out, the fine-tuned models' names start with "ft-" instead of a colon. This means that the method is not correctly identifying fine-tuned models and is therefore not correctly standardizing the model names for these models. This could lead to the cost calculation always being zero.

To fix this issue, the condition to check for fine-tuned models should be updated to check for names starting with "ft-" instead of containing a colon. The corrected code would look like this:

if model_name.startswith("ft-"):
    return model_name.split(":")[0] + "-finetuned"

This change will ensure that fine-tuned models are correctly identified and their names are correctly standardized, which should fix the issue with the cost calculation always being zero.

I will create a pull request with this fix and link it to this issue for tracking. Once the pull request is merged, you can update your LangChain version to include this fix.

Thank you for your patience and for helping us improve LangChain.

Best,
Dosu

Sources

• get_openai_callback total_cost BROKEN
• Tracking total cost for gpt-3.5-turbo-0613 yields $0.0
• LangChain openAI callback doesn't allow finetuned models
• libs/langchain/langchain/llms/openai.py
• libs/langchain/langchain/callbacks/openai_info.py
• libs/langchain/tests/integration_tests/callbacks/test_openai_callback.py
• libs/langchain/langchain/callbacks/manager.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[robatwilliams]

There are several things wrong in the bot message above. I think a person would understand from the information I provided.

[dosubot]

Hi, @robatwilliams,

I'm helping the LangChain team manage their backlog and am marking this issue as stale. From what I understand, you encountered a problem with cost calculation always returning zero for fine-tuned OpenAI models. The issue seems to stem from the code expecting the model names to start with "ft-" instead of a colon. There was a detailed explanation provided by a user named dosubot, along with a proposed code change. However, it seems there was dissatisfaction with my response and an emphasis on the need for human understanding of the issue.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!

[robatwilliams]

It appears, as you can see above, that #12190 has fixed this.

I haven't checked it but it's a safe assumption that it's safe to close this.

[dosubot]

🤖

Thank you, @robatwilliams, for closing the issue! Your contribution is much appreciated.

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.