Bug with "RetrievalQAWithSourcesChain" with AzureOpenAI - works as expected with OpenAI

[ventz]

I noticed that when I moved this solution from OpenAI to AzureOpenAI (same model), it produced non-expected results.
After digging into it, discovered that they may be a problem with the way RetrievalQAWithSourcesChain.from_chain_type utilizes the LLM specifically with the map_reduce chain.
(it does not seem to occur with the refine chain -- it seems to work as expected)

Here is the full example:
I ingested a website and embedded it into the ChromaDB vector database.
I am using the same DB for both of my tests side by side.

I created a "loader" (text interface) pointed at OpenAI, using the 'text-davinci-003' model, temperature of 0, and using the ChromaDB of embeddings.

Asking this question: "What paid holidays do staff get?"
I get this answer:

Answer: Staff get 12.5 paid holidays, including New Year's Day, Martin Luther King, Jr. Day, Presidents' Day, Memorial Day, Juneteenth, Independence Day, Labor Day, Columbus Day/Indigenous Peoples' Day, Veterans Day, Thanksgiving, the Friday after Thanksgiving, Christmas Eve 1/2 day, and Christmas Day, plus a Winter Recess during the week between Christmas and New Year.
Source: https://website-redacted/reference1, https://website-redacted/reference2

When moving this loader over to the AzureOpenAI side --

Asking the same question: "What paid holidays do staff get?"
And I get this answer:

I don't know what paid holidays staff get.

There are only 3 changes to move OpenAI -> AzureOpenAI

1.) Removing the:

oaikey = "..."
os.environ["OPENAI_API_KEY"] = oaikey

And switching it out for:

export OPENAI_API_TYPE="azure"
export OPENAI_API_VERSION="2023-03-15-preview"
export OPENAI_API_BASE="https://endpoit-redacted.openai.azure.com/"
export OPENAI_API_KEY="..."

2.) Changing my OpenAI initializer to use the deployment_name instead of model_name

temperature = 0
embedding_model="text-embedding-ada-002"
openai = OpenAI(
    model_name='text-davinci-003',
    temperature=temperature,
)

to:

deployment_id="davinci" #(note: see below for the screenshot - set correctly)
embedding_model="text-embedding-ada-002"
temperature = 0

openai = AzureOpenAI(
    deployment_name=deployment_id,
    temperature=temperature,
)

Here are the Azure models we have:

3.) Changing the langchain loader from:

from langchain.llms import OpenAI

to:

from langchain.llms import AzureOpenAI

Everything else stays the same.

The key part here is that it seems to fail when it comes to map_reduce with Azure:

qa = RetrievalQAWithSourcesChain.from_chain_type(llm=openai, chain_type="map_reduce", retriever=retriever, return_source_documents=True)

I have tried this with a custom chain_type_kwargs arguments overriding the question and combine prompt, and without (using the default).
It fails in both cases with Azure, but works exactly as expected with OpenAI.

Again, this seems to fail specifically around the map_reduce chain when it comes to Azure, and seems to produce results with refine.

If using with OpenAI -- it seems to work as expected in both cases.

[floleuerer]

Do you use the embeddings created with the OpenAI Embedding API? Azure uses an older version of the Embedding model - and the created embeddings differ from the OpenAI model. So make sure to use OpenAI Embeddings with the OpenAI Embedding API and Azure Embeddings with the Azure Embedding API.

As far as I know there will (maybe is already) be a new version 2 of the text-embedding-ada-002 model in Azure that is exactly the same as the OpenAI version and will give the same embeddings. If you use version 1 that could be the reason for the issue.

https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#embeddings-models

[ventz]

@floleuerer Thank you! -- I'll verify. I suspect the embeddings were created using the OpenAI engine.

[ventz]

@floleuerer Just some interesting information to follow up on this.

Moved the embeddings to Azure's OpenAI:

Azure AI's embedding model limits the document "submits" to 1 currently. (this seems insane!)

openai.error.InvalidRequestError: Too many inputs for model None. The max number of inputs is 1.  We hope to increase the number of inputs per request soon. Please contact us through an Azure support request at: https://go.microsoft.com/fwlink/?linkid=2213926 for further questions.

That is -- the docs in the Chroma from_documents is limited to a single one! Ex:

loader = UnstructuredURLLoader(urls=urls)
docs = loader.load()
db = Chroma.from_documents(documents=docs,...)

One solution to work around this would be something which embeds each document

docs = []
for url in urls:
    loader = UnstructuredURLLoader(urls=[url])
    d = loader.load()
    db = Chroma.from_documents(documents=d, embedding=embeddings, ...)

With that said -- if you do work around it to create the embeddings, I am seeing much "weaker" and not as rich answers, when using the text-embedding-ada-002 and davinci models.

So, either the embedding model, and/or the latest davinci models are simply not the same between Azure's OpenAI and OpenAI's.

Posting this here in case someone else comes up on it in the future.

[pm78]

Thank you. I experience the same issue with AzureOpenAI, with the text-embedding-ada-002. The embedding works only with 1 doc at a time...
openai.error.InvalidRequestError: Too many inputs. The max number of inputs is 1. We hope to increase the number of inputs per request soon. Please contact us through an Azure support request at: https://go.microsoft.com/fwlink/?linkid=2213926 for further questions.
But actually, I don't want to implement a work around for now. Will wait for the update by MSoft.

[ventz]

@pm78 Implementing a work-around is very simple -- it's effectively a loop where you just create a "Document" of each item and embbed it. I am just surprised by this limitation Azure imposes...it's pretty impacting.

[floleuerer]

The workaround for the Azure OpenAI Embedding Model to use Chroma.from_documents is to set chunk_size=1 in the embedding model.

embeddings = OpenAIEmbeddings(chunk_size=1)
...
db = Chroma.from_documents(documents=docs,...)

[pm78]

@floleuerer Thank you for the tip, it works!... until I try to run an LLMChain.
I get the following error: openai.error.InvalidRequestError: Must provide an 'engine' or 'deployment_id' parameter to create a <class 'openai.api_resources.chat_completion.ChatCompletion'>
I wonder if it is because I use a gpt-4 instance.
@ventz @floleuerer your thoughts/advices?

[ventz]

@pm78 See my post here: #2377 (comment)

Specifically # 2 -- you have to change a few things to go from OpenAI -> Azure's OpenAI.

[pm78]

I've done all this:

in my ".env"
OPENAI_API_TYPE="azure"
OPENAI_API_BASE="https://wlgpt4us.openai.azure.com/"
OPENAI_API_VERSION="2023-03-15-preview"
LLM_DEPLOYMENT_ID="gpt4test"
LLM_DEPLOYMENT_TYPE="gpt-4"

in my pthon code:
(...)
##########################################
# initializing the llm type
##########################################
openai_api_type = os.environ.get('OPENAI_API_TYPE', "openai")
llm_deployment_id = os.environ.get('LLM_DEPLOYMENT_ID', "gpt4test")
llm_deployment_type = os.environ.get('LLM_DEPLOYMENT_TYPE', "gpt-4")
if openai_api_type == "azure":
llm = AzureOpenAI(deployment_name=llm_deployment_id, model_name=llm_deployment_type, openai_api_base="https://xxxx.openai.azure.com/", openai_api_version="2023-03-15-preview")
else:
llm = ChatOpenAI(model_name="gpt-4") # or gpt-3.5-turbo
print (f'API type >> {MAGENTA}{openai_api_type}{RESET}')
print (f'llm_deployment_id >> {MAGENTA}{llm_deployment_id}{RESET}')
print (f'llm_deployment_type >> {MAGENTA}{llm_deployment_type}{RESET}')
(...)

And the variable are ok:
API type >> azure
llm_deployment_id >> gpt4test
llm_deployment_type >> gpt-4

Did I forget anything?
The embeddings work ok.
But the LLMChain seems to call "completion" instead of "chat", and generate a warning: UserWarning: You are trying to use a chd, please use: from langchain.chat_models import ChatOpenAI

Anyway the engine and deployment_id are not passed properly.

Any other idea?

[ventz]

@pm78 For your AzureOpenAI -- initialize it with with just deployment_name. There is no model_name for that.

If you initialize these as env variables, you don't need to pass them to the constructor:

os.environ["OPENAI_API_TYPE"] = "azure"
os.environ["OPENAI_API_VERSION"] = "2023-03-15-preview"
os.environ["OPENAI_API_BASE"] = "https://wlgpt4us.openai.azure.com"
os.environ["OPENAI_API_KEY"] = "..."

See here for the ChatOpenAI: https://python.langchain.com/en/latest/use_cases/agents/wikibase_agent.html?highlight=ChatOpenAI#specify-the-llm-model

For OpenAI it's just "model".

But if you are using Azure's version, you want AzureChatOpenAI, which takes different parameters -- see:
https://python.langchain.com/en/latest/modules/models/chat/integrations/azure_chat_openai.html?highlight=ChatOpenAI%20azure#azure

[pm78]

Great. But that's exactly what I was doing.
I just checked: my code works perfectly with GPT-3, but not with GPT-4.
It seems that langchain has not implemented LLMChain with Azure + GPT-4 (or GPT-3.5).
I guess it will be corrected soon.

[dosubot]

Hi, @ventz. I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

Based on my understanding of the issue, there seems to be a bug with the "RetrievalQAWithSourcesChain" function in AzureOpenAI. It appears that the function is not producing the expected results when using the "map_reduce" chain type. There have been some discussions in the comments about the issue possibly being related to the use of embeddings created with the OpenAI Embedding API instead of the Azure Embedding API. It seems that the limitation of the Azure OpenAI embedding model to only allow 1 document "submit" is also impacting the results.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project.