Clarification on Azure OpenAI usage, CSV file support, and troubleshooting a clustering error in graphrag/graphrag-accelerator

[shipaleks]

Hello, I'm new to graphrag and graphrag-accelerator, and I have a few questions after going through the advanced_getting_started.ipynb notebook and attempting to use graphrag with my own data:

1. I'm a bit confused about the relationship between graphrag and graphrag-accelerator, especially regarding Azure OpenAI usage. In the base graphrag package, I can set my Azure OpenAI credentials in a .env or yaml file instead of using an OpenAI API key. Does this mean I can use my Azure OpenAI subscription without necessarily using graphrag-accelerator? If so, are there any examples showing how to correctly configure these parameters for Azure OpenAI (assuming I already have the necessary models deployed in Azure OpenAI)?

2. My main question is about file format support: Can I use CSV files with graphrag-accelerator instead of TXT files, as is possible with the base graphrag package? All the example notebooks I've seen only mention TXT files. With CSV files, I could potentially set parameters for each chunk (like date or source) in separate columns. Is this possible with graphrag-accelerator, or is it limited to TXT files only?

3. I also encountered an error while processing a large CSV file (about 30k rows) using graphrag on my Mac. The initial processing completed successfully (the progress bar reached 100%, and my OpenAI API usage increased noticeably). However, in the final stage, I received this error:

   Error executing verb "cluster_graph" in create_base_entity_graph: Columns must be same length as key
   

   I suspect this error might be related to how line breaks are handled in the CSV (I replaced '\n' with actual line breaks), or possibly because I'm using Cyrillic data (though I'm not sure why this would be a problem). Has anyone encountered a similar issue? How can I resolve this?

Thank you in advance for any clarification you can provide!

[timothymeyers]

Hi @shipaleks - first and foremost, thanks for checking out graphrag and kicking the tires!

I'll try to answer what I can here or steer you in the right direction.

(1) Think of the graphrag-accelerator as the reference implementation for how you could deploy an end-to-end graphrag solution in Azure. It is a set of Infra as Code (IaC) and other scripts that deploy additional cloud services that help flesh out a full solution and host GraphRAG.

It serves as an example of how you could do it, but many of the pieces could be swapped or even removed (e.g., CosmosDB is used to store metadata and status info, among other things, but another datastore could be used). The Accelerator puts an APIM service (and API) in front of the GraphRAG library, and this APIM API key is what the Accelerator notebooks use to access GraphRAG.

You can absolutely use Azure OpenAI and graphrag itself without using the graphrag-accelerator. The graphrag repository and docs have examples for doing everything in python locally with calls out to Azure OpenAI.

(2) The accelerator is hardcoded to support TXT files but as you noted the library itself can support CSV. I'm not sure all of the places where you'd need to adjust things, but I would take a look at the backend container configuration in pipeline-settings.yaml and logic in the data.py file. Then the notebooks would need to be adjusted. I haven't attempted any of this yet, so if you do please report back!

(3) This question is probably best suited to be a Discussion (possibly an issue) in the graphrag library repo.

Hope this helps put you on a useful path!