Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Chroma to Vector Database examples #262

Merged
merged 7 commits into from
Apr 6, 2023

Conversation

atroyn
Copy link
Contributor

@atroyn atroyn commented Mar 21, 2023

This PR adds Chroma to the vector database examples in the OpenAI cookbook.

Chroma is a lightweight, developer-friendly embeddings store which aims to get A.I application developers up and running with embeddings as fast as possible.

This PR:

  • Adds Chroma as another example to the generic Using_vector_databases_for_embeddings_search.ipynb
  • Adds an example which shows some of the different tradeoffs in using document retrieval with an embeddings store for LLM-based document Q/A. This uses the gpt3.5-turbo api as well as the ada-002 embeddings API.
  • Adds a scientific question/answering dataset for use with the above example, in jsonl format.

@ted-at-openai
Copy link
Collaborator

Thanks for the PR; I'll take a look. Always happy to include more solutions for end users.

@ted-at-openai ted-at-openai self-requested a review March 22, 2023 19:56
Copy link

@csvoss csvoss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a fun read! I feel like I actually understand how the hallucinated abstracts improve the overall performance, thanks to the notes and commentary.

This code is outside my regular scope but I happened to be here looking this notebook over anyways, so I left notes on a few minor spelling nits that I noticed while reading.

@atroyn
Copy link
Contributor Author

atroyn commented Mar 23, 2023

Thanks @csvoss for catching those spelling nits!

@atroyn
Copy link
Contributor Author

atroyn commented Mar 27, 2023

@ted-at-openai - let me know if there are any changes you'd like me to make here!

@ted-at-openai
Copy link
Collaborator

I'll take a look - thanks for the ping.

@atroyn
Copy link
Contributor Author

atroyn commented Apr 5, 2023

@ted-at-openai following up here again - i'm concerned that this will grow stale soon, we'd like to land this ahead of the retrieval plugin integration.

@ted-at-openai
Copy link
Collaborator

Thanks for following up. Here is my current thinking:

Right now the PR can't be merged because due to merge conflicts stemming from another update made to the notebook.

Ideally I'd like to have one notebook per provider, so we don't have a huge blob of concatenated examples, which is (a) hard to read and (b) hard to have multiple people merge to.

Option 1:

  • You can fix the merge conflicts by applying your changes to the updated notebook (sorry for the extra work)

Option 2:

  • You can wait for us to disentangle the notebook and try to apply your commits to that new framework (this will take longer).

Option 1 will go fastest, but take more work from your side.

I'm sorry I haven't had the time to do this; unfortunately it's not my at the top of my priority list right now. I still really appreciate your contributions, am happy to merge them as soon as ready, and am happy to continue discussing.

@atroyn
Copy link
Contributor Author

atroyn commented Apr 5, 2023

Happy to resolve the conflict on our end

@atroyn atroyn force-pushed the chroma/add_chroma_examples branch from 92a7efa to 7d9d3b9 Compare April 5, 2023 22:21
@atroyn
Copy link
Contributor Author

atroyn commented Apr 5, 2023

All set.

@ted-at-openai ted-at-openai merged commit 1deea48 into openai:main Apr 6, 2023
@ted-at-openai
Copy link
Collaborator

Merged! Thanks for your patience during the long delay. Appreciate your quick changes. Glad to you have your example in the repo.

@atroyn
Copy link
Contributor Author

atroyn commented Apr 6, 2023

Thanks for reviewing! I know it's a busy time, much appreciated.

syusuke9999 pushed a commit to syusuke9999/openai-cookbook that referenced this pull request May 12, 2023
* Vector store notebook

* HyDE with Chroma

* Cleaner text

* add swyx edits

* Cleaned up text / outputs

* Spelling nits

* Fixed comment format

---------

Co-authored-by: swyx <shawnthe1@gmail.com>
syusuke9999 pushed a commit to syusuke9999/openai-cookbook that referenced this pull request May 12, 2023
* Vector store notebook

* HyDE with Chroma

* Cleaner text

* add swyx edits

* Cleaned up text / outputs

* Spelling nits

* Fixed comment format

---------

Co-authored-by: swyx <shawnthe1@gmail.com>
syusuke9999 pushed a commit to syusuke9999/openai-cookbook that referenced this pull request May 12, 2023
* Vector store notebook

* HyDE with Chroma

* Cleaner text

* add swyx edits

* Cleaned up text / outputs

* Spelling nits

* Fixed comment format

---------

Co-authored-by: swyx <shawnthe1@gmail.com>
syusuke9999 pushed a commit to syusuke9999/openai-cookbook that referenced this pull request May 12, 2023
* Vector store notebook

* HyDE with Chroma

* Cleaner text

* add swyx edits

* Cleaned up text / outputs

* Spelling nits

* Fixed comment format

---------

Co-authored-by: swyx <shawnthe1@gmail.com>
syusuke9999 pushed a commit to syusuke9999/openai-cookbook that referenced this pull request May 12, 2023
* Vector store notebook

* HyDE with Chroma

* Cleaner text

* add swyx edits

* Cleaned up text / outputs

* Spelling nits

* Fixed comment format

---------

Co-authored-by: swyx <shawnthe1@gmail.com>
katia-openai pushed a commit that referenced this pull request Feb 29, 2024
* Vector store notebook

* HyDE with Chroma

* Cleaner text

* add swyx edits

* Cleaned up text / outputs

* Spelling nits

* Fixed comment format

---------

Co-authored-by: swyx <shawnthe1@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants