chore: performing sparse search with query expansion in langchain

[bruvduroiu]

Problem

Showcase the use of Pinecone sparse embeddings and sparse indexes for in situations where dense vector search might miss specific technical terms.

Solution

• use Pinecone sparse embeddings
• use query expansion technique
• allow Langchain agent to search for specific technical terms from general queries ("Which tech companies were top performers this week?" transforms into vector search about "GOOGL performance", "NVIDIA Performance", ...

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
• Infrastructure change (CI configs, etc)
• Non-code change (docs, etc)
• None of the above: (updates to documentation)

Test Plan

Describe specific steps for validating this change.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[jamescalam]

• move imports to the cells that they're used — it makes it easier for the user to see where various objects etc of coming from as they're working through the code
• swap to using HF datasets
• the cell where you're setting up the index is too complicated — it needs breaking apart and simplifying, see other code examples (such as rag-chatbot.ipynb) — try to avoid doing multiple actions in a single cell and if we do, explain each new step with a comment above/next to the code
• explanation in markdown on "here we are setting up the index, dimension is None because X, metric, etc" — also need to point people to where to get their API keys and setup as we do in other notebooks
• missing explanation of what @tool is doing and why we need it
• random from langchain.memory import ConversationBufferMemory and from langchain.agents import create_tool_calling_agent, AgentExecutor imports mid-cell
• expansion_llm = ChatOpenAI(temperature=0.7) used in query expansion tool can be a smaller LLM imo, either gpt-4.1-nano or gpt-4.1-mini — using smaller model also shows another reason why we might use query expansion as a separate tool rather than just built in to the prompting etc
• we shouldn't use create_tool_calling_agent — better use LCEL, can refer to aurelio langchain course code
• might be a good idea to explicitly define which model we're using for llm = ChatOpenAI()

[jamescalam]

moving to in-repo PR