Additional chunks based on references and definitions in code and proc tools

[rmuller-ml]

Some queries benefit from basic navigation (go to ref, go to def). Since adding a new tool to the agent makes the agent too complicated, we expand chunks retrieved by code search and proc with chunks containing definitions and references to symbols in the original chunks. To decide which symbol to add def/refs, we list all of the symbols in the original chunks, filter by some heuristics and use a basic LLM classifier to select a single symbol.

Pipeline

• code search retrieves 10 chunks

• we call get_related_chunks to get 3 additional chunks

  • we list all symbols from the original chunks in extract_symbols
  • we filter local symbols, symbols with no ref/defs and symbol with too many (>5) in extract_symbols
  • we use a llm classifier in filter_stmbols to select a symbol to expand
  • we expand the symbol into snippets in expand_symbol_into_chunks

• same things is done in proc

Implementation

agent/symbol.rs is the main file with the changes
hoverable endpoint is used to collect all symbols from a file
token-info endpoint is used to get refs/defs for a given symbol

Remaining things to do

• revert path filter in proc back to text search (instead of keyword search)
• handle errors by cleaning the code from unwraps
  • llm call fails
  • function call json fails
  • selected symbol alias not in the set of symbols
  • list of symbols is empty
• [x ] make it faster (load documents a single time per chunk)

[gitpod-io]

[ggordonhall]

Can you add a detailed description of the approach in a comment in symbol.rs?