Connectivity metrics tool (obione)

[jankrepl]

Closes #550

TODO

• Wait to get a proper description from the scientists on how to fill out the request body
  • Make description of the get circuits metrics better and metion that for level of node detail = 1 you get all the possible "dropdowns" basically
• Override the autogen class to avoid passing null values - decided not to do it
• Explain the sonata syntax to the LLM (unless it knows it already)
• The LLM confuses the obi tools with the entitycore get one circuit
• The LLM seems to be also confused about the output format (even though you mentioned it was the pandas to_dict`

[WonderPG]

I tried a couple of times with the question you gave in the eval, but I could not get the tool to work. It kept returning a non 200 status code from obi-one. Not sure if the issue is the input of the tools from the LLM of their endpoint.

[WonderPG]

Crazy that we have to add a new rule specifically to describe how to use obi-tools. Do you think they are intrinsically complicated and this is unavoidable, or the complexity comes from the decisions made in the repo ?

[jankrepl]

I was guided by the /eval and just iterated until the Tool Correctness was 1 among other thing. Not claiming it is pretty. Honestly, we should definitely refactor all the circuit related tools in the near future because it is getting messy.

[jankrepl]

I changed this rule file name to circuit-tools.mdc and explain all the different circuit related endpoints across entitycore and obi-one.

[WonderPG]

I guess you have tried before, but wasn't it enough to have these explanations in the tool description ?

[jankrepl]

As mentioned in another comment, I was guided by the eval. But very likely it is possible to make things simpler. I just feel like we should rethink how we do these circuit related tools. Because right now there are so many of them.

[jankrepl]

Just for reference, the evaluate_agent.py sometimes errors out

  File "Desktop/Projects/neuroagent/backend/src/neuroagent/scripts/evaluate_agent.py", line 481, in eval_sample                                                                                                                                                                                                  
    input_class = tool_class.__annotations__["input_schema"](        
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Desktop/Projects/neuroagent/backend/.venv/lib/python3.12/site-packages/pydantic/main.py", line 253, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 4 validation errors for CircuitConnectivityMetricsGetOneToolInput
pre_selection.layer.str                                                                                                                                                                                                                                                                                                       
  Input should be a valid string [type=string_type, input_value=2, input_type=int]
    For further information visit https://errors.pydantic.dev/2.11/v/string_type
pre_selection.layer.list[str]                                                 
  Input should be a valid list [type=list_type, input_value=2, input_type=int]
    For further information visit https://errors.pydantic.dev/2.11/v/list_type
post_selection.layer.str                                                                                                                                       
  Input should be a valid string [type=string_type, input_value=2, input_type=int]
    For further information visit https://errors.pydantic.dev/2.11/v/string_type        
post_selection.layer.list[str]                                                                                                                                                                                                                                                                                                
  Input should be a valid list [type=list_type, input_value=2, input_type=int]
    For further information visit https://errors.pydantic.dev/2.11/v/list_type

EDIT: I handled it in evaluate_agent.py.