Initial Proposal Draft for AI Agent for API Testing & Tool Generation #629
Conversation
|
@ashitaprasad, looking for feedback and answers for clarification. |
|
Sure @akshayw1 |
| 6. **Tool Definition Generator**: This component converts API specifications into properly structured tool definitions for various AI frameworks, handling the specific requirements and patterns of each target framework. | ||
|
|
||
| 7. **Benchmark Framework**: The evaluation system that assesses LLM performance on standardized tasks with detailed metrics for accuracy, coverage, relevance, and efficiency. | ||
|
|
There was a problem hiding this comment.
The design is for a standalone service and currently it is not aligned with API Dash.
You will also have to think of UI/UX for this feature not just backend.
There was a problem hiding this comment.
For these I have UI UX in my mind, Will provide figma for that soon
|
|
||
| 7. **Benchmark Framework**: The evaluation system that assesses LLM performance on standardized tasks with detailed metrics for accuracy, coverage, relevance, and efficiency. | ||
|
|
||
| All components will be implemented in Python with comprehensive test coverage and documentation. The architecture will be modular, allowing for component reuse and independent scaling as needs evolve. |
There was a problem hiding this comment.
API Dash is a Flutter project.
Everything has to be implemented in Flutter. LLM interactions will happen via Ollama (local)/ChatGPT/Claude APIs.
|
|
||
| I have some questions for more understanding: | ||
|
|
||
| 1. Which AI frameworks are highest priority for tool definition generation? Is there a specific order of importance for crewAI, langchain, pydantic-ai, and langgraph? |
There was a problem hiding this comment.
Tool definition generation is not a complex task. Once it is done for a framework, it will be easy to replicate for others.
|
|
||
| 1. Which AI frameworks are highest priority for tool definition generation? Is there a specific order of importance for crewAI, langchain, pydantic-ai, and langgraph? | ||
|
|
||
| 2. Do you have preferred LLM providers that should be prioritized for integration, or should the system be designed to work with any provider through a common interface? |
There was a problem hiding this comment.
Ollama API (local)
ChatGPT, Anthropic, Gemini APIs (if the users wants to connect to some API providers)
|
|
||
| 2. Do you have preferred LLM providers that should be prioritized for integration, or should the system be designed to work with any provider through a common interface? | ||
|
|
||
| 3. Are there specific types of APIs that should be given special focus in the benchmark dataset (e.g., e-commerce, financial, IoT)? |
There was a problem hiding this comment.
No. The benchmark must be a good mix.
|
|
||
| 3. Are there specific types of APIs that should be given special focus in the benchmark dataset (e.g., e-commerce, financial, IoT)? | ||
|
|
||
| 4. How will the frontend be planned? Will it be a standalone interface, an extension of an existing dashboard, or fully integrated into an API testing - API Dash client ? |
There was a problem hiding this comment.
It will be fully integrated to API Dash.
|
@ashitaprasad Based on your feedback provided for proposal AI Agent for API Testing and Automated Tool Integration Idea (Refined Proposal)Changes and RefinementsArchitecture and Implementation
System Architecture UpdatesArchitectural Considerations
Benchmark Framework Refinements
UI/UX Considerations
Please review these changes so I can update them in my idea document. Let me know if I'm missing anything. |
|
@akshayw1 any updates you make have to be sent as a PR |
|
Okay sure @ashitaprasad |
PR Description
This PR adds an initial proposal draft for the AI Agent for API Testing & Tool Generation (#620). The project aims to automate API testing using Large Language Models (LLMs), enabling intelligent test case generation, response validation, and seamless integration with AI agent frameworks like crewAI, smolagents, and pydantic-ai.
Related Issues
Feedback
Any insights or suggestions on the architecture, integration strategy, or additional features would be greatly appreciated to refine the proposal further. Looking forward to feedback