diff --git a/notebook/agentchat_memory_using_mem0.ipynb b/notebook/agentchat_memory_using_mem0.ipynb
new file mode 100644
index 00000000000..d590002164b
--- /dev/null
+++ b/notebook/agentchat_memory_using_mem0.ipynb
@@ -0,0 +1,353 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "A3guIYLG6PnX"
+ },
+ "source": [
+ "# Agent with memory using Mem0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "di1opljX6sQT"
+ },
+ "source": [
+ "This notebook demonstrates an intelligent customer service chatbot system that combines:\n",
+ "\n",
+ "- PyAutoGen for conversational agents\n",
+ "- Mem0 for memory management\n",
+ "\n",
+ "[Mem0](https://www.mem0.ai/) provides a smart, self-improving memory layer for Large Language Models (LLMs), enabling developers to create personalized AI experiences that evolve with each user interaction. Refer [docs](https://docs.mem0.ai/overview) for more information.\n",
+ "\n",
+ "\n",
+ "Mem0 uses a hybrid database approach, combining vector, key-value, and graph databases to efficiently store and retrieve different types of information. It associates memories with unique identifiers, extracts relevant facts and preferences when storing, and uses a sophisticated retrieval process that considers relevance, importance, and recency.\n",
+ "\n",
+ "Key features of Mem0 include:\n",
+ "1. Comprehensive Memory Management: Easily manage long-term, short-term, semantic, and episodic memories for individual users, agents, and sessions through robust APIs.\n",
+ "2. Self-Improving Memory: An adaptive system that continuously learns from user interactions, refining its understanding over time.\n",
+ "3. Cross-Platform Consistency: Ensures a unified user experience across various AI platforms and applications.\n",
+ "4. Centralized Memory Control: Simplifies storing, updating, and deleting memories.\n",
+ "\n",
+ "This approach allows for maintaining context across sessions, adaptive personalization, and dynamic updates, making it more powerful than traditional Retrieval-Augmented Generation (RAG) approaches for creating context-aware AI applications.\n",
+ "\n",
+ "\n",
+ "**The implementation showcases how to initialize agents, manage conversation memory, and facilitate multi-agent conversations for enhanced problem-solving in customer support scenarios.**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "EAKSBJcLAf0h"
+ },
+ "source": [
+ "## Requirements\n",
+ "\n",
+ "````{=mdx}\n",
+ ":::info Requirements\n",
+ "Some extra dependencies are needed for this notebook, which can be installed via pip:\n",
+ "\n",
+ "```bash\n",
+ "pip install pyautogen mem0ai\n",
+ "```\n",
+ "\n",
+ "For more information, please refer to the [installation guide](/docs/installation/).\n",
+ ":::\n",
+ "````"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "TA1-sGxD9-zZ"
+ },
+ "source": [
+ "## Get API Keys\n",
+ "\n",
+ "Please get `MEM0_API_KEY` from [Mem0 Platform](https://app.mem0.ai/)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "4nQWyJ-n9qOB",
+ "outputId": "de897f4f-32fd-4359-f021-c3510467b69c"
+ },
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "\n",
+ "from mem0 import MemoryClient\n",
+ "\n",
+ "from autogen import ConversableAgent\n",
+ "\n",
+ "os.environ[\"OPENAI_API_KEY\"] = \"your_api_key\"\n",
+ "os.environ[\"MEM0_API_KEY\"] = \"your_api_key\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "6GAZAkGmAjsT"
+ },
+ "source": [
+ "## Initialize Agent and Memory\n",
+ "\n",
+ "The conversational agent is set up using the 'gpt-4o' model and a mem0 client. We'll utilize the client's methods for storing and accessing memories.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "id": "xWSEHMKT9qLz"
+ },
+ "outputs": [],
+ "source": [
+ "agent = ConversableAgent(\n",
+ " \"chatbot\",\n",
+ " llm_config={\"config_list\": [{\"model\": \"gpt-4o\", \"api_key\": os.environ.get(\"OPENAI_API_KEY\")}]},\n",
+ " code_execution_config=False,\n",
+ " function_map=None,\n",
+ " human_input_mode=\"NEVER\",\n",
+ ")\n",
+ "\n",
+ "memory = MemoryClient()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "c7vGQoMg_KhS"
+ },
+ "source": [
+ "Initialize a conversation history for a Best Buy customer service chatbot. It contains a list of message exchanges between the user and the assistant, structured as dictionaries with 'role' and 'content' keys. The entire conversation is then stored in memory using the `memory.add()` method, associated with the identifier \"customer_service_bot\"."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "b6ghSobs-mYo",
+ "outputId": "18a6e657-49ba-41cd-8e51-2e76cf668967"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'message': 'ok'}"
+ ]
+ },
+ "execution_count": 18,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "conversation = [\n",
+ " {\n",
+ " \"role\": \"assistant\",\n",
+ " \"content\": \"Hi, I'm Best Buy's chatbot!\\n\\nThanks for being a My Best Buy TotalTM member.\\n\\nWhat can I help you with?\",\n",
+ " },\n",
+ " {\n",
+ " \"role\": \"user\",\n",
+ " \"content\": 'Seeing horizontal lines on our tv. TV model: Sony - 77\" Class BRAVIA XR A80K OLED 4K UHD Smart Google TV',\n",
+ " },\n",
+ " {\n",
+ " \"role\": \"assistant\",\n",
+ " \"content\": \"Thanks for being a My Best Buy Total™ member. I can connect you to an expert immediately - just one perk of your membership!\\n\\nSelect the button below when you're ready to chat.\",\n",
+ " },\n",
+ " {\n",
+ " \"role\": \"assistant\",\n",
+ " \"content\": \"Good evening, thank you for choosing Best Buy, Fnu. My name is Lovely. I hope you are doing well. I'm sorry to hear that you're seeing horizontal lines on your TV.\\n\\nI'm absolutely committed to exploring all possible ways to assist you to fix this issue.\\n\\nTo ensure that we are on the right account, may I please have your email address registered with your Best Buy account?\",\n",
+ " },\n",
+ " {\"role\": \"user\", \"content\": \"dd@gmail.com\"},\n",
+ " {\n",
+ " \"role\": \"assistant\",\n",
+ " \"content\": \"Perfect! Thank you for providing all the details, surely you have made my job easier by doing this. I really appreciate it.\\n\\nI also want to take a moment to express our heartfelt appreciation for your trust and loyalty. Thank you for being an amazing customer of BestBuy Total.\\n\\nCould you please help me with the order number or product's details to check it quickly?\\n\\nSamsung - 49\\\" Odyssey OLED G9 (G95SC) DQHD 240Hz 0.03ms G-Sync Compatible Curved Smart Gaming Monitor - Silver - just to confirm this is the item, right?\",\n",
+ " },\n",
+ " {\"role\": \"user\", \"content\": \"Order number: 112217629\"},\n",
+ " {\n",
+ " \"role\": \"assistant\",\n",
+ " \"content\": \"Superb! Thank you for confirmation.\\n\\nThank you for your patience. After exploring all possible solutions, I can help you to arrange a home repair appointment for your device. Our Geek Squad experts will visit your home to inspect and fix your device.\\n\\nIt's great that you have a protection plan - rest assured, we've got your back! As a valued Total member, you can avail this service at a minimal service fee. This fee, applicable to all repairs, covers the cost of diagnosing the issue and any small parts needed for the repair. It's part of our 24-month free protection plan.\\n\\nPlease click here to review the service fee and plan coverage details -\\n\\nhttps://www.bestbuy.com/site/best-buy-membership/best-buy-protection/pcmcat1608643232014.c?id=pcmcat1608643232014#jl-servicefees\\n\\nFnu - just to confirm shall I proceed to schedule the appointment?\",\n",
+ " },\n",
+ " {\"role\": \"user\", \"content\": \"Yes please\"},\n",
+ " {\"role\": \"assistant\", \"content\": \"When should I schedule the appointment?\"},\n",
+ " {\"role\": \"user\", \"content\": \"Schedule it for tomorrow please\"},\n",
+ "]\n",
+ "\n",
+ "memory.add(messages=conversation, user_id=\"customer_service_bot\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "hlxxncxkAvck"
+ },
+ "source": [
+ "## Agent Inference\n",
+ "\n",
+ "We ask a question to the agent, utilizing mem0 to retrieve relevant memories. The agent then formulates a response based on both the question and the retrieved contextual information."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "wYdDkuKc-Xc8",
+ "outputId": "b50f55d3-a280-4c0d-82e5-43fc0589109b"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Sure, your order number is 112217629.\n"
+ ]
+ }
+ ],
+ "source": [
+ "data = \"I forgot the order numnber, can you quickly tell me?\"\n",
+ "\n",
+ "relevant_memories = memory.search(data, user_id=\"customer_service_bot\")\n",
+ "flatten_relevant_memories = \"\\n\".join([m[\"memory\"] for m in relevant_memories])\n",
+ "\n",
+ "prompt = f\"\"\"Answer the user question considering the memories. Keep answers clear and concise.\n",
+ "Memories:\n",
+ "{flatten_relevant_memories}\n",
+ "\\n\\n\n",
+ "Question: {data}\n",
+ "\"\"\"\n",
+ "\n",
+ "reply = agent.generate_reply(messages=[{\"content\": prompt, \"role\": \"user\"}])\n",
+ "print(reply)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "sryX0gfdBGGD"
+ },
+ "source": [
+ "## Multi Agent Conversation\n",
+ "\n",
+ "Initialize two AI agents: a \"manager\" for resolving customer issues and a \"customer_bot\" for gathering information on customer problems, both using GPT-4. It then retrieves relevant memories for a given question, combining them with the question into a prompt. This prompt can be used by either the manager or customer_bot to generate a contextually informed response."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {
+ "id": "Vq5BFL2l-XZl"
+ },
+ "outputs": [],
+ "source": [
+ "manager = ConversableAgent(\n",
+ " \"manager\",\n",
+ " system_message=\"You are a manager who helps in resolving customer issues.\",\n",
+ " llm_config={\"config_list\": [{\"model\": \"gpt-4\", \"temperature\": 0, \"api_key\": os.environ.get(\"OPENAI_API_KEY\")}]},\n",
+ " human_input_mode=\"NEVER\",\n",
+ ")\n",
+ "\n",
+ "customer_bot = ConversableAgent(\n",
+ " \"customer_bot\",\n",
+ " system_message=\"You are a customer service bot who gathers information on issues customers are facing. Keep answers clear and concise.\",\n",
+ " llm_config={\"config_list\": [{\"model\": \"gpt-4\", \"temperature\": 0, \"api_key\": os.environ.get(\"OPENAI_API_KEY\")}]},\n",
+ " human_input_mode=\"NEVER\",\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {
+ "id": "iX4ehmc6Fbib"
+ },
+ "outputs": [],
+ "source": [
+ "data = \"When is the appointment?\"\n",
+ "\n",
+ "relevant_memories = memory.search(data, user_id=\"customer_service_bot\")\n",
+ "flatten_relevant_memories = \"\\n\".join([m[\"memory\"] for m in relevant_memories])\n",
+ "\n",
+ "prompt = f\"\"\"\n",
+ "Context:\n",
+ "{flatten_relevant_memories}\n",
+ "\\n\\n\n",
+ "Question: {data}\n",
+ "\"\"\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "D3paRZWvCIzt",
+ "outputId": "15eadb7c-5973-44f1-de43-5e6cdebe88de"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "manager (to customer_bot):\n",
+ "\n",
+ "\n",
+ "Context:\n",
+ "Scheduled an appointment for a home repair for tomorrow\n",
+ "Order number is 112217629\n",
+ "TV model is Sony - 77\" Class BRAVIA XR A80K OLED 4K UHD Smart Google TV\n",
+ "User's email address is dd@gmail.com\n",
+ "\n",
+ "\n",
+ "\n",
+ "Question: When is the appointment?\n",
+ "\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "customer_bot (to manager):\n",
+ "\n",
+ "The appointment is scheduled for tomorrow.\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n"
+ ]
+ }
+ ],
+ "source": [
+ "result = manager.send(prompt, customer_bot, request_reply=True)"
+ ]
+ }
+ ],
+ "metadata": {
+ "front_matter": {
+ "description": "Use Mem0 to create agents with memory.",
+ "tags": [
+ "memory"
+ ]
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/notebook/autobuild_agent_library.ipynb b/notebook/autobuild_agent_library.ipynb
index 43521a1d25f..bde50355319 100644
--- a/notebook/autobuild_agent_library.ipynb
+++ b/notebook/autobuild_agent_library.ipynb
@@ -1,926 +1,927 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "6264276d39875995",
- "metadata": {
- "collapsed": false
- },
- "source": [
- "# Automatically Build Multi-agent System from Agent Library\n",
- "\n",
- "By: [Linxin Song](https://linxins97.github.io/), [Jieyu Zhang](https://jieyuz2.github.io/)\n",
- "\n",
- "In this notebook, we introduce a new feature for AutoBuild, `build_from_library`, which help users build an automatic task-solving process powered by a multi-agent system from a pre-defined agent library. \n",
- "Specifically, in `build_from_library`, we prompt an LLM to explore useful agents from a pre-defined agent library, generating configurations for those agents for a group chat to solve the user's task."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ec78dda8e3826d8a",
- "metadata": {
- "collapsed": false
- },
- "source": [
- "## Requirement\n",
- "\n",
- "AutoBuild require `pyautogen[autobuild]`, which can be installed by the following command:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "e8e9ae50658be975",
- "metadata": {
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "%pip install pyautogen[autobuild]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "176c200804af63f3",
- "metadata": {
- "collapsed": false
- },
- "source": [
- "## Preparation and useful tools\n",
- "We need to specify a `config_path`, `default_llm_config` that include backbone LLM configurations."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "2505f029423b21ab",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2023-12-26T16:58:02.762702600Z",
- "start_time": "2023-12-26T16:58:02.472073Z"
- },
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "import json\n",
- "\n",
- "import autogen\n",
- "from autogen.agentchat.contrib.agent_builder import AgentBuilder\n",
- "\n",
- "config_file_or_env = \"OAI_CONFIG_LIST\" # modify path\n",
- "llm_config = {\"temperature\": 0}\n",
- "config_list = autogen.config_list_from_json(config_file_or_env, filter_dict={\"model\": [\"gpt-4-1106-preview\", \"gpt-4\"]})\n",
- "\n",
- "def start_task(execution_task: str, agent_list: list):\n",
- " group_chat = autogen.GroupChat(agents=agent_list, messages=[], max_round=12)\n",
- " manager = autogen.GroupChatManager(groupchat=group_chat, llm_config={\"config_list\": config_list, **llm_config})\n",
- " agent_list[0].initiate_chat(manager, message=execution_task)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "5fb3db8885dd6ee6",
- "metadata": {
- "collapsed": false
- },
- "source": [
- "## Example for generating an agent library\n",
- "Here, we show an example of generating an agent library from a pre-defined list of agents' names by prompting a `gpt-4`. You can also prepare a handcrafted library yourself.\n",
- "\n",
- "A Library contains each agent's name, description and system_message. The description is a brief introduction about agent's characteristics. As we will feed all agents' names and description to gpt-4 and let it choose the best agents for us, each agent's description should be simple but informative. \n",
- "\n",
- "First, we define a prompt template for description and system_message generation and a list of agents' name:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "68315f6ec912c58a",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2023-12-23T07:39:03.317527600Z",
- "start_time": "2023-12-23T07:39:03.276859600Z"
- },
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "AGENT_SYS_MSG_PROMPT = \"\"\"Acccording to the following postion name, write a high quality instruction for the position following a given example. You should only return the instruction.\n",
- "\n",
- "# Position Name\n",
- "{position}\n",
- "\n",
- "# Example instruction for Data Analyst\n",
- "\n",
- "As Data Analyst, you are tasked with leveraging your extensive knowledge in data analysis to recognize and extract meaningful features from vast datasets. Your expertise in machine learning, specifically with the Random Forest Classifier, allows you to construct robust predictive models adept at handling both classification and regression tasks. You excel in model evaluation and interpretation, ensuring that the performance of your algorithms is not just assessed with precision, but also understood in the context of the data and the problem at hand. With a command over Python and proficiency in using the pandas library, you manipulate and preprocess data with ease.\n",
- "\"\"\"\n",
- "\n",
- "AGENT_DESC_PROMPT = \"\"\"According to position name and the instruction, summarize the position into a high quality one sentence description.\n",
- "\n",
- "# Position Name\n",
- "{position}\n",
- "\n",
- "# Instruction\n",
- "{instruction}\n",
- "\"\"\"\n",
- "\n",
- "position_list = [\n",
- " \"Environmental_Scientist\",\n",
- " \"Astronomer\",\n",
- " \"Software_Developer\",\n",
- " \"Data_Analyst\",\n",
- " \"Journalist\",\n",
- " \"Teacher\",\n",
- " \"Lawyer\",\n",
- " \"Programmer\",\n",
- " \"Accountant\",\n",
- " \"Mathematician\",\n",
- " \"Physicist\",\n",
- " \"Biologist\",\n",
- " \"Chemist\",\n",
- " \"Statistician\",\n",
- " \"IT_Specialist\",\n",
- " \"Cybersecurity_Expert\",\n",
- " \"Artificial_Intelligence_Engineer\",\n",
- " \"Financial_Analyst\",\n",
- "]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "72b8e7d9d334a5c2",
- "metadata": {
- "collapsed": false
- },
- "source": [
- "Then we can prompt a `gpt-4` model to generate each agent's system message as well as the description:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "8fbfef9268fc5191",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2023-12-23T07:40:01.703372Z",
- "start_time": "2023-12-23T07:39:04.472589200Z"
- },
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "build_manager = autogen.OpenAIWrapper(config_list=config_list)\n",
- "sys_msg_list = []\n",
- "\n",
- "for pos in position_list:\n",
- " resp_agent_sys_msg = (\n",
- " build_manager.create(\n",
- " messages=[\n",
- " {\n",
- " \"role\": \"user\",\n",
- " \"content\": AGENT_SYS_MSG_PROMPT.format(\n",
- " position=pos,\n",
- " ),\n",
- " }\n",
- " ]\n",
- " )\n",
- " .choices[0]\n",
- " .message.content\n",
- " )\n",
- " resp_desc_msg = (\n",
- " build_manager.create(\n",
- " messages=[\n",
- " {\n",
- " \"role\": \"user\",\n",
- " \"content\": AGENT_DESC_PROMPT.format(\n",
- " position=pos,\n",
- " instruction=resp_agent_sys_msg,\n",
- " ),\n",
- " }\n",
- " ]\n",
- " )\n",
- " .choices[0]\n",
- " .message.content\n",
- " )\n",
- " sys_msg_list.append({\"name\": pos, \"system_message\": resp_agent_sys_msg, \"description\": resp_desc_msg})"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9e26c6db4befacc5",
- "metadata": {
- "collapsed": false
- },
- "source": [
- "The generated profile will have the following format:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "8ede1d7088eb183d",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2023-12-23T07:40:01.712399300Z",
- "start_time": "2023-12-23T07:40:01.707400200Z"
- },
- "collapsed": false
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "[{'name': 'Environmental_Scientist',\n",
- " 'system_message': 'As an Environmental Scientist, you are responsible for applying your profound knowledge of environmental science to analyze ecological data and assess the impact of human activities on natural resources and ecosystems. Your proficiency in environmental assessment techniques enables you to design and conduct field studies, collect samples, and monitor environmental parameters effectively. Utilizing Geographic Information Systems (GIS), you spatially analyze and visualize environmental data to better understand patterns and changes in the landscape. You are adept at interpreting the results and communicating your findings clearly to stakeholders, policymakers, and the public, thereby contributing to informed decision-making on environmental issues. Your role is essential in developing sustainable practices and recommending mitigation measures to minimize environmental degradation and promote conservation.',\n",
- " 'description': 'As an Environmental Scientist, you are tasked with analyzing and assessing the impact of human activities on ecosystems by conducting field studies, using GIS for spatial analysis, and communicating your findings to inform sustainable practices and conservation efforts.'},\n",
- " {'name': 'Astronomer',\n",
- " 'system_message': 'As an Astronomer, your duty involves diligent observation and analysis of celestial phenomena across the universe. Utilize cutting-edge telescopes and instruments to gather astronomical data, looking for patterns and irregularities that can lead to groundbreaking discoveries. Your profound knowledge in astrophysics is pivotal in interpreting these findings, which may include identifying new celestial objects, scrutinizing the properties and behaviors of stars, planets, and galaxies, and understanding cosmic events. Mastery of complex astronomical software and advanced mathematics is crucial for modeling astronomical phenomena and processing the vast amounts of data. Your role is essential in advancing our understanding of the cosmos, contributing to the broader scientific community by publishing your findings in reputable journals and engaging in peer collaboration to further space exploration and research.',\n",
- " 'description': 'An Astronomer is a professional who meticulously observes, analyzes, and interprets celestial phenomena using advanced telescopes and instruments, requiring a deep knowledge of astrophysics, proficiency in mathematical modeling, and collaboration in scientific communities to enhance our comprehension of the universe.'},\n",
- " {'name': 'Software_Developer',\n",
- " 'system_message': 'As a Software Developer, your objective is to craft, test, and maintain the software that will meet the needs of our users and clients. Your proficiency in programming languages such as Java, C#, or JavaScript is essential, enabling you to write clean, efficient, and maintainable code. You will design algorithms and flowcharts to create systems that are logical and user-friendly. Collaboration with cross-functional teams, including product managers and designers, is crucial in order to understand software requirements and deliver innovative solutions. With your understanding of the software development life cycle, you will work through the processes of coding, debugging, testing, and deployment. You will employ industry best practices such as version control with Git and conduct code reviews to maintain high standards of software quality. Your role places you at the heart of our development efforts, where your technical prowess advances the functionality, scalability, and reliability of our software products.',\n",
- " 'description': 'A Software Developer is responsible for designing, coding, testing, and maintaining software that meets client needs using languages like Java, C#, or JavaScript, collaborating with teams, adhering to best practices like Git for version control, and ensuring quality and innovation throughout the development life cycle.'},\n",
- " {'name': 'Data_Analyst',\n",
- " 'system_message': 'As a Data Analyst, your role is pivotal in interpreting complex data and providing insights that inform strategic decision-making. Utilize your analytical skills to cleanse and organize large sets of structured and unstructured data, ensuring its accuracy and readiness for in-depth analysis. Apply statistical analysis and predictive modeling to uncover trends, patterns, and correlations that drive operational improvements and innovative solutions. Use your proficiency in SQL for database interactions, and harness visualization tools such as Tableau or Power BI to craft compelling stories from data, aiding stakeholders in visualizing the implications of your findings. Stay abreast with the latest analytics techniques and continuously refine your models for enhanced performance, contributing significantly to the data-driven culture of our organization.',\n",
- " 'description': 'The Data Analyst interprets complex datasets to provide strategic insights, cleanses and organizes data, performs statistical analysis and predictive modeling to identify trends and inform improvements, utilizes SQL for database management, and employs visualization tools like Tableau or Power BI to effectively communicate findings to stakeholders.'},\n",
- " {'name': 'Journalist',\n",
- " 'system_message': 'As a Journalist, you are responsible for identifying and pursuing newsworthy stories with the utmost ethical standards and a commitment to factual reporting. Your innate curiosity and excellent communication skills enable you to conduct thorough research and interviews, uncovering the details that make each story compelling and informative. Skilled in both written and verbal storytelling, you craft articles, reports, and features that engage and inform the public, adhering to strict deadlines without compromising on the integrity and accuracy of your work. Proficient in multimedia journalism, you adeptly use digital tools and social media to reach a wider audience, ensuring that your stories have the maximum impact.',\n",
- " 'description': 'A Journalist is tasked with ethically sourcing and meticulously reporting newsworthy events, utilizing strong research and storytelling abilities across multiple platforms to accurately inform and engage a diverse audience.'},\n",
- " {'name': 'Teacher',\n",
- " 'system_message': 'As a Teacher, you are entrusted with the essential responsibility of fostering knowledge and encouraging academic and personal growth in your students. Your deep understanding of pedagogy, coupled with your expertise in the subject matter, enables you to create and deliver curricula that are both engaging and educational. Your adeptness at differentiated instruction allows you to tailor your teaching methods to suit the varied learning styles and needs within your classroom. By skillfully blending traditional teaching techniques with modern educational technology, you facilitate a dynamic and interactive learning environment. You excel in assessment and feedback, not only to gauge student progress but also to continuously improve your own teaching strategies. With strong interpersonal skills, you maintain open lines of communication with students, parents, and colleagues, fostering a collaborative and supportive school community.',\n",
- " 'description': \"A Teacher is responsible for cultivating students' knowledge and growth through expertise in pedagogical practices and subject matter, designing engaging curricula, adapting teaching methods to diverse learning needs, integrating technology, and using assessment for continuous improvement while nurturing a cooperative school community.\"},\n",
- " {'name': 'Lawyer',\n",
- " 'system_message': 'As a Lawyer, you are required to uphold the highest standards of legal proficiency and ethical practice. Your role involves advising clients on their legal rights and responsibilities, as well as representing them in civil and criminal proceedings. You must possess a strong understanding of the law, paired with the ability to analyze case law and legislate history, to construct compelling arguments in support of your client’s position. Your keen attention to detail and dedication to thorough research are crucial in identifying legal precedents and crafting legal documents that adhere to the strictest of procedural standards. Moreover, you must exhibit exceptional negotiation skills to achieve favorable outcomes, whether in the courtroom or at the settlement table. With your articulate verbal and written communication, you clearly and persuasively present cases, explaining complex legal concepts in understandable terms to clients, judges, and juries. Your commitment to confidentiality and upholding justice is paramount and reflected in all aspects of your professional conduct.',\n",
- " 'description': 'A Lawyer is a professionally trained legal advocate responsible for representing clients in legal proceedings, providing expert advice on legal matters, constructing persuasive arguments through meticulous research and analysis of law, and negotiating settlements, all while adhering to the highest ethical standards and maintaining strict confidentiality.'},\n",
- " {'name': 'Programmer',\n",
- " 'system_message': 'As a Programmer, you are responsible for the design, development, and implementation of software programs. Utilize your comprehensive understanding of programming languages, including but not limited to Java, C++, and Python, to create efficient and innovative software solutions. Your role involves writing clean, maintainable code while adhering to best practices in software development. You are expected to troubleshoot, debug, and upgrade existing software, as well as collaborate with cross-functional teams to define and design new product features. Your ability to think algorithmically and solve problems systematically will be integral in creating software that is not only functional but also scalable and secure.',\n",
- " 'description': 'A Programmer designs, develops, and implements innovative and efficient software solutions using languages like Java, C++, and Python, ensuring code maintainability, collaborating on new features, and enhancing existing applications with a strong focus on scalability and security.'},\n",
- " {'name': 'Accountant',\n",
- " 'system_message': 'As Accountant, you are charged with the meticulous management and analysis of financial records, ensuring accuracy and compliance with relevant laws and regulations. Utilize your comprehensive understanding of accounting principles to prepare, examine, and maintain financial reports and statements, including balance sheets and income statements. Your role involves the reconciliation of accounts, evaluating financial operations to recommend best practices, identifying issues, and strategizing solutions for fiscal efficiency and profitability. Mastery in accounting software such as QuickBooks or Sage, alongside proficiency in Microsoft Excel, enables you to efficiently process and analyze financial data. You must ensure proper financial documentation and control systems are in place, providing comprehensive support to the organization’s financial health and integrity.',\n",
- " 'description': 'As an Accountant, you are responsible for the accurate and compliant management, analysis, and reporting of financial data, along with recommending strategies to enhance fiscal efficiency and profitability, supported by proficiency in accounting software and Microsoft Excel.'},\n",
- " {'name': 'Mathematician',\n",
- " 'system_message': 'As a Mathematician, you are responsible for utilizing your profound understanding of mathematical theories and methodologies to solve complex theoretical and practical problems across various domains. Your proficiency in abstract reasoning enables you to develop new mathematical principles and to recognize and articulate the underlying mathematical relationships within real-world scenarios. You apply your expertise in calculus, algebra, statistics, and other mathematical branches to conduct rigorous analyses and to model systems for prediction and optimization. With a strong foundation in logic and quantitative reasoning, you perform peer reviews and contribute to interdisciplinary research projects, ensuring accuracy and consistency in mathematical arguments and results. Your role is crucial in advancing mathematical knowledge and providing innovative solutions to scientific and engineering challenges.',\n",
- " 'description': 'As a Mathematician, you apply advanced mathematical theories and analytical skills to solve theoretical and practical problems in various industries, develop new principles, and provide innovative solutions to complex scientific and engineering challenges.'},\n",
- " {'name': 'Physicist',\n",
- " 'system_message': 'As a Physicist, you are charged with applying your profound understanding of the physical laws that govern the universe to unravel complex scientific phenomena. Your proficiency in theoretical and experimental physics enables you to develop models and conduct experiments that explore fundamental forces and particles. With exceptional analytical skills, you interpret empirical data to validate existing theories or propose new explanations for unexplained observations. Mastery in the use of mathematical tools such as differential equations and linear algebra is crucial for you to simulate physical processes. You are also adept at using specialized software and equipment for data acquisition and analysis, contributing to advancements in fields ranging from quantum mechanics to cosmology. Your strong critical thinking abilities empower you to solve intricate problems, and your commitment to scientific rigor ensures the integrity and accuracy of your research outcomes.',\n",
- " 'description': 'A Physicist applies deep knowledge of physical laws to investigate scientific phenomena through theoretical modeling and experimental research, utilizing advanced mathematical techniques and specialized equipment to advance understanding in areas such as quantum mechanics and cosmology.'},\n",
- " {'name': 'Biologist',\n",
- " 'system_message': 'As a Biologist, you are entrusted with the study and understanding of living organisms, applying your expertise to investigate their functions, genetics, evolution, and ecosystems. Your skills in experimental design empower you to conduct research and experiments that can unlock new biological insights and improve our comprehension of life processes. Utilizing advanced microscopy techniques and molecular biology methods, you should meticulously analyze cell structures and DNA sequences to uncover the intricacies of life at a microscopic level. Demonstrate proficiency in bioinformatics tools to analyze genetic data and contribute valuable findings to the scientific community. Furthermore, as a communicator of science, ensure that your research findings are effectively documented and presented in scientific journals and at conferences, thereby enhancing the collective knowledge in your field.',\n",
- " 'description': 'A Biologist meticulously studies and understands living organisms, conducting advanced research to decode genetics and ecosystems and sharing findings through scientific publications and presentations.'},\n",
- " {'name': 'Chemist',\n",
- " 'system_message': 'As a Chemist, you are charged with applying your profound understanding of chemical principles to conduct complex experiments, synthesize new compounds, and analyze the molecular and atomic structure of materials. Your proficiency in utilizing sophisticated analytical techniques - such as chromatography, spectroscopy, and mass spectrometry - enables you to decipher the composition and properties of substances. The knowledge you hold in chemical safety and handling procedures ensures a secure laboratory environment. With an adeptness in maintaining accurate records and an insightful approach to interpreting data, you transform raw experimental results into valuable scientific insights. Your ability to communicate complex chemical information clearly makes you essential in collaborative research efforts and in driving innovation within the field.',\n",
- " 'description': 'As a Chemist, you are responsible for conducting advanced experiments, synthesizing compounds, deciphering substance compositions with techniques like chromatography and mass spectrometry, and transforming experimental data into scientific insights, while maintaining safety and clear communication in research collaborations.'},\n",
- " {'name': 'Statistician',\n",
- " 'system_message': 'As a Statistician, your primary duty is to apply mathematical and statistical methods to collect, analyze, and interpret numerical data to make informed decisions. Your strong grounding in probability theory will be essential for designing surveys and experiments to generate data. You are adept at constructing and applying sophisticated statistical models and methods, such as linear regression, ANOVA, or time-series analysis, ensuring that you accurately capture trends and relationships within the data. You possess an in-depth understanding of statistical software such as R or SAS, allowing you to perform complex analyses with efficiency and precision. Your ability to communicate complex statistical concepts to non-experts will be crucial; hence, your role includes presenting findings in a clear, actionable manner, with data visualizations and reports that drive strategic planning and policy development.',\n",
- " 'description': 'A Statistician employs and interprets advanced statistical techniques to design data-collection processes, analyze data, and present findings in a comprehensible manner, supporting evidence-based decision-making and policy formation.'},\n",
- " {'name': 'IT_Specialist',\n",
- " 'system_message': 'As an IT Specialist, your primary responsibility is to maintain the integrity and functionality of all our computer systems and networks. Your comprehensive understanding of hardware and software is crucial for diagnosing and resolving technical issues. You are adept at implementing network security measures to protect data and systems from cyber threats. You also play a significant role in systems and software upgrades, ensuring a seamless transition without disrupting workflow. Utilizing your strong problem-solving skills and proficiency in scripting languages, you automate repetitive tasks, enhancing system efficiency. Your ability to communicate effectively with team members and non-technical staff allows you to provide clear guidance and end-user support.',\n",
- " 'description': 'An IT Specialist is responsible for upholding and optimizing our computer systems and networks through maintenance, security, upgrades, issue resolution, automation, and providing support and clear communication to both technical and non-technical personnel.'},\n",
- " {'name': 'Cybersecurity_Expert',\n",
- " 'system_message': \"As a Cybersecurity Expert, you are charged with the responsibility of safeguarding the organization's computer networks and systems. Your deep understanding of cyber threats and mitigation techniques is critical in identifying vulnerabilities and protecting against malicious attacks. Employing your experience with tools such as firewalls, antivirus software, and intrusion detection systems, you will continuously monitor and defend our digital infrastructure. You are expected to conduct regular security audits and penetration testing to simulate cyber attacks and find potential weaknesses before they can be exploited. Your proficiency in risk management frameworks and incident response protocols ensures that you are prepared to swiftly handle and mitigate any security incidents that occur. With your expertise in encryption technologies and network protocols, you protect sensitive data and ensure compliance with relevant security standards and regulations. Your foresight in staying up-to-date with the latest cybersecurity trends and threats is paramount to maintaining the organization's digital defense at its peak.\",\n",
- " 'description': \"As a Cybersecurity Expert, you are responsible for the proactive protection and defense of an organization's computer networks and systems against cyber threats through continuous monitoring, conducting security audits, penetrating testing, and swiftly mitigating security incidents, while ensuring compliance with security regulations.\"},\n",
- " {'name': 'Artificial_Intelligence_Engineer',\n",
- " 'system_message': 'As an Artificial Intelligence Engineer, you are responsible for conceptualizing, designing, and implementing intelligent systems that simulate human cognitive processes. Your role demands a deep understanding of neural networks, particularly Convolutional Neural Networks (CNNs) for image recognition tasks and Recurrent Neural Networks (RNNs) for natural language processing. With your expertise in TensorFlow or PyTorch, you develop complex models that can learn, adapt, and make decisions. You prioritize the ethical design and deployment of AI systems, conscious of the implications your work may have on society. Mastery of algorithms and a proficiency in a high-level programming language, preferably Python, enable you to transform theoretical AI concepts into practical solutions that drive innovation and efficiency.',\n",
- " 'description': 'An Artificial Intelligence Engineer specializes in creating and implementing advanced intelligent systems, with a mastery of neural networks, machine learning frameworks, and ethical AI principles, to develop innovative solutions that emulate human cognition.'},\n",
- " {'name': 'Financial_Analyst',\n",
- " 'system_message': 'As a Financial Analyst, you are entrusted with utilizing your in-depth understanding of financial principles to assess investment opportunities, analyze financial data, and forecast economic trends. Your proficiency in financial modeling is paramount, enabling you to develop complex models that underpin the valuation of stocks, bonds, and other financial instruments. With a sharp eye for detail, you scrutinize company financial statements to derive actionable insights and recommend strategies to optimize financial performance. Your expertise in Excel, especially with advanced functions and formulas, allows you to efficiently manipulate and analyze large financial datasets. You are a whiz at creating compelling visualizations and delivering presentations to communicate your findings and influence strategic decisions. Your role is crucial in guiding investment decisions and driving the fiscal prudence of the organization.',\n",
- " 'description': \"A Financial Analyst performs in-depth financial analysis and modeling to evaluate investments, forecast economic trends, and deliver strategic recommendations, leveraging advanced Excel skills to inform and guide the organization's financial decisions.\"}]"
- ]
- },
- "execution_count": 5,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "sys_msg_list"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "256dd32b03a7a172",
- "metadata": {
- "collapsed": false
- },
- "source": [
- "We can save the generated agents' information into a json file."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "53111125938845cf",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2023-12-23T07:40:01.750855900Z",
- "start_time": "2023-12-23T07:40:01.710399600Z"
- },
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "json.dump(sys_msg_list, open(\"./agent_library_example.json\", \"w\"), indent=4)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "cfd883b79a3bd932",
- "metadata": {
- "collapsed": false
- },
- "source": [
- "## Build agents from library (by LLM)\n",
- "Here, we introduce how to build agents from the generated library. As in the previous `build`, we also need to specify a `building_task` that lets the build manager know which agents should be selected from the library according to the task. \n",
- "\n",
- "We also need to specify a `library_path_or_json`, which can be a path of library or a JSON string with agents' configs. Here, we use the previously saved path as the library path."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "8963a8709c8e92e2",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2023-12-23T07:40:01.752918500Z",
- "start_time": "2023-12-23T07:40:01.735461Z"
- },
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "library_path_or_json = \"./agent_library_example.json\"\n",
- "building_task = \"Find a paper on arxiv by programming, and analyze its application in some domain. For example, find a recent paper about gpt-4 on arxiv and find its potential applications in software.\""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "72656a8d0c1a9b12",
- "metadata": {
- "collapsed": false
- },
- "source": [
- "Then, we can call the `build_from_library` from the AgentBuilder to generate a list of agents from the library and let them complete the user's `execution_task` in a group chat."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "5c669b76b2c9b750",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2023-12-23T07:46:02.075542200Z",
- "start_time": "2023-12-23T07:43:55.489042900Z"
- },
- "collapsed": false
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\u001b[32m==> Looking for suitable agents in the library...\u001b[0m\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "['Programmer', 'Mathematician'] are selected.\n",
- "\u001b[32m==> Creating agents...\u001b[0m\n",
- "Creating agent Programmer...\n",
- "Creating agent Mathematician...\n",
- "Adding user console proxy...\n",
- "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
- "\n",
- "Find a recent paper about explainable AI on arxiv and find its potential applications in medical.\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Computer_terminal\n",
- "\u001b[0m\n",
- "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
- "\n",
- "There is no code from the last 1 message for me to execute. Group chat manager should let other participants to continue the conversation. If the group chat manager want to end the conversation, you should let other participant reply me only with \"TERMINATE\"\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Programmer\n",
- "\u001b[0m\n",
- "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
- "\n",
- "To find a recent paper about explainable AI on arXiv, we can use the arXiv API to search for papers that match the query. However, since I cannot directly access external APIs, I suggest that one of us manually searches for the paper on the arXiv website using relevant search terms such as \"explainable AI\" and \"medical applications\". Once we find a suitable paper, we can discuss its potential applications in the medical field. \n",
- "\n",
- "Mathematician, would you like to perform the search, or shall I provide a Python script that could be used to perform the search programmatically?\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Mathematician\n",
- "\u001b[0m\n",
- "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
- "\n",
- "As a Mathematician, I can certainly appreciate the value of a programmatic approach to searching for academic papers. However, since I do not have direct access to execute code or APIs, I would suggest that you, as the Programmer, provide the Python script that could be used to perform the search on arXiv. Once we have identified a paper, I can then assist in discussing its potential applications in the medical field from a mathematical perspective.\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Programmer\n",
- "\u001b[0m\n",
- "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
- "\n",
- "Understood. I will provide a Python script that can be used to search for recent papers about explainable AI on arXiv with potential applications in the medical field. The script will use the `arxiv` Python package, which is a wrapper for the arXiv API. If the package is not installed, we will need to install it first.\n",
- "\n",
- "Let's start by checking if the `arxiv` package is installed and if not, we will install it. Computer_terminal, please execute the following command to check for the `arxiv` package and install it if necessary.\n",
- "\n",
- "```sh\n",
- "pip show arxiv || pip install arxiv\n",
- "```\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Computer_terminal\n",
- "\u001b[0m\n",
- "\u001b[31m\n",
- ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is sh)...\u001b[0m\n",
- "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
- "\n",
- "exitcode: 0 (execution succeeded)\n",
- "Code output: \n",
- "Name: arxiv\n",
- "Version: 2.1.0\n",
- "Summary: Python wrapper for the arXiv API: https://arxiv.org/help/api/\n",
- "Home-page: https://github.com/lukasschwab/arxiv.py\n",
- "Author: Lukas Schwab\n",
- "Author-email: lukas.schwab@gmail.com\n",
- "License: MIT\n",
- "Location: /home/vscode/.local/lib/python3.10/site-packages\n",
- "Requires: feedparser, requests\n",
- "Required-by: \n",
- "\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Programmer\n",
- "\u001b[0m\n",
- "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
- "\n",
- "Great, the `arxiv` package is already installed. Now, I will provide a Python script that uses the `arxiv` package to search for recent papers related to explainable AI with potential applications in the medical field. The script will query the arXiv API for papers with relevant keywords and print out the title and summary of the most recent paper found.\n",
- "\n",
- "Computer_terminal, please execute the following Python script.\n",
- "\n",
- "```python\n",
- "import arxiv\n",
- "\n",
- "# Define the search query\n",
- "search_query = 'all:explainable AI AND all:medical'\n",
- "\n",
- "# Search for papers on arXiv\n",
- "search = arxiv.Search(\n",
- " query = search_query,\n",
- " max_results = 1,\n",
- " sort_by = arxiv.SortCriterion.SubmittedDate\n",
- ")\n",
- "\n",
- "# Fetch the most recent paper\n",
- "for paper in search.results():\n",
- " print(\"Title:\", paper.title)\n",
- " print(\"Summary:\", paper.summary)\n",
- " # Only print the most recent paper\n",
- " break\n",
- "```\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Computer_terminal\n",
- "\u001b[0m\n",
- "\u001b[31m\n",
- ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
- "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
- "\n",
- "exitcode: 0 (execution succeeded)\n",
- "Code output: \n",
- "Title: Automated Information Extraction from Thyroid Operation Narrative: A Comparative Study of GPT-4 and Fine-tuned KoELECTRA\n",
- "Summary: In the rapidly evolving field of healthcare, the integration of artificial\n",
- "intelligence (AI) has become a pivotal component in the automation of clinical\n",
- "workflows, ushering in a new era of efficiency and accuracy. This study focuses\n",
- "on the transformative capabilities of the fine-tuned KoELECTRA model in\n",
- "comparison to the GPT-4 model, aiming to facilitate automated information\n",
- "extraction from thyroid operation narratives. The current research landscape is\n",
- "dominated by traditional methods heavily reliant on regular expressions, which\n",
- "often face challenges in processing free-style text formats containing critical\n",
- "details of operation records, including frozen biopsy reports. Addressing this,\n",
- "the study leverages advanced natural language processing (NLP) techniques to\n",
- "foster a paradigm shift towards more sophisticated data processing systems.\n",
- "Through this comparative study, we aspire to unveil a more streamlined,\n",
- "precise, and efficient approach to document processing in the healthcare\n",
- "domain, potentially revolutionizing the way medical data is handled and\n",
- "analyzed.\n",
- "\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Mathematician\n",
- "\u001b[0m\n",
- "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
- "\n",
- "The paper titled \"Automated Information Extraction from Thyroid Operation Narrative: A Comparative Study of GPT-4 and Fine-tuned KoELECTRA\" presents a study on the use of artificial intelligence for automating the extraction of information from thyroid operation narratives. This is a clear example of explainable AI being applied in the medical field, specifically in the area of clinical workflows and document processing.\n",
- "\n",
- "The potential applications of such technology in medicine are vast. By automating the extraction of information from operation narratives, healthcare professionals can save time and reduce the likelihood of human error. This can lead to more accurate patient records, improved patient care, and streamlined administrative processes. Additionally, the ability to quickly and accurately process operation records can facilitate better data analysis, which can be used for medical research, trend analysis, and improving healthcare outcomes.\n",
- "\n",
- "The use of advanced natural language processing (NLP) techniques, as mentioned in the summary, is particularly important for processing free-style text formats that contain critical medical information. This technology could be further explored to extend its application to other types of medical documents and records, enhancing the overall efficiency of the healthcare system.\n",
- "\n",
- "The study's focus on comparing the performance of the fine-tuned KoELECTRA model with GPT-4 also highlights the importance of evaluating different AI models to determine the most effective approach for specific medical applications. This comparative analysis can lead to the development of more specialized AI tools tailored to the needs of the healthcare industry.\n",
- "\n",
- "In conclusion, the research presented in this paper has significant implications for the future of medical document processing and the broader integration of AI in healthcare.\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Programmer\n",
- "\u001b[0m\n",
- "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
- "\n",
- "The insights provided by the Mathematician are indeed valuable. The application of AI for automated information extraction from medical documents like thyroid operation narratives can greatly enhance efficiency and accuracy in healthcare. The use of models like GPT-4 and KoELECTRA for natural language processing tasks shows the potential of AI to handle complex, unstructured data which is common in medical records.\n",
- "\n",
- "From a programming perspective, the implementation of such AI systems would involve training models on large datasets of medical documents to learn the context and semantics specific to medical terminology. Ensuring the explainability of AI in this context is crucial, as healthcare professionals need to understand and trust the AI's decision-making process, especially when it directly affects patient care.\n",
- "\n",
- "Moreover, the integration of explainable AI into healthcare systems must adhere to strict privacy and security regulations to protect sensitive patient data. This requires careful design and implementation of data handling procedures within the AI system.\n",
- "\n",
- "The potential applications extend beyond just document processing to diagnostic assistance, personalized treatment plans, and predictive analytics for patient outcomes. As AI technology continues to evolve, its role in supporting and enhancing the capabilities of healthcare professionals will undoubtedly expand.\n",
- "\n",
- "Given the importance of the topic and the potential impact on healthcare, it would be beneficial to keep an eye on further developments in this field. If there are no further questions or points to discuss, we can conclude our conversation on this topic.\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Mathematician\n",
- "\u001b[0m\n",
- "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
- "\n",
- "I agree with the Programmer's assessment. The implementation of AI in healthcare does indeed require careful consideration of the models used, the training data, and the explainability of the AI's decisions. The ethical implications, particularly concerning patient privacy and data security, are paramount and must be addressed with the utmost rigor.\n",
- "\n",
- "The potential for AI to assist in diagnostics, treatment planning, and predictive analytics is a promising development for the future of medicine. It is essential that these systems are developed in collaboration with healthcare professionals to ensure they meet the real-world needs of the field.\n",
- "\n",
- "The interdisciplinary nature of this work, combining expertise in mathematics, computer science, and medicine, is a testament to the collaborative efforts needed to advance healthcare technology. It has been a pleasure discussing the potential applications of explainable AI in medicine with you.\n",
- "\n",
- "If there are no further points to add, I believe we have reached a natural conclusion to our conversation.\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Computer_terminal\n",
- "\u001b[0m\n",
- "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
- "\n",
- "There is no code from the last 1 message for me to execute. Group chat manager should let other participants to continue the conversation. If the group chat manager want to end the conversation, you should let other participant reply me only with \"TERMINATE\"\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[33mAll agents have been cleared.\u001b[0m\n"
- ]
- }
- ],
- "source": [
- "new_builder = AgentBuilder(\n",
- " config_file_or_env=config_file_or_env, builder_model=\"gpt-4-1106-preview\", agent_model=\"gpt-4-1106-preview\"\n",
- ")\n",
- "agent_list, _ = new_builder.build_from_library(building_task, library_path_or_json, llm_config)\n",
- "start_task(\n",
- " execution_task=\"Find a recent paper about explainable AI on arxiv and find its potential applications in medical.\",\n",
- " agent_list=agent_list,\n",
- ")\n",
- "new_builder.clear_all_agents()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c7a10e6fa00a5a0d",
- "metadata": {
- "collapsed": false
- },
- "source": [
- "## Build agents from library (by description-task similarity)\n",
- "We also support using embedding similarity to select agents. You can use a [Sentence-Transformers model](https://www.sbert.net/docs/pretrained_models.html) as an embedding extractor, and AgentBuilder will select agents with profiles that are the most similar to the building task from the library by comparing their embedding similarity. This will reduce the use of LLMs but may have less accuracy."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "id": "521dc5f961efde59",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2023-12-26T17:01:29.333975100Z",
- "start_time": "2023-12-26T16:58:11.070813500Z"
- },
- "collapsed": false
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\u001b[32m==> Looking for suitable agents in the library...\u001b[0m\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "['Programmer', 'Mathematician'] are selected.\n",
- "\u001b[32m==> Creating agents...\u001b[0m\n",
- "Creating agent Programmer...\n",
- "Creating agent Mathematician...\n",
- "Adding user console proxy...\n",
- "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
- "\n",
- "Find a recent paper about gpt-4 on arxiv and find its potential applications in software.\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Computer_terminal\n",
- "\u001b[0m\n",
- "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
- "\n",
- "There is no code from the last 1 message for me to execute. Group chat manager should let other participants to continue the conversation. If the group chat manager want to end the conversation, you should let other participant reply me only with \"TERMINATE\"\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Programmer\n",
- "\u001b[0m\n",
- "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
- "\n",
- "To find a recent paper about GPT-4 on arXiv, we can use the arXiv API to search for papers. However, since I can't directly access external APIs, I can write a Python script that you can run on your local machine to perform this search. Would you like me to provide you with such a script?\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Computer_terminal\n",
- "\u001b[0m\n",
- "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
- "\n",
- "There is no code from the last 1 message for me to execute. Group chat manager should let other participants to continue the conversation. If the group chat manager want to end the conversation, you should let other participant reply me only with \"TERMINATE\"\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Programmer\n",
- "\u001b[0m\n",
- "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
- "\n",
- "I apologize for the confusion. I will provide a Python script that can be executed by the Computer_terminal to search for recent papers about GPT-4 on arXiv. Let's proceed with that.\n",
- "\n",
- "```python\n",
- "import requests\n",
- "from xml.etree import ElementTree\n",
- "\n",
- "# Define the search parameters and URL for the arXiv API\n",
- "search_query = 'all:gpt-4'\n",
- "start = 0\n",
- "max_results = 5\n",
- "sort_by = 'submittedDate'\n",
- "sort_order = 'descending'\n",
- "url = f'http://export.arxiv.org/api/query?search_query={search_query}&start={start}&max_results={max_results}&sortBy={sort_by}&sortOrder={sort_order}'\n",
- "\n",
- "# Send a GET request to the arXiv API\n",
- "response = requests.get(url)\n",
- "\n",
- "# Parse the response if it was successful\n",
- "if response.status_code == 200:\n",
- " root = ElementTree.fromstring(response.content)\n",
- " # Find and print the entries (papers)\n",
- " for entry in root.findall('{http://www.w3.org/2005/Atom}entry'):\n",
- " title = entry.find('{http://www.w3.org/2005/Atom}title').text\n",
- " summary = entry.find('{http://www.w3.org/2005/Atom}summary').text\n",
- " published = entry.find('{http://www.w3.org/2005/Atom}published').text\n",
- " print(f\"Title: {title}\\nSummary: {summary}\\nPublished Date: {published}\\n\")\n",
- "else:\n",
- " print(f\"Failed to fetch data from arXiv. Status code: {response.status_code}\")\n",
- "```\n",
- "\n",
- "This script will fetch the most recent papers related to GPT-4 from the arXiv API and print out their titles, summaries, and publication dates. Please execute this script to find the information we need.\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Computer_terminal\n",
- "\u001b[0m\n",
- "\u001b[31m\n",
- ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
- "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
- "\n",
- "exitcode: 0 (execution succeeded)\n",
- "Code output: \n",
- "Title: What If We Recaption Billions of Web Images with LLaMA-3?\n",
- "Summary: Web-crawled image-text pairs are inherently noisy. Prior studies demonstrate\n",
- "that semantically aligning and enriching textual descriptions of these pairs\n",
- "can significantly enhance model training across various vision-language tasks,\n",
- "particularly text-to-image generation. However, large-scale investigations in\n",
- "this area remain predominantly closed-source. Our paper aims to bridge this\n",
- "community effort, leveraging the powerful and \\textit{open-sourced} LLaMA-3, a\n",
- "GPT-4 level LLM. Our recaptioning pipeline is simple: first, we fine-tune a\n",
- "LLaMA-3-8B powered LLaVA-1.5 and then employ it to recaption 1.3 billion images\n",
- "from the DataComp-1B dataset. Our empirical results confirm that this enhanced\n",
- "dataset, Recap-DataComp-1B, offers substantial benefits in training advanced\n",
- "vision-language models. For discriminative models like CLIP, we observe\n",
- "enhanced zero-shot performance in cross-modal retrieval tasks. For generative\n",
- "models like text-to-image Diffusion Transformers, the generated images exhibit\n",
- "a significant improvement in alignment with users' text instructions,\n",
- "especially in following complex queries. Our project page is\n",
- "https://www.haqtu.me/Recap-Datacomp-1B/\n",
- "\n",
- "Published Date: 2024-06-12T17:59:07Z\n",
- "\n",
- "Title: DafnyBench: A Benchmark for Formal Software Verification\n",
- "Summary: We introduce DafnyBench, the largest benchmark of its kind for training and\n",
- "evaluating machine learning systems for formal software verification. We test\n",
- "the ability of LLMs such as GPT-4 and Claude 3 to auto-generate enough hints\n",
- "for the Dafny formal verification engine to successfully verify over 750\n",
- "programs with about 53,000 lines of code. The best model and prompting scheme\n",
- "achieved 68% success rate, and we quantify how this rate improves when retrying\n",
- "with error message feedback and how it deteriorates with the amount of required\n",
- "code and hints. We hope that DafnyBench will enable rapid improvements from\n",
- "this baseline as LLMs and verification techniques grow in quality.\n",
- "\n",
- "Published Date: 2024-06-12T17:53:31Z\n",
- "\n",
- "Title: A Sociotechnical Lens for Evaluating Computer Vision Models: A Case\n",
- " Study on Detecting and Reasoning about Gender and Emotion\n",
- "Summary: In the evolving landscape of computer vision (CV) technologies, the automatic\n",
- "detection and interpretation of gender and emotion in images is a critical area\n",
- "of study. This paper investigates social biases in CV models, emphasizing the\n",
- "limitations of traditional evaluation metrics such as precision, recall, and\n",
- "accuracy. These metrics often fall short in capturing the complexities of\n",
- "gender and emotion, which are fluid and culturally nuanced constructs. Our\n",
- "study proposes a sociotechnical framework for evaluating CV models,\n",
- "incorporating both technical performance measures and considerations of social\n",
- "fairness. Using a dataset of 5,570 images related to vaccination and climate\n",
- "change, we empirically compared the performance of various CV models, including\n",
- "traditional models like DeepFace and FER, and generative models like GPT-4\n",
- "Vision. Our analysis involved manually validating the gender and emotional\n",
- "expressions in a subset of images to serve as benchmarks. Our findings reveal\n",
- "that while GPT-4 Vision outperforms other models in technical accuracy for\n",
- "gender classification, it exhibits discriminatory biases, particularly in\n",
- "response to transgender and non-binary personas. Furthermore, the model's\n",
- "emotion detection skew heavily towards positive emotions, with a notable bias\n",
- "towards associating female images with happiness, especially when prompted by\n",
- "male personas. These findings underscore the necessity of developing more\n",
- "comprehensive evaluation criteria that address both validity and discriminatory\n",
- "biases in CV models. Our proposed framework provides guidelines for researchers\n",
- "to critically assess CV tools, ensuring their application in communication\n",
- "research is both ethical and effective. The significant contribution of this\n",
- "study lies in its emphasis on a sociotechnical approach, advocating for CV\n",
- "technologies that support social good and mitigate biases rather than\n",
- "perpetuate them.\n",
- "\n",
- "Published Date: 2024-06-12T13:52:30Z\n",
- "\n",
- "Title: Supportiveness-based Knowledge Rewriting for Retrieval-augmented\n",
- " Language Modeling\n",
- "Summary: Retrieval-augmented language models (RALMs) have recently shown great\n",
- "potential in mitigating the limitations of implicit knowledge in LLMs, such as\n",
- "untimely updating of the latest expertise and unreliable retention of long-tail\n",
- "knowledge. However, since the external knowledge base, as well as the\n",
- "retriever, can not guarantee reliability, potentially leading to the knowledge\n",
- "retrieved not being helpful or even misleading for LLM generation. In this\n",
- "paper, we introduce Supportiveness-based Knowledge Rewriting (SKR), a robust\n",
- "and pluggable knowledge rewriter inherently optimized for LLM generation.\n",
- "Specifically, we introduce the novel concept of \"supportiveness\"--which\n",
- "represents how effectively a knowledge piece facilitates downstream tasks--by\n",
- "considering the perplexity impact of augmented knowledge on the response text\n",
- "of a white-box LLM. Based on knowledge supportiveness, we first design a\n",
- "training data curation strategy for our rewriter model, effectively identifying\n",
- "and filtering out poor or irrelevant rewrites (e.g., with low supportiveness\n",
- "scores) to improve data efficacy. We then introduce the direct preference\n",
- "optimization (DPO) algorithm to align the generated rewrites to optimal\n",
- "supportiveness, guiding the rewriter model to summarize augmented content that\n",
- "better improves the final response. Comprehensive evaluations across six\n",
- "popular knowledge-intensive tasks and four LLMs have demonstrated the\n",
- "effectiveness and superiority of SKR. With only 7B parameters, SKR has shown\n",
- "better knowledge rewriting capability over GPT-4, the current state-of-the-art\n",
- "general-purpose LLM.\n",
- "\n",
- "Published Date: 2024-06-12T11:52:35Z\n",
- "\n",
- "Title: Automated Information Extraction from Thyroid Operation Narrative: A\n",
- " Comparative Study of GPT-4 and Fine-tuned KoELECTRA\n",
- "Summary: In the rapidly evolving field of healthcare, the integration of artificial\n",
- "intelligence (AI) has become a pivotal component in the automation of clinical\n",
- "workflows, ushering in a new era of efficiency and accuracy. This study focuses\n",
- "on the transformative capabilities of the fine-tuned KoELECTRA model in\n",
- "comparison to the GPT-4 model, aiming to facilitate automated information\n",
- "extraction from thyroid operation narratives. The current research landscape is\n",
- "dominated by traditional methods heavily reliant on regular expressions, which\n",
- "often face challenges in processing free-style text formats containing critical\n",
- "details of operation records, including frozen biopsy reports. Addressing this,\n",
- "the study leverages advanced natural language processing (NLP) techniques to\n",
- "foster a paradigm shift towards more sophisticated data processing systems.\n",
- "Through this comparative study, we aspire to unveil a more streamlined,\n",
- "precise, and efficient approach to document processing in the healthcare\n",
- "domain, potentially revolutionizing the way medical data is handled and\n",
- "analyzed.\n",
- "\n",
- "Published Date: 2024-06-12T06:44:05Z\n",
- "\n",
- "\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Programmer\n",
- "\u001b[0m\n",
- "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
- "\n",
- "Based on the search results from the arXiv API, we have found several papers that discuss potential applications of GPT-4 in software:\n",
- "\n",
- "1. **Recaptioning Web Images with LLaMA-3 and GPT-4**: This paper discusses the use of GPT-4 level LLMs for recaptioning web images, which can enhance model training across various vision-language tasks. This has implications for improving the quality of datasets used in machine learning and could be particularly beneficial for text-to-image generation and cross-modal retrieval tasks.\n",
- "\n",
- "2. **DafnyBench: A Benchmark for Formal Software Verification**: This paper introduces a benchmark for training and evaluating machine learning systems for formal software verification. It tests the ability of LLMs such as GPT-4 to auto-generate hints for the Dafny formal verification engine to successfully verify programs. This application could significantly impact the field of software verification by automating the generation of verification hints, potentially improving the efficiency and reliability of the verification process.\n",
- "\n",
- "3. **Automated Information Extraction from Thyroid Operation Narrative**: This study compares the GPT-4 model with the fine-tuned KoELECTRA model for automated information extraction from thyroid operation narratives. The application of GPT-4 in this context could revolutionize document processing in healthcare by providing a more efficient and accurate method for extracting information from medical records.\n",
- "\n",
- "These papers suggest that GPT-4 has the potential to be applied in various software-related fields, including enhancing datasets for machine learning, formal software verification, and healthcare document processing. The applications in these papers could lead to more efficient, accurate, and reliable software systems across different domains.\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Mathematician\n",
- "\u001b[0m\n",
- "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
- "\n",
- "The applications mentioned indeed highlight the versatility of GPT-4 in different domains. To further understand the potential impact of GPT-4 on software, let's delve into the mathematical and algorithmic principles that could be at play in these applications:\n",
- "\n",
- "1. **Recaptioning Web Images**: The process of recaptioning images with a language model like GPT-4 involves understanding the context of an image and generating descriptive text that accurately reflects its content. This task likely involves a combination of computer vision techniques to interpret the image and natural language processing to generate the caption. From a mathematical perspective, this would involve optimization algorithms to fine-tune the language model on a specific dataset, ensuring that the generated captions are both semantically and syntactically correct.\n",
- "\n",
- "2. **Formal Software Verification**: The use of GPT-4 to auto-generate hints for formal verification engines like Dafny involves the model understanding the logic and structure of the code. This requires a deep understanding of formal logic, proof theory, and possibly type theory if the language being verified is statically typed. The success rate of auto-generated hints would depend on the model's ability to reason about the correctness of code and the underlying mathematical properties that ensure its validity.\n",
- "\n",
- "3. **Automated Information Extraction from Medical Records**: For GPT-4 to extract information from medical narratives, it must process unstructured text and identify relevant medical terms and their relationships. This task involves natural language understanding, which from a mathematical standpoint, can be seen as a form of pattern recognition and classification. The model would need to be trained on a large corpus of medical texts, and its performance would be measured by its precision and recall in identifying and extracting the correct information.\n",
- "\n",
- "In each of these applications, GPT-4's effectiveness would be influenced by the underlying mathematical models, such as neural networks, and the optimization techniques used during training, such as gradient descent. The quality of the training data and the model's architecture (e.g., attention mechanisms, transformer layers) also play a crucial role in its performance.\n",
- "\n",
- "To verify the potential of GPT-4 in these applications, one could set up experiments to measure the performance of GPT-4 against specific benchmarks or metrics relevant to each domain. For example, in the case of formal software verification, one could measure the percentage of programs that are successfully verified with the hints generated by GPT-4 compared to a baseline or human-generated hints.\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Mathematician\n",
- "\u001b[0m\n",
- "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
- "\n",
- "To further verify the potential applications of GPT-4 in software, we can consider the following:\n",
- "\n",
- "1. **Recaptioning Web Images**: The effectiveness of GPT-4 in this application can be measured by the accuracy of the captions it generates. This can be quantified using metrics such as BLEU (Bilingual Evaluation Understudy) or ROUGE (Recall-Oriented Understudy for Gisting Evaluation), which compare the machine-generated text to a set of reference captions. A high score on these metrics would indicate that GPT-4 is capable of understanding and describing the content of images accurately.\n",
- "\n",
- "2. **Formal Software Verification**: To verify the application of GPT-4 in software verification, we could measure the success rate of the model in generating verification hints that lead to successful program verification. This could be done by comparing the verification success rate with and without the hints provided by GPT-4. Additionally, the complexity of the programs and the hints required could be analyzed to understand the model's limitations and strengths.\n",
- "\n",
- "3. **Automated Information Extraction from Medical Records**: The performance of GPT-4 in extracting information from medical narratives can be assessed by comparing the extracted information to a gold standard set of annotations. Precision, recall, and F1-score are common metrics used to evaluate information extraction systems. A high F1-score would suggest that GPT-4 is effective in identifying and extracting relevant information from unstructured medical texts.\n",
- "\n",
- "For each of these applications, it would be important to conduct a thorough analysis of the results to ensure that the model's performance is not only statistically significant but also practically relevant. Additionally, it would be crucial to assess the model's performance on diverse datasets to ensure that it generalizes well across different contexts and does not exhibit biases that could lead to incorrect or unfair outcomes.\n",
- "\n",
- "If we want to simulate or test these applications, we could propose specific experiments or data analysis procedures that align with the above verification strategies. However, such experiments would require access to the relevant datasets and computational resources to run the models, which may not be feasible within the scope of this conversation.\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Mathematician\n",
- "\u001b[0m\n",
- "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
- "\n",
- "To simulate or test these applications, we could propose specific experiments or data analysis procedures that align with the above verification strategies. However, such experiments would require access to the relevant datasets and computational resources to run the models, which may not be feasible within the scope of this conversation.\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[32m\n",
- "Next speaker: Mathematician\n",
- "\u001b[0m\n",
- "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
- "\n",
- "TERMINATE\n",
- "\n",
- "--------------------------------------------------------------------------------\n",
- "\u001b[33mAll agents have been cleared.\u001b[0m\n"
- ]
- }
- ],
- "source": [
- "new_builder = AgentBuilder(\n",
- " config_file_or_env=config_file_or_env, builder_model=\"gpt-4-1106-preview\", agent_model=\"gpt-4-1106-preview\"\n",
- ")\n",
- "agent_list, _ = new_builder.build_from_library(\n",
- " building_task, library_path_or_json, llm_config, embedding_model=\"all-mpnet-base-v2\"\n",
- ")\n",
- "start_task(\n",
- " execution_task=\"Find a recent paper about gpt-4 on arxiv and find its potential applications in software.\",\n",
- " agent_list=agent_list,\n",
- ")\n",
- "new_builder.clear_all_agents()"
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "6264276d39875995",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "# Automatically Build Multi-agent System from Agent Library\n",
+ "\n",
+ "By: [Linxin Song](https://linxins97.github.io/), [Jieyu Zhang](https://jieyuz2.github.io/)\n",
+ "\n",
+ "In this notebook, we introduce a new feature for AutoBuild, `build_from_library`, which help users build an automatic task-solving process powered by a multi-agent system from a pre-defined agent library. \n",
+ "Specifically, in `build_from_library`, we prompt an LLM to explore useful agents from a pre-defined agent library, generating configurations for those agents for a group chat to solve the user's task."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ec78dda8e3826d8a",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## Requirement\n",
+ "\n",
+ "AutoBuild require `pyautogen[autobuild]`, which can be installed by the following command:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "e8e9ae50658be975",
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "%pip install pyautogen[autobuild]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "176c200804af63f3",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## Preparation and useful tools\n",
+ "We need to specify a `config_path`, `default_llm_config` that include backbone LLM configurations."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "2505f029423b21ab",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2023-12-26T16:58:02.762702600Z",
+ "start_time": "2023-12-26T16:58:02.472073Z"
+ },
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "\n",
+ "import autogen\n",
+ "from autogen.agentchat.contrib.agent_builder import AgentBuilder\n",
+ "\n",
+ "config_file_or_env = \"OAI_CONFIG_LIST\" # modify path\n",
+ "llm_config = {\"temperature\": 0}\n",
+ "config_list = autogen.config_list_from_json(config_file_or_env, filter_dict={\"model\": [\"gpt-4-1106-preview\", \"gpt-4\"]})\n",
+ "\n",
+ "\n",
+ "def start_task(execution_task: str, agent_list: list):\n",
+ " group_chat = autogen.GroupChat(agents=agent_list, messages=[], max_round=12)\n",
+ " manager = autogen.GroupChatManager(groupchat=group_chat, llm_config={\"config_list\": config_list, **llm_config})\n",
+ " agent_list[0].initiate_chat(manager, message=execution_task)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5fb3db8885dd6ee6",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## Example for generating an agent library\n",
+ "Here, we show an example of generating an agent library from a pre-defined list of agents' names by prompting a `gpt-4`. You can also prepare a handcrafted library yourself.\n",
+ "\n",
+ "A Library contains each agent's name, description and system_message. The description is a brief introduction about agent's characteristics. As we will feed all agents' names and description to gpt-4 and let it choose the best agents for us, each agent's description should be simple but informative. \n",
+ "\n",
+ "First, we define a prompt template for description and system_message generation and a list of agents' name:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "68315f6ec912c58a",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2023-12-23T07:39:03.317527600Z",
+ "start_time": "2023-12-23T07:39:03.276859600Z"
+ },
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "AGENT_SYS_MSG_PROMPT = \"\"\"Acccording to the following postion name, write a high quality instruction for the position following a given example. You should only return the instruction.\n",
+ "\n",
+ "# Position Name\n",
+ "{position}\n",
+ "\n",
+ "# Example instruction for Data Analyst\n",
+ "\n",
+ "As Data Analyst, you are tasked with leveraging your extensive knowledge in data analysis to recognize and extract meaningful features from vast datasets. Your expertise in machine learning, specifically with the Random Forest Classifier, allows you to construct robust predictive models adept at handling both classification and regression tasks. You excel in model evaluation and interpretation, ensuring that the performance of your algorithms is not just assessed with precision, but also understood in the context of the data and the problem at hand. With a command over Python and proficiency in using the pandas library, you manipulate and preprocess data with ease.\n",
+ "\"\"\"\n",
+ "\n",
+ "AGENT_DESC_PROMPT = \"\"\"According to position name and the instruction, summarize the position into a high quality one sentence description.\n",
+ "\n",
+ "# Position Name\n",
+ "{position}\n",
+ "\n",
+ "# Instruction\n",
+ "{instruction}\n",
+ "\"\"\"\n",
+ "\n",
+ "position_list = [\n",
+ " \"Environmental_Scientist\",\n",
+ " \"Astronomer\",\n",
+ " \"Software_Developer\",\n",
+ " \"Data_Analyst\",\n",
+ " \"Journalist\",\n",
+ " \"Teacher\",\n",
+ " \"Lawyer\",\n",
+ " \"Programmer\",\n",
+ " \"Accountant\",\n",
+ " \"Mathematician\",\n",
+ " \"Physicist\",\n",
+ " \"Biologist\",\n",
+ " \"Chemist\",\n",
+ " \"Statistician\",\n",
+ " \"IT_Specialist\",\n",
+ " \"Cybersecurity_Expert\",\n",
+ " \"Artificial_Intelligence_Engineer\",\n",
+ " \"Financial_Analyst\",\n",
+ "]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "72b8e7d9d334a5c2",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "Then we can prompt a `gpt-4` model to generate each agent's system message as well as the description:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "8fbfef9268fc5191",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2023-12-23T07:40:01.703372Z",
+ "start_time": "2023-12-23T07:39:04.472589200Z"
+ },
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "build_manager = autogen.OpenAIWrapper(config_list=config_list)\n",
+ "sys_msg_list = []\n",
+ "\n",
+ "for pos in position_list:\n",
+ " resp_agent_sys_msg = (\n",
+ " build_manager.create(\n",
+ " messages=[\n",
+ " {\n",
+ " \"role\": \"user\",\n",
+ " \"content\": AGENT_SYS_MSG_PROMPT.format(\n",
+ " position=pos,\n",
+ " ),\n",
+ " }\n",
+ " ]\n",
+ " )\n",
+ " .choices[0]\n",
+ " .message.content\n",
+ " )\n",
+ " resp_desc_msg = (\n",
+ " build_manager.create(\n",
+ " messages=[\n",
+ " {\n",
+ " \"role\": \"user\",\n",
+ " \"content\": AGENT_DESC_PROMPT.format(\n",
+ " position=pos,\n",
+ " instruction=resp_agent_sys_msg,\n",
+ " ),\n",
+ " }\n",
+ " ]\n",
+ " )\n",
+ " .choices[0]\n",
+ " .message.content\n",
+ " )\n",
+ " sys_msg_list.append({\"name\": pos, \"system_message\": resp_agent_sys_msg, \"description\": resp_desc_msg})"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9e26c6db4befacc5",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "The generated profile will have the following format:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "8ede1d7088eb183d",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2023-12-23T07:40:01.712399300Z",
+ "start_time": "2023-12-23T07:40:01.707400200Z"
+ },
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[{'name': 'Environmental_Scientist',\n",
+ " 'system_message': 'As an Environmental Scientist, you are responsible for applying your profound knowledge of environmental science to analyze ecological data and assess the impact of human activities on natural resources and ecosystems. Your proficiency in environmental assessment techniques enables you to design and conduct field studies, collect samples, and monitor environmental parameters effectively. Utilizing Geographic Information Systems (GIS), you spatially analyze and visualize environmental data to better understand patterns and changes in the landscape. You are adept at interpreting the results and communicating your findings clearly to stakeholders, policymakers, and the public, thereby contributing to informed decision-making on environmental issues. Your role is essential in developing sustainable practices and recommending mitigation measures to minimize environmental degradation and promote conservation.',\n",
+ " 'description': 'As an Environmental Scientist, you are tasked with analyzing and assessing the impact of human activities on ecosystems by conducting field studies, using GIS for spatial analysis, and communicating your findings to inform sustainable practices and conservation efforts.'},\n",
+ " {'name': 'Astronomer',\n",
+ " 'system_message': 'As an Astronomer, your duty involves diligent observation and analysis of celestial phenomena across the universe. Utilize cutting-edge telescopes and instruments to gather astronomical data, looking for patterns and irregularities that can lead to groundbreaking discoveries. Your profound knowledge in astrophysics is pivotal in interpreting these findings, which may include identifying new celestial objects, scrutinizing the properties and behaviors of stars, planets, and galaxies, and understanding cosmic events. Mastery of complex astronomical software and advanced mathematics is crucial for modeling astronomical phenomena and processing the vast amounts of data. Your role is essential in advancing our understanding of the cosmos, contributing to the broader scientific community by publishing your findings in reputable journals and engaging in peer collaboration to further space exploration and research.',\n",
+ " 'description': 'An Astronomer is a professional who meticulously observes, analyzes, and interprets celestial phenomena using advanced telescopes and instruments, requiring a deep knowledge of astrophysics, proficiency in mathematical modeling, and collaboration in scientific communities to enhance our comprehension of the universe.'},\n",
+ " {'name': 'Software_Developer',\n",
+ " 'system_message': 'As a Software Developer, your objective is to craft, test, and maintain the software that will meet the needs of our users and clients. Your proficiency in programming languages such as Java, C#, or JavaScript is essential, enabling you to write clean, efficient, and maintainable code. You will design algorithms and flowcharts to create systems that are logical and user-friendly. Collaboration with cross-functional teams, including product managers and designers, is crucial in order to understand software requirements and deliver innovative solutions. With your understanding of the software development life cycle, you will work through the processes of coding, debugging, testing, and deployment. You will employ industry best practices such as version control with Git and conduct code reviews to maintain high standards of software quality. Your role places you at the heart of our development efforts, where your technical prowess advances the functionality, scalability, and reliability of our software products.',\n",
+ " 'description': 'A Software Developer is responsible for designing, coding, testing, and maintaining software that meets client needs using languages like Java, C#, or JavaScript, collaborating with teams, adhering to best practices like Git for version control, and ensuring quality and innovation throughout the development life cycle.'},\n",
+ " {'name': 'Data_Analyst',\n",
+ " 'system_message': 'As a Data Analyst, your role is pivotal in interpreting complex data and providing insights that inform strategic decision-making. Utilize your analytical skills to cleanse and organize large sets of structured and unstructured data, ensuring its accuracy and readiness for in-depth analysis. Apply statistical analysis and predictive modeling to uncover trends, patterns, and correlations that drive operational improvements and innovative solutions. Use your proficiency in SQL for database interactions, and harness visualization tools such as Tableau or Power BI to craft compelling stories from data, aiding stakeholders in visualizing the implications of your findings. Stay abreast with the latest analytics techniques and continuously refine your models for enhanced performance, contributing significantly to the data-driven culture of our organization.',\n",
+ " 'description': 'The Data Analyst interprets complex datasets to provide strategic insights, cleanses and organizes data, performs statistical analysis and predictive modeling to identify trends and inform improvements, utilizes SQL for database management, and employs visualization tools like Tableau or Power BI to effectively communicate findings to stakeholders.'},\n",
+ " {'name': 'Journalist',\n",
+ " 'system_message': 'As a Journalist, you are responsible for identifying and pursuing newsworthy stories with the utmost ethical standards and a commitment to factual reporting. Your innate curiosity and excellent communication skills enable you to conduct thorough research and interviews, uncovering the details that make each story compelling and informative. Skilled in both written and verbal storytelling, you craft articles, reports, and features that engage and inform the public, adhering to strict deadlines without compromising on the integrity and accuracy of your work. Proficient in multimedia journalism, you adeptly use digital tools and social media to reach a wider audience, ensuring that your stories have the maximum impact.',\n",
+ " 'description': 'A Journalist is tasked with ethically sourcing and meticulously reporting newsworthy events, utilizing strong research and storytelling abilities across multiple platforms to accurately inform and engage a diverse audience.'},\n",
+ " {'name': 'Teacher',\n",
+ " 'system_message': 'As a Teacher, you are entrusted with the essential responsibility of fostering knowledge and encouraging academic and personal growth in your students. Your deep understanding of pedagogy, coupled with your expertise in the subject matter, enables you to create and deliver curricula that are both engaging and educational. Your adeptness at differentiated instruction allows you to tailor your teaching methods to suit the varied learning styles and needs within your classroom. By skillfully blending traditional teaching techniques with modern educational technology, you facilitate a dynamic and interactive learning environment. You excel in assessment and feedback, not only to gauge student progress but also to continuously improve your own teaching strategies. With strong interpersonal skills, you maintain open lines of communication with students, parents, and colleagues, fostering a collaborative and supportive school community.',\n",
+ " 'description': \"A Teacher is responsible for cultivating students' knowledge and growth through expertise in pedagogical practices and subject matter, designing engaging curricula, adapting teaching methods to diverse learning needs, integrating technology, and using assessment for continuous improvement while nurturing a cooperative school community.\"},\n",
+ " {'name': 'Lawyer',\n",
+ " 'system_message': 'As a Lawyer, you are required to uphold the highest standards of legal proficiency and ethical practice. Your role involves advising clients on their legal rights and responsibilities, as well as representing them in civil and criminal proceedings. You must possess a strong understanding of the law, paired with the ability to analyze case law and legislate history, to construct compelling arguments in support of your client’s position. Your keen attention to detail and dedication to thorough research are crucial in identifying legal precedents and crafting legal documents that adhere to the strictest of procedural standards. Moreover, you must exhibit exceptional negotiation skills to achieve favorable outcomes, whether in the courtroom or at the settlement table. With your articulate verbal and written communication, you clearly and persuasively present cases, explaining complex legal concepts in understandable terms to clients, judges, and juries. Your commitment to confidentiality and upholding justice is paramount and reflected in all aspects of your professional conduct.',\n",
+ " 'description': 'A Lawyer is a professionally trained legal advocate responsible for representing clients in legal proceedings, providing expert advice on legal matters, constructing persuasive arguments through meticulous research and analysis of law, and negotiating settlements, all while adhering to the highest ethical standards and maintaining strict confidentiality.'},\n",
+ " {'name': 'Programmer',\n",
+ " 'system_message': 'As a Programmer, you are responsible for the design, development, and implementation of software programs. Utilize your comprehensive understanding of programming languages, including but not limited to Java, C++, and Python, to create efficient and innovative software solutions. Your role involves writing clean, maintainable code while adhering to best practices in software development. You are expected to troubleshoot, debug, and upgrade existing software, as well as collaborate with cross-functional teams to define and design new product features. Your ability to think algorithmically and solve problems systematically will be integral in creating software that is not only functional but also scalable and secure.',\n",
+ " 'description': 'A Programmer designs, develops, and implements innovative and efficient software solutions using languages like Java, C++, and Python, ensuring code maintainability, collaborating on new features, and enhancing existing applications with a strong focus on scalability and security.'},\n",
+ " {'name': 'Accountant',\n",
+ " 'system_message': 'As Accountant, you are charged with the meticulous management and analysis of financial records, ensuring accuracy and compliance with relevant laws and regulations. Utilize your comprehensive understanding of accounting principles to prepare, examine, and maintain financial reports and statements, including balance sheets and income statements. Your role involves the reconciliation of accounts, evaluating financial operations to recommend best practices, identifying issues, and strategizing solutions for fiscal efficiency and profitability. Mastery in accounting software such as QuickBooks or Sage, alongside proficiency in Microsoft Excel, enables you to efficiently process and analyze financial data. You must ensure proper financial documentation and control systems are in place, providing comprehensive support to the organization’s financial health and integrity.',\n",
+ " 'description': 'As an Accountant, you are responsible for the accurate and compliant management, analysis, and reporting of financial data, along with recommending strategies to enhance fiscal efficiency and profitability, supported by proficiency in accounting software and Microsoft Excel.'},\n",
+ " {'name': 'Mathematician',\n",
+ " 'system_message': 'As a Mathematician, you are responsible for utilizing your profound understanding of mathematical theories and methodologies to solve complex theoretical and practical problems across various domains. Your proficiency in abstract reasoning enables you to develop new mathematical principles and to recognize and articulate the underlying mathematical relationships within real-world scenarios. You apply your expertise in calculus, algebra, statistics, and other mathematical branches to conduct rigorous analyses and to model systems for prediction and optimization. With a strong foundation in logic and quantitative reasoning, you perform peer reviews and contribute to interdisciplinary research projects, ensuring accuracy and consistency in mathematical arguments and results. Your role is crucial in advancing mathematical knowledge and providing innovative solutions to scientific and engineering challenges.',\n",
+ " 'description': 'As a Mathematician, you apply advanced mathematical theories and analytical skills to solve theoretical and practical problems in various industries, develop new principles, and provide innovative solutions to complex scientific and engineering challenges.'},\n",
+ " {'name': 'Physicist',\n",
+ " 'system_message': 'As a Physicist, you are charged with applying your profound understanding of the physical laws that govern the universe to unravel complex scientific phenomena. Your proficiency in theoretical and experimental physics enables you to develop models and conduct experiments that explore fundamental forces and particles. With exceptional analytical skills, you interpret empirical data to validate existing theories or propose new explanations for unexplained observations. Mastery in the use of mathematical tools such as differential equations and linear algebra is crucial for you to simulate physical processes. You are also adept at using specialized software and equipment for data acquisition and analysis, contributing to advancements in fields ranging from quantum mechanics to cosmology. Your strong critical thinking abilities empower you to solve intricate problems, and your commitment to scientific rigor ensures the integrity and accuracy of your research outcomes.',\n",
+ " 'description': 'A Physicist applies deep knowledge of physical laws to investigate scientific phenomena through theoretical modeling and experimental research, utilizing advanced mathematical techniques and specialized equipment to advance understanding in areas such as quantum mechanics and cosmology.'},\n",
+ " {'name': 'Biologist',\n",
+ " 'system_message': 'As a Biologist, you are entrusted with the study and understanding of living organisms, applying your expertise to investigate their functions, genetics, evolution, and ecosystems. Your skills in experimental design empower you to conduct research and experiments that can unlock new biological insights and improve our comprehension of life processes. Utilizing advanced microscopy techniques and molecular biology methods, you should meticulously analyze cell structures and DNA sequences to uncover the intricacies of life at a microscopic level. Demonstrate proficiency in bioinformatics tools to analyze genetic data and contribute valuable findings to the scientific community. Furthermore, as a communicator of science, ensure that your research findings are effectively documented and presented in scientific journals and at conferences, thereby enhancing the collective knowledge in your field.',\n",
+ " 'description': 'A Biologist meticulously studies and understands living organisms, conducting advanced research to decode genetics and ecosystems and sharing findings through scientific publications and presentations.'},\n",
+ " {'name': 'Chemist',\n",
+ " 'system_message': 'As a Chemist, you are charged with applying your profound understanding of chemical principles to conduct complex experiments, synthesize new compounds, and analyze the molecular and atomic structure of materials. Your proficiency in utilizing sophisticated analytical techniques - such as chromatography, spectroscopy, and mass spectrometry - enables you to decipher the composition and properties of substances. The knowledge you hold in chemical safety and handling procedures ensures a secure laboratory environment. With an adeptness in maintaining accurate records and an insightful approach to interpreting data, you transform raw experimental results into valuable scientific insights. Your ability to communicate complex chemical information clearly makes you essential in collaborative research efforts and in driving innovation within the field.',\n",
+ " 'description': 'As a Chemist, you are responsible for conducting advanced experiments, synthesizing compounds, deciphering substance compositions with techniques like chromatography and mass spectrometry, and transforming experimental data into scientific insights, while maintaining safety and clear communication in research collaborations.'},\n",
+ " {'name': 'Statistician',\n",
+ " 'system_message': 'As a Statistician, your primary duty is to apply mathematical and statistical methods to collect, analyze, and interpret numerical data to make informed decisions. Your strong grounding in probability theory will be essential for designing surveys and experiments to generate data. You are adept at constructing and applying sophisticated statistical models and methods, such as linear regression, ANOVA, or time-series analysis, ensuring that you accurately capture trends and relationships within the data. You possess an in-depth understanding of statistical software such as R or SAS, allowing you to perform complex analyses with efficiency and precision. Your ability to communicate complex statistical concepts to non-experts will be crucial; hence, your role includes presenting findings in a clear, actionable manner, with data visualizations and reports that drive strategic planning and policy development.',\n",
+ " 'description': 'A Statistician employs and interprets advanced statistical techniques to design data-collection processes, analyze data, and present findings in a comprehensible manner, supporting evidence-based decision-making and policy formation.'},\n",
+ " {'name': 'IT_Specialist',\n",
+ " 'system_message': 'As an IT Specialist, your primary responsibility is to maintain the integrity and functionality of all our computer systems and networks. Your comprehensive understanding of hardware and software is crucial for diagnosing and resolving technical issues. You are adept at implementing network security measures to protect data and systems from cyber threats. You also play a significant role in systems and software upgrades, ensuring a seamless transition without disrupting workflow. Utilizing your strong problem-solving skills and proficiency in scripting languages, you automate repetitive tasks, enhancing system efficiency. Your ability to communicate effectively with team members and non-technical staff allows you to provide clear guidance and end-user support.',\n",
+ " 'description': 'An IT Specialist is responsible for upholding and optimizing our computer systems and networks through maintenance, security, upgrades, issue resolution, automation, and providing support and clear communication to both technical and non-technical personnel.'},\n",
+ " {'name': 'Cybersecurity_Expert',\n",
+ " 'system_message': \"As a Cybersecurity Expert, you are charged with the responsibility of safeguarding the organization's computer networks and systems. Your deep understanding of cyber threats and mitigation techniques is critical in identifying vulnerabilities and protecting against malicious attacks. Employing your experience with tools such as firewalls, antivirus software, and intrusion detection systems, you will continuously monitor and defend our digital infrastructure. You are expected to conduct regular security audits and penetration testing to simulate cyber attacks and find potential weaknesses before they can be exploited. Your proficiency in risk management frameworks and incident response protocols ensures that you are prepared to swiftly handle and mitigate any security incidents that occur. With your expertise in encryption technologies and network protocols, you protect sensitive data and ensure compliance with relevant security standards and regulations. Your foresight in staying up-to-date with the latest cybersecurity trends and threats is paramount to maintaining the organization's digital defense at its peak.\",\n",
+ " 'description': \"As a Cybersecurity Expert, you are responsible for the proactive protection and defense of an organization's computer networks and systems against cyber threats through continuous monitoring, conducting security audits, penetrating testing, and swiftly mitigating security incidents, while ensuring compliance with security regulations.\"},\n",
+ " {'name': 'Artificial_Intelligence_Engineer',\n",
+ " 'system_message': 'As an Artificial Intelligence Engineer, you are responsible for conceptualizing, designing, and implementing intelligent systems that simulate human cognitive processes. Your role demands a deep understanding of neural networks, particularly Convolutional Neural Networks (CNNs) for image recognition tasks and Recurrent Neural Networks (RNNs) for natural language processing. With your expertise in TensorFlow or PyTorch, you develop complex models that can learn, adapt, and make decisions. You prioritize the ethical design and deployment of AI systems, conscious of the implications your work may have on society. Mastery of algorithms and a proficiency in a high-level programming language, preferably Python, enable you to transform theoretical AI concepts into practical solutions that drive innovation and efficiency.',\n",
+ " 'description': 'An Artificial Intelligence Engineer specializes in creating and implementing advanced intelligent systems, with a mastery of neural networks, machine learning frameworks, and ethical AI principles, to develop innovative solutions that emulate human cognition.'},\n",
+ " {'name': 'Financial_Analyst',\n",
+ " 'system_message': 'As a Financial Analyst, you are entrusted with utilizing your in-depth understanding of financial principles to assess investment opportunities, analyze financial data, and forecast economic trends. Your proficiency in financial modeling is paramount, enabling you to develop complex models that underpin the valuation of stocks, bonds, and other financial instruments. With a sharp eye for detail, you scrutinize company financial statements to derive actionable insights and recommend strategies to optimize financial performance. Your expertise in Excel, especially with advanced functions and formulas, allows you to efficiently manipulate and analyze large financial datasets. You are a whiz at creating compelling visualizations and delivering presentations to communicate your findings and influence strategic decisions. Your role is crucial in guiding investment decisions and driving the fiscal prudence of the organization.',\n",
+ " 'description': \"A Financial Analyst performs in-depth financial analysis and modeling to evaluate investments, forecast economic trends, and deliver strategic recommendations, leveraging advanced Excel skills to inform and guide the organization's financial decisions.\"}]"
]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
},
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.14"
- }
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sys_msg_list"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "256dd32b03a7a172",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "We can save the generated agents' information into a json file."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "53111125938845cf",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2023-12-23T07:40:01.750855900Z",
+ "start_time": "2023-12-23T07:40:01.710399600Z"
+ },
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "json.dump(sys_msg_list, open(\"./agent_library_example.json\", \"w\"), indent=4)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cfd883b79a3bd932",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## Build agents from library (by LLM)\n",
+ "Here, we introduce how to build agents from the generated library. As in the previous `build`, we also need to specify a `building_task` that lets the build manager know which agents should be selected from the library according to the task. \n",
+ "\n",
+ "We also need to specify a `library_path_or_json`, which can be a path of library or a JSON string with agents' configs. Here, we use the previously saved path as the library path."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "8963a8709c8e92e2",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2023-12-23T07:40:01.752918500Z",
+ "start_time": "2023-12-23T07:40:01.735461Z"
+ },
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "library_path_or_json = \"./agent_library_example.json\"\n",
+ "building_task = \"Find a paper on arxiv by programming, and analyze its application in some domain. For example, find a recent paper about gpt-4 on arxiv and find its potential applications in software.\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "72656a8d0c1a9b12",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "Then, we can call the `build_from_library` from the AgentBuilder to generate a list of agents from the library and let them complete the user's `execution_task` in a group chat."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "5c669b76b2c9b750",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2023-12-23T07:46:02.075542200Z",
+ "start_time": "2023-12-23T07:43:55.489042900Z"
+ },
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\u001b[32m==> Looking for suitable agents in the library...\u001b[0m\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "['Programmer', 'Mathematician'] are selected.\n",
+ "\u001b[32m==> Creating agents...\u001b[0m\n",
+ "Creating agent Programmer...\n",
+ "Creating agent Mathematician...\n",
+ "Adding user console proxy...\n",
+ "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
+ "\n",
+ "Find a recent paper about explainable AI on arxiv and find its potential applications in medical.\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Computer_terminal\n",
+ "\u001b[0m\n",
+ "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
+ "\n",
+ "There is no code from the last 1 message for me to execute. Group chat manager should let other participants to continue the conversation. If the group chat manager want to end the conversation, you should let other participant reply me only with \"TERMINATE\"\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Programmer\n",
+ "\u001b[0m\n",
+ "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
+ "\n",
+ "To find a recent paper about explainable AI on arXiv, we can use the arXiv API to search for papers that match the query. However, since I cannot directly access external APIs, I suggest that one of us manually searches for the paper on the arXiv website using relevant search terms such as \"explainable AI\" and \"medical applications\". Once we find a suitable paper, we can discuss its potential applications in the medical field. \n",
+ "\n",
+ "Mathematician, would you like to perform the search, or shall I provide a Python script that could be used to perform the search programmatically?\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Mathematician\n",
+ "\u001b[0m\n",
+ "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
+ "\n",
+ "As a Mathematician, I can certainly appreciate the value of a programmatic approach to searching for academic papers. However, since I do not have direct access to execute code or APIs, I would suggest that you, as the Programmer, provide the Python script that could be used to perform the search on arXiv. Once we have identified a paper, I can then assist in discussing its potential applications in the medical field from a mathematical perspective.\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Programmer\n",
+ "\u001b[0m\n",
+ "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
+ "\n",
+ "Understood. I will provide a Python script that can be used to search for recent papers about explainable AI on arXiv with potential applications in the medical field. The script will use the `arxiv` Python package, which is a wrapper for the arXiv API. If the package is not installed, we will need to install it first.\n",
+ "\n",
+ "Let's start by checking if the `arxiv` package is installed and if not, we will install it. Computer_terminal, please execute the following command to check for the `arxiv` package and install it if necessary.\n",
+ "\n",
+ "```sh\n",
+ "pip show arxiv || pip install arxiv\n",
+ "```\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Computer_terminal\n",
+ "\u001b[0m\n",
+ "\u001b[31m\n",
+ ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is sh)...\u001b[0m\n",
+ "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
+ "\n",
+ "exitcode: 0 (execution succeeded)\n",
+ "Code output: \n",
+ "Name: arxiv\n",
+ "Version: 2.1.0\n",
+ "Summary: Python wrapper for the arXiv API: https://arxiv.org/help/api/\n",
+ "Home-page: https://github.com/lukasschwab/arxiv.py\n",
+ "Author: Lukas Schwab\n",
+ "Author-email: lukas.schwab@gmail.com\n",
+ "License: MIT\n",
+ "Location: /home/vscode/.local/lib/python3.10/site-packages\n",
+ "Requires: feedparser, requests\n",
+ "Required-by: \n",
+ "\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Programmer\n",
+ "\u001b[0m\n",
+ "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
+ "\n",
+ "Great, the `arxiv` package is already installed. Now, I will provide a Python script that uses the `arxiv` package to search for recent papers related to explainable AI with potential applications in the medical field. The script will query the arXiv API for papers with relevant keywords and print out the title and summary of the most recent paper found.\n",
+ "\n",
+ "Computer_terminal, please execute the following Python script.\n",
+ "\n",
+ "```python\n",
+ "import arxiv\n",
+ "\n",
+ "# Define the search query\n",
+ "search_query = 'all:explainable AI AND all:medical'\n",
+ "\n",
+ "# Search for papers on arXiv\n",
+ "search = arxiv.Search(\n",
+ " query = search_query,\n",
+ " max_results = 1,\n",
+ " sort_by = arxiv.SortCriterion.SubmittedDate\n",
+ ")\n",
+ "\n",
+ "# Fetch the most recent paper\n",
+ "for paper in search.results():\n",
+ " print(\"Title:\", paper.title)\n",
+ " print(\"Summary:\", paper.summary)\n",
+ " # Only print the most recent paper\n",
+ " break\n",
+ "```\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Computer_terminal\n",
+ "\u001b[0m\n",
+ "\u001b[31m\n",
+ ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+ "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
+ "\n",
+ "exitcode: 0 (execution succeeded)\n",
+ "Code output: \n",
+ "Title: Automated Information Extraction from Thyroid Operation Narrative: A Comparative Study of GPT-4 and Fine-tuned KoELECTRA\n",
+ "Summary: In the rapidly evolving field of healthcare, the integration of artificial\n",
+ "intelligence (AI) has become a pivotal component in the automation of clinical\n",
+ "workflows, ushering in a new era of efficiency and accuracy. This study focuses\n",
+ "on the transformative capabilities of the fine-tuned KoELECTRA model in\n",
+ "comparison to the GPT-4 model, aiming to facilitate automated information\n",
+ "extraction from thyroid operation narratives. The current research landscape is\n",
+ "dominated by traditional methods heavily reliant on regular expressions, which\n",
+ "often face challenges in processing free-style text formats containing critical\n",
+ "details of operation records, including frozen biopsy reports. Addressing this,\n",
+ "the study leverages advanced natural language processing (NLP) techniques to\n",
+ "foster a paradigm shift towards more sophisticated data processing systems.\n",
+ "Through this comparative study, we aspire to unveil a more streamlined,\n",
+ "precise, and efficient approach to document processing in the healthcare\n",
+ "domain, potentially revolutionizing the way medical data is handled and\n",
+ "analyzed.\n",
+ "\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Mathematician\n",
+ "\u001b[0m\n",
+ "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
+ "\n",
+ "The paper titled \"Automated Information Extraction from Thyroid Operation Narrative: A Comparative Study of GPT-4 and Fine-tuned KoELECTRA\" presents a study on the use of artificial intelligence for automating the extraction of information from thyroid operation narratives. This is a clear example of explainable AI being applied in the medical field, specifically in the area of clinical workflows and document processing.\n",
+ "\n",
+ "The potential applications of such technology in medicine are vast. By automating the extraction of information from operation narratives, healthcare professionals can save time and reduce the likelihood of human error. This can lead to more accurate patient records, improved patient care, and streamlined administrative processes. Additionally, the ability to quickly and accurately process operation records can facilitate better data analysis, which can be used for medical research, trend analysis, and improving healthcare outcomes.\n",
+ "\n",
+ "The use of advanced natural language processing (NLP) techniques, as mentioned in the summary, is particularly important for processing free-style text formats that contain critical medical information. This technology could be further explored to extend its application to other types of medical documents and records, enhancing the overall efficiency of the healthcare system.\n",
+ "\n",
+ "The study's focus on comparing the performance of the fine-tuned KoELECTRA model with GPT-4 also highlights the importance of evaluating different AI models to determine the most effective approach for specific medical applications. This comparative analysis can lead to the development of more specialized AI tools tailored to the needs of the healthcare industry.\n",
+ "\n",
+ "In conclusion, the research presented in this paper has significant implications for the future of medical document processing and the broader integration of AI in healthcare.\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Programmer\n",
+ "\u001b[0m\n",
+ "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
+ "\n",
+ "The insights provided by the Mathematician are indeed valuable. The application of AI for automated information extraction from medical documents like thyroid operation narratives can greatly enhance efficiency and accuracy in healthcare. The use of models like GPT-4 and KoELECTRA for natural language processing tasks shows the potential of AI to handle complex, unstructured data which is common in medical records.\n",
+ "\n",
+ "From a programming perspective, the implementation of such AI systems would involve training models on large datasets of medical documents to learn the context and semantics specific to medical terminology. Ensuring the explainability of AI in this context is crucial, as healthcare professionals need to understand and trust the AI's decision-making process, especially when it directly affects patient care.\n",
+ "\n",
+ "Moreover, the integration of explainable AI into healthcare systems must adhere to strict privacy and security regulations to protect sensitive patient data. This requires careful design and implementation of data handling procedures within the AI system.\n",
+ "\n",
+ "The potential applications extend beyond just document processing to diagnostic assistance, personalized treatment plans, and predictive analytics for patient outcomes. As AI technology continues to evolve, its role in supporting and enhancing the capabilities of healthcare professionals will undoubtedly expand.\n",
+ "\n",
+ "Given the importance of the topic and the potential impact on healthcare, it would be beneficial to keep an eye on further developments in this field. If there are no further questions or points to discuss, we can conclude our conversation on this topic.\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Mathematician\n",
+ "\u001b[0m\n",
+ "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
+ "\n",
+ "I agree with the Programmer's assessment. The implementation of AI in healthcare does indeed require careful consideration of the models used, the training data, and the explainability of the AI's decisions. The ethical implications, particularly concerning patient privacy and data security, are paramount and must be addressed with the utmost rigor.\n",
+ "\n",
+ "The potential for AI to assist in diagnostics, treatment planning, and predictive analytics is a promising development for the future of medicine. It is essential that these systems are developed in collaboration with healthcare professionals to ensure they meet the real-world needs of the field.\n",
+ "\n",
+ "The interdisciplinary nature of this work, combining expertise in mathematics, computer science, and medicine, is a testament to the collaborative efforts needed to advance healthcare technology. It has been a pleasure discussing the potential applications of explainable AI in medicine with you.\n",
+ "\n",
+ "If there are no further points to add, I believe we have reached a natural conclusion to our conversation.\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Computer_terminal\n",
+ "\u001b[0m\n",
+ "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
+ "\n",
+ "There is no code from the last 1 message for me to execute. Group chat manager should let other participants to continue the conversation. If the group chat manager want to end the conversation, you should let other participant reply me only with \"TERMINATE\"\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[33mAll agents have been cleared.\u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "new_builder = AgentBuilder(\n",
+ " config_file_or_env=config_file_or_env, builder_model=\"gpt-4-1106-preview\", agent_model=\"gpt-4-1106-preview\"\n",
+ ")\n",
+ "agent_list, _ = new_builder.build_from_library(building_task, library_path_or_json, llm_config)\n",
+ "start_task(\n",
+ " execution_task=\"Find a recent paper about explainable AI on arxiv and find its potential applications in medical.\",\n",
+ " agent_list=agent_list,\n",
+ ")\n",
+ "new_builder.clear_all_agents()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c7a10e6fa00a5a0d",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## Build agents from library (by description-task similarity)\n",
+ "We also support using embedding similarity to select agents. You can use a [Sentence-Transformers model](https://www.sbert.net/docs/pretrained_models.html) as an embedding extractor, and AgentBuilder will select agents with profiles that are the most similar to the building task from the library by comparing their embedding similarity. This will reduce the use of LLMs but may have less accuracy."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "521dc5f961efde59",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2023-12-26T17:01:29.333975100Z",
+ "start_time": "2023-12-26T16:58:11.070813500Z"
+ },
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\u001b[32m==> Looking for suitable agents in the library...\u001b[0m\n"
+ ]
},
- "nbformat": 4,
- "nbformat_minor": 5
- }
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "['Programmer', 'Mathematician'] are selected.\n",
+ "\u001b[32m==> Creating agents...\u001b[0m\n",
+ "Creating agent Programmer...\n",
+ "Creating agent Mathematician...\n",
+ "Adding user console proxy...\n",
+ "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
+ "\n",
+ "Find a recent paper about gpt-4 on arxiv and find its potential applications in software.\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Computer_terminal\n",
+ "\u001b[0m\n",
+ "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
+ "\n",
+ "There is no code from the last 1 message for me to execute. Group chat manager should let other participants to continue the conversation. If the group chat manager want to end the conversation, you should let other participant reply me only with \"TERMINATE\"\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Programmer\n",
+ "\u001b[0m\n",
+ "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
+ "\n",
+ "To find a recent paper about GPT-4 on arXiv, we can use the arXiv API to search for papers. However, since I can't directly access external APIs, I can write a Python script that you can run on your local machine to perform this search. Would you like me to provide you with such a script?\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Computer_terminal\n",
+ "\u001b[0m\n",
+ "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
+ "\n",
+ "There is no code from the last 1 message for me to execute. Group chat manager should let other participants to continue the conversation. If the group chat manager want to end the conversation, you should let other participant reply me only with \"TERMINATE\"\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Programmer\n",
+ "\u001b[0m\n",
+ "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
+ "\n",
+ "I apologize for the confusion. I will provide a Python script that can be executed by the Computer_terminal to search for recent papers about GPT-4 on arXiv. Let's proceed with that.\n",
+ "\n",
+ "```python\n",
+ "import requests\n",
+ "from xml.etree import ElementTree\n",
+ "\n",
+ "# Define the search parameters and URL for the arXiv API\n",
+ "search_query = 'all:gpt-4'\n",
+ "start = 0\n",
+ "max_results = 5\n",
+ "sort_by = 'submittedDate'\n",
+ "sort_order = 'descending'\n",
+ "url = f'http://export.arxiv.org/api/query?search_query={search_query}&start={start}&max_results={max_results}&sortBy={sort_by}&sortOrder={sort_order}'\n",
+ "\n",
+ "# Send a GET request to the arXiv API\n",
+ "response = requests.get(url)\n",
+ "\n",
+ "# Parse the response if it was successful\n",
+ "if response.status_code == 200:\n",
+ " root = ElementTree.fromstring(response.content)\n",
+ " # Find and print the entries (papers)\n",
+ " for entry in root.findall('{http://www.w3.org/2005/Atom}entry'):\n",
+ " title = entry.find('{http://www.w3.org/2005/Atom}title').text\n",
+ " summary = entry.find('{http://www.w3.org/2005/Atom}summary').text\n",
+ " published = entry.find('{http://www.w3.org/2005/Atom}published').text\n",
+ " print(f\"Title: {title}\\nSummary: {summary}\\nPublished Date: {published}\\n\")\n",
+ "else:\n",
+ " print(f\"Failed to fetch data from arXiv. Status code: {response.status_code}\")\n",
+ "```\n",
+ "\n",
+ "This script will fetch the most recent papers related to GPT-4 from the arXiv API and print out their titles, summaries, and publication dates. Please execute this script to find the information we need.\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Computer_terminal\n",
+ "\u001b[0m\n",
+ "\u001b[31m\n",
+ ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...\u001b[0m\n",
+ "\u001b[33mComputer_terminal\u001b[0m (to chat_manager):\n",
+ "\n",
+ "exitcode: 0 (execution succeeded)\n",
+ "Code output: \n",
+ "Title: What If We Recaption Billions of Web Images with LLaMA-3?\n",
+ "Summary: Web-crawled image-text pairs are inherently noisy. Prior studies demonstrate\n",
+ "that semantically aligning and enriching textual descriptions of these pairs\n",
+ "can significantly enhance model training across various vision-language tasks,\n",
+ "particularly text-to-image generation. However, large-scale investigations in\n",
+ "this area remain predominantly closed-source. Our paper aims to bridge this\n",
+ "community effort, leveraging the powerful and \\textit{open-sourced} LLaMA-3, a\n",
+ "GPT-4 level LLM. Our recaptioning pipeline is simple: first, we fine-tune a\n",
+ "LLaMA-3-8B powered LLaVA-1.5 and then employ it to recaption 1.3 billion images\n",
+ "from the DataComp-1B dataset. Our empirical results confirm that this enhanced\n",
+ "dataset, Recap-DataComp-1B, offers substantial benefits in training advanced\n",
+ "vision-language models. For discriminative models like CLIP, we observe\n",
+ "enhanced zero-shot performance in cross-modal retrieval tasks. For generative\n",
+ "models like text-to-image Diffusion Transformers, the generated images exhibit\n",
+ "a significant improvement in alignment with users' text instructions,\n",
+ "especially in following complex queries. Our project page is\n",
+ "https://www.haqtu.me/Recap-Datacomp-1B/\n",
+ "\n",
+ "Published Date: 2024-06-12T17:59:07Z\n",
+ "\n",
+ "Title: DafnyBench: A Benchmark for Formal Software Verification\n",
+ "Summary: We introduce DafnyBench, the largest benchmark of its kind for training and\n",
+ "evaluating machine learning systems for formal software verification. We test\n",
+ "the ability of LLMs such as GPT-4 and Claude 3 to auto-generate enough hints\n",
+ "for the Dafny formal verification engine to successfully verify over 750\n",
+ "programs with about 53,000 lines of code. The best model and prompting scheme\n",
+ "achieved 68% success rate, and we quantify how this rate improves when retrying\n",
+ "with error message feedback and how it deteriorates with the amount of required\n",
+ "code and hints. We hope that DafnyBench will enable rapid improvements from\n",
+ "this baseline as LLMs and verification techniques grow in quality.\n",
+ "\n",
+ "Published Date: 2024-06-12T17:53:31Z\n",
+ "\n",
+ "Title: A Sociotechnical Lens for Evaluating Computer Vision Models: A Case\n",
+ " Study on Detecting and Reasoning about Gender and Emotion\n",
+ "Summary: In the evolving landscape of computer vision (CV) technologies, the automatic\n",
+ "detection and interpretation of gender and emotion in images is a critical area\n",
+ "of study. This paper investigates social biases in CV models, emphasizing the\n",
+ "limitations of traditional evaluation metrics such as precision, recall, and\n",
+ "accuracy. These metrics often fall short in capturing the complexities of\n",
+ "gender and emotion, which are fluid and culturally nuanced constructs. Our\n",
+ "study proposes a sociotechnical framework for evaluating CV models,\n",
+ "incorporating both technical performance measures and considerations of social\n",
+ "fairness. Using a dataset of 5,570 images related to vaccination and climate\n",
+ "change, we empirically compared the performance of various CV models, including\n",
+ "traditional models like DeepFace and FER, and generative models like GPT-4\n",
+ "Vision. Our analysis involved manually validating the gender and emotional\n",
+ "expressions in a subset of images to serve as benchmarks. Our findings reveal\n",
+ "that while GPT-4 Vision outperforms other models in technical accuracy for\n",
+ "gender classification, it exhibits discriminatory biases, particularly in\n",
+ "response to transgender and non-binary personas. Furthermore, the model's\n",
+ "emotion detection skew heavily towards positive emotions, with a notable bias\n",
+ "towards associating female images with happiness, especially when prompted by\n",
+ "male personas. These findings underscore the necessity of developing more\n",
+ "comprehensive evaluation criteria that address both validity and discriminatory\n",
+ "biases in CV models. Our proposed framework provides guidelines for researchers\n",
+ "to critically assess CV tools, ensuring their application in communication\n",
+ "research is both ethical and effective. The significant contribution of this\n",
+ "study lies in its emphasis on a sociotechnical approach, advocating for CV\n",
+ "technologies that support social good and mitigate biases rather than\n",
+ "perpetuate them.\n",
+ "\n",
+ "Published Date: 2024-06-12T13:52:30Z\n",
+ "\n",
+ "Title: Supportiveness-based Knowledge Rewriting for Retrieval-augmented\n",
+ " Language Modeling\n",
+ "Summary: Retrieval-augmented language models (RALMs) have recently shown great\n",
+ "potential in mitigating the limitations of implicit knowledge in LLMs, such as\n",
+ "untimely updating of the latest expertise and unreliable retention of long-tail\n",
+ "knowledge. However, since the external knowledge base, as well as the\n",
+ "retriever, can not guarantee reliability, potentially leading to the knowledge\n",
+ "retrieved not being helpful or even misleading for LLM generation. In this\n",
+ "paper, we introduce Supportiveness-based Knowledge Rewriting (SKR), a robust\n",
+ "and pluggable knowledge rewriter inherently optimized for LLM generation.\n",
+ "Specifically, we introduce the novel concept of \"supportiveness\"--which\n",
+ "represents how effectively a knowledge piece facilitates downstream tasks--by\n",
+ "considering the perplexity impact of augmented knowledge on the response text\n",
+ "of a white-box LLM. Based on knowledge supportiveness, we first design a\n",
+ "training data curation strategy for our rewriter model, effectively identifying\n",
+ "and filtering out poor or irrelevant rewrites (e.g., with low supportiveness\n",
+ "scores) to improve data efficacy. We then introduce the direct preference\n",
+ "optimization (DPO) algorithm to align the generated rewrites to optimal\n",
+ "supportiveness, guiding the rewriter model to summarize augmented content that\n",
+ "better improves the final response. Comprehensive evaluations across six\n",
+ "popular knowledge-intensive tasks and four LLMs have demonstrated the\n",
+ "effectiveness and superiority of SKR. With only 7B parameters, SKR has shown\n",
+ "better knowledge rewriting capability over GPT-4, the current state-of-the-art\n",
+ "general-purpose LLM.\n",
+ "\n",
+ "Published Date: 2024-06-12T11:52:35Z\n",
+ "\n",
+ "Title: Automated Information Extraction from Thyroid Operation Narrative: A\n",
+ " Comparative Study of GPT-4 and Fine-tuned KoELECTRA\n",
+ "Summary: In the rapidly evolving field of healthcare, the integration of artificial\n",
+ "intelligence (AI) has become a pivotal component in the automation of clinical\n",
+ "workflows, ushering in a new era of efficiency and accuracy. This study focuses\n",
+ "on the transformative capabilities of the fine-tuned KoELECTRA model in\n",
+ "comparison to the GPT-4 model, aiming to facilitate automated information\n",
+ "extraction from thyroid operation narratives. The current research landscape is\n",
+ "dominated by traditional methods heavily reliant on regular expressions, which\n",
+ "often face challenges in processing free-style text formats containing critical\n",
+ "details of operation records, including frozen biopsy reports. Addressing this,\n",
+ "the study leverages advanced natural language processing (NLP) techniques to\n",
+ "foster a paradigm shift towards more sophisticated data processing systems.\n",
+ "Through this comparative study, we aspire to unveil a more streamlined,\n",
+ "precise, and efficient approach to document processing in the healthcare\n",
+ "domain, potentially revolutionizing the way medical data is handled and\n",
+ "analyzed.\n",
+ "\n",
+ "Published Date: 2024-06-12T06:44:05Z\n",
+ "\n",
+ "\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Programmer\n",
+ "\u001b[0m\n",
+ "\u001b[33mProgrammer\u001b[0m (to chat_manager):\n",
+ "\n",
+ "Based on the search results from the arXiv API, we have found several papers that discuss potential applications of GPT-4 in software:\n",
+ "\n",
+ "1. **Recaptioning Web Images with LLaMA-3 and GPT-4**: This paper discusses the use of GPT-4 level LLMs for recaptioning web images, which can enhance model training across various vision-language tasks. This has implications for improving the quality of datasets used in machine learning and could be particularly beneficial for text-to-image generation and cross-modal retrieval tasks.\n",
+ "\n",
+ "2. **DafnyBench: A Benchmark for Formal Software Verification**: This paper introduces a benchmark for training and evaluating machine learning systems for formal software verification. It tests the ability of LLMs such as GPT-4 to auto-generate hints for the Dafny formal verification engine to successfully verify programs. This application could significantly impact the field of software verification by automating the generation of verification hints, potentially improving the efficiency and reliability of the verification process.\n",
+ "\n",
+ "3. **Automated Information Extraction from Thyroid Operation Narrative**: This study compares the GPT-4 model with the fine-tuned KoELECTRA model for automated information extraction from thyroid operation narratives. The application of GPT-4 in this context could revolutionize document processing in healthcare by providing a more efficient and accurate method for extracting information from medical records.\n",
+ "\n",
+ "These papers suggest that GPT-4 has the potential to be applied in various software-related fields, including enhancing datasets for machine learning, formal software verification, and healthcare document processing. The applications in these papers could lead to more efficient, accurate, and reliable software systems across different domains.\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Mathematician\n",
+ "\u001b[0m\n",
+ "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
+ "\n",
+ "The applications mentioned indeed highlight the versatility of GPT-4 in different domains. To further understand the potential impact of GPT-4 on software, let's delve into the mathematical and algorithmic principles that could be at play in these applications:\n",
+ "\n",
+ "1. **Recaptioning Web Images**: The process of recaptioning images with a language model like GPT-4 involves understanding the context of an image and generating descriptive text that accurately reflects its content. This task likely involves a combination of computer vision techniques to interpret the image and natural language processing to generate the caption. From a mathematical perspective, this would involve optimization algorithms to fine-tune the language model on a specific dataset, ensuring that the generated captions are both semantically and syntactically correct.\n",
+ "\n",
+ "2. **Formal Software Verification**: The use of GPT-4 to auto-generate hints for formal verification engines like Dafny involves the model understanding the logic and structure of the code. This requires a deep understanding of formal logic, proof theory, and possibly type theory if the language being verified is statically typed. The success rate of auto-generated hints would depend on the model's ability to reason about the correctness of code and the underlying mathematical properties that ensure its validity.\n",
+ "\n",
+ "3. **Automated Information Extraction from Medical Records**: For GPT-4 to extract information from medical narratives, it must process unstructured text and identify relevant medical terms and their relationships. This task involves natural language understanding, which from a mathematical standpoint, can be seen as a form of pattern recognition and classification. The model would need to be trained on a large corpus of medical texts, and its performance would be measured by its precision and recall in identifying and extracting the correct information.\n",
+ "\n",
+ "In each of these applications, GPT-4's effectiveness would be influenced by the underlying mathematical models, such as neural networks, and the optimization techniques used during training, such as gradient descent. The quality of the training data and the model's architecture (e.g., attention mechanisms, transformer layers) also play a crucial role in its performance.\n",
+ "\n",
+ "To verify the potential of GPT-4 in these applications, one could set up experiments to measure the performance of GPT-4 against specific benchmarks or metrics relevant to each domain. For example, in the case of formal software verification, one could measure the percentage of programs that are successfully verified with the hints generated by GPT-4 compared to a baseline or human-generated hints.\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Mathematician\n",
+ "\u001b[0m\n",
+ "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
+ "\n",
+ "To further verify the potential applications of GPT-4 in software, we can consider the following:\n",
+ "\n",
+ "1. **Recaptioning Web Images**: The effectiveness of GPT-4 in this application can be measured by the accuracy of the captions it generates. This can be quantified using metrics such as BLEU (Bilingual Evaluation Understudy) or ROUGE (Recall-Oriented Understudy for Gisting Evaluation), which compare the machine-generated text to a set of reference captions. A high score on these metrics would indicate that GPT-4 is capable of understanding and describing the content of images accurately.\n",
+ "\n",
+ "2. **Formal Software Verification**: To verify the application of GPT-4 in software verification, we could measure the success rate of the model in generating verification hints that lead to successful program verification. This could be done by comparing the verification success rate with and without the hints provided by GPT-4. Additionally, the complexity of the programs and the hints required could be analyzed to understand the model's limitations and strengths.\n",
+ "\n",
+ "3. **Automated Information Extraction from Medical Records**: The performance of GPT-4 in extracting information from medical narratives can be assessed by comparing the extracted information to a gold standard set of annotations. Precision, recall, and F1-score are common metrics used to evaluate information extraction systems. A high F1-score would suggest that GPT-4 is effective in identifying and extracting relevant information from unstructured medical texts.\n",
+ "\n",
+ "For each of these applications, it would be important to conduct a thorough analysis of the results to ensure that the model's performance is not only statistically significant but also practically relevant. Additionally, it would be crucial to assess the model's performance on diverse datasets to ensure that it generalizes well across different contexts and does not exhibit biases that could lead to incorrect or unfair outcomes.\n",
+ "\n",
+ "If we want to simulate or test these applications, we could propose specific experiments or data analysis procedures that align with the above verification strategies. However, such experiments would require access to the relevant datasets and computational resources to run the models, which may not be feasible within the scope of this conversation.\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Mathematician\n",
+ "\u001b[0m\n",
+ "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
+ "\n",
+ "To simulate or test these applications, we could propose specific experiments or data analysis procedures that align with the above verification strategies. However, such experiments would require access to the relevant datasets and computational resources to run the models, which may not be feasible within the scope of this conversation.\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[32m\n",
+ "Next speaker: Mathematician\n",
+ "\u001b[0m\n",
+ "\u001b[33mMathematician\u001b[0m (to chat_manager):\n",
+ "\n",
+ "TERMINATE\n",
+ "\n",
+ "--------------------------------------------------------------------------------\n",
+ "\u001b[33mAll agents have been cleared.\u001b[0m\n"
+ ]
+ }
+ ],
+ "source": [
+ "new_builder = AgentBuilder(\n",
+ " config_file_or_env=config_file_or_env, builder_model=\"gpt-4-1106-preview\", agent_model=\"gpt-4-1106-preview\"\n",
+ ")\n",
+ "agent_list, _ = new_builder.build_from_library(\n",
+ " building_task, library_path_or_json, llm_config, embedding_model=\"all-mpnet-base-v2\"\n",
+ ")\n",
+ "start_task(\n",
+ " execution_task=\"Find a recent paper about gpt-4 on arxiv and find its potential applications in software.\",\n",
+ " agent_list=agent_list,\n",
+ ")\n",
+ "new_builder.clear_all_agents()"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.14"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/notebook/lats_search.ipynb b/notebook/lats_search.ipynb
index 01b4449890e..fe7d362aa04 100644
--- a/notebook/lats_search.ipynb
+++ b/notebook/lats_search.ipynb
@@ -132,7 +132,6 @@
"metadata": {},
"outputs": [],
"source": [
- "\n",
"from pydantic import BaseModel, Field\n",
"\n",
"\n",
@@ -680,7 +679,6 @@
"metadata": {},
"outputs": [],
"source": [
- "\n",
"# Define the function to generate the initial response\n",
"\n",
"\n",
diff --git a/website/docs/ecosystem/mem0.md b/website/docs/ecosystem/mem0.md
new file mode 100644
index 00000000000..d9a9139df82
--- /dev/null
+++ b/website/docs/ecosystem/mem0.md
@@ -0,0 +1,165 @@
+# Mem0: Empower your AI applications with long-term memory and personalization
+
+
+
+[Mem0 Platform](https://www.mem0.ai/) provides a smart, self-improving memory layer for Large Language Models (LLMs), enabling developers to create personalized AI experiences that evolve with each user interaction.
+
+At a high level, Mem0 Platform offers comprehensive memory management, self-improving memory capabilities, cross-platform consistency, and centralized memory control for AI applications. For more info, check out the [Mem0 Platform Documentation](https://docs.mem0.ai).
+
+| | |
+| ---------------------------------------- | ----------------------------------------------------------------- |
+| 🧠 **Comprehensive Memory Management** | Manage long-term, short-term, semantic, and episodic memories |
+| 🔄 **Self-Improving Memory** | Adaptive system that learns from user interactions |
+| 🌐 **Cross-Platform Consistency** | Unified user experience across various AI platforms |
+| 🎛️ **Centralized Memory Control** | Effortless storage, updating, and deletion of memories |
+| 🚀 **Simplified Development** | API-first approach for streamlined integration |
+
+
+ Activity Dashboard
+
+ 
+
+
+
+## Installation
+
+Mem0 Platform works seamlessly with various AI applications.
+
+1. **Sign Up:**
+Create an account at [Mem0 Platform](https://app.mem0.ai/)
+
+2. **Generate API Key:**
+Create an API key in your Mem0 dashboard
+
+3. **Install Mem0 SDK:**
+```bash
+pip install mem0ai
+```
+
+4. **Configure Your Environment:**
+Add your API key to your environment variables
+
+```
+MEM0_API_KEY=
+```
+
+5. **Initialize Mem0:**
+
+```python
+from mem0ai import MemoryClient
+memory = MemoryClient(api_key=os.getenv("MEM0_API_KEY"))
+```
+
+After initializing Mem0, you can start using its memory management features in your AI application.
+
+## Features
+
+- **Long-term Memory**: Store and retrieve information persistently across sessions
+- **Short-term Memory**: Manage temporary information within a single interaction
+- **Semantic Memory**: Organize and retrieve conceptual knowledge
+- **Episodic Memory**: Store and recall specific events or experiences
+- **Self-Improving System**: Continuously refine understanding based on user interactions
+
+## Common Use Cases
+
+- Personalized Learning Assistants
+- Customer Support AI Agents
+- Healthcare Assistants
+- Virtual Companions
+
+## Mem0 Platform Examples
+
+### AutoGen with Mem0 Example
+
+This example demonstrates how to use Mem0 with AutoGen to create a conversational AI system with memory capabilities.
+
+```python
+import os
+from autogen import ConversableAgent
+from mem0 import MemoryClient
+
+# Set up environment variables
+os.environ["OPENAI_API_KEY"] = "your_openai_api_key"
+os.environ["MEM0_API_KEY"] = "your_mem0_api_key"
+
+# Initialize Agent and Memory
+agent = ConversableAgent(
+ "chatbot",
+ llm_config={"config_list": [{"model": "gpt-4", "api_key": os.environ.get("OPENAI_API_KEY")}]},
+ code_execution_config=False,
+ function_map=None,
+ human_input_mode="NEVER",
+)
+
+memory = MemoryClient(api_key=os.environ.get("MEM0_API_KEY"))
+
+# Insert a conversation into memory
+conversation = [
+ {
+ "role": "assistant",
+ "content": "Hi, I'm Best Buy's chatbot!\n\nThanks for being a My Best Buy TotalTM member.\n\nWhat can I help you with?"
+ },
+ {
+ "role": "user",
+ "content": "Seeing horizontal lines on our tv. TV model: Sony - 77\" Class BRAVIA XR A80K OLED 4K UHD Smart Google TV"
+ },
+ ........
+]
+
+memory.add(messages=conversation, user_id="customer_service_bot")
+
+# Agent Inference
+data = "Which TV am I using?"
+
+relevant_memories = memory.search(data, user_id="customer_service_bot")
+flatten_relevant_memories = "\n".join([m["memory"] for m in relevant_memories])
+
+prompt = f"""Answer the user question considering the memories.
+Memories:
+{flatten_relevant_memories}
+\n\n
+Question: {data}
+"""
+
+reply = agent.generate_reply(messages=[{"content": prompt, "role": "user"}])
+print("Reply :", reply)
+
+# Multi Agent Conversation
+manager = ConversableAgent(
+ "manager",
+ system_message="You are a manager who helps in resolving customer issues.",
+ llm_config={"config_list": [{"model": "gpt-4", "temperature": 0, "api_key": os.environ.get("OPENAI_API_KEY")}]},
+ human_input_mode="NEVER"
+)
+
+customer_bot = ConversableAgent(
+ "customer_bot",
+ system_message="You are a customer service bot who gathers information on issues customers are facing.",
+ llm_config={"config_list": [{"model": "gpt-4", "temperature": 0, "api_key": os.environ.get("OPENAI_API_KEY")}]},
+ human_input_mode="NEVER"
+)
+
+data = "What appointment is booked?"
+
+relevant_memories = memory.search(data, user_id="customer_service_bot")
+flatten_relevant_memories = "\n".join([m["memory"] for m in relevant_memories])
+
+prompt = f"""
+Context:
+{flatten_relevant_memories}
+\n\n
+Question: {data}
+"""
+
+result = manager.send(prompt, customer_bot, request_reply=True)
+```
+
+Access the complete code from this notebook: [Mem0 with AutoGen](https://colab.research.google.com/drive/1NZEwC9w6V2S6hYmK7l2SQ9jhQrG1uKk8?usp=sharing)
+
+This example showcases:
+1. Setting up AutoGen agents and Mem0 memory
+2. Adding a conversation to Mem0 memory
+3. Using Mem0 to retrieve relevant memories for agent inference
+4. Implementing a multi-agent conversation with memory-augmented context
+
+For more Mem0 examples, visit our [documentation](https://docs.mem0.ai/examples).