tutorials/11_System_Prompt_Optimizer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prompt Optimizer\n",
    "\n",
    "In this notebook we are building a prompt optimizer based upon the research whitepaper:  Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4  download from here https://arxiv.org/pdf/2312.16171.pdf\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if needed, install and/or upgrade to the latest version of the OpenAI Python library\n",
    "%pip install --upgrade openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import os module & the OpenAI Python library for calling the OpenAI API\n",
    "import os\n",
    "from openai import AzureOpenAI\n",
    "import json\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Expecting an .env file with the following content:\n",
    "# OPENAI_API_KEY=your_openai_api_key_here\n",
    "# AZURE_OPENAI_ENDPOINT =\"https://your_open_ai_resource_name.openai.azure.com/\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# System prompt for reviewing user prompts, adapted for the conversational format\n",
    "reviwer_system_prompt = \"\"\"\n",
    "##Objective##:\n",
    "To review user prompts critically, ensuring they align with the 26 key principles for effective interaction with Large Language Models (LLMs), and to provide constructive feedback for improvement.\n",
    "\n",
    "##Instructions for the Assistant##\n",
    "#Directness and Conciseness: Confirm the user prompt is straightforward, avoiding unnecessary politeness or filler words.\n",
    "#Intended Audience: Ensure the user prompt clearly identifies who the audience is, tailoring the complexity and style accordingly.\n",
    "#Simplifying Complex Tasks: Check if complex tasks are broken down into simpler, sequential prompts.\n",
    "#Affirmative Directives: Assess the use of clear, action-oriented language, avoiding negative constructions.\n",
    "#Clarity in Requests: Verify if the prompt asks for explanations in simple terms, suitable for laypersons or specific age groups.\n",
    "#Incentive Mention: Note if the prompt includes unnecessary incentives, which are not applicable here.\n",
    "#Example-Driven: Ensure the prompt encourages or includes specific examples or few-shot learning approaches.\n",
    "#Structured Formatting: Check for clear, structured formatting, including instructional headers and separation of sections.\n",
    "#Directive Phrasing: Confirm the use of explicit directives like “Your task is” to guide the LLM's response.\n",
    "#Consequence Awareness: Look for language indicating awareness of the importance of accuracy and the potential consequences of misinformation.\n",
    "#Natural, Human-Like Responses: Assess whether the prompt requests responses in a natural, conversational manner.\n",
    "#Step-by-Step Guidance: Ensure the prompt encourages thinking or explaining processes step by step.\n",
    "#Bias Avoidance: Check for explicit instructions to avoid bias and rely on stereotypes.\n",
    "#Detail Elicitation: Confirm if the prompt is open-ended, allowing the LLM to ask follow-up questions for clarification.\n",
    "#Learning and Testing: Verify if the prompt encourages an educational approach, including learning checks or tests.\n",
    "#Role Assignment: Assess whether the prompt assigns a specific role or perspective to the LLM.\n",
    "#Use of Delimiters: Check for the use of delimiters for clarity in separating sections or ideas.\n",
    "#Keyword Repetition: Look for intentional repetition of key words or phrases for emphasis.\n",
    "#Combining CoT with Few-Shot: Ensure the prompt integrates a Chain-of-Thought approach with example-driven or few-shot learning for complex tasks.\n",
    "#Output Primers: Confirm if the prompt concludes with an output primer, setting up the expected format or beginning of the response.\n",
    "#Detailed Writing Requests: Assess if the prompt asks for detailed, comprehensive coverage on a topic.\n",
    "#Style Preservation: Ensure the prompt asks for stylistic consistency when revising or responding to user inputs.\n",
    "#Complex Coding Requests: Note if the prompt properly guides the creation of multi-file coding solutions or complex coding tasks.\n",
    "#Content Initiation: Verify if the prompt effectively sets the stage or beginning for creative or continuation tasks.\n",
    "#Clear Requirements: Ensure the prompt clearly outlines all requirements, including format, style, and specific content expectations.\n",
    "#Style Mimicry: Check if the prompt asks for mimicry of a provided sample's style or language for text generation tasks.\n",
    "\n",
    "##Feedback and Recommendations##\n",
    "\n",
    "#Provide specific, actionable feedback for each principle, noting where the user prompt excels and where it falls short.\n",
    "#Offer suggestions for improvement, ensuring the user understands how to align their prompt better with these principles.\n",
    "#YOU MUST ALWAYS GENERATE AN IMPROVED VERSION OF THE PROMPT AND SUGGEST IT TO THE USER\n",
    "##Final Evaluation##\n",
    "\n",
    "Give an overall evaluation of the prompt's effectiveness and adherence to the 26 principles.\n",
    "Encourage the user to refine their prompting skills based on the feedback.\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "prompt_to_review = \"\"\"\n",
    "Develop an executive summary that encapsulates the potential investment in a point-of-care testing company within the in-vitro diagnostics market. The summary should:\n",
    "\n",
    "Present an overview of the company's business model, product offerings, market reach, and financial highlights without specifying exact sales figures or margins.\n",
    "Give an account of the company's historical background, current ownership, and ongoing bid process, mentioning interest from other equity firms without naming them.\n",
    "Formulate an investment thesis that emphasizes the company's market position, growth strategy, and business model strengths, such as recurring revenue streams and proprietary technology.\n",
    "List key investment considerations and due diligence items that an investor should evaluate, focusing on market share, product portfolio, and technological innovation.\n",
    "Conclude with a preliminary valuation outlook and recommendations for proceeding with an investment procedure, without including specific multiples or revenue projections.\"\n",
    "One-shot Example for LLM Output:\n",
    "\n",
    "\"The executive summary should highlight an investment opportunity in a dynamic IVD company with a strong presence in point-of-care testing. The company, leveraging a recurring revenue model, has established significant market access in Europe and plans for global expansion. Originally part of a larger pharmaceutical entity, it is now in the midst of a strategic acquisition process. The investment thesis should underscore the firm's solid market standing, the potential for growth via technological advancements, and its strategic plans for market expansion. Considerations for investment should cover market analysis, risk assessment, and evaluation of proprietary technologies. The summary concludes with a recommendation for investment proceedings based on a positive valuation outlook, considering the company's robust business model and future potential.\"\n",
    "\"\"\"\n",
    "messages =[\n",
    "    {\n",
    "        \"role\": \"system\",\n",
    "        \"content\": reviwer_system_prompt\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": prompt_to_review\n",
    "    }\n",
    "]\n",
    "   \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "assistant: The user prompt has been drafted with the intent to create an outline for an executive summary regarding an investment opportunity. Here is a critique of the prompt based on the 26 key principles along with suggestions for improvement:\n",
      "\n",
      "1. Directness and Conciseness: The prompt is relatively straightforward and on point. There is little to no filler content, meeting the principle well.\n",
      "\n",
      "2. Intended Audience: It is implied the audience is potential investors or stakeholders. Adding a line specifying the audience could make it more explicit.\n",
      "\n",
      "3. Simplifying Complex Tasks: The task could benefit from being broken down into parts, especially for the section on investment considerations. \n",
      "\n",
      "4. Affirmative Directives: The prompt uses affirmative language effectively, with clear directives such as \"Present,\" \"Give,\" and \"Conclude.\"\n",
      "\n",
      "5. Clarity in Requests: While the request is fairly clear, it could be simplified by specifying that technical jargon should be minimized.\n",
      "\n",
      "6. Incentive Mention: There is no unnecessary incentive mentioned, which aligns with the principles.\n",
      "\n",
      "7. Example-Driven: The prompt could benefit from including an example to guide the LLM.\n",
      "\n",
      "8. Structured Formatting: Organizing the prompt into bullet points or separated sections could enhance clarity.\n",
      "\n",
      "9. Directive Phrasing: The prompt could use more explicit directive phrasing, such as “Your task is to develop an executive summary...”\n",
      "\n",
      "10. Consequence Awareness: The prompt lacks an expression of the importance of accuracy or potential consequences of misinformation.\n",
      "\n",
      "11. Natural, Human-Like Responses: There is no specific request for a natural, conversational tone. It may not be necessary in this case due to the nature of the expected content.\n",
      "\n",
      "12. Step-by-Step Guidance: There's no step-by-step process indicated for the LLM, which might make the task complex.\n",
      "\n",
      "13. Bias Avoidance: The prompt does not explicitly instruct on avoiding bias.\n",
      "\n",
      "14. Detail Elicitation: There could be more encouragement for the LLM to ask for specifics if necessary.\n",
      "\n",
      "15. Learning and Testing: This principle is not applicable here as the prompt doesn’t involve a learning aspect.\n",
      "\n",
      "16. Role Assignment: No specific role is assigned to the LLM. It should consider taking on the role of an investment analyst.\n",
      "\n",
      "17. Use of Delimiters: The use of delimiters would improve clarity by structuring the response.\n",
      "\n",
      "18. Keyword Repetition: The prompt does not repeat key words or phrases for emphasis, but it could to reinforce the task's key points.\n",
      "\n",
      "19. Combining CoT with Few-Shot: The prompt could integrate a Chain-of-Thought approach with an example-driven method for better clarity.\n",
      "\n",
      "20. Output Primers: An output primer is not provided, which could assist in setting up the expectation for the response's format.\n",
      "\n",
      "21. Detailed Writing Requests: The prompt asks for detailed content appropriately.\n",
      "\n",
      "22. Style Preservation: The prompt should mention to maintain the executive summary style.\n",
      "\n",
      "23. Complex Coding Requests: This principle is not applicable to the task at hand.\n",
      "\n",
      "24. Content Initiation: The prompt could benefit from a more precise content initiation.\n",
      "\n",
      "25. Clear Requirements: The requirements are mostly clear but could specify expectations around style and content better.\n",
      "\n",
      "26. Style Mimicry: There’s no request for style mimicry, which seems appropriate given the task. \n",
      "\n",
      "Suggested Improved Prompt:\n",
      "\"Develop an executive summary focused on a potential investment opportunity in a point-of-care testing company in the in-vitro diagnostics market. Your task is to: \n",
      "\n",
      "- Outline the company's business model, including product offerings and market reach, providing financial highlights in broad terms.\n",
      "- Share the company's history, its journey from inception to its current status, and its ongoing acquisition process, hinting at interests from unnamed equity firms.\n",
      "- Present an investment thesis demonstrating the company's strategic market position, growth avenues, and business model merits like recurring revenues and unique technologies.\n",
      "- Enumerate key considerations investors should explore during due diligence, such as market share, product mix, and technological innovations.\n",
      "- Conclude with a preliminary valuation outlook and suggest next steps in the investment process without disclosing specific financial predictions.\"\n",
      "\n",
      "Overall, the given prompt is effective but could be enhanced for clarity, directive language, and structured formatting, among other points as detailed in the critique. The improved prompt addresses these areas, maintaining a clear, concise, and structured approach for generating the executive summary.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "try:\n",
    "    \n",
    "    client = AzureOpenAI(\n",
    "        api_key=os.getenv(\"AZURE_OPENAI_KEY\"),  \n",
    "        api_version=\"2024-02-15-preview\",\n",
    "        azure_endpoint = f\"https://{os.getenv('AZURE_OPENAI_RESOURCE')}.openai.azure.com/\"\n",
    "    )\n",
    "\n",
    "    deployment_name='gpt-4' #This will correspond to the custom name you chose for your deployment when you deployed a model. Use a gpt-35-turbo-instruct deployment. \n",
    "        \n",
    "    # Send a completion call to generate an answer\n",
    "    completion = client.chat.completions.create(\n",
    "        messages=messages,\n",
    "        model=deployment_name,\n",
    "        max_tokens=3000\n",
    "    )\n",
    "\n",
    "    print(f\"{completion.choices[0].message.role}: {completion.choices[0].message.content}\")\n",
    "except Exception as e:\n",
    "    # Handles all other exceptions\n",
    "    print(f\"An exception has occurred: {e}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mmdoc",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}