Refactor Stepwise Planner handler, added logic to allow use of stepwi…

…se plan result as bot response (microsoft#514)

### Motivation and Context

<!-- Thank you for your contribution to the chat-copilot repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

This pull request refactors the use of Stepwise Planner in CC. The
changes include a fix to embed the result of the Stepwise Planner into
the meta prompt if a valid result is returned and adds a new property to
PlannerOptions to indicate whether to use the planner result as the bot
response.

Additionally, the ChatSkill class has been updated to handle the new
changes, including adding the ability to pass a response string to the
HandleBotResponseAsync method and changing the StepwiseThoughtProcess
property to a PlanExecutionMetadata object. The PromptDialog component
has also been updated to display the Stepwise Planner supplement and raw
view when appropriate, and the BotResponsePrompt interface has been
updated to include a rawView property (to show the Stepwise Thought
Process if the webapi is configured to return the stepwise response
directly).

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

Logic:
1. If Stepwise Planner doesn't return a valid result (i.e., "Result not
found"), continue with regular response generation. Planner result will
not be included in meta prompt.

![image](https://github.com/microsoft/chat-copilot/assets/125500434/d25168b4-3bb2-4148-ad3a-1ded6d93fe75)

2. If Stepwise Planner returns a valid result, that result will be
supplemented with complementary text to guide the model in using this
response in the meta prompt.

![image](https://github.com/microsoft/chat-copilot/assets/125500434/61622a4c-61f4-4b10-8af0-36d9aed014b2)

3. If Stepwise Planner returns a valid result and the webapi is
configured to use this result directly, it will be immediately returned
to the client.

![image](https://github.com/microsoft/chat-copilot/assets/125500434/8f4cdf8e-67e7-4fde-9bcb-e2fdc42fbc3b)

Change Details:
- Update ExternalInformation property in BotResponsePrompt to use
SemanticDependency of PlanExecutionMetadata instead of
ISemanticDependency.
- Rename StepwiseThoughtProcess class to PlanExecutionMetadata.
- Add RawResult property to PlanExecutionMetadata to store the raw
result of the planner.
- Add UseStepwiseResultAsBotResponse property to PlannerOptions to
indicate whether to use the planner result as the bot response.
- Add StepwisePlannerSupplement property to PromptsOptions to help guide
model in using a response from StepwisePlanner.
- Update ChatSkill to use SemanticDependency of PlanExecutionMetadata
instead of StepwiseThoughtProcess.
- Add logic to handle using plan result as bot response if
UseStepwiseResultAsBotResponse is true.
- Update HandleBotResponseAsync method to accept rawResult parameter.

Stepwise Thought Proceses available in Prompt details anytime the
Stepwise planner is used, even if no result is found.

![image](https://github.com/microsoft/chat-copilot/assets/125500434/6b14dc70-f79e-47be-a0b3-0833c9b14117)


### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [Contribution
Guidelines](https://github.com/microsoft/chat-copilot/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/chat-copilot/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
~- [ ] All unit tests pass, and I have added new tests where possible~
- [x] I didn't break anyone 😄

webapi/Models/Response/BotResponsePrompt.cs

@@ -44,7 +44,7 @@ public class BotResponsePrompt
     /// Relevant additional knowledge extracted using a planner.
     /// </summary>
     [JsonPropertyName("externalInformation")]
-    public ISemanticDependency ExternalInformation { get; set; }
+    public SemanticDependency<PlanExecutionMetadata> ExternalInformation { get; set; }
 
     /// <summary>
     /// The collection of context messages associated with this chat completions request.
@@ -58,7 +58,7 @@ public BotResponsePrompt(
         string audience,
         string userIntent,
         string chatMemories,
-        ISemanticDependency externalInformation,
+        SemanticDependency<PlanExecutionMetadata> externalInformation,
         string chatHistory,
         ChatCompletionContextMessages metaPromptTemplate
     )

webapi/Models/Response/StepwiseThoughtProcess.cs → webapi/Models/Response/PlanExecutionMetadata.cs

@@ -5,9 +5,9 @@
 namespace CopilotChat.WebApi.Models.Response;
 
 /// <summary>
-/// Information about a pass through stepwise planner.
+/// Metadata about plan execution.
 /// </summary>
-public class StepwiseThoughtProcess
+public class PlanExecutionMetadata
 {
     /// <summary>
     /// Steps taken execution stat.
@@ -34,10 +34,17 @@ public class StepwiseThoughtProcess
     [JsonPropertyName("plannerType")]
     public PlanType PlannerType { get; set; } = PlanType.Stepwise;
 
-    public StepwiseThoughtProcess(string stepsTaken, string timeTaken, string skillsUsed)
+    /// <summary>
+    /// Raw result of the planner.
+    /// </summary>
+    [JsonIgnore]
+    public string RawResult { get; set; } = string.Empty;
+
+    public PlanExecutionMetadata(string stepsTaken, string timeTaken, string skillsUsed, string rawResult)
     {
         this.StepsTaken = stepsTaken;
         this.TimeTaken = timeTaken;
         this.SkillsUsed = skillsUsed;
+        this.RawResult = rawResult;
     }
 }

webapi/Options/PlannerOptions.cs

@@ -66,6 +66,12 @@ public class ErrorOptions
     /// </summary>
     public ErrorOptions ErrorHandling { get; set; } = new ErrorOptions();
 
+    /// <summary>
+    /// Optional flag to indicate whether to use the planner result as the bot response.
+    /// </summary>
+    [RequiredOnPropertyValue(nameof(Type), PlanType.Stepwise)]
+    public bool UseStepwiseResultAsBotResponse { get; set; } = false;
+
     /// <summary>
     /// The configuration for the stepwise planner.
     /// </summary>

webapi/Options/PromptsOptions.cs

@@ -71,6 +71,11 @@ public class PromptsOptions
     /// </summary>
     [Required, NotEmptyOrWhitespace] public string PlanResultsDescription { get; set; } = string.Empty;
 
+    /// <summary>
+    /// Supplement to help guide model in using a response from StepwisePlanner.
+    /// </summary>
+    [Required, NotEmptyOrWhitespace] public string StepwisePlannerSupplement { get; set; } = string.Empty;
+
     internal string[] SystemAudiencePromptComponents => new string[]
     {
         this.SystemAudience,

webapi/Skills/ChatSkills/ChatSkill.cs

@@ -20,6 +20,7 @@
 using Microsoft.AspNetCore.SignalR;
 using Microsoft.Extensions.Logging;
 using Microsoft.Extensions.Options;
+using Microsoft.IdentityModel.Tokens;
 using Microsoft.SemanticKernel;
 using Microsoft.SemanticKernel.AI.ChatCompletion;
 using Microsoft.SemanticKernel.AI.TextCompletion;
@@ -476,7 +477,7 @@ await this.SaveNewResponseAsync(
                 chatContext.Variables.Set(TokenUtils.GetFunctionKey(this._logger, "SystemMetaPrompt")!, TokenUtils.GetContextMessagesTokenCount(promptTemplate).ToString(CultureInfo.CurrentCulture));
 
                 // TODO: [Issue #150, sk#2106] Accommodate different planner contexts once core team finishes work to return prompt and token usage.
-                var plannerDetails = new SemanticDependency<object>(planResult, null, deserializedPlan.Type.ToString());
+                var plannerDetails = new SemanticDependency<PlanExecutionMetadata>(planResult, null, deserializedPlan.Type.ToString());
 
                 // Get bot response and stream to client
                 var promptView = new BotResponsePrompt(systemInstructions, "", deserializedPlan.UserIntent, "", plannerDetails, chatHistoryString, promptTemplate);
@@ -564,8 +565,8 @@ private async Task<ChatMessage> GetChatResponseAsync(string chatId, string userI
             () => this.AcquireExternalInformationAsync(chatContext, userIntent, externalInformationTokenLimit, cancellationToken: cancellationToken), nameof(AcquireExternalInformationAsync));
 
         // Extract additional details about stepwise planner execution in chat context
-        var plannerDetails = new SemanticDependency<StepwiseThoughtProcess>(
-                planResult,
+        var plannerDetails = new SemanticDependency<PlanExecutionMetadata>(
+                this._externalInformationSkill.StepwiseThoughtProcess?.RawResult ?? planResult,
                 this._externalInformationSkill.StepwiseThoughtProcess
             );
 
@@ -585,6 +586,13 @@ private async Task<ChatMessage> GetChatResponseAsync(string chatId, string userI
             );
         }
 
+        // If plan result is to be used as bot response, save the Stepwise result as a new response to the chat history and return.
+        if (this._externalInformationSkill.UseStepwiseResultAsBotResponse(planResult))
+        {
+            var promptDetails = new BotResponsePrompt("", "", userIntent, "", plannerDetails, "", new ChatHistory());
+            return await this.HandleBotResponseAsync(chatId, userId, chatContext, promptDetails, cancellationToken, null, this._externalInformationSkill.StepwiseThoughtProcess!.RawResult);
+        }
+
         // Query relevant semantic and document memories
         await this.UpdateBotResponseStatusOnClientAsync(chatId, "Extracting semantic and document memories", cancellationToken);
         var chatMemoriesTokenLimit = (int)(remainingTokenBudget * this._promptOptions.MemoriesResponseContextWeight);
@@ -614,9 +622,7 @@ private async Task<ChatMessage> GetChatResponseAsync(string chatId, string userI
 
         // Stream the response to the client
         var promptView = new BotResponsePrompt(systemInstructions, audience, userIntent, memoryText, plannerDetails, chatHistory, promptTemplate);
-        ChatMessage chatMessage = await this.HandleBotResponseAsync(chatId, userId, chatContext, promptView, cancellationToken, citationMap.Values.AsEnumerable());
-
-        return chatMessage;
+        return await this.HandleBotResponseAsync(chatId, userId, chatContext, promptView, cancellationToken, citationMap.Values.AsEnumerable());
     }
 
     /// <summary>
@@ -650,16 +656,32 @@ private async Task<ChatMessage> HandleBotResponseAsync(
         SKContext chatContext,
         BotResponsePrompt promptView,
         CancellationToken cancellationToken,
-        IEnumerable<CitationSource>? citations = null)
+        IEnumerable<CitationSource>? citations = null,
+        string? responseContent = null)
     {
-        // Get bot response and stream to client
-        await this.UpdateBotResponseStatusOnClientAsync(chatId, "Generating bot response", cancellationToken);
-        ChatMessage chatMessage = await AsyncUtils.SafeInvokeAsync(
-            () => this.StreamResponseToClientAsync(chatId, userId, promptView, cancellationToken, citations), nameof(StreamResponseToClientAsync));
+        ChatMessage chatMessage;
+        if (responseContent.IsNullOrEmpty())
+        {
+            // Get bot response and stream to client
+            await this.UpdateBotResponseStatusOnClientAsync(chatId, "Generating bot response", cancellationToken);
+            chatMessage = await AsyncUtils.SafeInvokeAsync(
+                () => this.StreamResponseToClientAsync(chatId, userId, promptView, cancellationToken, citations), nameof(StreamResponseToClientAsync));
+        }
+        else
+        {
+            chatMessage = await this.CreateBotMessageOnClient(
+                chatId,
+                userId,
+                JsonSerializer.Serialize(promptView),
+                responseContent!,
+                cancellationToken,
+                citations
+            );
+        }
 
         // Save the message into chat history
         await this.UpdateBotResponseStatusOnClientAsync(chatId, "Saving message to chat history", cancellationToken);
-        await this._chatMessageRepository.UpsertAsync(chatMessage);
+        await this._chatMessageRepository.UpsertAsync(chatMessage!);
 
         // Extract semantic chat memory
         await this.UpdateBotResponseStatusOnClientAsync(chatId, "Generating semantic chat memory", cancellationToken);
@@ -675,7 +697,7 @@ await AsyncUtils.SafeInvokeAsync(
 
         // Calculate total token usage for dependency functions and prompt template
         await this.UpdateBotResponseStatusOnClientAsync(chatId, "Calculating token usage", cancellationToken);
-        chatMessage.TokenUsage = this.GetTokenUsages(chatContext, chatMessage.Content);
+        chatMessage!.TokenUsage = this.GetTokenUsages(chatContext, chatMessage.Content);
 
         // Update the message on client and in chat history with final completion token usage
         await this.UpdateMessageOnClient(chatMessage, cancellationToken);

webapi/Skills/ChatSkills/ExternalInformationSkill.cs

@@ -17,9 +17,12 @@
 using CopilotChat.WebApi.Skills.Utils;
 using Microsoft.Extensions.Logging;
 using Microsoft.Extensions.Options;
+using Microsoft.IdentityModel.Tokens;
 using Microsoft.SemanticKernel.Orchestration;
 using Microsoft.SemanticKernel.Planning;
+using Microsoft.SemanticKernel.Planning.Stepwise;
 using Microsoft.SemanticKernel.SkillDefinition;
+using Microsoft.SemanticKernel.TemplateEngine.Prompt;
 
 namespace CopilotChat.WebApi.Skills.ChatSkills;
 
@@ -47,7 +50,13 @@ public class ExternalInformationSkill
     ///  Options for the planner.
     /// </summary>
     private readonly PlannerOptions? _plannerOptions;
-    public PlannerOptions? PlannerOptions => this._plannerOptions;
+    public PlannerOptions? PlannerOptions
+    {
+        get
+        {
+            return this._plannerOptions;
+        }
+    }
 
     /// <summary>
     /// Proposed plan to return for approval.
@@ -57,7 +66,7 @@ public class ExternalInformationSkill
     /// <summary>
     /// Stepwise thought process to return for view.
     /// </summary>
-    public StepwiseThoughtProcess? StepwiseThoughtProcess { get; private set; }
+    public PlanExecutionMetadata? StepwiseThoughtProcess { get; private set; }
 
     /// <summary>
     /// Header to indicate plan results.
@@ -101,15 +110,11 @@ public async Task<string> InvokePlannerAsync(
 
         var contextString = this.GetChatContextString(context);
         var goal = $"Given the following context, accomplish the user intent.\nContext:\n{contextString}\n{userIntent}";
+
+        // Run stepwise planner if PlannerOptions.Type == Stepwise
         if (this._planner.PlannerOptions?.Type == PlanType.Stepwise)
         {
-            var plannerContext = context.Clone();
-            plannerContext = await this._planner.RunStepwisePlannerAsync(goal, context, cancellationToken);
-            this.StepwiseThoughtProcess = new StepwiseThoughtProcess(
-                plannerContext.Variables["stepsTaken"],
-                plannerContext.Variables["timeTaken"],
-                plannerContext.Variables["skillCount"]);
-            return $"{plannerContext.Variables.Input.Trim()}\n";
+            return await this.RunStepwisePlannerAsync(goal, context, cancellationToken);
         }
 
         // Create a plan and set it in context for approval.
@@ -193,8 +198,84 @@ public async Task<string> ExecutePlanAsync(
         return $"{functionsUsed}\n{ResultHeader}{planResult.Trim()}";
     }
 
+    /// <summary>
+    /// Determines whether to use the stepwise planner result as the bot response, thereby bypassing meta prompt generation and completion.
+    /// </summary>
+    /// <param name="planResult">The result obtained from the stepwise planner.</param>
+    /// <returns>
+    /// True if the stepwise planner result should be used as the bot response,
+    /// false otherwise.
+    /// </returns>
+    /// <remarks>
+    /// This method checks the following conditions:
+    /// 1. The plan result is not null, empty, or whitespace.
+    /// 2. The planner options are specified, and the plan type is set to Stepwise.
+    /// 3. The UseStepwiseResultAsBotResponse option is enabled.
+    /// 4. The StepwiseThoughtProcess is not null.
+    /// </remarks>
+    public bool UseStepwiseResultAsBotResponse(string planResult)
+    {
+        return !string.IsNullOrWhiteSpace(planResult)
+            && this._plannerOptions?.Type == PlanType.Stepwise
+            && this._plannerOptions.UseStepwiseResultAsBotResponse
+            && this.StepwiseThoughtProcess != null;
+    }
+
     #region Private
 
+    /// <summary>
+    /// Executes the stepwise planner with a given goal and context, and returns the result along with descriptive text.
+    /// Also sets any metadata associated with stepwise planner execution.
+    /// </summary>
+    /// <param name="goal">The goal to be achieved by the stepwise planner.</param>
+    /// <param name="context">The SKContext containing the necessary information for the planner.</param>
+    /// <param name="cancellationToken">A CancellationToken to observe while waiting for the task to complete.</param>
+    /// <returns>
+    /// A formatted string containing the result of the stepwise planner and a supplementary message to guide the model in using the result.
+    /// </returns>
+    private async Task<string> RunStepwisePlannerAsync(string goal, SKContext context, CancellationToken cancellationToken)
+    {
+        var plannerContext = context.Clone();
+        plannerContext = await this._planner.RunStepwisePlannerAsync(goal, context, cancellationToken);
+
+        // Populate the execution metadata.
+        var plannerResult = plannerContext.Variables.Input.Trim();
+        this.StepwiseThoughtProcess = new PlanExecutionMetadata(
+            plannerContext.Variables["stepsTaken"],
+            plannerContext.Variables["timeTaken"],
+            plannerContext.Variables["skillCount"],
+            plannerResult);
+
+        // Return empty string if result was not found so it's omitted from the meta prompt.
+        if (plannerResult.Contains("Result not found, review 'stepsTaken' to see what happened.", StringComparison.OrdinalIgnoreCase))
+        {
+            return string.Empty;
+        }
+
+        // Parse the steps taken to determine which functions were used.
+        if (plannerContext.Variables.TryGetValue("stepsTaken", out var stepsTaken))
+        {
+            var steps = JsonSerializer.Deserialize<List<SystemStep>>(stepsTaken);
+            var functionsUsed = new HashSet<string>();
+            steps?.ForEach(step =>
+            {
+                if (!step.Action.IsNullOrEmpty()) { functionsUsed.Add(step.Action); }
+            });
+
+            var planFunctions = string.Join(", ", functionsUsed);
+            plannerContext.Variables.Set("planFunctions", functionsUsed.Count > 0 ? planFunctions : "N/A");
+        }
+
+        // Render the supplement to guide the model in using the result.
+        var promptRenderer = new PromptTemplateEngine();
+        var resultSupplement = await promptRenderer.RenderAsync(
+            this._promptOptions.StepwisePlannerSupplement,
+            plannerContext,
+            cancellationToken);
+
+        return $"{resultSupplement}\n\nResult:\n\"{plannerResult}\"";
+    }
+
     /// <summary>
     /// Merge any variables from context into plan parameters.
     /// </summary>

webapi/appsettings.json

@@ -48,11 +48,14 @@
   // - Set Planner:Type to "Sequential" to enable the multi-step SequentialPlanner
   //     Note: SequentialPlanner works best with `gpt-4`. See the "Enabling Sequential Planner" section in webapi/README.md for configuration instructions.
   // - Set Planner:Type to "Stepwise" to enable MRKL style planning
+  // - Set Planner:UseStepwiseResultAsBotResponse to "true" to use the result of the planner as the bot response. If false, planner result will be imbedded in meta prompt.
+  //      - This is helpful because the Stepwise Planner returns a sensical chat response as its result.
   // - Set Planner:RelevancyThreshold to a decimal between 0 and 1.0.
   // - Set Planner:Model to a chat completion model (e.g., gpt-35-turbo, gpt-4).
   //
   "Planner": {
     "Type": "Sequential",
+    "UseStepwiseResultAsBotResponse": true,
     // The minimum relevancy score for a function to be considered.
     // Set RelevancyThreshold to a value between 0 and 1 if using the SequentialPlanner or Stepwise planner with gpt-3.5-turbo.
     // Ignored when Planner:Type is "Action"
@@ -166,6 +169,7 @@
     "SystemResponse": "Either return [silence] or provide a response to the last message. ONLY PROVIDE A RESPONSE IF the last message WAS ADDRESSED TO THE 'BOT' OR 'COPILOT'. If it appears the last message was not for you, send [silence] as the bot response.",
     "InitialBotMessage": "Hello, thank you for democratizing AI's productivity benefits with open source! How can I help you today?",
     "ProposedPlanBotMessage": "As an AI language model, my knowledge is based solely on the data that was used to train me, but I can use the following functions to get fresh information: {{$planFunctions}}. Do you agree to proceed?",
+    "StepwisePlannerSupplement": "This result was obtained using the Stepwise Planner, which used a series of thoughts and actions to fulfill the user intent. The planner attempted to use the following functions to gather necessary information: {{$planFunctions}}.",
     "PlanResultsDescription": "This is the result of invoking the functions listed after \"FUNCTIONS USED:\" to retrieve additional information outside of the data you were trained on. This information was retrieved on {{TimeSkill.Now}}. You can use this data to help answer the user's query.",
     "KnowledgeCutoffDate": "Saturday, January 1, 2022",
     "SystemAudience": "Below is a chat history between an intelligent AI bot named Copilot with one or more participants.",