Features/add image llm config

[iceljc]

PR Type

Enhancement, Other

---

Description

• Introduced agent-level LLM configuration for image, audio, and realtime processing

• Migrated image handling functionality from FileHandler to dedicated ImageHandler plugin

• Refactored LLM provider/model settings from state-based to agent configuration-based

• Enhanced code script management with agent service methods and update options

---

Diagram Walkthrough

flowchart LR
  AgentLlmConfig["AgentLlmConfig"] -- "adds" --> ImageGeneration["ImageGeneration config"]
  AgentLlmConfig -- "adds" --> ImageEdit["ImageEdit config"]
  AgentLlmConfig -- "adds" --> AudioTranscription["AudioTranscription config"]
  AgentLlmConfig -- "adds" --> Realtime["Realtime config"]
  FileHandlerPlugin["FileHandlerPlugin"] -- "extracts" --> ImageHandlerPlugin["ImageHandlerPlugin"]
  ImageHandlerPlugin -- "contains" --> ImageFunctions["Image functions"]
  InstructService["InstructService"] -- "adds" --> FileLlmProcessor["IFileLlmProcessor support"]
  AgentService["AgentService"] -- "adds" --> CodeScriptMethods["Code script methods"]

Loading

File Walkthrough

	Relevant files
Enhancement	18 files AgentLlmConfig.cs Add image, audio, and realtime LLM configurations                +45/-0    AgentLlmConfigMongoModel.cs Create MongoDB models for agent LLM configurations              +193/-0  ImageHandlerPlugin.cs Create new ImageHandler plugin with DI registration            +26/-0    GenerateImageFn.cs Migrate image generation function from FileHandler plugin +14/-44  EditImageFn.cs Migrate image editing function from FileHandler plugin      +20/-46  ReadImageFn.cs Migrate image reading function from FileHandler plugin      +13/-30  InstructService.Execute.cs Add file LLM processor support and refactor completions    +78/-20  IFileLlmProcessor.cs Define interface for file LLM processing                                  +9/-0      IAgentService.cs Add code script management methods to interface                    +10/-0    AgentService.CodeScripts.cs Implement agent code script management methods                      +46/-0    LlmConfigBase.cs Rename LlmBase to LlmProviderModel and update properties  +8/-10    FileHandlerSettings.cs Remove image settings from FileHandler configuration          +1/-26    ReadAudioFn.cs Use agent LLM config for audio transcription                          +15/-15  RealtimeHub.cs Apply agent realtime LLM configuration                                      +35/-6    MongoRepository.Agent.cs Update agent LLM config model references                                  +4/-4      ChatCompletionProvider.cs Replace RoleContent with LlmContent property                          +3/-3      RoleDialogModel.cs Rename RoleContent property to LlmContent                                +1/-5      IBotSharpRepository.cs Add update options parameter to code script method              +2/-1
Configuration changes	1 files ImageHandlerSettings.cs Create settings for image handler operations                          +25/-0
Additional files	53 files BotSharp.sln +11/-0    AgentCodeScriptUpdateOptions.cs +8/-0      FileLlmProcessOptions.cs +34/-0    IInstructService.cs +4/-1      FileInstructOptions.cs +6/-0      ProgramCodeTemplateMessage.cs +3/-0      AgentCodeScriptDbUpdateOptions.cs +6/-0      TokenStatistics.cs +7/-7      FileInstructService.SelectFile.cs +4/-4      FileRepository.AgentCodeScript.cs +6/-2      InstructModeController.cs +2/-1      InstructMessageModel.cs +1/-0      ChatCompletionProvider.cs +3/-3      AudioHandlerSettings.cs +1/-1      ChatCompletionProvider.cs +3/-3      PlotChartFn.cs +12/-16  ChatCompletionProvider.cs +3/-3      HandleEmailSenderFn.cs +2/-2      BotSharp.Plugin.FileHandler.csproj +0/-28    UtilityName.cs +0/-3      FileHandlerPlugin.cs +2/-5      ReadPdfFn.cs +9/-25    FileHandlerUtilityHook.cs +0/-39    GeminiChatCompletionProvider.cs +4/-4      PalmChatCompletionProvider.cs +3/-3      RealTimeCompletionProvider.cs +14/-12  BotSharp.Plugin.ImageHandler.csproj +42/-0    ImageHandlerImageConverter.cs +6/-6      UtilityName.cs +8/-0      AiResponseHelper.cs +31/-0    ImageHandlerUtilityHook.cs +53/-0    LlmContextIn.cs +18/-0    LlmContextOut.cs +10/-0    Using.cs +34/-0    util-file-edit_image.json [link]    util-file-generate_image.json [link]    util-file-read_image.json [link]    util-file-edit_image.fn.liquid [link]    util-file-generate_image.fn.liquid [link]    util-file-read_image.fn.liquid [link]    ChatCompletionProvider.cs +2/-2      ChatCompletionProvider.cs +1/-1      ChatCompletionProvider.cs +4/-4      MicrosoftExtensionsAIChatCompletionProvider.cs +3/-3      AgentDocument.cs +1/-1      AgentLlmConfigMongoElement.cs +0/-44    LlmProviderModelMongoModel.cs +38/-0    MongoRepository.AgentCodeScript.cs +3/-2      RealTimeCompletionProvider.cs +4/-4      PyProgrammerFn.cs +12/-14  ChatCompletionProvider.cs +2/-2      WebStarter.csproj +1/-0      appsettings.json +30/-28

---

[qodo-merge-pro]

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
🟢	No security concerns identified No security vulnerabilities detected by AI analysis. Human verification advised for critical code.
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
⚪	No custom compliance provided Follow the guide to enable custom compliance check.
Update

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[qodo-merge-pro]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Fix incorrect state key for model In EditImageFn.cs, correct the state key used for retrieving the LLM model from "image_edit_llm_provider" to "image_edit_llm_model" to fix a bug. src/Plugins/BotSharp.Plugin.ImageHandler/Functions/EditImageFn.cs [155] -var model = state.GetState("image_edit_llm_provider"); +var model = state.GetState("image_edit_llm_model"); [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a copy-paste bug where the wrong state key is used to retrieve the model name, which would cause runtime failures.	High
	Prevent potential null reference exception In ReadImageFn.cs, add a null-conditional operator (?.) when accessing _settings.Reading.ImageDetailLevel inside the if block to prevent a potential NullReferenceException. src/Plugins/BotSharp.Plugin.ImageHandler/Functions/ReadImageFn.cs [168-171] if (string.IsNullOrWhiteSpace(level) && !string.IsNullOrWhiteSpace(_settings.Reading?.ImageDetailLevel)) { - state.SetState(key, _settings.Reading.ImageDetailLevel); + state.SetState(key, _settings.Reading?.ImageDetailLevel); } Apply / Chat Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies a potential NullReferenceException due to accessing _settings.Reading without a null check, even though the condition implies it could be null.	Medium
Learned best practice	Initialize safe LLM defaults Provide a non-null default for LLM provider and model when fromAgent?.LlmConfig is null to avoid downstream null access. Initialize LlmConfig with safe defaults. src/Plugins/BotSharp.Plugin.ImageHandler/Functions/ReadImageFn.cs [43-49] var agent = new Agent { Id = fromAgent?.Id ?? BuiltInAgentId.FileAssistant, Name = fromAgent?.Name ?? "File Assistant", Instruction = fromAgent?.Instruction ?? args?.UserRequest ?? "Please describe the image(s).", - LlmConfig = fromAgent?.LlmConfig ?? new() + LlmConfig = fromAgent?.LlmConfig ?? new AgentLlmConfig + { + Provider = "openai", + Model = "gpt-4o-mini" + } }; Apply / Chat Suggestion importance[1-10]: 6 __ Why: Relevant best practice - Ensure null-safety on optional inputs and object properties, returning safe defaults to avoid NullReferenceExceptions.	Low
Update

[qodo-merge-pro]

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
⚪	Insecure file write Description: Updating agent code scripts writes arbitrary script contents to disk with optional upsert without sanitizing names or paths, risking path traversal or unintended overwrite if untrusted input reaches script.Name. FileRepository.AgentCodeScript.cs [96-103] Referred Code } var file = Path.Combine(dir, script.Name); if (options?.IsUpsert == true || File.Exists(file)) { File.WriteAllText(file, script.Content); } }
	Insecure upsert criteria Description: Bulk upsert of agent code scripts uses client-provided Name and CodePath in filters without validation, which could enable overwriting unintended records if identifiers are not constrained or sanitized. MongoRepository.AgentCodeScript.cs [74-80] Referred Code }), Builders<AgentCodeScriptDocument>.Update.Set(y => y.Content, x.Content) .Set(x => x.UpdatedTime, DateTime.UtcNow) ) { IsUpsert = options?.IsUpsert ?? false }) .ToList(); var result = _dc.AgentCodeScripts.BulkWrite(ops, new BulkWriteOptions { IsOrdered = false });
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
⚪	No custom compliance provided Follow the guide to enable custom compliance check.

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[qodo-merge-pro]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Restore fallback for assistant content Restore the fallback logic for non-user roles in the LlmContent property to use RichContent?.Message?.Text when Content is empty, preventing potential issues with empty assistant messages. src/Infrastructure/BotSharp.Abstraction/Conversations/Models/RoleDialogModel.cs [144-156] public string LlmContent { get { var text = Content; if (Role == AgentRole.User) { text = !string.IsNullOrWhiteSpace(Payload) ? Payload : Content; } + else + { + text = !string.IsNullOrWhiteSpace(Content) ? Content : RichContent?.Message?.Text; + } return text; } } Apply / Chat Suggestion importance[1-10]: 8 __ Why: This suggestion correctly identifies a potential regression where the fallback logic for assistant message content was removed during a refactoring, which could lead to sending empty content to the LLM.	Medium
General	Simplify agent data passing to helper Refactor the GetImageGenerationResponse method to pass only the necessary LlmConfig from the agent object to the helper method, or ensure the helper can handle a null agent gracefully. src/Plugins/BotSharp.Plugin.ImageHandler/Functions/GenerateImageFn.cs [81-84] +private async Task<string> GetImageGenerationResponse(Agent agent, string description) +{ + return await AiResponseHelper.GetImageGenerationResponse(_services, agent, description); +} - Apply / Chat Suggestion importance[1-10]: 7 __ Why: The suggestion is valid as the agent object is passed to a helper that only uses a small part of it. However, the improved_code is identical to the existing_code, making the suggestion a request for verification rather than a direct code change.	Medium
	Use a specific response helper Create and use a dedicated helper method, like AiResponseHelper.GetImageEditResponse, for image editing responses to provide more contextually accurate feedback instead of the generic image generation response. src/Plugins/BotSharp.Plugin.ImageHandler/Functions/EditImageFn.cs [121-124] private async Task<string> GetImageEditResponse(Agent agent, string description) { - return await AiResponseHelper.GetImageGenerationResponse(_services, agent, description); + return await AiResponseHelper.GetImageEditResponse(_services, agent, description); } Apply / Chat Suggestion importance[1-10]: 6 __ Why: The suggestion correctly identifies that a generic response helper is used for a specific action (image editing), which could lead to a suboptimal user experience. Proposing a dedicated helper improves contextual accuracy.	Low
	✅ Fix typos in property names Suggestion Impact: The commit changed LlModel to Model (not to LlmModel as suggested) and also renamed LlmProvider to Provider; it did not show ReasoningEfforLevel being fixed. The change indicates the suggestion influenced renaming away from the typoed LlModel, though with a different final name. code diff: /// llm model /// </summary> - public string? LlModel { get; set; } + public string? Model { get; set; } + In the FileLlmProcessOptions class, correct the typos in property names from LlModel to LlmModel and ReasoningEfforLevel to ReasoningEffortLevel for consistency and clarity. src/Infrastructure/BotSharp.Abstraction/Files/Models/FileLlmProcessOptions.cs [10-18] /// <summary> /// llm model /// </summary> -public string? LlModel { get; set; } +public string? LlmModel { get; set; } /// <summary> /// Reasoning effort level /// </summary> -public string? ReasoningEfforLevel { get; set; } +public string? ReasoningEffortLevel { get; set; } [Suggestion processed] Suggestion importance[1-10]: 4 __ Why: The suggestion correctly points out two typos (LlModel and ReasoningEfforLevel) in a newly added class, and fixing them improves code consistency and maintainability.	Low
More