This repo contains examples of the patterns described in the senction of II Patterns of Application Development Using AI book.
- Chain of Thought
- Mode Switch
- Prompt Chaining
- Prompt Object
- Prompt Rewriter
- Prompt Template
- Query Analyzer
- Query Rewriter
- Response Fencing
- Role Assignment
- Structured IO
- Ventriloquist
HITL patterns are broader and more descriptive than code-related patterns. Here is a summary:
As its name implies, this pattern is used when you want to escalate a task or conversation to a human in the loop.
To implement this, you will need to:
- Define escalation criteria => The set of rules that will trigger the escalation.
- Implement escalation triggers => Monitoring and trigger functions, notifications, etc.
- Notification and task assignment.
- Human review and decision.
- Feedback and model improvements => Use the feedback to improve the model.
The feedback loop pattern is best described as a conversation between the LLM and the user. It could take the form of a chat, a survey, or a conversation between a writer and an editor. In all its forms, the feedback loop is a way to improve the model output by providing new feedback and adjustments.
Areas where this pattern can be used:
- Content creation
- Summarization
- Question-answering conversations, like chatbots or teaching assistants.
This pattern describes an AI system where the AI provides information-feedback to the user without the user having to ask for it. This pattern is useful when the AI has information that the user may not know they need.
Examples:
- Desktop assistant that provides reminders and notifications.
- Monitoring system that alerts the user to potential issues.
- AI that monitors financial transactions, assessing risk and sending alerts.
This pattern uses user feedback to improve the model, similar to the "ok" button you see in some chat applications when they give you a response.
In this pattern, the feedback is structured and stored for later use, which could be:
- Improving the model during training.
- Improving prompts and adjusting inference parameters.
- Storing in a RAG system for future reference.
Where this pattern can be used:
- Recommendation systems.
- Chatbots and virtual assistants.
- Predictive maintenance systems.
Note: One possibility is to store the feedback in a RAG system and use it to improve prompts.
This chapter is also somewhat theoretical, in the same way that HITL discussion is more architectural than pattern-based.
Here, the author describes how a system could use AI to gather the most relevant information when an error occurs. By saving all this contextual data and utilizing a Retrieval-Augmented Generation (RAG) system to access historical data, logs, or documentation, the AI could analyze the information to identify root causes or potential fixes. This approach could provide developers with a strong starting point during debugging.
Idea => With this, we could, for example, capture context about files, variable values, stack traces, and more, store that in a background job, enhance it with logs, and use AI to generate an analysis. This analysis could then create a Jira bug ticket with a substantial amount of information and a hypothesis to begin with.
This involves using AI in code-system analysis to try to prevent errors and also monitoring for anomalies or deviations from the standards.
Eval is essentially the same approach that RAGAS uses to monitor and evaluate the quality of the information provided by the RAG system. However, it is applied to the results of our system by checking reference-free attributes such as coherence, relevance, fluency, factual consistency, and safety.
The Guardrail pattern is similar to Eval. The logic is basically the same but uses evaluation to make decisions about how appropriate or relevant a response is. Here, it is possible to use a different model that is faster and less prone to being creative.
- Maybe a generalization to parse responses is needed.