Based on: https://github.com/Patotricks15/amazon-bedrock-workshop
- The InvokeModel API is used to send requests to foundation models in Amazon Bedrock.
- Boto3 is used to interact with Amazon Bedrock and send API requests.
- Boto3 provides direct control over request parameters, whereas LangChain simplifies API calls by wrapping Boto3 operations.
- It sets the maximum number of tokens in the generated response.
- It defines where the model should stop generating text, preventing unwanted continuations.
-
It controls the creativity of the response:
-
0 = Minimal variability, useful for reproducibility.
-
1 = High variability, useful for creative tasks.
-
- It limits the selection of tokens based on probability, focusing on the most likely words when set to a lower value.
- The prompt should start with a clear instruction (e.g., “Summarize the following text”) and the text should be enclosed within tags.
- If the text exceeds the model’s token limit, it may get truncated, leading to incomplete summaries.
-
Applications include:
-
Summarizing technical documents and research papers.
-
Generating reports and study notes.
-
Extracting knowledge from large text sources.
-
- Below:
-
Low temperature & high topP = More structured and predictable responses (good for summarization).
-
High temperature & lower topP = More creative and diverse text (good for content generation).
-
- Boto3 allows users to send API requests to Amazon Bedrock to invoke foundation models for various tasks like text generation and summarization.
- modelId – Specifies the foundation model to use.
- accept – Defines the expected response format.
- contentType – Specifies the request body format.
- It specifies which foundation model should be used for processing the request.
- "application/json", which indicates that the response should be returned in JSON format.
- It defines the format of the request body, typically set to "application/json".
- The model analyzes the input prompt and generates a response based on its pretrained knowledge and configuration parameters.
- The foundation model is used to summarize a given text.
A1: It is a serverless option in Amazon OpenSearch Service that allows developers to run petabyte-scale workloads without managing OpenSearch clusters.
- Automatic scaling based on workload demand.
- Interactive millisecond response times.
- Pay-per-use pricing model.
- No need to manually configure or manage clusters.
- Initialize OpenSearch Serverless configuration (collection ARN, index name, vector field, text field, metadata field).
- Define chunking strategy to split documents based on chunk size.
- Configure S3 data source for document storage.
- Set up Titan embeddings model ARN to generate vector embeddings.
- Chunking splits documents into smaller pieces of a defined size, making it easier to process and store embeddings for efficient retrieval.
- It is used to convert text chunks into embeddings, which are then stored in the OpenSearch Serverless index for efficient retrieval.
- KB fetches documents from the data source.
- It extracts text and chunks it based on the chunking strategy.
- The Titan model creates embeddings for each chunk.
- The embeddings are stored in the vector database (OSS).
- It automatically scales, efficiently processes large-scale text data, and optimizes searches for fast retrieval.
- Converts user queries into embeddings.
- Searches the knowledge base for relevant context.
- Augments the foundation model prompt with retrieved information.
- Generates a contextual response based on the results.
- The generated response.
- Source attribution of retrieved data.
- Retrieved text chunks for reference.
- Retrieve API: Only fetches relevant text chunks and metadata for custom workflows.
- RetrieveAndGenerate API: Enhances queries, searches the KB, and returns an AI-generated response with contextual augmentation.
- Retrieved text chunks.
- Location type and URI of the source data.
- Relevance scores for search results.
- It converts queries into embeddings, searches the knowledge base, augments the foundation model prompt with retrieved context, and generates a response using a foundation model.
- It retrieves relevant knowledge from a database and uses it as context to enhance the foundation model's response.
- It manages short-term memory of the conversation to provide more contextual and coherent responses across multiple interactions.
- It allows the system to remember past exchanges, making responses more relevant and context-aware over time.
- Generated response from the foundation model.
- Source attribution (where the retrieved data came from).
- Retrieved text chunks used for context in the response.
- It provides transparency by showing where the model retrieved the supporting information from.
- It feeds the retrieved context into a prompt and LLM by "stuffing" all retrieved content into the prompt without summarization or processing.
- It adds a retrieval step, propagates the retrieved context through the chain, and outputs input, context, and answer together.
- epochs: Number of training iterations (1-10, default: 5).
- batchSize: Number of samples processed per update (1-64, default: 1).
- learningRate: Step size for updating model parameters (0.0-1.0, default: 1.00E-5).
- learningRateWarmupSteps: Steps over which learning rate increases (0-250, default: 5).
- It helps adapt the model to specific use cases by improving response accuracy and relevance based on training data.
- BERTScore, which measures similarity between a generated summary and reference text using BERT embeddings.
- It compares the fine-tuned model’s output with a base model to check if fine-tuning improved text quality.
- It captures semantic meaning similarity rather than relying on exact word matching, making it more robust.
A1: Inline Agents are dynamically configurable AI assistants that can be modified in real time to adjust their behavior, capabilities, and knowledge base.
A2: Unlike static agents, Inline Agents can be modified at runtime, allowing real-time adjustments to instructions, integrations, and responses.
- Dynamic configuration: Modify agent behavior on the fly.
- Flexible integration: Connect to external APIs and services.
- Contextual adaptation: Adjust responses based on user roles and scenarios.
- Rapid prototyping: Test different configurations without redeploying.
- Personalization: Customize responses and behavior in real time.
- Scalability: Manage a single agent across multiple functions.
- Cost-effectiveness: Optimize resource usage dynamically.
- They use only the necessary tools and knowledge per interaction, reducing computational costs.
- By using the InvokeInlineAgent API via the boto3 bedrock-agent-runtime client.
- Processes agent trace requests for better readability.
- Displays the code used by the interpreter.
- Shows function invocations and knowledge base content.
- Invocation time (how long the request took).
- Input and output tokens used during the interaction.
- It is a binary metric (0 or 1) that evaluates how well an LLM identifies and achieves user goals.
- Using an evaluator LLM (e.g., Amazon Nova Pro) to assess whether the AI successfully accomplished the intended goal.
- AgentGoalAccuracyWithoutReference – No predefined reference; evaluates based on inferred success from conversation context.
- AgentGoalAccuracyWithReference – Compares AI’s response against a predefined reference outcome.
- When explicit reference outcomes are unavailable, and the success criteria can be inferred from the conversation.
####What additional input does AgentGoalAccuracyWithReference require?
- user_input (the query or task).
- reference outcome (the expected ideal result).
- The MultiTurnSample object in the ragas ecosystem.
- Andrew’s conversations → High accuracy (1.0) due to focused, goal-oriented approach.
- Jane’s conversations → Low accuracy (0.0) due to lack of historical information. The system fails to provide suggestions in a single conversation turn.
- A human-in-the-loop approach asking for her interests could help provide better suggestions.
- By adjusting either the conversation flow or the reference outcome, allowing greater flexibility in evaluation.
- A metric that evaluates how accurately the LLM identifies and invokes necessary tools to complete a task.
- user_input (task or query).
- reference_tool_calls (expected tool invocations).
- By comparing actual tool calls made by the AI against the annotated reference tool calls.
- Because it uses direct comparison between reference and actual tool calls instead of semantic evaluation.
- Using the MultiTurnSample object in ragas to handle multi-turn conversations.
- The NonLLMStringSimilarity metric, which leverages string distance measures like:
- Levenshtein
- Hamming
- Jaro
- Because it compares tool arguments effectively without relying on an LLM for semantic understanding.