aws-solutions-library-samples · garystafford-aws · Feb 22, 2024 · Feb 22, 2024 · Feb 22, 2024 · Feb 22, 2024
diff --git a/README.md b/README.md
@@ -21,13 +21,11 @@ performing NLQ. In addition to model choice, NLQ accuracy also relies heavily on
 prompt, prompt template, labeled sample queries used for in-context learning (_aka few-shot prompting_), and the naming
 conventions used for your database schema (tables and columns).
 
-The NLQ Application was tested on a variety of open source and commercial FMs. As a baseline both Anthropic Claude v1.3
-and v2, and OpenAI's GPT-3 and GPT-4 series models were capable of accurately answering most or all sample questions
-included in this README file and the NLQ Application's web UI. Anthropic's Claude-series models,
-including `anthropic.claude-v1` and `anthropic.claude-v2`, as well as OpenAI's Generative Pre-trained Transformer GPT-3
-and GPT-4 series models, including `text-davinci-003` (Legacy), `gpt-3.5-turbo`, and `gpt-4`, all provide accurate
-responses to a wide range of complex natural language queries using an average amount of in-context learning and prompt
-engineering.
+The NLQ Application was tested on a variety of open source and commercial FMs. As a baseline OpenAI's Generative 
+Pre-trained Transformer GPT-3 and GPT-4 series models, including `gpt-3.5-turbo`, and 
+`gpt-4`, all provide accurate responses to a wide range of complex natural language queries using an average amount of 
+in-context learning and prompt engineering. In addition, the Amazon Titan Text G1 - Express, `amazon.titan-text-express-v1`, 
+was able to answer most of the Simple questions with some model specific prompt optimization.
 
 Open source NLQ-capable models, such as `google/flan-t5-xxl` and `google/flan-t5-xxl-fp16` (half-precision
 floating-point format (FP16) version of the full model), are available through Amazon SageMaker JumpStart Foundation
@@ -45,20 +43,20 @@ the `NlqSageMakerEndpointStack.yaml` CloudFormation template file. You will firs
 the `NlqSageMakerEndpointStack.yaml` file and update the deployed CloudFormation stack, `NlqSageMakerEndpointStack`.
 Additionally, you may need to make adjustments to the NLQ Application, the `app_sagemaker.py` file, modifying
 the `ContentHandler` Class to match the response payload of the chosen model. Then, rebuild the Amazon ECR Docker Image,
-incrementing the version, e.g., `nlq-genai-1.0.1-sm`, using the `Dockerfile_SageMaker` Dockerfile and push to the Amazon
+incrementing the version, e.g., `nlq-genai-2.0.1-sm`, using the `Dockerfile_SageMaker` Dockerfile and push to the Amazon
 ECR repository. Lastly, you will need to update the deployed ECS task and service, which are part of
 the `NlqEcsSageMakerStack` CloudFormation stack.
 
 ### Option 2: Switching to Amazon Bedrock
 
 Switching from the solution's default Amazon SageMaker JumpStart Foundation Model to Amazon Bedrock model's, such as
-Anthropic `anthropic.claude-v1` and `anthropic.claude-v2`, will provide superior results. Using Amazon Bedrock
+Amazon Titan Text G1 - Express (`amazon.titan-text-express-v1`), will provide better results than JumpStarts's `google/flan-t5-xxl-fp16`. Using Amazon Bedrock
 eliminates the need for the deployment of the `NlqSageMakerEndpointStack` CloudFormation stack. If the stack has already
 been deployed, it can be deleted. Next, build the Amazon ECR Docker Image using the `Dockerfile_Bedrock` Dockerfile and
-push the resulting image, e.g., `nlq-genai-1.0.0-bedrock`, to the Amazon ECR repository. Finally, deploy
+push the resulting image, e.g., `nlq-genai-2.0.0-bedrock`, to the Amazon ECR repository. Finally, deploy
 the `NlqEcsBedrockStack.yaml` CloudFormation template file. This stack replaces the the `NlqEcsSageMakerStack`
-CloudFormation stack, designed for use JumpStart Foundation Models. The default model used for this option is Anthropic
-Claude Instant v1, `anthropic.claude-instant-v1`.
+CloudFormation stack, designed for use JumpStart Foundation Models. The default Amazon Bedrock model used for this option is 
+Amazon Titan Text G1 - Express (`amazon.titan-text-express-v1`).
 
 ### Option 3: Switching to a Third-party Model Provider's API
 
@@ -67,7 +65,7 @@ such as OpenAI, Cohere, and Anthropic is straightforward. To utilize OpenAI's mo
 OpenAI account and obtain your own personal API key. Using a third-party model via an API eliminates the need for the
 deployment of the `NlqSageMakerEndpointStack` CloudFormation stack. If the stack has already been deployed, it can be
 deleted. Next, build the Amazon ECR Docker Image using the `Dockerfile_OpenAI` Dockerfile and push the resulting image,
-e.g., `nlq-genai-1.0.0-oai`, to the Amazon ECR repository. Finally, deploy the `NlqEcsOpenAIStack.yaml` CloudFormation
+e.g., `nlq-genai-2.0.0-oai`, to the Amazon ECR repository. Finally, deploy the `NlqEcsOpenAIStack.yaml` CloudFormation
 template file. This stack replaces the the `NlqEcsSageMakerStack` CloudFormation stack, designed for use JumpStart
 Foundation Models.
 
@@ -121,8 +119,8 @@ the choice of model. Not all models are capable of NLQ, while others will not re
    one in your account, or the `AWSServiceRoleForECS` Service-Linked Role will not yet exist and the stack will fail.
    Check the `AWSServiceRoleForECS` Service-Linked Role before deploying the `NlqMainStack` stack. This role is
    auto-created the first time you create an ECS cluster in your account.
-4. If you use Option 1: SageMaker JumpStart FM Endpoint, build and push the `nlq-genai:1.0.0-sm` Docker image to the new
-   Amazon ECR repository. Alternately, build and push the `nlq-genai:1.0.0-bedrock` or `nlq-genai:1.0.0-oai` Docker
+4. If you use Option 1: SageMaker JumpStart FM Endpoint, build and push the `nlq-genai:2.0.0-sm` Docker image to the new
+   Amazon ECR repository. Alternately, build and push the `nlq-genai:2.0.0-bedrock` or `nlq-genai:2.0.0-oai` Docker
    image for use with Option 2: Bedrock or Option 3: OpenAI API.
 5. Import the included sample data into the Amazon RDS MoMA database.
 6. Add the `nlqapp` user to the MoMA database.
@@ -134,14 +132,9 @@ the choice of model. Not all models are capable of NLQ, while others will not re
 
 ### Step 2: Create AWS Secret Manager Secrets
 
-Make sure you update the secret values before continuing.
+Make sure you update the secret values below before continuing. This step will create secrets for the credentials for the NLQ application. Optionally, this step will create a secret to store your OpenAI API key. Master User credentials for the RDS instance are set automatically and stored in AWS Secret Manager as part of the `NlqMainStack` CloudFormation template.
 
 ```sh
-aws secretsmanager create-secret \
-    --name /nlq/MasterUsername \
-    --description "Master username for RDS instance." \
-    --secret-string "<your_master_username>"
-
 aws secretsmanager create-secret \
     --name /nlq/NLQAppUsername \
     --description "NLQ Application username for MoMA database." \
@@ -189,23 +182,23 @@ aws ecr get-login-password --region us-east-1 | \
 Option 1: SageMaker JumpStart FM Endpoint
 
 ```sh
-TAG="1.0.0-sm"
+TAG="2.0.0-sm"
 docker build -f Dockerfile_SageMaker -t $ECS_REPOSITORY:$TAG .
 docker push $ECS_REPOSITORY:$TAG
 ```
 
 Option 2: Amazon Bedrock
 
 ```sh
-TAG="1.0.0-bedrock"
+TAG="2.0.0-bedrock"
 docker build -f Dockerfile_Bedrock -t $ECS_REPOSITORY:$TAG .
 docker push $ECS_REPOSITORY:$TAG
 ```
 
 Option 3: OpenAI API
 
 ```sh
-TAG="1.0.0-oai"
+TAG="2.0.0-oai"
 docker build -f Dockerfile_OpenAI -t $ECS_REPOSITORY:$TAG .
 docker push $ECS_REPOSITORY:$TAG
 ```

diff --git a/cloudformation/NlqEcsBedrockStack.yaml b/cloudformation/NlqEcsBedrockStack.yaml
@@ -3,7 +3,7 @@ Description: "SO9250 Guidance for Natural Language Queries of Relational Databas
 Parameters:
   BedrockModelName:
     Type: String
-    Default: "anthropic.claude-instant-v1"
+    Default: "amazon.titan-text-express-v1"
     Description: The Bedrock Model.
 
   # BedrockModelTemperature:
@@ -50,7 +50,7 @@ Parameters:
 
   ECRImageTag:
     Type: String
-    Default: "1.0.0-bedrock"
+    Default: "2.0.0-bedrock"
     Description: The name of the ECR Image tag to use with ECS/Fargate.
 
   TaskName:

diff --git a/cloudformation/NlqEcsOpenAIStack.yaml b/cloudformation/NlqEcsOpenAIStack.yaml
@@ -3,12 +3,12 @@ Description: "SO9250 Guidance for Natural Language Queries of Relational Databas
 Parameters:
   OpenAIModelName:
     Type: String
-    Default: "gpt-3.5-turbo"
+    Default: "gpt-4"
     Description: The OpenAI LLM to call via their API.
 
   ECRImageTag:
     Type: String
-    Default: "1.0.0-oai"
+    Default: "2.0.0-oai"
     Description: The name of the ECR Image tag to use with ECS/Fargate.
 
   TaskName:

diff --git a/cloudformation/NlqEcsSageMakerStack.yaml b/cloudformation/NlqEcsSageMakerStack.yaml
@@ -3,7 +3,7 @@ Description: "SO9250 Guidance for Natural Language Queries of Relational Databas
 Parameters:
   ECRImageTag:
     Type: String
-    Default: "1.0.0-sm"
+    Default: "2.0.0-sm"
     Description: The name of the ECR Image tag to use with ECS/Fargate.
 
   TaskName:

diff --git a/cloudformation/NlqMainStack.yaml b/cloudformation/NlqMainStack.yaml
@@ -30,6 +30,11 @@ Parameters:
     Default: 5432
     Description: The port RDS is listening on.
 
+  RDSEngineVersion:
+    Type: String
+    Default: "16.1"
+    Description: The RDS PostgreSQL engine version.
+
   ALBPort:
     Type: Number
     Default: 80
@@ -226,7 +231,7 @@ Resources:
       BackupRetentionPeriod: 7
       PreferredMaintenanceWindow: "fri:04:30-fri:05:00"
       MultiAZ: true
-      EngineVersion: "15.3"
+      EngineVersion: !Ref RDSEngineVersion
       AutoMinorVersionUpgrade: true
       LicenseModel: "postgresql-license"
       PubliclyAccessible: false

diff --git a/docker/Dockerfile_Bedrock b/docker/Dockerfile_Bedrock
@@ -1,11 +1,11 @@
 # Natural Language Query (NLQ) demo using Amazon RDS for PostgreSQL and Amazon Bedrock.
 # Author: Gary A. Stafford (garystaf@amazon.com)
-# Date: 2023-10-10
+# Date: 2024-01-21
 
-FROM python:3.11.6-slim
+FROM python:3.12.2-slim
 
 LABEL name="nlq-genai" \
-    version="1.0.0-bedrock" \
+    version="2.0.0-bedrock" \
     maintainer="Gary A. Stafford (garystaf@amazon.com)"
 
 ENV PIP_DEFAULT_TIMEOUT=100 \

diff --git a/docker/Dockerfile_OpenAI b/docker/Dockerfile_OpenAI
@@ -1,11 +1,11 @@
 # Natural Language Query (NLQ) demo using Amazon RDS for PostgreSQL and OpenAI's LLM models via their API.
 # Author: Gary A. Stafford (garystaf@amazon.com)
-# Date: 2023-10-10
+# Date: 2024-01-21
 
-FROM python:3.11.6-slim
+FROM python:3.12.2-slim
 
 LABEL name="nlq-genai" \
-    version="1.0.0-oai" \
+    version="2.0.0-oai" \
     maintainer="Gary A. Stafford (garystaf@amazon.com)"
 
 ENV PIP_DEFAULT_TIMEOUT=100 \

diff --git a/docker/Dockerfile_SageMaker b/docker/Dockerfile_SageMaker
@@ -1,11 +1,11 @@
 # Natural Language Query (NLQ) demo using Amazon RDS for PostgreSQL and Amazon SageMaker JumpStart Foundation Models.
 # Author: Gary A. Stafford (garystaf@amazon.com)
-# Date: 2023-10-10
+# Date: 2024-01-21
 
-FROM python:3.11.6-slim
+FROM python:3.12.2-slim
 
 LABEL name="nlq-genai" \
-    version="1.0.0-sm" \
+    version="2.0.0-sm" \
     maintainer="Gary A. Stafford (garystaf@amazon.com)"
 
 ENV PIP_DEFAULT_TIMEOUT=100 \

diff --git a/docker/app_bedrock.py b/docker/app_bedrock.py
@@ -1,37 +1,33 @@
 # Natural Language Query (NLQ) demo using Amazon RDS for PostgreSQL and Amazon Bedrock.
 # Author: Gary A. Stafford (garystaf@amazon.com)
-# Date: 2023-08-12
+# Date: 2024-02-21
 # Usage: streamlit run app_bedrock.py --server.runOnSave true
 
 import ast
+import boto3
 import json
 import logging
 import os
-
-import boto3
 import pandas as pd
 import streamlit as st
 import yaml
 from botocore.exceptions import ClientError
-from langchain.sql_database import SQLDatabase
-from langchain.prompts import FewShotPromptTemplate, PromptTemplate
 from langchain.chains.sql_database.prompt import PROMPT_SUFFIX, _postgres_prompt
 from langchain.embeddings.huggingface import HuggingFaceEmbeddings
-from langchain.llms import Bedrock
+from langchain.prompts import FewShotPromptTemplate, PromptTemplate
 from langchain.prompts.example_selector.semantic_similarity import (
     SemanticSimilarityExampleSelector,
 )
-from langchain.vectorstores import Chroma
+from langchain.sql_database import SQLDatabase
+from langchain_community.llms import Bedrock
+from langchain_community.vectorstores import Chroma
 from langchain_experimental.sql import SQLDatabaseChain
 
 # ***** CONFIGURABLE PARAMETERS *****
 REGION_NAME = os.environ.get("REGION_NAME", "us-east-1")
-MODEL_NAME = os.environ.get("MODEL_NAME", "anthropic.claude-instant-v1")
+MODEL_NAME = os.environ.get("MODEL_NAME", "amazon.titan-text-express-v1")
 TEMPERATURE = os.environ.get("TEMPERATURE", 0.3)
-MAX_TOKENS_TO_SAMPLE = os.environ.get("MAX_TOKENS_TO_SAMPLE", 4096)
-TOP_K = os.environ.get("TOP_K", 250)
 TOP_P = os.environ.get("TOP_P", 1)
-STOP_SEQUENCES = os.environ.get("STOP_SEQUENCES", ["\n\nHuman"])
 BASE_AVATAR_URL = (
     "https://raw.githubusercontent.com/garystafford-aws/static-assets/main/static"
 )
@@ -40,7 +36,6 @@
 HUGGING_FACE_EMBEDDINGS_MODEL = os.environ.get(
     "HUGGING_FACE_EMBEDDINGS_MODEL", "sentence-transformers/all-MiniLM-L6-v2"
 )
-# ******************************************************************
 
 
 def main():
@@ -66,11 +61,8 @@ def main():
     NO_ANSWER_MSG = "Sorry, I was unable to answer your question."
 
     parameters = {
-        "max_tokens_to_sample": MAX_TOKENS_TO_SAMPLE,
         "temperature": TEMPERATURE,
-        "top_k": TOP_K,
-        "top_p": TOP_P,
-        "stop_sequences": STOP_SEQUENCES,
+        "topP": TOP_P,
     }
 
     llm = Bedrock(
@@ -101,10 +93,13 @@ def main():
         st.session_state["past"] = []
 
     if "query" not in st.session_state:
-        st.session_state["query"] = []
+        st.session_state["query"] = ""
 
     if "query_text" not in st.session_state:
-        st.session_state["query_text"] = []
+        st.session_state["query_text"] = ""
+
+    if "query_error" not in st.session_state:
+        st.session_state["query_error"] = ""
 
     tab1, tab2, tab3 = st.tabs(["Chatbot", "Details", "Technologies"])
 
@@ -124,10 +119,10 @@ def main():
                         - Simple
                             - How many artists are there in the collection?
                             - How many pieces of artwork are there?
-                            - How many artists are there whose nationality is Italian?
-                            - How many artworks are by the artist Claude Monet?
+                            - How many artists are there whose nationality is 'Italian'?
+                            - How many artworks are by the artist 'Claude Monet'?
                             - How many artworks are classified as paintings?
-                            - How many artworks were created by Spanish artists?
+                            - How many artworks were created by 'Spanish' artists?
                             - How many artist names start with the letter 'M'?
                         - Moderate
                             - How many artists are deceased as a percentage of all artists?
@@ -170,33 +165,34 @@ def main():
                         except Exception as exc:
                             st.session_state.generated.append(NO_ANSWER_MSG)
                             logging.error(exc)
+                            st.session_state["query_error"] = exc
 
                 # https://discuss.streamlit.io/t/streamlit-chat-avatars-not-working-on-cloud/46713/2
                 if st.session_state["generated"]:
                     with col1:
                         for i in range(len(st.session_state["generated"]) - 1, -1, -1):
                             if (i >= 0) and (
-                                st.session_state["generated"][i] != NO_ANSWER_MSG
+                                    st.session_state["generated"][i] != NO_ANSWER_MSG
                             ):
                                 with st.chat_message(
-                                    "assistant",
-                                    avatar=f"{BASE_AVATAR_URL}/{ASSISTANT_ICON}",
+                                        "assistant",
+                                        avatar=f"{BASE_AVATAR_URL}/{ASSISTANT_ICON}",
                                 ):
                                     st.write(st.session_state["generated"][i]["result"])
                                 with st.chat_message(
-                                    "user",
-                                    avatar=f"{BASE_AVATAR_URL}/{USER_ICON}",
+                                        "user",
+                                        avatar=f"{BASE_AVATAR_URL}/{USER_ICON}",
                                 ):
                                     st.write(st.session_state["past"][i])
                             else:
                                 with st.chat_message(
-                                    "assistant",
-                                    avatar=f"{BASE_AVATAR_URL}/{ASSISTANT_ICON}",
+                                        "assistant",
+                                        avatar=f"{BASE_AVATAR_URL}/{ASSISTANT_ICON}",
                                 ):
                                     st.write(NO_ANSWER_MSG)
                                 with st.chat_message(
-                                    "user",
-                                    avatar=f"{BASE_AVATAR_URL}/{USER_ICON}",
+                                        "user",
+                                        avatar=f"{BASE_AVATAR_URL}/{USER_ICON}",
                                 ):
                                     st.write(st.session_state["past"][i])
         with col2:
@@ -210,7 +206,7 @@ def main():
 
             position = len(st.session_state["generated"]) - 1
             if (position >= 0) and (
-                st.session_state["generated"][position] != NO_ANSWER_MSG
+                    st.session_state["generated"][position] != NO_ANSWER_MSG
             ):
                 st.markdown("Question:")
                 st.code(
@@ -242,6 +238,10 @@ def main():
                     st.markdown("Pandas DataFrame:")
                     df = pd.DataFrame(data)
                     df
+            st.markdown("Query Error:")
+            st.code(
+                st.session_state["query_error"], language="text"
+            )
     with tab3:
         with st.container():
             st.markdown("### Technologies")
@@ -371,7 +371,7 @@ def load_few_shot_chain(llm, db, examples):
     few_shot_prompt = FewShotPromptTemplate(
         example_selector=example_selector,
         example_prompt=example_prompt,
-        prefix=_postgres_prompt + "Here are some examples:",
+        prefix=_postgres_prompt + " Here are some examples:",
         suffix=PROMPT_SUFFIX,
         input_variables=["table_info", "input", "top_k"],
     )
@@ -389,6 +389,7 @@ def load_few_shot_chain(llm, db, examples):
 def clear_text():
     st.session_state["query"] = st.session_state["query_text"]
     st.session_state["query_text"] = ""
+    st.session_state["query_error"] = ""
 
 
 def clear_session():