Skip to content

schmitech/user-feedback-simulator

BranchesTags

Repository files navigation

AWS Customer Feedback Analysis System

Project Overview

This project implements a real-time customer feedback analysis system using AWS services. It helps businesses automatically collect, analyze, and visualize customer feedback data, enabling faster response times and data-driven decision making. The system is particularly valuable for organizations dealing with high volumes of customer feedback that require immediate insights.

Business Value

  • Real-time Analysis: Automatically process customer feedback as it arrives, eliminating manual analysis bottlenecks
  • Consistent Evaluation: Apply standardized sentiment analysis across all feedback
  • Quick Issue Detection: Identify and escalate critical customer concerns immediately
  • Data-Driven Insights: Generate actionable business intelligence through automated dashboard visualizations
  • Scalable Solution: Handle varying volumes of feedback with AWS's elastic infrastructure

Architecture

flowchart TB
    subgraph Client["Client Layer"]
        WebApp["Web Application
        or Mobile App"]
    end

    subgraph API["API Layer"]
        APIGW["API Gateway
        /feedback endpoint"]
    end

    subgraph Processing["Processing Layer"]
        Lambda["AWS Lambda
        Process Feedback Function"]
        subgraph Analysis["Analysis Services"]
            Comprehend["Amazon Comprehend
            Sentiment Analysis"]
        end
    end

    subgraph Storage["Storage Layer"]
        DDB["DynamoDB
        Feedback Table"]
    end

    subgraph Analytics["Analytics Layer"]
        subgraph Dashboards["Dashboard Views"]
            Overview["Overview Dashboard"]
            DetailedAnalysis["Detailed Analysis"]
        end
    end

    Client -->|"POST /feedback"| APIGW
    APIGW -->|Trigger| Lambda
    Lambda -->|"Analyze Text"| Comprehend
    Comprehend -->|"Return Sentiment"| Lambda
    Lambda -->|"Store Feedback"| DDB
    DDB -->|"Real-time\nData"| Elasticsearch
    Elasticsearch --> Overview
    Elasticsearch --> DetailedAnalysis

    classDef aws fill:#FF9900,stroke:#232F3E,stroke-width:2px,color:black;
    classDef dashboard fill:#00A1C9,stroke:#232F3E,stroke-width:2px,color:white;
    classDef client fill:#3B48CC,stroke:#232F3E,stroke-width:2px,color:white;

    class APIGW,Lambda,Comprehend,DDB,Elasticsearch aws;
    class Overview,DetailedAnalysis dashboard;
    class WebApp client;
Loading

AWS Services Used

  • Amazon API Gateway: Handles feedback submission and review retrieval through RESTful APIs
  • AWS Lambda: Processes incoming feedback and orchestrates the analysis workflow
  • Amazon DynamoDB: Provides fast, scalable storage for structured feedback data
  • Amazon Comprehend: Performs natural language processing for sentiment analysis and key phrase extraction
  • Elastic Stack: Creates interactive dashboards for data visualization
  • AWS CloudFormation: Manages infrastructure as code for consistent deployments

Prerequisites

Before starting, ensure you have:

  • An active AWS Account with appropriate permissions
  • Node.js and npm installed (for CDK development and Real-Time Reviews Web App)
  • AWS CDK CLI installed
  • Python 3.9 or higher
  • Elastic Stack deployment (Elasticsearch and Kibana)
  • AWS SSO configured (for deployment)

Deployment Instructions

1. Deploy AWS Infrastructure

Navigate to the CDK project directory:

cd /cdk-user-feedback

Configure environment:

cp .env.example .env
# Edit .env to set your Elasticsearch endpoint

Set up AWS SSO:

# Configure SSO profile
aws sso configure --profile my-sso-profile

# Verify configuration
aws configure list-profiles
cat ~/.aws/config
aws sts get-caller-identity --profile your-aws-sso-profile

# Log in to AWS
aws sso login --profile your-aws-sso-profile

Deploy the stack:

# Bootstrap CDK
./bootstrap.sh your-aws-sso-profile dev

# Deploy
./deploy.sh your-aws-sso-profile dev

2. Load Sample Data

Navigate to the data loader directory:

cd /user-feedback-simulator/data-loader

# Configure environment
cp .env.example .env
# Edit .env to set AWS profile, region, and DynamoDB table name if needed

# Set up Python environment
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Load sample data
python load_reviews.py

3. Create ES Index with Mappings

curl -X PUT "es-endpoint:port/sentiment" \
-H "Content-Type: application/json" \
-d '{
  "mappings": {
    "properties": {
      "reviewId": { "type": "keyword" },
      "timestamp": { "type": "long" },
      "reviewDateTime": { "type": "date" },
      "review": { "type": "text" },
      "title": { "type": "text" },
      "rating": { "type": "integer" },
      "recommended": { "type": "boolean" },
      "age": { "type": "integer" },
      "department": { "type": "keyword" },
      "division": { "type": "keyword" },
      "class": { "type": "keyword" },
      "clothingId": { "type": "keyword" },
      "randomBucket": { "type": "integer" },
      "sentiment": { "type": "keyword" },
      "sentimentScore": {
        "properties": {
          "Positive": { "type": "double" },
          "Negative": { "type": "double" },
          "Neutral": { "type": "double" },
          "Mixed": { "type": "double" }
        }
      },
      "keyPhrases": { "type": "keyword" },
      "processingTimestamp": { "type": "date" },
      "source": { "type": "keyword" }
    }
  }
}'

4. Run the Feedback Simulator

cd /user-feedback-simulator
cp .env.example .env
# Edit .env to set your API GW Endpoints
npm install
npm run dev

Access the simulator at http://localhost:5173/

4. Verify Data Flow

  1. Check DynamoDB Records:

    • Go to AWS Console > DynamoDB
    • Find the 'FeedbackTable' table
    • Select "Explore table items"
    • Verify that review records are present

  2. Verify Elasticsearch Integration:

    • Access your Elasticsearch cluster endpoint
    • Use Kibana Dev Tools or curl to check the 'sentiment' index: 
      # Replace with your Elasticsearch endpoint and index name
curl -X GET "https://es-endpoint/sentiment/_count"
    • Verify records are being indexed

  3. Set Up Kibana Dashboards:

    • Log in to your Kibana instance
    • Navigate to Stack Management > Index Patterns
    • Create an index pattern for your feedback data
    • Go to Dashboard section to create visualizations
    • Suggested visualizations:
      • Sentiment trends over time
      • Feedback volume by category
      • Top keywords from reviews
      • Average ratings distribution

Sample Data

This project uses the Women's E-Commerce Clothing Reviews dataset from Kaggle for demonstration purposes. You can find the dataset at: https://www.kaggle.com/datasets/nicapotato/womens-ecommerce-clothing-reviews

Troubleshooting

  • Verify AWS SSO login status if deployment fails
  • Check AWS Console for service deployment status and CloudWatch logs
  • Ensure all environment variables are properly set in .env files
  • Confirm Elasticsearch endpoint is accessible from your VPC

Elasticsearch Commands Reference

Get Document Count

GET /sentiment/_count

Get Statistics

GET /sentiment/_stats

Delete Index

DELETE /sentiment

Create Index with Mappings

PUT /sentiment
{
  "mappings": {
    "properties": {
      "reviewId": { "type": "keyword" },
      "timestamp": { "type": "long" },
      "reviewDateTime": { "type": "date" },
      "review": { "type": "text" },
      "title": { "type": "text" },
      "rating": { "type": "integer" },
      "recommended": { "type": "boolean" },
      "age": { "type": "integer" },
      "department": { "type": "keyword" },
      "division": { "type": "keyword" },
      "class": { "type": "keyword" },
      "clothingId": { "type": "keyword" },
      "randomBucket": { "type": "integer" },
      "sentiment": { "type": "keyword" },
      "sentimentScore": {
        "properties": {
          "Positive": { "type": "double" },
          "Negative": { "type": "double" },
          "Neutral": { "type": "double" },
          "Mixed": { "type": "double" }
        }
      },
      "keyPhrases": { "type": "keyword" },
      "processingTimestamp": { "type": "date" },
      "source": { "type": "keyword" }
    }
  }
}

Queries

Delete Documents Matching a Query

POST /sentiment/_delete_by_query
{
  "query": {
    "match_all": {}
  }
}

Search by Specific Field

GET /sentiment/_search
{
  "query": {
    "term": {
      "reviewId": "1234567-abcd-defgb-abcd-123456789"
    }
  }
}

Match All Documents

GET /sentiment/_search
{
  "query": {
    "match_all": {}
  }
}

Search by Range

GET /sentiment/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "sentimentScore.Negative": {
              "gt": 0.95
            }
          }
        },
        {
          "range": {
            "processingTimestamp": {
              "gte": "now-1h/h"
            }
          }
        }
      ]
    }
  }
}

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright 2024 Schmitech Inc.

About

Application to simulate user reviews in real-time.

Resources

Readme

Code of conduct

Code of conduct
Activity

Stars

7 stars

Watchers

1 watching

Forks

3 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages