Serverless Middleware

A serverless middleware solution for integrating Enterprise applications like crm, Billing, payment gateway etc.

Overview

This project provides a set of microservices that handle the integration between CRM and Billing systems. Each service is designed to handle specific business operations and is deployed as an AWS Lambda function.

Services

1. AccountService

• Handles account synchronization between CRM and Billing
• Manages account creation, updates, and retrieval
• Uses AWS Secrets Manager for secure configuration

2. OrderService

• Manages order processing and synchronization
• Handles order creation, updates, and status tracking
• Integrates with CRM Orders and Billing Subscriptions

3. ProductsService

• Manages product catalog synchronization
• Handles product pricing and availability
• Maintains product mapping between systems

4. CorrelationService

• Manages correlation IDs between systems
• Tracks relationships between CRM and Billing objects
• Ensures data consistency across systems

5. AccessManager

• Manages authentication tokens for both systems
• Handles token refresh and rotation
• Secures API access credentials

Architecture

• AWS Lambda: Serverless compute service
• AWS Secrets Manager: Secure storage for credentials and configuration
• AWS Parameter Store: Secure storage for non-sensitive configuration
• AWS SQS: Message queue for asynchronous processing
• AWS SAM: Serverless Application Model for deployment

Security

• All sensitive data is stored in AWS Secrets Manager
• Configuration is managed through AWS Parameter Store
• IAM roles with least privilege principle
• KMS encryption for sensitive data
• Secure token management

Secrets Management Best Practices

Ideal Path for Secrets Management

1. Store Credentials

   • Store all credentials in AWS Secrets Manager or Parameter Store
   • Ensure all secrets are encrypted using KMS
   • Use appropriate naming conventions (e.g., /apiTokens/<service>)

2. Token Refresh Strategy

   • Implement a scheduled program (e.g., using AWS EventBridge) to refresh tokens
   • Update tokens in Secrets Manager/Parameter Store after refresh
   • Use appropriate TTL settings based on token expiration policies

3. Lambda Integration

   • Use AWS Parameters and Secrets Lambda Extension for efficient secret retrieval
   • The extension caches secrets for Lambda functions
   • Reduces API calls to Secrets Manager/Parameter Store
   • Improves performance and reduces costs

# Environment variables set by Lambda
aws_session_token = os.environ.get('AWS_SESSION_TOKEN')
port = os.environ.get('PARAMETERS_SECRETS_EXTENSION_HTTP_PORT')
param_name = '/apiTokens/<service>'

def lambda_handler(event, context):
    url = f'http://localhost:{port}/systemsmanager/parameters/get?name={param_name}&withDecryption=true'    
    headers = {'X-Aws-Parameters-Secrets-Token': aws_session_token}
    response = requests.get(url, headers=headers)
    if response.status_code != 200:
        raise Exception(f"Error fetching parameter: {response.status_code} - {response.text}")
    
    return response.json()

Enable Lambda Extension

• Add the extension layer to your Lambda function
• Set appropriate environment variables
• Configure IAM permissions

IAM Permissions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "ssm:GetParameter"
            ],
            "Resource": [
                "arn:aws:secretsmanager:*:*:secret:/apiTokens/*",
                "arn:aws:ssm:*:*:parameter:/apiTokens/*"
            ]
        }
    ]
}

Environment Variables

PARAMETERS_SECRETS_EXTENSION_HTTP_PORT=2773

Benefits

• Reduced latency for secret retrieval
• Lower costs due to caching
• Improved security through centralized management
• Automatic token refresh handling
• Simplified error handling and retry logic

Prerequisites

• AWS CLI installed and configured
• AWS SAM CLI installed
• Python 3.8 or later
• Git

Installation

1. Clone the repository:

git clone https://github.com/PuneetChandel/ServerLessMiddleware.git
cd ServerLessMiddleware

2. Install dependencies for each service:

cd <service_name>
pip install -r requirements.txt

3. Configure AWS credentials:

aws configure

Deployment

Deploy All Services

./deploy.sh

Deploy Specific Service

./deploy.sh <service_name>

Available services:

• AccountService
• OrderService
• ProductsService
• CorrelationService
• AccessManager

Configuration

Each service uses AWS Secrets Manager and Parameter Store for configuration. To set up the configuration:

1. Run the migration script for each service:

cd <service_name>/scripts
python migrate_secrets.py --service <service_name>

2. Update the IAM role for Lambda functions with necessary permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:UpdateSecret",
                "ssm:GetParameter",
                "ssm:PutParameter"
            ],
            "Resource": [
                "arn:aws:secretsmanager:*:*:secret:/<service_name>/*",
                "arn:aws:ssm:*:*:parameter:/<service_name>/*"
            ]
        }
    ]
}

Development

Project Structure

ServerLessMiddleware/
├── AccountService/
├── OrderService/
├── ProductsService/
├── CorrelationService/
├── AccessManager/
├── deploy.sh
└── README.md

Each service follows the same structure:

<service_name>/
├── utils/
│   ├── logger.py
│   └── secrets_manager.py
├── scripts/
│   └── migrate_secrets.py
├── template.yaml
├── requirements.txt
└── lambda_function.py

Monitoring

• CloudWatch Logs for each Lambda function
• CloudWatch Metrics for performance monitoring
• X-Ray for tracing and debugging

Components

CRM System

1. Create Platform Events → OrderEvent__e
2. Create Named credentials → AWS_US_East_1
3. Create a Channel (One channel can have multiple channel members, which means that you can add multiple platform events to a channel)

/services/data/v58.0/tooling/sobjects/PlatformEventChannel
{
  "FullName": "SF_event__chn",
  "Metadata": {
     "channelType": "event",
     "label": "crm event channel"
  }
}

4. Custom Platform Event in a New Channel Member→ Add a member to channel that is the PE
5. A channel (PlatformEventChannel) can have multiple channel members (PlatformEventChannelMember), which means that you can add multiple platform events to a channel. This example adds only one platform event, Carbon_Comparison__e. To add another event to the channel, create another PlatformEventChannelMember.

/services/data/v58.0/tooling/sobjects/PlatformEventChannelMember

{
  "FullName": "SF_event_chn_OrderEvent_e",
  "Metadata": {
     "eventChannel": "SF_event__chn",
     "selectedEntity": "OrderEvent__e"
  }
}

{
  "FullName": "SF_event_chn_Fulfillment_e",
  "Metadata": {
     "eventChannel": "SF_event__chn",
     "selectedEntity": "FulfillmentEvent__e"
  }
}

6. Create an Event Relay → SF_Events_Relay
7. Activate Partner Event Source in Amazon EventBridge
8. Reference

• https://help.crm.com/s/articleView?id=sf.ev_relay_events_section.htm&type=5
• https://help.crm.com/s/articleView?id=sf.ev_relay_create_channel_section.htm&type=5
• https://help.crm.com/s/articleView?id=sf.ev_relay_activate_eventbridge_bus.htm&type=5

8. Start Event relay

AWS Components

1. AWS Event bridge, Event bus & Event Rules - Receive events from CRM and Billing and route them to the SQS and CloudWatch
2. Scheduler - This is for rotating the credential using the AccessManager
3. SQS - Separate Queues for Events that trigger the Lambda functions
4. Lambda Functions
5. DynamoDB
6. Parameter store - Store the API tokens encrypted using KMS
7. KMS - KMS key used as encryption key for DynamoDB and Parameter store
8. CloudWatch