AWS Lambda SFTP-to-S3 CSV to JSON Pipeline

Description

This project automates the process of moving CSV files from an SFTP server to AWS S3, then converting those CSV files to JSON using AWS Lambda functions. It’s designed to simplify data ingestion and transformation for downstream applications or analytics.

How It Works

1. Lambda 1 (SFTP to S3)

   • Connects to an SFTP server (such as SFTP To Go).
   • Downloads CSV files.
   • Uploads each CSV file to a designated S3 bucket (raw data).

2. S3 Event Trigger

   • When a new CSV file is uploaded to the raw S3 bucket, an S3 event automatically triggers the next Lambda.

3. Lambda 2 (CSV to JSON)

   • Reads the new CSV file from S3.
   • Converts the CSV data to JSON.
   • Uploads the resulting JSON file to another S3 bucket (processed data).

Project Structure

• lambda1/ – Code for Lambda function 1 (SFTP to S3)
• lambda2/ – Code for Lambda function 2 (CSV to JSON)
• README.md – Project documentation
• .env.example – Example environment variable configuration

Setup

1. Configure environment variables for SFTP credentials and S3 bucket names (see .env.example).
2. Deploy both Lambda functions to AWS.
3. Set up S3 event notifications to trigger Lambda 2 when new files are added.
4. Assign appropriate IAM roles to each Lambda for S3 access.

Use Cases

• Automating data ingestion from external partners/vendors.
• Building data pipelines for reporting or analytics.
• Any workflow requiring CSV-to-JSON transformation in the cloud.

---

Feel free to fork, modify, or extend this project for your own data integration needs!