Batch Streamer #20
Description
Description
Create an example of the "Batch Streamer" serverless microservices pattern.
The "Batch Streamer" pattern represents the scenario where a file containing one to many records, i.e. the batch, requires ingestion and processing.
First, the file is read and a record is created to track the overall status of the file processing, e.g. "RECEIVED", "VALIDATED", "PROCESSING", "COMPLETE", "INVALID", "ERROR". This record tracks the total number of records to be processed, the remaining records to be processed, and the count of records which have been successfully processed. These counts facilitate the "Fan In" pattern if needed. Any other metadata attributes such as the file name, uploader information, receipt timestamp, processing timestamp(s), etc. may be persisted on the tracking record.
The file may optionally be validated holistically upon receipt to fail fast and provide quick feedback before proceeding with individual record processing.
The file is read with each individual record sent to a SQS queue to be processed individually and in parallel using the "Fan Out" pattern.
As each record is processed, the tracking record updates the counter(s) when a record is completes processing successfully. When the count of remaining records to be processed reaches 0, an event is published via SNS to indicate that the file processing has completed successfully. Listeners receiving this event may initiate the "Fan In" process if one exists.