|
1 | 1 | --- |
2 | 2 | layout: default |
3 | | -title: AWS Lambda integration for Data Prepper |
| 3 | +title: aws_lambda |
4 | 4 | parent: Processors |
5 | 5 | grand_parent: Pipelines |
6 | 6 | nav_order: 10 |
7 | 7 | --- |
8 | 8 |
|
9 | | -# AWS Lambda integration for Data Prepper |
| 9 | +# aws_lambda integration for Data Prepper |
10 | 10 |
|
11 | | -The AWS Lambda integration allows developers to use serverless computing capabilities within their Data Prepper pipelines for flexible event processing and data routing. |
| 11 | +The [AWS Lambda](https://aws.amazon.com/lambda/) integration allows developers to use serverless computing capabilities within their Data Prepper pipelines for flexible event processing and data routing. |
12 | 12 |
|
13 | 13 | ## AWS Lambda processor configuration |
14 | 14 |
|
15 | | -The `aws_lambda processor` enables invocation of an AWS Lambda function within your Data Prepper pipeline to process events. It supports both synchronous and asynchronous invocations based on your use case. |
| 15 | +The `aws_lambda` processor enables invocation of an AWS Lambda function within your Data Prepper pipeline in order to process events. It supports both synchronous and asynchronous invocations based on your use case. |
16 | 16 |
|
17 | 17 | ## Configuration fields |
18 | 18 |
|
@@ -61,8 +61,8 @@ The processor supports the following invocation types: |
61 | 61 |
|
62 | 62 | - `request-response`: The processor waits for Lambda function completion before proceeding. |
63 | 63 | - `event`: The function is triggered asynchronously without waiting for a response. |
64 | | -- `Batching`: When enabled, events are aggregated and sent in bulk to optimize Lambda invocations. Batch thresholds control the event count, size limit, and timeout. |
65 | | -- `Codec`: JSON is used for both request and response codecs. Lambda must return JSON array outputs. |
| 64 | +- `batch`: When enabled, events are aggregated and sent in bulk to optimize Lambda invocations. Batch thresholds control the event count, size limit, and timeout. |
| 65 | +- `codec`: JSON is used for both request and response codecs. Lambda must return JSON array outputs. |
66 | 66 | - `tags_on_match_failure`: Custom tags can be applied to events when Lambda processing fails or encounters unexpected issues. |
67 | 67 |
|
68 | 68 | ## Behavior |
@@ -90,71 +90,5 @@ Integration tests for this plugin are executed separately from the main Data Pre |
90 | 90 | ``` |
91 | 91 | ./gradlew :data-prepper-plugins:aws-lambda:integrationTest -Dtests.processor.lambda.region="us-east-1" -Dtests.processor.lambda.functionName="lambda_test_function" -Dtests.processor.lambda.sts_role_arn="arn:aws:iam::123456789012:role/dataprepper-role |
92 | 92 | ``` |
93 | | -{% include copy-curl.html %} |
94 | | - |
95 | | -## AWS Lambda sink |
96 | | - |
97 | | -You can configure the sink using the following configuration options. |
98 | | - |
99 | | -Field | Type | Required | Description |
100 | | ------------------ | ------- | -------- | ---------------------------------------------------------------------------- |
101 | | -`function_name` | String | Required | The name of the AWS Lambda function to invoke. |
102 | | -`invocation_type` | String | Optional | Specifies the invocation type. Default is `event`. |
103 | | -`aws.region` | String | Required | The AWS Region in which the Lambda function is located. |
104 | | -`aws.sts_role_arn`| String | Optional | The ARN of the role to assume before invoking the Lambda function. |
105 | | -`max_retries` | Integer | Optional | The maximum number of retries for failed invocations. Default is `3`. |
106 | | -`batch` | Object | Optional | The batch settings for the Lambda invocations. Default is `key_name = "events"`. Default threshold is `event_count=100`, `maximum_size="5mb"`, and `event_collect_timeout = 10s`. |
107 | | -`lambda_when` | String | Optional | A conditional expression that determines when to invoke the Lambda processor. |
108 | | -`dlq` | Object | Optional | A dead-letter queue (DLQ) configuration for failed invocations. |
109 | | - |
110 | | -#### Example configuration |
111 | | - |
112 | | -``` |
113 | | -sink: |
114 | | - - aws_lambda: |
115 | | - function_name: "my-lambda-sink" |
116 | | - invocation_type: "event" |
117 | | - aws: |
118 | | - region: "us-west-2" |
119 | | - sts_role_arn: "arn:aws:iam::123456789012:role/my-lambda-sink-role" |
120 | | - max_retries: 5 |
121 | | - batch: |
122 | | - key_name: "events" |
123 | | - threshold: |
124 | | - event_count: 50 |
125 | | - maximum_size: "3mb" |
126 | | - event_collect_timeout: PT5S |
127 | | - lambda_when: "event['type'] == 'log'" |
128 | | - dlq: |
129 | | - region: "us-east-1" |
130 | | - sts_role_arn: "arn:aws:iam::123456789012:role/my-sqs-role" |
131 | | - bucket: "<<your-dlq-bucket-name>>" |
132 | | -``` |
133 | | -{% include copy-curl.html %} |
134 | 93 |
|
135 | | -## Usage |
136 | | - |
137 | | -The sink supports the following invocation types: |
138 | | - |
139 | | -- `event`: The function is triggered asynchronously without waiting for a response. |
140 | | -- `request-response`: Not supported for sink operations. |
141 | | -- `Batching`: When enabled, events are aggregated and sent in bulk to optimize Lambda invocations. Default is `enabled`. |
142 | | -- `DLQ`: A setup available for routing and processing events that persistently fail Lambda invocations after multiple retry attempts. |
143 | | - |
144 | | -## Advanced configurations |
145 | | - |
146 | | -The AWS Lambda processor and sink provide the following advanced options for security and performance optimization: |
147 | | - |
148 | | -- AWS Identity and Access Management (IAM) role assumption: The processor and sink support assuming the specified IAM role `aws.sts_role_arn` before Lambda invocation. This enhances secure handling by providing access control to AWS resources. |
149 | | -- Concurrency management: When using the `event` invocation type, consider Lambda concurrency limits to avoid throttling. |
150 | | - |
151 | | -For more information about AWS Lambda integration with Data Prepper, see the [AWS Lambda documentation](https://docs.aws.amazon.com/lambda). |
152 | | - |
153 | | -## Integration testing |
154 | | - |
155 | | -Integration tests for this plugin are executed separately from the main Data Prepper build process. Use the following Gradle command to run these tests: |
156 | | - |
157 | | -``` |
158 | | -./gradlew :data-prepper-plugins:aws-lambda:integrationTest -Dtests.sink.lambda.region="us-east-1" -Dtests.sink.lambda.functionName="lambda_test_function" -Dtests.sink.lambda.sts_role_arn="arn:aws:iam::123456789012:role/dataprepper-role |
159 | | -``` |
160 | 94 | {% include copy-curl.html %} |
0 commit comments