Skip to content

Commit bf2d664

Browse files
github-actions[bot]vagimelinatebower
committed
Modify lambda processor and sink documentation (#8697)
* Modify lambda processor and sink documentation Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Update _data-prepper/pipelines/configuration/processors/aws-lambda.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _data-prepper/pipelines/configuration/processors/aws-lambda.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update aws-lambda.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _data-prepper/pipelines/configuration/sinks/aws-lambda.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _data-prepper/pipelines/configuration/processors/aws-lambda.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _data-prepper/pipelines/configuration/processors/aws-lambda.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _data-prepper/pipelines/configuration/sinks/aws-lambda.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _data-prepper/pipelines/configuration/sinks/aws-lambda.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _data-prepper/pipelines/configuration/sinks/aws-lambda.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _data-prepper/pipelines/configuration/sinks/aws-lambda.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _data-prepper/pipelines/configuration/sinks/aws-lambda.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _data-prepper/pipelines/configuration/sinks/aws-lambda.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _data-prepper/pipelines/configuration/sinks/aws-lambda.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _data-prepper/pipelines/configuration/sinks/aws-lambda.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> --------- Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com> (cherry picked from commit 4b376b5) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
1 parent 6c1e06b commit bf2d664

File tree

2 files changed

+79
-72
lines changed

2 files changed

+79
-72
lines changed

‎_data-prepper/pipelines/configuration/processors/aws-lambda.md‎

Lines changed: 6 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,18 @@
11
---
22
layout: default
3-
title: AWS Lambda integration for Data Prepper
3+
title: aws_lambda
44
parent: Processors
55
grand_parent: Pipelines
66
nav_order: 10
77
---
88

9-
# AWS Lambda integration for Data Prepper
9+
# aws_lambda integration for Data Prepper
1010

11-
The AWS Lambda integration allows developers to use serverless computing capabilities within their Data Prepper pipelines for flexible event processing and data routing.
11+
The [AWS Lambda](https://aws.amazon.com/lambda/) integration allows developers to use serverless computing capabilities within their Data Prepper pipelines for flexible event processing and data routing.
1212

1313
## AWS Lambda processor configuration
1414

15-
The `aws_lambda processor` enables invocation of an AWS Lambda function within your Data Prepper pipeline to process events. It supports both synchronous and asynchronous invocations based on your use case.
15+
The `aws_lambda` processor enables invocation of an AWS Lambda function within your Data Prepper pipeline in order to process events. It supports both synchronous and asynchronous invocations based on your use case.
1616

1717
## Configuration fields
1818

@@ -61,8 +61,8 @@ The processor supports the following invocation types:
6161

6262
- `request-response`: The processor waits for Lambda function completion before proceeding.
6363
- `event`: The function is triggered asynchronously without waiting for a response.
64-
- `Batching`: When enabled, events are aggregated and sent in bulk to optimize Lambda invocations. Batch thresholds control the event count, size limit, and timeout.
65-
- `Codec`: JSON is used for both request and response codecs. Lambda must return JSON array outputs.
64+
- `batch`: When enabled, events are aggregated and sent in bulk to optimize Lambda invocations. Batch thresholds control the event count, size limit, and timeout.
65+
- `codec`: JSON is used for both request and response codecs. Lambda must return JSON array outputs.
6666
- `tags_on_match_failure`: Custom tags can be applied to events when Lambda processing fails or encounters unexpected issues.
6767

6868
## Behavior
@@ -90,71 +90,5 @@ Integration tests for this plugin are executed separately from the main Data Pre
9090
```
9191
./gradlew :data-prepper-plugins:aws-lambda:integrationTest -Dtests.processor.lambda.region="us-east-1" -Dtests.processor.lambda.functionName="lambda_test_function" -Dtests.processor.lambda.sts_role_arn="arn:aws:iam::123456789012:role/dataprepper-role
9292
```
93-
{% include copy-curl.html %}
94-
95-
## AWS Lambda sink
96-
97-
You can configure the sink using the following configuration options.
98-
99-
Field | Type | Required | Description
100-
----------------- | ------- | -------- | ----------------------------------------------------------------------------
101-
`function_name` | String | Required | The name of the AWS Lambda function to invoke.
102-
`invocation_type` | String | Optional | Specifies the invocation type. Default is `event`.
103-
`aws.region` | String | Required | The AWS Region in which the Lambda function is located.
104-
`aws.sts_role_arn`| String | Optional | The ARN of the role to assume before invoking the Lambda function.
105-
`max_retries` | Integer | Optional | The maximum number of retries for failed invocations. Default is `3`.
106-
`batch` | Object | Optional | The batch settings for the Lambda invocations. Default is `key_name = "events"`. Default threshold is `event_count=100`, `maximum_size="5mb"`, and `event_collect_timeout = 10s`.
107-
`lambda_when` | String | Optional | A conditional expression that determines when to invoke the Lambda processor.
108-
`dlq` | Object | Optional | A dead-letter queue (DLQ) configuration for failed invocations.
109-
110-
#### Example configuration
111-
112-
```
113-
sink:
114-
- aws_lambda:
115-
function_name: "my-lambda-sink"
116-
invocation_type: "event"
117-
aws:
118-
region: "us-west-2"
119-
sts_role_arn: "arn:aws:iam::123456789012:role/my-lambda-sink-role"
120-
max_retries: 5
121-
batch:
122-
key_name: "events"
123-
threshold:
124-
event_count: 50
125-
maximum_size: "3mb"
126-
event_collect_timeout: PT5S
127-
lambda_when: "event['type'] == 'log'"
128-
dlq:
129-
region: "us-east-1"
130-
sts_role_arn: "arn:aws:iam::123456789012:role/my-sqs-role"
131-
bucket: "<<your-dlq-bucket-name>>"
132-
```
133-
{% include copy-curl.html %}
13493

135-
## Usage
136-
137-
The sink supports the following invocation types:
138-
139-
- `event`: The function is triggered asynchronously without waiting for a response.
140-
- `request-response`: Not supported for sink operations.
141-
- `Batching`: When enabled, events are aggregated and sent in bulk to optimize Lambda invocations. Default is `enabled`.
142-
- `DLQ`: A setup available for routing and processing events that persistently fail Lambda invocations after multiple retry attempts.
143-
144-
## Advanced configurations
145-
146-
The AWS Lambda processor and sink provide the following advanced options for security and performance optimization:
147-
148-
- AWS Identity and Access Management (IAM) role assumption: The processor and sink support assuming the specified IAM role `aws.sts_role_arn` before Lambda invocation. This enhances secure handling by providing access control to AWS resources.
149-
- Concurrency management: When using the `event` invocation type, consider Lambda concurrency limits to avoid throttling.
150-
151-
For more information about AWS Lambda integration with Data Prepper, see the [AWS Lambda documentation](https://docs.aws.amazon.com/lambda).
152-
153-
## Integration testing
154-
155-
Integration tests for this plugin are executed separately from the main Data Prepper build process. Use the following Gradle command to run these tests:
156-
157-
```
158-
./gradlew :data-prepper-plugins:aws-lambda:integrationTest -Dtests.sink.lambda.region="us-east-1" -Dtests.sink.lambda.functionName="lambda_test_function" -Dtests.sink.lambda.sts_role_arn="arn:aws:iam::123456789012:role/dataprepper-role
159-
```
16094
{% include copy-curl.html %}

‎_data-prepper/pipelines/configuration/sinks/aws-lambda.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
---
2+
layout: default
3+
title: aws_lambda
4+
parent: Sinks
5+
grand_parent: Pipelines
6+
nav_order: 10
7+
---
8+
9+
----------------------------------------------------------------------------------------
10+
# `aws_lambda` sink for Data Prepper
11+
12+
This page explains how to configure and use [AWS Lambda](https://aws.amazon.com/lambda/) with Data Prepper, enabling Lambda functions to serve as both processors and sinks.
13+
14+
## `aws_lambda` sink
15+
16+
Configure the Lambda sink using the following parameters.
17+
18+
Field | Type | Required | Description
19+
--------------------| ------- | -------- | ----------------------------------------------------------------------------
20+
`function_name` | String | Yes | The name of the AWS Lambda function to invoke.
21+
`invocation_type` | String | No | Specifies the invocation type. Default is `event`.
22+
`aws.region` | String | Yes | The AWS Region in which the Lambda function is located.
23+
`aws.sts_role_arn` | String | No | The Amazon Resource Name (ARN) of the role to assume before invoking the Lambda function.
24+
`max_retries` | Integer | No | The maximum number of retries if the invocation fails. Default is `3`.
25+
`batch` | Object | No | Optional batch settings for Lambda invocations. Default is `key_name = events`. Default threshold is `event_count=100`, `maximum_size="5mb"`, and `event_collect_timeout = 10s`.
26+
`lambda_when` | String | No | A conditional expression that determines when to invoke the Lambda sink.
27+
`dlq` | Object | No | The dead-letter queue (DLQ) configuration for failed invocations.
28+
29+
#### Example configuration
30+
31+
```
32+
sink:
33+
- aws_lambda:
34+
function_name: "my-lambda-sink"
35+
invocation_type: "event"
36+
aws:
37+
region: "us-west-2"
38+
sts_role_arn: "arn:aws:iam::123456789012:role/my-lambda-sink-role"
39+
max_retries: 5
40+
batch:
41+
key_name: "events"
42+
threshold:
43+
event_count: 50
44+
maximum_size: "3mb"
45+
event_collect_timeout: PT5S
46+
lambda_when: "event['type'] == 'log'"
47+
dlq:
48+
region: "us-east-1"
49+
sts_role_arn: "arn:aws:iam::123456789012:role/my-sqs-role"
50+
bucket: "<<your-dlq-bucket-name>>"
51+
```
52+
{% include copy-curl.html %}
53+
54+
## Usage
55+
56+
The invocation types are as follows:
57+
58+
- `event` (Default): Executes functions asynchronously without waiting for responses.
59+
- `request-response` (Sink only): Executes functions synchronously, though responses are not processed.
60+
- `batch`: Automatically groups events based on configured thresholds.
61+
- `dlq`: Supports the DLQ configuration for failed invocations after retry attempts.
62+
63+
Data Prepper components use an AWS Identity and Access Management (IAM) role assumption, `aws.sts_role_arn`, for secure Lambda function invocation and respect Lambda's concurrency limits during event processing. For more information, see the [AWS Lambda documentation](https://docs.aws.amazon.com/lambda).
64+
{: .note}
65+
66+
## Developer guide
67+
68+
Integration tests must be executed separately from the main Data Prepper build. Execute them with the following command:
69+
70+
```
71+
./gradlew :data-prepper-plugins:aws-lambda:integrationTest -Dtests.sink.lambda.region="us-east-1" -Dtests.sink.lambda.functionName="lambda_test_function" -Dtests.sink.lambda.sts_role_arn="arn:aws:iam::123456789012:role/dataprepper-role
72+
```
73+
{% include copy-curl.html %}

0 commit comments

Comments
 (0)