Skip to content

Add example of partial batch failure handling in Destination Function [FUNK] #6922

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: develop
Choose a base branch
from
87 changes: 74 additions & 13 deletions src/connections/functions/destination-functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,65 @@ The [Public API](/docs/api/public-api) Functions/Preview endpoint also supports

### Handling batching errors

Standard [function error types](/docs/connections/functions/destination-functions/#destination-functions-error-types) apply to batch handlers. Segment attempts to retry the batch in the case of Timeout or Retry errors. For all other error types, Segment discards the batch. It's also possible to report a partial failure by returning status of each event in the batch. Segment retries only the failed events in a batch until those events are successful or until they result in a permanent error.
Standard [function error types](/docs/connections/functions/destination-functions/#destination-functions-error-types) apply to batch handlers. Segment handles errors differently depending on whether they affect the entire batch or individual events:

#### Handling partial batch failures

If your destination can process some events while rejecting others, you can return a structured response indicating the status of each event. Segment retries only the failed events.

The following example shows how to process each event separately and return a structured response:

```js
async function onBatch(events, settings) {
const responseArray = [];
for (let i = 0; i < events.length; i++) {
try {
await processEvent(events[i], settings);
responseArray.push({
status: 200 // success
});
} catch (error) {
let response;
switch (error.name) {
case 'ValidationError':
response = {
status: 400, // Bad request - non-retryable
errormessage: error.message
};
break;
case 'RetryError':
response = {
status: 500, // Retry error
errormessage: error.message
};
break;
default:
response = {
status: 500, // Internal server error
errormessage: 'An unexpected error occurred'
};
break;
}

responseArray.push(response);
}
}
console.log(responseArray);
return responseArray;
}

async function processEvent(event, settings) {
// business logic
if (event.properties.shouldFail) {
throw new ValidationError('Validation Error: invalid field');
}
if (event.properties.retry) {
throw new RetryError('Retry Error');
}
}
```

Here's an example response from the `onBatch` handler:

```json
[
Expand All @@ -287,36 +345,39 @@ Standard [function error types](/docs/connections/functions/destination-function
},
{
"status": 400,
"errormessage": "Bad Request"
"errormessage": "Validation Error: invalid field"
},
{
"status": 200
},
{
"status": 500,
"errormessage": "Error processing request"
"errormessage": "Retry Error"
},
{
"status": 500,
"errormessage": "Error processing request"
"errormessage": "An unexpected error occurred"
},
{
"status": 200
},
]
```

For example, after receiving the responses above from the `onBatch` handler, Segment only retries **event_4** and **event_5**.
After receiving these responses from `onBatch`, Segment only retries **event_4** and **event_5**.

The following table shows how Segment processes different event statuses. Segment retries events with a retryable error and discards others or marks them as successfully processed:

| Event Status | Result |
| ----------------------- | ------- |
| `200` (Success) | Success |
| `400` (ValidationError) | Discard |
| `200` (Success) | Success |
| `500` (`RetryError`) | Retry |
| `500` (Internal Error) | Retry |

| Error Type | Result |
| ---------------------- | ------- |
| Bad Request | Discard |
| Invalid Settings | Discard |
| Message Rejected | Discard |
| RetryError | Retry |
| Timeout | Retry |
| Unsupported Event Type | Discard |

This approach lets you handle errors at the event level while still benefiting from batching efficiency.

## Save and deploy the function

Expand Down
2 changes: 1 addition & 1 deletion src/unify/data-graph/data-graph.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[---
---
title: Data Graph
plan: unify
beta: true
Expand Down