Skip to content

Commit

Permalink
Merge pull request n8n-io#1660 from n8n-io/NODE-798-split-loop
Browse files Browse the repository at this point in the history
split in batches -> loop over items
  • Loading branch information
Deborah authored Oct 16, 2023
2 parents 3ac6bad + a5cf763 commit 61494db
Show file tree
Hide file tree
Showing 13 changed files with 33 additions and 35 deletions.
Binary file modified docs/_images/flow-logic/looping/example_workflow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions docs/code/builtin/current-node-input.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Methods for working with the input of the current node. Some methods and variabl
| `$input.last()` | Last input item in current node. | :white_check_mark: |
| `$input.params` | Object containing the query settings of the previous node. This includes data such as the operation it ran, result limits, and so on. | :white_check_mark: |
| `$json` | Shorthand for `$input.item.json`. Incoming JSON data from a node. Refer to [Data structure](/data/data-structure/) for information on item structure. | :white_check_mark: (when running once for each item) |
| `$input.context.noItemsLeft` | Boolean. Only available when working with the Split in Batches node. Provides information about what's happening in the node, allowing you to see if the node is still processing items. | :white_check_mark: |
| `$input.context.noItemsLeft` | Boolean. Only available when working with the Loop Over Items node. Provides information about what's happening in the node, allowing you to see if the node is still processing items. | :white_check_mark: |
=== "Python"
| Method | Description |
| ------ | ----------- |
Expand All @@ -30,4 +30,4 @@ Methods for working with the input of the current node. Some methods and variabl
| `_input.last()` | Last input item in current node. |
| `_input.params` | Object containing the query settings of the previous node. This includes data such as the operation it ran, result limits, and so on. |
| `_json` | Shorthand for `_input.item.json`. Incoming JSON data from a node. Refer to [Data structure](/data/data-structure/) for information on item structure. Available when you set **Mode** to **Run Once for Each Item**. |
| `_input.context.noItemsLeft` | Boolean. Only available when working with the Split in Batches node. Provides information about what's happening in the node, allowing you to see if the node is still processing items. |
| `_input.context.noItemsLeft` | Boolean. Only available when working with the Loop Over Items node. Provides information about what's happening in the node, allowing you to see if the node is still processing items. |
4 changes: 2 additions & 2 deletions docs/code/builtin/output-other-nodes.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Methods for working with the output of other nodes. Some methods and variables a
| `$("<node-name>").last(branchIndex?, run Index?)` | The last item output by the given node. | :white_check_mark: |
| `$("<node-name>").item` | The linked item. This is the item in the specified node used to produce the current item. Refer to [Item linking](/data/data-mapping/data-item-linking/) for more information on item linking. | :x: |
| `$("<node-name>").params` | Object containing the query settings of the given node. This includes data such as the operation it ran, result limits, and so on. | :white_check_mark: |
| `$("<node-name>").context` | Only available when working with the Split in Batches node. Provides information about what's happening in the node, allowing you to see if the node is still processing items. | :white_check_mark: |
| `$("<node-name>").context` | Only available when working with the Loop Over Items node. Provides information about what's happening in the node, allowing you to see if the node is still processing items. | :white_check_mark: |
| `$("<node-name>").itemMatching(currentNodeinputIndex)` | Use instead of `$("<node-name>").item` in the Code node if you need to trace back from an input item. | :white_check_mark: |
=== "Python"
| Method | Description | Available in Code node? |
Expand All @@ -28,5 +28,5 @@ Methods for working with the output of other nodes. Some methods and variables a
| `_("<node-name>").last(branchIndex?, run Index?)` | The last item output by the given node. | :white_check_mark: |
| `_("<node-name>").item` | The linked item. This is the item in the specified node used to produce the current item. Refer to [Item linking](/data/data-mapping/data-item-linking/) for more information on item linking. | :x: |
| `_("<node-name>").params` | Object containing the query settings of the given node. This includes data such as the operation it ran, result limits, and so on. | :white_check_mark: |
| `_("<node-name>").context` | Only available when working with the Split in Batches node. Provides information about what's happening in the node, allowing you to see if the node is still processing items. | :white_check_mark: |
| `_("<node-name>").context` | Only available when working with the Loop Over Items node. Provides information about what's happening in the node, allowing you to see if the node is still processing items. | :white_check_mark: |
| `_("<node-name>").itemMatching(currentNodeinputIndex)` | Use instead of `_("<node-name>").item` in the Code node if you need to trace back from an input item. | :white_check_mark: |
10 changes: 5 additions & 5 deletions docs/courses/level-two/chapter-3.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,20 +79,20 @@ To [create a loop in an n8n workflow](/flow-logic/looping/#using-loops-in-n8n){:

## Splitting data in batches

If you need to process large incoming data, execute the Code node multiple times, or avoid API rate limits, it's best to split the data into batches (groups) and process these batches. You can do this with the [Split in Batches node](/integrations/builtin/core-nodes/n8n-nodes-base.splitinbatches/){:target="_blank" .external}. This node splits input data into a specified batch size and, with each iteration, returns a predefined amount of data.
If you need to process large incoming data, execute the Code node multiple times, or avoid API rate limits, it's best to split the data into batches (groups) and process these batches. You can do this with the [Loop Over Items node](/integrations/builtin/core-nodes/n8n-nodes-base.splitinbatches/){:target="_blank" .external}. This node splits input data into a specified batch size and, with each iteration, returns a predefined amount of data.

!!! warning "Execution of Split in Batches node"
!!! warning "Execution of Loop Over Items node"

The Split In Batches node stops executing after all the incoming items get divided into batches and passed on to the next node in the workflow, so it is not necessary to add an IF node to stop the loop.
The Loop Over Items node stops executing after all the incoming items get divided into batches and passed on to the next node in the workflow, so it is not necessary to add an IF node to stop the loop.


### Exercise

Build a workflow that reads the RSS feed from Medium and dev.to. The workflow should consist of three nodes:

- A Code node that returns the URLs of the RSS feeds of Medium (`https://medium.com/feed/n8n-io`) and dev.to (`https://dev.to/feed/n8n`)
- A Split In Batches node with `Batch Size: 1`, that takes in the inputs from the Code node and RSS node and iterates over the items.
- An RSS Read node that gets the URL of the Medium RSS feed, passed as an expression: `{{$node["SplitInBatches"].json["url"]}}`. The RSS Read node is one of the exception nodes which processes only the first item it receives, so the Split in Batches node is necessary for iterating over multiple items.
- A Loop Over Items node with `Batch Size: 1`, that takes in the inputs from the Code node and RSS node and iterates over the items.
- An RSS Read node that gets the URL of the Medium RSS feed, passed as an expression: `{{$node["SplitInBatches"].json["url"]}}`. The RSS Read node is one of the exception nodes which processes only the first item it receives, so the Loop Over Items node is necessary for iterating over multiple items.

??? note "Show me the solution"

Expand Down
2 changes: 1 addition & 1 deletion docs/courses/level-two/chapter-5/chapter-5.2.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ The third part of the workflow consists of seven nodes:

<figure><img src="/_images/courses/level-two/chapter-five/workflow2_3.png" alt="" style="width:100%"><figcaption align = "center"><i>Workflow 3 – Generating files for total sales</i></figcaption></figure>

1. Use the [Split In Batches node](/integrations/builtin/core-nodes/n8n-nodes-base.splitinbatches/){:target="_blank" .external} to split data from the Item Lists node into batches of 5.
1. Use the [Loop Over Items node](/integrations/builtin/core-nodes/n8n-nodes-base.splitinbatches/){:target="_blank" .external} to split data from the Item Lists node into batches of 5.
2. Use the [Set node](/integrations/builtin/core-nodes/n8n-nodes-base.set/){:target="_blank" .external} to set four values, referenced with expressions from the previous node: `customerEmail`, `customerRegion`, `customerSince`, and `orderPrice`.
3. Use the [Date & Time node](/integrations/builtin/core-nodes/n8n-nodes-base.datetime/){:target="_blank" .external} to change the date format of the field `customerSince` to the format MM/DD/YYYY.
4. Use the [Spreadsheet File node](/integrations/builtin/core-nodes/n8n-nodes-base.spreadsheetfile/){:target="_blank" .external} to create a CSV spreadsheet with the file name set as the expression: `{{$runIndex > 0 ? 'file_low_orders':'file_high_orders'}}`.
Expand Down
2 changes: 1 addition & 1 deletion docs/flow-logic/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ When building your logic, you'll use n8n's [Core nodes](/integrations/builtin/co

* Splitting: [IF](/integrations/builtin/core-nodes/n8n-nodes-base.if/) and [Switch](/integrations/builtin/core-nodes/n8n-nodes-base.switch/).
* Merging: [Merge](/integrations/builtin/core-nodes/n8n-nodes-base.merge/), [Compare Datasets](/integrations/builtin/core-nodes/n8n-nodes-base.comparedatasets/), and [Code](/integrations/builtin/core-nodes/n8n-nodes-base.code/).
* Looping: [IF](/integrations/builtin/core-nodes/n8n-nodes-base.if/) and [Split In Batches](/integrations/builtin/core-nodes/n8n-nodes-base.splitinbatches/).
* Looping: [IF](/integrations/builtin/core-nodes/n8n-nodes-base.if/) and [Loop Over Items](/integrations/builtin/core-nodes/n8n-nodes-base.splitinbatches/).
* Waiting: [Wait](/integrations/builtin/core-nodes/n8n-nodes-base.wait/).
* Creating sub-workflows: [Execute Workflow](/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflow/) and [Execute Workflow Trigger](/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflowtrigger/).
* Error handling: [Stop And Error](/integrations/builtin/core-nodes/n8n-nodes-base.stopanderror/) and [Error Trigger](/integrations/builtin/core-nodes/n8n-nodes-base.errortrigger/).
4 changes: 2 additions & 2 deletions docs/flow-logic/looping.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,11 @@ Here is an [example workflow](https://n8n.io/workflows/1130) that implements a l

### Loop until all items are processed

Use the [Split In Batches](/integrations/builtin/core-nodes/n8n-nodes-base.splitinbatches/) node when you want to loop until all items are processed. To process each item individually, set **Batch Size** to `1`.
Use the [Loop Over Items](/integrations/builtin/core-nodes/n8n-nodes-base.splitinbatches/) node when you want to loop until all items are processed. To process each item individually, set **Batch Size** to `1`.

You can batch the data in groups and process these batches. This approach is useful for avoiding API rate limits when processing large incoming data or when you want to process a specific group of returned items.

The Split In Batches node stops executing after all the incoming items get divided into batches and passed on to the next node in the workflow so it's not necessary to add an IF node to stop the loop.
The Loop Over Items node stops executing after all the incoming items get divided into batches and passed on to the next node in the workflow so it's not necessary to add an IF node to stop the loop.

## Node exceptions

Expand Down
2 changes: 1 addition & 1 deletion docs/flow-logic/merging.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Use the Code node to merge data from multiple node executions. This is useful in
!!! note "Node executions and workflow executions"
This section describes merging data from multiple node executions. This is when a node executes multiple times during a single workflow execution.

Refer to this [example workflow](https://n8n.io/workflows/1814-merge-multiple-runs-into-one/){:target=_blank .external-link} with an RSS feed use case, and [another example workflow](https://n8n.io/workflows/1814-merge-multiple-runs-into-one/){:target=_blank .external-link} using Split In Batches and Wait to artificially create multiple executions.
Refer to this [example workflow](https://n8n.io/workflows/1814-merge-multiple-runs-into-one/){:target=_blank .external-link} using Loop Over Items and Wait to artificially create multiple executions.

## Compare, merge, and split again

Expand Down
2 changes: 1 addition & 1 deletion docs/hosting/scaling/memory-errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ This approach is more complex and means re-building the workflows causing the is
* Avoid manual executions when processing larger amounts of data.
* Split the workflow up into sub-workflows and ensure each sub-workflow returns a limited amount of data to its parent workflow.

Splitting the workflow might seem counter-intuitive at first as it usually requires adding at least two additional nodes: the [Split In Batches](/integrations/builtin/core-nodes/n8n-nodes-base.splitinbatches/) node to split up the items into smaller batches and the [Execute Workflow](/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflow/) node to start the sub-workflow.
Splitting the workflow might seem counter-intuitive at first as it usually requires adding at least two additional nodes: the [Loop Over Items](/integrations/builtin/core-nodes/n8n-nodes-base.splitinbatches/) node to split up the items into smaller batches and the [Execute Workflow](/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflow/) node to start the sub-workflow.

However, as long as your sub-workflow does the heavy lifting for each batch and then returns only a very small result set to the main workflow, the memory consumption is significantly reduced. This is because the sub-workflow only holds the data for the current batch in memory, after which the memory is freed again.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,9 @@ View [example workflows and related content](https://n8n.io/integrations/telegra

The Telegram API has a [limitation](https://core.telegram.org/bots/faq#broadcasting-to-users){:target=_blank .external-link} of sending only 30 messages per second. Follow these steps to send more than 30 messages:

1. Split In Batches node: Use the [Split in Batches](/integrations/builtin/core-nodes/n8n-nodes-base.splitinbatches/) node to get at most 30 chat IDs from your database.
2. Telegram node: Connect the Telegram node with the Split In Batches node. Use the **Expression Editor** to select the Chat IDs from the Split in Batches node.
3. Code node: Connect the [Code](/integrations/builtin/core-nodes/n8n-nodes-base.code/) node with the Telegram node. Use the Code node to wait for a few seconds before fetching the next batch of chat IDs. Connect this node with the Split In Batches node.
1. Loop Over Items node: Use the [Loop Over Items](/integrations/builtin/core-nodes/n8n-nodes-base.splitinbatches/) node to get at most 30 chat IDs from your database.
2. Telegram node: Connect the Telegram node with the Loop Over Items node. Use the **Expression Editor** to select the Chat IDs from the Loop Over Items node.
3. Code node: Connect the [Code](/integrations/builtin/core-nodes/n8n-nodes-base.code/) node with the Telegram node. Use the Code node to wait for a few seconds before fetching the next batch of chat IDs. Connect this node with the Loop Over Items node.

You can also use this [workflow](https://n8n.io/workflows/772){:target=_blank .external-link}.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
---
title: Split In Batches
description: Documentation for the Split In Batches node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
title: Loop Over Items
description: Documentation for the Loop Over Items node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
contentType: integration
---

# Split In Batches
# Loop Over Items

The Split In Batches node helps you loop through data.
The Loop Over Items node helps you loop through data.

The node saves the original incoming data, and with each iteration, returns a predefined amount of data through the **loop** output.

Expand All @@ -19,18 +19,18 @@ When the node execution completes, it combines all the data and returns it throu
- **Reset:** if set to true, the node will reset.

!!! note "Check if you need this node"
n8n automatically processes incoming items. You may not need the SplitInBatches node in your workflow. To learn more about how n8n handles multiple items, refer to the documentation on [Looping in n8n](/flow-logic/looping/).
n8n automatically processes incoming items. You may not need the Loop Over Items node in your workflow. To learn more about how n8n handles multiple items, refer to the documentation on [Looping in n8n](/flow-logic/looping/).


## Example usage: Read RSS feed from two different sources

This workflow allows you to read an RSS feed from two different sources using the Split In Batches node. You need the Split in Batches node in the workflow as the RSS Feed Read node only processes the first item it receives. You can also find the [workflow](https://n8n.io/workflows/687){:target=_blank .external-link} on n8n.io.
This workflow allows you to read an RSS feed from two different sources using the Loop Over Items node. You need the Loop Over Items node in the workflow as the RSS Feed Read node only processes the first item it receives. You can also find the [workflow](https://n8n.io/workflows/687-read-rss-feed-from-two-different-sources/){:target=_blank .external-link} on n8n.io.

The example walks through building the workflow, but assumes you are already familiar with n8n. To build your first workflow, including learning how to add nodes to a workflow, refer to [Try it out](/try-it-out/).

The final workflow looks like this:

![A workflow with the Split In Batches node](/_images/integrations/builtin/core-nodes/splitinbatches/workflow.png)
![A workflow with the Loop Over Items node](/_images/integrations/builtin/core-nodes/splitinbatches/workflow.png)

1. Add the manual trigger.
2. Add the Code node.
Expand All @@ -49,8 +49,8 @@ The final workflow looks like this:
}
];
```
4. Add the SplitInBatches node.
5. Configure SplitInBatches: set the batch size to `1` in the **Batch Size** field.
4. Add the Loop Over Items node.
5. Configure Loop Over Items: set the batch size to `1` in the **Batch Size** field.
6. Add the RSS Feed Read node.
7. Select **Execute Workflow**. This runs the workflow to load data into the RSS Feed Read node.
8. Configure RSS Feed Read: map `url` from the input to the **URL** field. You can do this by dragging and dropping from the **INPUT** panel, or using this expression: `{{ $json.url }}`.
Expand All @@ -59,9 +59,9 @@ The final workflow looks like this:

## Check that the node has processed all items

To check if the node still has items to process, use the following expression: `{{$node["SplitInBatches"].context["noItemsLeft"]}}`. This expression returns a boolean value. If the node still has data to process, the expression returns `false`, otherwise it returns `true`.
To check if the node still has items to process, use the following expression: `{{$node["Loop Over Items"].context["noItemsLeft"]}}`. This expression returns a boolean value. If the node still has data to process, the expression returns `false`, otherwise it returns `true`.

## Get the current running index of the node

To get the current running index of the node, use the following expression: `{{$node["SplitInBatches"].context["currentRunIndex"];}}`.
To get the current running index of the node, use the following expression: `{{$node["Loop Over Items"].context["currentRunIndex"];}}`.

Loading

0 comments on commit 61494db

Please sign in to comment.