Skip to content

Commit

Permalink
Merge pull request n8n-io#1771 from n8n-io/NODE-967-item-lists
Browse files Browse the repository at this point in the history
add new data transformation actions; deprecate item lists
  • Loading branch information
Deborah authored Dec 13, 2023
2 parents 8cfdd49 + f3e1db6 commit 44e4ec4
Show file tree
Hide file tree
Showing 17 changed files with 225 additions and 15 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Learn more about [data structure and data flow](/data/) in n8n workflows.
2 changes: 0 additions & 2 deletions docs/_redirects
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@


/integrations/builtin/cluster-nodes/sub-nodes/n8n-nodes-langchain.documentbinaryinputloader/ /integrations/builtin/cluster-nodes/sub-nodes/n8n-nodes-langchain.documentdefaultdataloader/
/integrations/builtin/cluster-nodes/sub-nodes/n8n-nodes-langchain.documentjsoninputloader/ /integrations/builtin/cluster-nodes/sub-nodes/n8n-nodes-langchain.documentdefaultdataloader/

Expand Down
2 changes: 1 addition & 1 deletion docs/courses/level-two/chapter-1.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ The two most common operations for data transformation are:

There are several ways to transform data for the purposes mentioned above:

- With the [Item Lists node](/integrations/builtin/core-nodes/n8n-nodes-base.itemLists){:target="_blank" .external}, you can `Split Out Items` or `Aggregate Items`. This node is the easy way to modify the structure of incoming data that contain lists (arrays), without needing to use JavaScript code in the Code node.
- Using n8n's [data transformation nodes](/data/#data-transformation-nodes). This the easy way to modify the structure of incoming data that contain lists (arrays), without needing to use JavaScript code in the Code node. Use [Split Out](/integrations/builtin/core-nodes/n8n-nodes-base.splitout/) to separate a single data item containing a list into multiple items, and [Aggregate](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/) to take separate items, or portions of them, and group them together into individual items.
- With the Code node, you can write JavaScript functions to modify the data structure of incoming data using the *Run Once for All Items* mode:

To create multiple items from a single item, you can use this JavaScript code:
Expand Down
2 changes: 1 addition & 1 deletion docs/courses/level-two/chapter-5/chapter-5.2.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ The first part of the workflow consists of five nodes:

2. Use the [Airtable node](/integrations/builtin/app-nodes/n8n-nodes-base.airtable/){:target="_blank" .external} to list data from the `customers` table (where you updated the fields `region` and `subregion`).
3. Use the [Merge node](/integrations/builtin/core-nodes/n8n-nodes-base.merge/){:target="_blank" .external} to merge data from the Airtable and HTTP Request node, based on the common key `customer ID`.
4. Use the [Item Lists node](/integrations/builtin/core-nodes/n8n-nodes-base.itemlists/){:target="_blank" .external} to sort data by orderPrice in descending order.
4. Use the [Sort](/integrations/builtin/core-nodes/n8n-nodes-base.sort/) node to sort data by orderPrice in descending order.

/// question | Quiz questions
* What is the name of the employee assigned to customer 1?
Expand Down
2 changes: 1 addition & 1 deletion docs/data/binary-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ And nodes for performing common tasks:

You can trigger a workflow based on changes to a local file using the [Local File trigger](/integrations/builtin/core-nodes/n8n-nodes-base.localfiletrigger/).

To split or concatenate binary data items, use the [Item Lists](/integrations/builtin/core-nodes/n8n-nodes-base.itemlists/) node.
To split or concatenate binary data items, use the [data transformation nodes](/data/#data-transformation-nodes).

### Code

Expand Down
13 changes: 13 additions & 0 deletions docs/data/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,16 @@ This section covers:
* [Process data using code](/data/code/)
* [Pinning](/data/data-pinning/) and [editing](/data/data-editing/) data during workflow development.
* [Data mapping](/data/data-mapping/data-mapping-ui/) and [Item linking](/data/data-mapping/data-item-linking/): how data items link to each other.

## Related resources

### Data transformation nodes

n8n provides a collection of nodes to transform data:

* [Aggregate](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/): take separate items, or portions of them, and group them together into individual items.
* [Limit](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/): remove items beyond a defined maximum number.
* [Remove Duplicates](/integrations/builtin/core-nodes/n8n-nodes-base.removeduplicates/): identify items that are identical across all fields or a subset of fields.
* [Sort](/integrations/builtin/core-nodes/n8n-nodes-base.sort/): organize lists of in a desired ordering, or generate a random selection.
* [Split Out](/integrations/builtin/core-nodes/n8n-nodes-base.splitout/): separate a single data item containing a list into multiple items.
* [Summarize](/integrations/builtin/core-nodes/n8n-nodes-base.summarize/): aggregate items together, in a manner similar to Excel pivot tables.
13 changes: 8 additions & 5 deletions docs/data/transforming-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,13 @@ For example, the image below shows the output of an [HTTP Request](/integrations

![HTTP Request node output](/_images/data/transforming-data/HTTPRequest_output.png)

To transform this kind of structure into the n8n data structure you can use the [Item Lists](/integrations/builtin/core-nodes/n8n-nodes-base.itemlists/) node.

/// note
If you're using the HTTP Request node, you should use the Split Into items option to transform the data. You don't have to use a Code node in that case.
///
To transform this kind of structure into the n8n data structure you can use the data transformation nodes:

* [Aggregate](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/): take separate items, or portions of them, and group them together into individual items.
* [Limit](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/): remove items beyond a defined maximum number.
* [Remove Duplicates](/integrations/builtin/core-nodes/n8n-nodes-base.removeduplicates/): identify items that are identical across all fields or a subset of fields.
* [Sort](/integrations/builtin/core-nodes/n8n-nodes-base.sort/): organize lists of in a desired ordering, or generate a random selection.
* [Split Out](/integrations/builtin/core-nodes/n8n-nodes-base.splitout/): separate a single data item containing a list into multiple items.
* [Summarize](/integrations/builtin/core-nodes/n8n-nodes-base.summarize/): aggregate items together, in a manner similar to Excel pivot tables.


35 changes: 35 additions & 0 deletions docs/integrations/builtin/core-nodes/n8n-nodes-base.aggregate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
title: Aggregate
description: Documentation for the Aggregate node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
---

# Aggregate

Use the Aggregate node to take separate items, or portions of them, and group them together into individual items.


/// note | Examples and templates
For usage examples and templates to help you get started, refer to n8n's [Aggregate integrations](https://n8n.io/integrations/aggregate/){:target=_blank .external-link} page.
///

## Node parameters

* **Aggregate**: choose whether to aggregate **Individual Fields** or **All Item Data**.
* If you choose **Individual Fields**, you can then configure the fields you want to aggregat with the following parameters:
* **Input Field Name**: the name of the field in the input data to be aggregated together.
* **Rename Field**: enable this toggle to enter a field name for the aggregated output data. When aggregating multiple fields you must provide new output field names. You can't leave multiple fields undefined.
* **Output Field Name**: displayed when you enable **Rename Field**. The field name for the aggregated output data.
* **Options** > **Add Field**: use this to add more optional settings, including:
* **Disable Dot Notation**: when disabled, you can't reference child fields (in the format `parent.child`).
* **Merge Lists**: enable this if the field to aggregate is a list, and you want to output a single flat list rather than a list of lists.
* **Include Binaries**: include binary data from the input in the new output.
* **Keep Missing And Null Values**: enable this to add a null (empty) entry in the output list when there is a null or missing value in the input.
* If you choose **All Item Data**, you can then set:
* **Put Output in Field**: the name of the output field.
* **Include**: choose from **All fields**, **Specified Fields**, or **All Fields Except**.

## Related resources

View [example workflows and related content](https://n8n.io/integrations/aggregate/){:target=_blank .external-link} on n8n's website.

--8<-- "_snippets/integrations/builtin/core-nodes/data-transformation-actions/data-section-link.md"
11 changes: 11 additions & 0 deletions docs/integrations/builtin/core-nodes/n8n-nodes-base.itemlists.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,17 @@ contentType: integration

# Item Lists

/// warning | Removed in 1.21.0
n8n removed the Item Lists node in version 1.21.0. Use the following nodes instead:

* [Aggregate](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/): take separate items, or portions of them, and group them together into individual items.
* [Limit](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/): remove items beyond a defined maximum number.
* [Remove Duplicates](/integrations/builtin/core-nodes/n8n-nodes-base.removeduplicates/): identify items that are identical across all fields or a subset of fields.
* [Sort](/integrations/builtin/core-nodes/n8n-nodes-base.sort/): organize lists of in a desired ordering, or generate a random selection.
* [Split Out](/integrations/builtin/core-nodes/n8n-nodes-base.splitout/): separate a single data item containing a list into multiple items.
* [Summarize](/integrations/builtin/core-nodes/n8n-nodes-base.summarize/): aggregate items together, in a manner similar to Excel pivot tables.
///

The Item Lists node simplifies working with returned data that contain lists (arrays), enabling you to change the structure for further processing without the need to use [Code](/integrations/builtin/core-nodes/n8n-nodes-base.code/) nodes or write custom JavaScript.

## Operations
Expand Down
24 changes: 24 additions & 0 deletions docs/integrations/builtin/core-nodes/n8n-nodes-base.limit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
title: Limit
description: Documentation for the Limit node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
---

# Limit

Use the Limit node to remove items beyond a defined maximum number. You can choose whether n8n takes the items from the beginning or end of the input data.


/// note | Examples and templates
For usage examples and templates to help you get started, refer to n8n's [Limit integrations](https://n8n.io/integrations/limit/){:target=_blank .external-link} page.
///

## Node parameters

* **Max Items**: enter the maximum number of items that n8n should keep. If the input data contains more than this value, n8n removes the items.
* **Keep**: when items must be removed, select if n8n keeps the input items at the beginning or end.

## Related resources

View [example workflows and related content](https://n8n.io/integrations/limit/){:target=_blank .external-link} on n8n's website.

--8<-- "_snippets/integrations/builtin/core-nodes/data-transformation-actions/data-section-link.md"
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
title: Remove Duplicates
description: Documentation for the Remove Duplicates node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
---

# Remove Duplicates

Use the Remove Duplicates node to identify items that are identical across all fields or a subset of fields. This is helpful in situations where you can end up with duplicate data, such as a user creating multiple accounts, or a customer submitting the same order multiple times. When working with large datasets it becomes more difficult to spot and remove these items.

/// note | Examples and templates
For usage examples and templates to help you get started, refer to n8n's [Remove Duplicates integrations](https://n8n.io/integrations/remove-duplicates/){:target=_blank .external-link} page.
///

## Node parameters

* **Compare**: specify which fields of the input data n8n should compare to check if they're the same. The following options are available:
* **All Fields**: compares all fields of the input data.
* **All Fields Except**: enter which input data fields n8n should exclude from the comparison. You can provide multiple values separated by commas.
* **Selected Fields**: enter which input data fields n8n should include in the comparison. You can provide multiple values separated by commas.
* If you choose **All Fields Except** or **Selected Fields**, n8n displays **Options** > **Add Field**. Use this to add more optional settings, including:
* **Disable Dot Notation**: when disabled, you can't reference child fields (in the format `parent.child`).
* **Remove Other Fields**: keep the fields that you're comparing and remove the others.


## Related resources

View [example workflows and related content](https://n8n.io/integrations/remove-duplicates/){:target=_blank .external-link} on n8n's website.

--8<-- "_snippets/integrations/builtin/core-nodes/data-transformation-actions/data-section-link.md"
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,7 @@ The Respond to Webhook node runs once, using the first incoming data item. This
If you need to return more than one data item, you can either:

- Instead of using the Respond to Webhook node, use the **When Last Node Finishes** option in **Respond** in the Webhook node. Use this when you want to return the final data that the workflow outputs.
- Use the [Item Lists](/integrations/builtin/core-nodes/n8n-nodes-base.itemlists/) node to turn multiple items into a single item before passing the data to the Respond to Webhook node. Use the following parameter settings in the Item Lists node:
- **Operation** > **Concatenate Items**
- **Aggregate** > **All Item Data (Into a Single List)**
- Use the [Aggregate](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/) node to turn multiple items into a single item before passing the data to the Respond to Webhook node. Set **Aggregate** to **All Item Data (Into a Single List)**.

## Workflow behavior

Expand Down
31 changes: 31 additions & 0 deletions docs/integrations/builtin/core-nodes/n8n-nodes-base.sort.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
title: Sort
description: Documentation for the Sort node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
---

# Sort

Use the Sort node to organize lists of items in a desired ordering, or generate a random selection.

/// note | Examples and templates
For usage examples and templates to help you get started, refer to n8n's [Sort integrations](https://n8n.io/integrations/sort/){:target=_blank .external-link} page.
///

/// note | Array sort behavior
The Sort operation uses the default JavaScript operation where the elements to be sorted are converted into strings and their values compared. Refer to [Mozilla's guide to Array sort](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort){:target=_blank .external-link} to learn more.
///

## Node parameters

* **Type**: use the dropdown to select how you want to input the sorting. The following options are available:
* **Simple**: when you selected, you can use the **Add Field To Sort By** button to input the fields, and select whether to use **Ascending** or **Descending** order.
* **Random**: select to create a random order in the list.
* **Code**: when selected, displays a code input field where you can enter custom JavaScript code to perform the sort operation.
* **Options** > **Add Field**: use this to add more optional settings, including:
* **Disable Dot Notation**: when disabled, you can't reference child fields (in the format `parent.child`).

## Related resources

View [example workflows and related content](https://n8n.io/integrations/sort/){:target=_blank .external-link} on n8n's website.

--8<-- "_snippets/integrations/builtin/core-nodes/data-transformation-actions/data-section-link.md"
31 changes: 31 additions & 0 deletions docs/integrations/builtin/core-nodes/n8n-nodes-base.splitout.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
title: Split Out
description: Documentation for the Split Out node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
---

# Split Out

Use the Split Out node to separate a single data item containing a list into multiple items. For example, a list of customers, and you want to split them so that you have an item for each customer.

/// note | Examples and templates
For usage examples and templates to help you get started, refer to n8n's [Split Out integrations](https://n8n.io/integrations/split-out/){:target=_blank .external-link} page.
///

## Node parameters

* **Field to Split Out**: the field containing the list you want to separate out into individual items.
* If working with binary data inputs, use `$binary` in an expression to set the field to split out.
* **Include**: select if you want n8n to keep any other fields from the input data with each new individual item. You can select:
* **No Other Fields**
* **All Other Fields**
* **Selected Other Fields**: when selected, n8n displays **Fields to Include**. Enter a comma separated list of desired fields.
* **Options** > **Add Field**: use this to add more optional settings, including:
* **Disable Dot Notation**: when disabled, you can't reference child fields (in the format `parent.child`).
* **Destination Field Name**: optionally set the field name under which to put the new split contents.
* **Include Binary**: include binary data from the input in the new output.

## Related resources

View [example workflows and related content](https://n8n.io/integrations/split-out/){:target=_blank .external-link} on n8n's website.

--8<-- "_snippets/integrations/builtin/core-nodes/data-transformation-actions/data-section-link.md"
31 changes: 31 additions & 0 deletions docs/integrations/builtin/core-nodes/n8n-nodes-base.summarize.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
title: Summarize
description: Documentation for the Summarize node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
---

# Summarize

Use the Summarize node to aggregate items together, in a manner similar to Excel pivot tables.

/// note | Examples and templates
For usage examples and templates to help you get started, refer to n8n's [Summarize integrations](https://n8n.io/integrations/summarize/){:target=_blank .external-link} page.
///

## Node parameters

* **Fields to Summarize**:
* To combine values, select an **Aggregation** method, and enter a **Field** name.
* To split values, enter a field name or list of names in **Fields to Split By**.
* **Options** > **Add Field**: use this to add more optional settings, including:
* **Disable Dot Notation**: when disabled, you can't reference child fields (in the format `parent.child`).
* **Output Format**:
* **Each Split in a Separate Item**: splitting generates a separate output item for each split out field.
* **All Splits in a Single Item**: splitting generates a single item, which lists the split out fields.
* **Ignore items without valid fields to group by**: ignore input items that don't contain the field specified in **Fields to Summarize**.


## Related resources

View [example workflows and related content](https://n8n.io/integrations/summarize/){:target=_blank .external-link} on n8n's website.

--8<-- "_snippets/integrations/builtin/core-nodes/data-transformation-actions/data-section-link.md"
2 changes: 1 addition & 1 deletion document-templates/core-nodes.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# How to use this template
1. Make a new branch. If working on an internal ticket, include it at the start of the name. For example, DOC-123-feature-summary.
2. Create a new file, or find the file you want to edit, in integrations/builtin/core-nodes/. If creating a new file, pay attention to the naming conventions: it should match the node name in the codex file. For example, in the Item Lists node, the codex file (https://github.com/n8n-io/n8n/blob/master/packages/nodes-base/nodes/ItemLists/ItemLists.node.json) reads: `"node": "n8n-nodes-base.itemList"`. So the app node file name is n8n-nodes-base.itemList.
2. Create a new file, or find the file you want to edit, in integrations/builtin/core-nodes/. If creating a new file, pay attention to the naming conventions: it should match the node name in the codex file.
3. Copy the template into the file (don't copy this comment).
4. Placeholder text is in _italic_ or between <>. Make sure to replace it!
5. Before publishing, delete any comments.
Expand Down
Loading

0 comments on commit 44e4ec4

Please sign in to comment.