Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add new data transformation actions; deprecate item lists #1771

Merged
merged 4 commits into from
Dec 13, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Learn more about [data structure and data flow](/data/) in n8n workflows.
2 changes: 0 additions & 2 deletions docs/_redirects
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@


/integrations/builtin/cluster-nodes/sub-nodes/n8n-nodes-langchain.documentbinaryinputloader/ /integrations/builtin/cluster-nodes/sub-nodes/n8n-nodes-langchain.documentdefaultdataloader/
/integrations/builtin/cluster-nodes/sub-nodes/n8n-nodes-langchain.documentjsoninputloader/ /integrations/builtin/cluster-nodes/sub-nodes/n8n-nodes-langchain.documentdefaultdataloader/

Expand Down
2 changes: 1 addition & 1 deletion docs/courses/level-two/chapter-1.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ The two most common operations for data transformation are:

There are several ways to transform data for the purposes mentioned above:

- With the [Item Lists node](/integrations/builtin/core-nodes/n8n-nodes-base.itemLists){:target="_blank" .external}, you can `Split Out Items` or `Aggregate Items`. This node is the easy way to modify the structure of incoming data that contain lists (arrays), without needing to use JavaScript code in the Code node.
- Using n8n's [data transformation nodes](/data/#data-transformation-nodes). This the easy way to modify the structure of incoming data that contain lists (arrays), without needing to use JavaScript code in the Code node. Use [Split Out](/integrations/builtin/core-nodes/n8n-nodes-base.splitout/) to separate a single data item containing a list into multiple items, and [Aggregate](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/) to take separate items, or portions of them, and group them together into individual items.
- With the Code node, you can write JavaScript functions to modify the data structure of incoming data using the *Run Once for All Items* mode:

To create multiple items from a single item, you can use this JavaScript code:
Expand Down
2 changes: 1 addition & 1 deletion docs/courses/level-two/chapter-5/chapter-5.2.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ The first part of the workflow consists of five nodes:

2. Use the [Airtable node](/integrations/builtin/app-nodes/n8n-nodes-base.airtable/){:target="_blank" .external} to list data from the `customers` table (where you updated the fields `region` and `subregion`).
3. Use the [Merge node](/integrations/builtin/core-nodes/n8n-nodes-base.merge/){:target="_blank" .external} to merge data from the Airtable and HTTP Request node, based on the common key `customer ID`.
4. Use the [Item Lists node](/integrations/builtin/core-nodes/n8n-nodes-base.itemlists/){:target="_blank" .external} to sort data by orderPrice in descending order.
4. Use the [Sort](/integrations/builtin/core-nodes/n8n-nodes-base.sort/) node to sort data by orderPrice in descending order.

/// question | Quiz questions
* What is the name of the employee assigned to customer 1?
Expand Down
2 changes: 1 addition & 1 deletion docs/data/binary-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ And nodes for performing common tasks:

You can trigger a workflow based on changes to a local file using the [Local File trigger](/integrations/builtin/core-nodes/n8n-nodes-base.localfiletrigger/).

To split or concatenate binary data items, use the [Item Lists](/integrations/builtin/core-nodes/n8n-nodes-base.itemlists/) node.
To split or concatenate binary data items, use the [data transformation nodes](/data/#data-transformation-nodes).

### Code

Expand Down
13 changes: 13 additions & 0 deletions docs/data/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,16 @@ This section covers:
* [Process data using code](/data/code/)
* [Pinning](/data/data-pinning/) and [editing](/data/data-editing/) data during workflow development.
* [Data mapping](/data/data-mapping/data-mapping-ui/) and [Item linking](/data/data-mapping/data-item-linking/): how data items link to each other.

## Related resources

### Data transformation nodes

n8n provides a collection of nodes to transform data:

* [Aggregate](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/): take separate items, or portions of them, and group them together into individual items.
* [Limit](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/): remove items beyond a defined maximum number.
* [Remove Duplicates](/integrations/builtin/core-nodes/n8n-nodes-base.removeduplicates/): identify items that are identical across all fields or a subset of fields.
* [Sort](/integrations/builtin/core-nodes/n8n-nodes-base.sort/): organize lists of in a desired ordering, or generate a random selection.
* [Split Out](/integrations/builtin/core-nodes/n8n-nodes-base.splitout/): separate a single data item containing a list into multiple items.
* [Summarize](/integrations/builtin/core-nodes/n8n-nodes-base.summarize/): aggregate items together, in a manner similar to Excel pivot tables.
13 changes: 8 additions & 5 deletions docs/data/transforming-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,13 @@ For example, the image below shows the output of an [HTTP Request](/integrations

![HTTP Request node output](/_images/data/transforming-data/HTTPRequest_output.png)

To transform this kind of structure into the n8n data structure you can use the [Item Lists](/integrations/builtin/core-nodes/n8n-nodes-base.itemlists/) node.

/// note
If you're using the HTTP Request node, you should use the Split Into items option to transform the data. You don't have to use a Code node in that case.
///
To transform this kind of structure into the n8n data structure you can use the data transformation nodes:

* [Aggregate](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/): take separate items, or portions of them, and group them together into individual items.
* [Limit](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/): remove items beyond a defined maximum number.
* [Remove Duplicates](/integrations/builtin/core-nodes/n8n-nodes-base.removeduplicates/): identify items that are identical across all fields or a subset of fields.
* [Sort](/integrations/builtin/core-nodes/n8n-nodes-base.sort/): organize lists of in a desired ordering, or generate a random selection.
* [Split Out](/integrations/builtin/core-nodes/n8n-nodes-base.splitout/): separate a single data item containing a list into multiple items.
* [Summarize](/integrations/builtin/core-nodes/n8n-nodes-base.summarize/): aggregate items together, in a manner similar to Excel pivot tables.


35 changes: 35 additions & 0 deletions docs/integrations/builtin/core-nodes/n8n-nodes-base.aggregate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
title: Aggregate
description: Documentation for the Aggregate node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
---

# Aggregate

Use the Aggregate node to take separate items, or portions of them, and group them together into individual items.


/// note | Examples and templates
For usage examples and templates to help you get started, refer to n8n's [Aggregate integrations](https://n8n.io/integrations/aggregate/){:target=_blank .external-link} page.
///

## Node parameters

* **Aggregate**: choose whether to aggregate **Individual Fields** or **All Item Data**.
* If you choose **Individual Fields**, you can then configure the fields you want to aggregat with the following parameters:
* **Input Field Name**: the name of the field in the input data to be aggregated together.
* **Rename Field**: enable this toggle to enter a field name for the aggregated output data. When aggregating multiple fields you must provide new output field names. You can't leave multiple fields undefined.
* **Output Field Name**: displayed when you enable **Rename Field**. The field name for the aggregated output data.
* **Options** > **Add Field**: use this to add more optional settings, including:
* **Disable Dot Notation**: when disabled, you can't reference child fields (in the format `parent.child`).
* **Merge Lists**: enable this if the field to aggregate is a list, and you want to output a single flat list rather than a list of lists.
* **Include Binaries**: include binary data from the input in the new output.
* **Keep Missing And Null Values**: enable this to add a null (empty) entry in the output list when there is a null or missing value in the input.
* If you choose **All Item Data**, you can then set:
* **Put Output in Field**: the name of the output field.
* **Include**: choose from **All fields**, **Specified Fields**, or **All Fields Except**.

## Related resources

View [example workflows and related content](https://n8n.io/integrations/aggregate/){:target=_blank .external-link} on n8n's website.

--8<-- "_snippets/integrations/builtin/core-nodes/data-transformation-actions/data-section-link.md"
11 changes: 11 additions & 0 deletions docs/integrations/builtin/core-nodes/n8n-nodes-base.itemlists.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,17 @@ contentType: integration

# Item Lists

/// warning | Removed in 1.21.0
n8n removed the Item Lists node in version 1.21.0. Use the following nodes instead:

* [Aggregate](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/): take separate items, or portions of them, and group them together into individual items.
* [Limit](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/): remove items beyond a defined maximum number.
* [Remove Duplicates](/integrations/builtin/core-nodes/n8n-nodes-base.removeduplicates/): identify items that are identical across all fields or a subset of fields.
* [Sort](/integrations/builtin/core-nodes/n8n-nodes-base.sort/): organize lists of in a desired ordering, or generate a random selection.
* [Split Out](/integrations/builtin/core-nodes/n8n-nodes-base.splitout/): separate a single data item containing a list into multiple items.
* [Summarize](/integrations/builtin/core-nodes/n8n-nodes-base.summarize/): aggregate items together, in a manner similar to Excel pivot tables.
///

The Item Lists node simplifies working with returned data that contain lists (arrays), enabling you to change the structure for further processing without the need to use [Code](/integrations/builtin/core-nodes/n8n-nodes-base.code/) nodes or write custom JavaScript.

## Operations
Expand Down
24 changes: 24 additions & 0 deletions docs/integrations/builtin/core-nodes/n8n-nodes-base.limit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
title: Limit
description: Documentation for the Limit node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
---

# Limit

Use the Limit node to remove items beyond a defined maximum number. You can choose whether n8n takes the items from the beginning or end of the input data.


/// note | Examples and templates
For usage examples and templates to help you get started, refer to n8n's [Limit integrations](https://n8n.io/integrations/limit/){:target=_blank .external-link} page.
///

## Node parameters

* **Max Items**: enter the maximum number of items that n8n should keep. If the input data contains more than this value, n8n removes the items.
* **Keep**: when items must be removed, select if n8n keeps the input items at the beginning or end.

## Related resources

View [example workflows and related content](https://n8n.io/integrations/limit/){:target=_blank .external-link} on n8n's website.

--8<-- "_snippets/integrations/builtin/core-nodes/data-transformation-actions/data-section-link.md"
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
title: Remove Duplicates
description: Documentation for the Remove Duplicates node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
---

# Remove Duplicates

Use the Remove Duplicates node to identify items that are identical across all fields or a subset of fields. This is helpful in situations where you can end up with duplicate data, such as a user creating multiple accounts, or a customer submitting the same order multiple times. When working with large datasets it becomes more difficult to spot and remove these items.

/// note | Examples and templates
For usage examples and templates to help you get started, refer to n8n's [Remove Duplicates integrations](https://n8n.io/integrations/remove-duplicates/){:target=_blank .external-link} page.
///

## Node parameters

* **Compare**: specify which fields of the input data n8n should compare to check if they're the same. The following options are available:
* **All Fields**: compares all fields of the input data.
* **All Fields Except**: enter which input data fields n8n should exclude from the comparison. You can provide multiple values separated by commas.
* **Selected Fields**: enter which input data fields n8n should include in the comparison. You can provide multiple values separated by commas.
* If you choose **All Fields Except** or **Selected Fields**, n8n displays **Options** > **Add Field**. Use this to add more optional settings, including:
* **Disable Dot Notation**: when disabled, you can't reference child fields (in the format `parent.child`).
* **Remove Other Fields**: keep the fields that you're comparing and remove the others.


## Related resources

View [example workflows and related content](https://n8n.io/integrations/remove-duplicates/){:target=_blank .external-link} on n8n's website.

--8<-- "_snippets/integrations/builtin/core-nodes/data-transformation-actions/data-section-link.md"
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,7 @@ The Respond to Webhook node runs once, using the first incoming data item. This
If you need to return more than one data item, you can either:

- Instead of using the Respond to Webhook node, use the **When Last Node Finishes** option in **Respond** in the Webhook node. Use this when you want to return the final data that the workflow outputs.
- Use the [Item Lists](/integrations/builtin/core-nodes/n8n-nodes-base.itemlists/) node to turn multiple items into a single item before passing the data to the Respond to Webhook node. Use the following parameter settings in the Item Lists node:
- **Operation** > **Concatenate Items**
- **Aggregate** > **All Item Data (Into a Single List)**
- Use the [Aggregate](/integrations/builtin/core-nodes/n8n-nodes-base.aggregate/) node to turn multiple items into a single item before passing the data to the Respond to Webhook node. Set **Aggregate** to **All Item Data (Into a Single List)**.

## Workflow behavior

Expand Down
31 changes: 31 additions & 0 deletions docs/integrations/builtin/core-nodes/n8n-nodes-base.sort.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
title: Sort
description: Documentation for the Sort node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
---

# Sort

Use the Sort node to organize lists of items in a desired ordering, or generate a random selection.

/// note | Examples and templates
For usage examples and templates to help you get started, refer to n8n's [Sort integrations](https://n8n.io/integrations/sort/){:target=_blank .external-link} page.
///

/// note | Array sort behavior
The Sort operation uses the default JavaScript operation where the elements to be sorted are converted into strings and their values compared. Refer to [Mozilla's guide to Array sort](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort){:target=_blank .external-link} to learn more.
///

## Node parameters

* **Type**: use the dropdown to select how you want to input the sorting. The following options are available:
* **Simple**: when you selected, you can use the **Add Field To Sort By** button to input the fields, and select whether to use **Ascending** or **Descending** order.
* **Random**: select to create a random order in the list.
* **Code**: when selected, displays a code input field where you can enter custom JavaScript code to perform the sort operation.
* **Options** > **Add Field**: use this to add more optional settings, including:
* **Disable Dot Notation**: when disabled, you can't reference child fields (in the format `parent.child`).

## Related resources

View [example workflows and related content](https://n8n.io/integrations/sort/){:target=_blank .external-link} on n8n's website.

--8<-- "_snippets/integrations/builtin/core-nodes/data-transformation-actions/data-section-link.md"
31 changes: 31 additions & 0 deletions docs/integrations/builtin/core-nodes/n8n-nodes-base.splitout.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
title: Split Out
description: Documentation for the Split Out node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
---

# Split Out

Use the Split Out node to separate a single data item containing a list into multiple items. For example, a list of customers, and you want to split them so that you have an item for each customer.

/// note | Examples and templates
For usage examples and templates to help you get started, refer to n8n's [Split Out integrations](https://n8n.io/integrations/split-out/){:target=_blank .external-link} page.
///

## Node parameters

* **Field to Split Out**: the field containing the list you want to separate out into individual items.
* If working with binary data inputs, use `$binary` in an expression to set the field to split out.
* **Include**: select if you want n8n to keep any other fields from the input data with each new individual item. You can select:
* **No Other Fields**
* **All Other Fields**
* **Selected Other Fields**: when selected, n8n displays **Fields to Include**. Enter a comma separated list of desired fields.
* **Options** > **Add Field**: use this to add more optional settings, including:
* **Disable Dot Notation**: when disabled, you can't reference child fields (in the format `parent.child`).
* **Destination Field Name**: optionally set the field name under which to put the new split contents.
* **Include Binary**: include binary data from the input in the new output.

## Related resources

View [example workflows and related content](https://n8n.io/integrations/split-out/){:target=_blank .external-link} on n8n's website.

--8<-- "_snippets/integrations/builtin/core-nodes/data-transformation-actions/data-section-link.md"
31 changes: 31 additions & 0 deletions docs/integrations/builtin/core-nodes/n8n-nodes-base.summarize.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
title: Summarize
description: Documentation for the Summarize node in n8n, a workflow automation platform. Includes guidance on usage, and links to examples.
---

# Summarize

Use the Summarize node to aggregate items together, in a manner similar to Excel pivot tables.

/// note | Examples and templates
For usage examples and templates to help you get started, refer to n8n's [Summarize integrations](https://n8n.io/integrations/summarize/){:target=_blank .external-link} page.
///

## Node parameters

* **Fields to Summarize**:
* To combine values, select an **Aggregation** method, and enter a **Field** name.
* To split values, enter a field name or list of names in **Fields to Split By**.
* **Options** > **Add Field**: use this to add more optional settings, including:
* **Disable Dot Notation**: when disabled, you can't reference child fields (in the format `parent.child`).
* **Output Format**:
* **Each Split in a Separate Item**: splitting generates a separate output item for each split out field.
* **All Splits in a Single Item**: splitting generates a single item, which lists the split out fields.
* **Ignore items without valid fields to group by**: ignore input items that don't contain the field specified in **Fields to Summarize**.


## Related resources

View [example workflows and related content](https://n8n.io/integrations/summarize/){:target=_blank .external-link} on n8n's website.

--8<-- "_snippets/integrations/builtin/core-nodes/data-transformation-actions/data-section-link.md"
2 changes: 1 addition & 1 deletion document-templates/core-nodes.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# How to use this template

1. Make a new branch. If working on an internal ticket, include it at the start of the name. For example, DOC-123-feature-summary.
2. Create a new file, or find the file you want to edit, in integrations/builtin/core-nodes/. If creating a new file, pay attention to the naming conventions: it should match the node name in the codex file. For example, in the Item Lists node, the codex file (https://github.com/n8n-io/n8n/blob/master/packages/nodes-base/nodes/ItemLists/ItemLists.node.json) reads: `"node": "n8n-nodes-base.itemList"`. So the app node file name is n8n-nodes-base.itemList.
2. Create a new file, or find the file you want to edit, in integrations/builtin/core-nodes/. If creating a new file, pay attention to the naming conventions: it should match the node name in the codex file.
3. Copy the template into the file (don't copy this comment).
4. Placeholder text is in _italic_ or between <>. Make sure to replace it!
5. Before publishing, delete any comments.
Expand Down
Loading