Templates and examples

Here are some templates and examples for the Remove Duplicates node.

Templates

[[ templatesWidget(page.title, 'remove-duplicates') ]]

Set up sample data using the Code node

Create a workflow with some example input data to try out the Remove Duplicates node.

1. Add a Code node to the canvas and connect it to the Manual Trigger node.
2. In the Code node, set Mode to Run Once for Each Item and Language to JavaScript.
3. Paste the following JavaScript code snippet in the JavaScript field:

let data =[];

return {
  data: [
    { id: 1, name: 'Taylor Swift', job: 'Pop star', last_updated: '2024-09-20T10:12:43.493Z' },
    { id: 2, name: 'Ed Sheeran', job: 'Singer-songwriter', last_updated: '2024-10-05T08:30:59.493Z' },
    { id: 3, name: 'Adele', job: 'Singer-songwriter', last_updated: '2024-10-07T14:15:59.493Z' },
    { id: 4, name: 'Bruno Mars', job: 'Singer-songwriter', last_updated: '2024-08-25T17:45:12.493Z' },
    { id: 1, name: 'Taylor Swift', job: 'Pop star', last_updated: '2024-09-20T10:12:43.493Z' },  // duplicate
    { id: 5, name: 'Billie Eilish', job: 'Singer-songwriter', last_updated: '2024-09-10T09:30:12.493Z' },
    { id: 6, name: 'Katy Perry', job: 'Pop star', last_updated: '2024-10-08T12:30:45.493Z' },
    { id: 2, name: 'Ed Sheeran', job: 'Singer-songwriter', last_updated: '2024-10-05T08:30:59.493Z' },  // duplicate
    { id: 7, name: 'Lady Gaga', job: 'Pop star', last_updated: '2024-09-15T14:45:30.493Z' },
    { id: 8, name: 'Rihanna', job: 'Pop star', last_updated: '2024-10-01T11:50:22.493Z' },
    { id: 3, name: 'Adele', job: 'Singer-songwriter', last_updated: '2024-10-07T14:15:59.493Z' },  // duplicate
    //{ id: 9, name: 'Tom Hanks', job: 'Actor', last_updated: '2024-10-17T13:58:31.493Z' },
    //{ id: 0, name: 'Madonna', job: 'Pop star', last_updated: '2024-10-17T17:11:38.493Z' },
    //{ id: 15, name: 'Bob Dylan', job: 'Folk singer', last_updated: '2024-09-24T08:03:16.493Z'},
    //{ id: 10, name: 'Harry Nilsson', job: 'Singer-songwriter', last_updated: '2020-10-17T17:11:38.493Z' },
    //{ id: 11, name: 'Kylie Minogue', job: 'Pop star', last_updated: '2024-10-24T08:03:16.493Z'},
  ]
}

4. Add a Split Out node to the canvas and connect it to the Code node.
5. In the Split Out node, enter data in the Fields To Split Out field.

Removing duplicates from the current input

1. Add a Remove Duplicates node to the canvas and connect it to the Split Out node. Choose Remove items repeated within current input as the Action to start.
2. Open the Remove Duplicates node and ensure that the Operation is set to Remove Items Repeated Within Current Input.
3. Choose All fields in the Compare field.
4. Select Execute step to run the Remove Duplicates node, removing duplicated data in the current input.

n8n removes the items that have the same data across all fields. Your output in table view should look like this:

5. Open the Remove Duplicates node again and change the Compare parameter to Selected Fields.
6. In the Fields To Compare field, enter job.
7. Select Execute step to run the Remove Duplicates node, removing duplicated data in the current input.

n8n removes the items in the current input that have the same job data. Your output in table view should look like this:

Keep items where the value is new

1. Open the Remove Duplicates node and set the Operation to Remove Items Processed in Previous Executions.
2. Set the Keep Items Where parameter to Value Is New.
3. Set the Value to Dedupe On parameter to {{ $json.name }}.
4. On the canvas, select Execute workflow to run the workflow. Open the Remove Duplicates node to examine the results.

n8n compares the current input data to the items stored from previous executions. Since this is the first time running the Remove Duplicates node with this operation, n8n processes all data items and places them into the Kept output tab. The order of the items may be different than the order in the input data:

5. Open the Code node and uncomment (remove the // from) the line for "Tom Hanks."
6. On the canvas, select Execute workflow again. Open the Remove Duplicates node again to examine the results.

n8n compares the current input data to the items stored from previous executions. This time, the Kept tab contains the one new record from the Code node:

The Discarded tab contains the items processed by the previous execution:

Before continuing, clear the duplication history to get ready for the next example:

7. Open the Remove Duplicates node and set the Operation to Clear Deduplication History.
8. Select Execute step to clear the current duplication history.

Keep items where the value is higher than any previous value

1. Open the Remove Duplicates node and set the Operation to Remove Items Processed in Previous Executions.
2. Set the Keep Items Where parameter to Value Is Higher than Any Previous Value.
3. Set the Value to Dedupe On parameter to {{ $json.id }}.
4. On the canvas, select Execute workflow to run the workflow. Open the Remove Duplicates node to examine the results.

n8n compares the current input data to the items stored from previous executions. Since this is the first time running the Remove Duplicates node after clearing the history, n8n processes all data items and places them into the Kept output tab. The order of the items may be different than the order in the input data:

5. Open the Code node and uncomment (remove the // from) the lines for "Madonna" and "Bob Dylan."
6. On the canvas, select Execute workflow again. Open the Remove Duplicates node again to examine the results.

n8n compares the current input data to the items stored from previous executions. This time, the Kept tab contains a single entry for "Bob Dylan." n8n keeps this item because its id column value (15) is higher than any previous values (the previous maximum value was 9):

The Discarded tab contains the 13 items with an id column value equal to or less than the previous maximum value (9). Even though it's new, this table includes the entry for "Madonna" because its id value isn't larger than the previous maximum value:

Before continuing, clear the duplication history to get ready for the next example:

7. Open the Remove Duplicates node and set the Operation to Clear Deduplication History.
8. Select Execute step to clear the current duplication history.

Keep items where the value is a date later than any previous date

1. Open the Remove Duplicates node and set the Operation to Remove Items Processed in Previous Executions.
2. Set the Keep Items Where parameter to Value Is a Date Later than Any Previous Date.
3. Set the Value to Dedupe On parameter to {{ $json.last_updated }}.
4. On the canvas, select Execute workflow to run the workflow. Open the Remove Duplicates node to examine the results.

n8n compares the current input data to the items stored from previous executions. Since this is the first time running the Remove Duplicates node after clearing the history, n8n processes all data items and places them into the Kept output tab. The order of the items may be different than the order in the input data:

5. Open the Code node and uncomment (remove the // from) the lines for "Harry Nilsson" and "Kylie Minogue."

6. On the canvas, select Execute workflow again. Open the Remove Duplicates node again to examine the results.

n8n compares the current input data to the items stored from previous executions. This time, the Kept tab contains a single entry for "Kylie Minogue." n8n keeps this item because its last_updated column value (2024-10-24T08:03:16.493Z) is later than any previous values (the previous latest date was 2024-10-17T17:11:38.493Z):

The Discarded tab contains the 15 items with a last_updated column value equal to or earlier than the previous latest date (2024-10-17T17:11:38.493Z). Even though it's new, this table includes the entry for "Harry Nilsson" because its last_updated value isn't later than the previous maximum value: