Skip to content

Consent in Reverse ETL "soft" GA #6459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Jun 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
f14f9a7
init consent in retl docs
forstisabella Mar 20, 2024
7030e32
rmv retl from callout of unsupported dests
forstisabella Mar 20, 2024
4ea94d9
add consent in retl, consent faq
forstisabella Apr 29, 2024
db07872
yml
forstisabella Apr 29, 2024
a0e628a
only support one way of setting up
forstisabella May 1, 2024
60cfb9e
remove misc consent items
forstisabella May 2, 2024
f78b803
rmv more misc items
forstisabella May 2, 2024
608d9bf
Merge branch 'develop' into DOC-850
forstisabella May 7, 2024
2c40b39
Update consent-in-retl.md
forstisabella May 7, 2024
c9abf78
Merge branch 'DOC-850' of https://github.com/segmentio/segment-docs i…
forstisabella May 7, 2024
9a5b19a
Merge branch 'develop' into DOC-850
forstisabella May 30, 2024
619462d
complete first pass
forstisabella May 30, 2024
685fdb8
[netlify-build]
forstisabella May 31, 2024
76e58fa
add note to retl page
forstisabella Jun 3, 2024
3b997ae
[netlify-build]
forstisabella Jun 3, 2024
85c7aea
[netlify-build]
forstisabella Jun 3, 2024
4038e83
Add destinations procedure, "RETL" qa
forstisabella Jun 3, 2024
a176345
[netlify-build]
forstisabella Jun 3, 2024
fe7246d
fix doubles [netlify-build]
forstisabella Jun 5, 2024
242f2e2
addressing Atit's feedback [netlify-build]
forstisabella Jun 11, 2024
285eabd
adding missing item [netlify-build]
forstisabella Jun 11, 2024
e1aaa4e
[netlify-build]
forstisabella Jun 11, 2024
22cdc0c
rmv sidenav
forstisabella Jun 11, 2024
bcf3268
hidden: true
forstisabella Jun 11, 2024
ec76e8b
Merge branch 'master' into DOC-850
forstisabella Jun 11, 2024
927df01
Apply suggestions from code review
forstisabella Jun 12, 2024
af510ef
h3s, parallelism
forstisabella Jun 12, 2024
66964e7
Merge branch 'DOC-850' of https://github.com/segmentio/segment-docs i…
forstisabella Jun 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/_data/sidenav/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -515,9 +515,9 @@ sections:
title: Consent in Segment Connections
- path: /privacy/consent-management/configure-consent-management
title: Configure Consent Management
- path: /privacy/consent-management/consent-in-unify/
- path: /privacy/consent-management/consent-in-unify
title: Consent in Unify
- path: /privacy/consent-management/consent-faq/
- path: /privacy/consent-management/consent-faq
title: Consent FAQs
- path: /privacy/account-deletion
title: Account & Data Deletion
Expand Down
7 changes: 4 additions & 3 deletions src/connections/reverse-etl/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,9 @@ Follow these 4 steps to set up Reverse ETL and learn what each component is abou
3. [Add a destination](#step-3-add-a-destination)
4. [Create mappings](#step-4-create-mappings)

> info ""
> The UI navigation and interface will look different from what's presented in the docs until Reverse ETL rolls out to all users for GA.
> info "Enforce consent preferences stored in your warehouse with Consent Management"
> Use Consent Management to enforce your end users' consent preferences across your downstream Reverse ETL supported destinations. See the [Consent in Reverse ETL documentation](/docs/privacy/consent-management/consent-in-retl/) for more information.


### Step 1: Add a source
A source is where your data originates from. Traditionally in Segment, a [source](/docs/connections/sources/#what-is-a-source) is a website, server library, mobile SDK, or cloud application which can send data into Segment. In Reverse ETL, your data warehouse is the source.
Expand Down Expand Up @@ -95,7 +96,7 @@ To add your first destination:
After you’ve added a destination, you can create mappings from your warehouse to the destination. Mappings enable you to map the data you extract from your warehouse to the fields in your destination.

To create a mapping:
1. Navigate to **Conections > Destinations** and select the **Reverse ETL** tab.
1. Navigate to **Connections > Destinations** and select the **Reverse ETL** tab.
2. Select the destination that you want to create a mapping for.
3. Click **Add Mapping**.
4. Select the model to sync from.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
---
title: Configure Consent Management
plan: consent-management
related:
- "/privacy/consent-management/"
- "/privacy/consent-management/consent-in-segment-connections/"
- "/privacy/consent-management/consent-in-unify/"
redirect_from: "/privacy/configure-consent-management"
---

Expand All @@ -26,7 +22,7 @@ Before you can configure consent in Segment, take the following steps:
## Step 1: Create consent categories in the Segment app

> info "Limited availability of destinations"
> AWS S3, RETL, and Engage destinations do not enforce consent preferences.
> AWS S3 and Engage destinations do not enforce consent preferences.

1. From the [Segment homepage](https://app.segment.com/goto-my-workspace/){:target="_blank”}, select the Privacy tab and click **Consent Management**.
2. On the Consent management page, click **Create categories**.
Expand All @@ -35,8 +31,7 @@ Before you can configure consent in Segment, take the following steps:
- **Category name**: Enter a name that describes your use case for the data sent to this destination. This field only accepts category names that are 20 characters or less.
- **Category ID**: In OneTrust, this is a string of up to five alphanumeric characters, but other CMPs may have a different format. This field is case sensitive.
- **Mapped destinations**: Select one or more of your destinations to map to this category. Category mappings apply to all instances of a destination.
<br/><br/>**Optional**: Click **Add category** to create another category.
5. Once you've finished setting up your category or categories, click **Save**.
5. After you've finished setting up your category or categories, click **Save**.

> warning "Segment recommends mapping all destinations to a category"
> Segment assumes all destinations without a mapping do not require user consent and will receive all events containing a consent object. If a destination is mapped to multiple categories, a user must consent to all categories for data to flow to the destination.
Expand Down
153 changes: 153 additions & 0 deletions src/privacy/consent-management/consent-in-retl.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
---
title: Consent in Reverse ETL
plan: consent-management
hidden: true
---

With Consent Management in Reverse ETL, you can enforce your end-users' consent preferences that are captured by your consent management platform (CMP) and stored in your warehouse.

To enforce consent stored in your warehouse, build a Reverse ETL model that identifies consent categories. You can create a "consent to" column mapping in a new data model or update an existing data model to include a "consent to" mapping.

> info "Consent in Reverse ETL supports Reverse ETL-supported Actions destinations and Segment Connections"
> At this time, Consent in Reverse ETL does not support adding consent to Segment Profiles using the Segment Profiles destination. To enforce consent data in your classic Segment destinations, use the [Segment Connections destination](/docs/connections/destinations/catalog/actions-segment/).

## Prerequisites

> info "Consent management edit and update capabilities limited to Workspace Owners"
> Only users with the Workspace Owner role are able to create, edit, and disable consent categories. All other users have read-only access to Consent Management features.

Before you can enforce consent stored in your warehouse, take the following steps:
- **Set up your third-party consent management tool and create consent categories**. Take note of your consent categories and the key or ID associated with each category.
- **Know how your company uses each destination**. You need to know which destinations to map to each category.
- **Store your end user consent in a warehouse that [Segment supports for Reverse ETL](/docs/connections/reverse-etl/#step-1-add-a-source)**. Segment supports Reverse ETL capabilities in Azure, BigQuery, Databricks, Postgres, Snowflake, and Redshift data warehouses. Other data warehouses are not supported.

## Step 1: Create consent categories in the Segment app

> info "Limited availability of destinations"
> Reverse ETL supports the Actions destinations in the [Reverse ETL catalog](/docs/connections/reverse-etl/reverse-etl-catalog/) and [Segment Connections](/docs/connections/destinations/catalog/actions-segment/).

1. From the [Segment homepage](https://app.segment.com/goto-my-workspace/){:target="_blank”}, select the Privacy tab and click **Consent Management**.
2. On the Consent management page, click **Create categories**.
3. Confirm that you have completed the required prerequisites, and click **Next**.
4. On the Create consent categories page, add the following information to the category form:
- **Category name**: Enter a name that describes your use case for the data sent to this destination. This field only accepts category names that are 20 characters or less.
- **Category ID**: In OneTrust, this is a string of up to five alphanumeric characters, but other CMPs may have a different format. This field is case sensitive.
- **Mapped destinations**: Select one or more of your Reverse ETL destinations to map to this category. Category mappings apply to all instances of a destination.
5. After you've finished setting up your category or categories, click **Save**.

> warning "Segment recommends mapping all Reverse ETL destinations to a category"
> Segment assumes all destinations without a mapping do not require user consent and will receive all events containing a consent object. If a destination is mapped to multiple categories, a user must consent to all categories for data to flow to the destination.

To edit or disable consent categories, view the [Configure Consent Management](/docs/privacy/consent-management/configure-consent-management/) documentation.

## Step 2: Add your Reverse ETL source

> success ""
> If you've already added a [Reverse ETL source](/docs/connections/reverse-etl/#step-1-add-a-source) to your workspace, you can proceed to [Step 3: Identify consent columns](#step-3-identify-consent-columns).

If you haven't already configured a Reverse ETL source in your workspace, follow the instructions in the [Reverse ETL: Add a source](/docs/connections/reverse-etl/#step-1-add-a-source) documentation to add your warehouse as a data source. When you've configured your Reverse ETL source, proceed to [Step 3: Identify consent columns](#step-3-identify-consent-columns).

## Step 3: Identify consent columns

After you set up consent categories in the Segment app, you must identify the columns in your data warehouse that store end user consent by creating a *model*, or SQL query that defines the set of data you want to synchronize to your Reverse ETL destinations. When building your data model, Segment recommends that you represent consent as a boolean `true` or `false` value and map one consent category to one column.

> error "Creating a data model that does not include information about consent preferences results in no consent enforcement"
> If you create consent categories in your workspace but fail to identify columns that contain consent preferences in your data model, events flow to all destinations in your workspace regardless of end user consent preferences.

### Identify consent when building your model
To identify consent when building your model:
1. Navigate to **Connections > Sources** and select the Reverse ETL tab. Select your source and click **Add Model**.
2. Click **SQL Editor** as your modeling method.
3. Create the SQL query that’ll define your model. Your model is used to map data to your Reverse ETL destinations.
4. Choose a column to use as the unique identifier for each record in the Unique Identifier column field.
The Unique Identifier should be a column with unique values per record to ensure checkpointing works as expected. It can be a primary key. This column is used to detect new, updated, and deleted records.
5. Click **Preview** to see a preview of the results of your SQL query. The data from the preview is extracted from the first 10 records of your warehouse.
6. Click **Next**.
7. Enter your Model Name.
8. Click **Create Model**.
9. Select **Add consent mapping**.
10. On the **Add consent mapping** popup, identify the column in your model that holds the consent preferences for the consent category.
11. Select **Add consent mapping** to identify columns for all of your consent categories.
12. When you're satisfied with your consent mappings, click **Save**.

### Update your Reverse ETL mode to include consent
To update an existing Reverse ETL model to include consent enforcement:
1. Navigate to **Connections > Destinations** and select the **Reverse ETL** tab.
2. Select the source and the model you want to edit.
3. Select the **Query Builder** tab to edit your query. When you're editing your query, include columns that store information about end user consent preferences. When you've finished making changes, click **Save Query**.
4. Navigate to **Settings > Consent settings**.
5. Select **Add consent mapping**.
6. On the **Add consent mapping** popup, identify the column in your model that holds the consent preferences for the consent category.
7. Select **Add consent mapping** to identify columns for all of your consent categories.
8. When you're satisfied with your consent mappings, click **Save**.

You can select the **Settings** tab and click **Consent settings** to verify that the consent categories in your model match the consent categories you configured in your workspace.

You can store each consent category in its own column in your warehouse, or store your consent information in one single blob column. Segment requires your consent categories to be in their own column in your data model.

The following sample model maps consent categories from each column in your database:

``` sql
select
USERID,
name,
email,
distinctid,
Ads,
Personalization,
Analytics,

from CONSENT_PREFERENCES;
```

The following sample model maps consent categories from one blob column in your database:

```sql
select
USERID,
name,
email,
distinctid,
CAST(CONSENT_OBJ:consent.cookie.Advertising as Boolean) as Ads,
CAST(CONSENT_OBJ:consent.cookie.Personalization as Boolean) as Personalization,
CAST(CONSENT_OBJ:consent.cookie.Analytics as Boolean) as Analytics,

from CONSENT_PREFERENCES;
```

> warning "Failing to identify consent columns in your warehouse might lead to unintentional data loss"
> If you have destinations mapped to consent categories in the Segment app but fail to identify a column in your warehouse that stores consent for a category, then consent preference for that category will be considered to be false and **no data will flow to destinations mapped to the category**.

## Step 4: Connect your downstream destinations

After you set up categories in the Segment app and create a SQL model that extracts consent information, connect your downstream destinations to complete the consent enforcement process.

> info "Consent in Reverse ETL supports Reverse ETL-supported Actions destinations and Segment Connections"
> At this time, Consent in Reverse ETL does not support enforcing consent in the Segment Profiles destination. To enforce consent data in your classic Segment destinations, use the [Segment Connections destination](/docs/connections/destinations/catalog/actions-segment/).

To add your first destination:
1. Navigate to **Connections > Destinations** and select the **Reverse ETL** tab.
2. Click **Add Reverse ETL destination**.
3. Select the destination you want to connect to and click **Configure**.
4. Select the Reverse ETL source you want to connect the destination to.
5. Enter the **Destination name** and click **Create Destination**.
6. Enter the required information on the **Settings** tab of the destination.
7. Navigate to the destination settings tab and enable the destination. If the destination is disabled, then Segment won't be able to start a sync.

## Validate your consent mapping

You can validate that you successfully created your consent mapping in Segment Connections or supported Reverse ETL Actions destinations using the following methods.

### Segment Connections destination

Segment automatically adds the [consent object](/docs/privacy/consent-management/consent-in-segment-connections/#consent-object) to every event that's routed downstream to your Segment Connections destination. [Consent enforcement in Connections](/docs/privacy/consent-management/consent-in-segment-connections/) validates that only consenting data flows downstream to any classic Segment destinations connected to your Segment Connections instance.

Open the Source Debugger for your Reverse ETL source and confirm that the [consent object](/docs/privacy/consent-management/consent-in-segment-connections/#consent-object) appears on every event and that the consent object has the categories you mapped in [Step 2: Identify consent columns](#step-2-identify-consent-columns).

<!--- ### Segment Profiles
Open the [Profile Explorer](/docs/unify/#profile-explorer) and verify that your profiles contain the [Segment Consent Preference Updated](/docs/privacy/consent-management/consent-in-unify/#segment-consent-preference-updated-event) event. --->

### Reverse ETL Actions destinations
Segment automatically filters out data from users who have not consented to the category mapped to your destination.

To verify that this behavior is working as intended, open [Delivery Overview](/docs/connections/delivery-overview) for a RETL-supported Actions destination and view the events that were successfully delivered to the destination. The events in your destination should only come from users that consented to send data to the category that your supported Actions destination belongs to.
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
---
title: Consent in Segment Connections
plan: consent-management
related:
- "/privacy/consent-management/"
- "/privacy/consent-management/configure-consent-management/"
- "/privacy/consent-management/consent-in-unify/"
redirect_from: "/privacy/consent-in-segment-connections"
---

Expand Down
4 changes: 0 additions & 4 deletions src/privacy/consent-management/consent-in-unify.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
---
title: Consent in Unify
plan: consent-management
related:
- "/privacy/consent-management/"
- "/privacy/consent-management/configure-consent-management/"
- "/privacy/consent-management/consent-in-segment-connections/"
redirect_from: "/privacy/consent-in-unify"
---

Expand Down
4 changes: 0 additions & 4 deletions src/privacy/consent-management/index.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
---
title: Consent Management Overview
plan: consent-management
related:
- "/privacy/consent-management/configure-consent-management/"
- "/privacy/consent-management/consent-in-segment-connections/"
- "/privacy/consent-management/consent-in-unify/"
---

When an end user visits your web or mobile app, they set **consent preferences**, or make decisions about the types of data they want you to collect, use, and share. These consent preferences are typically presented as a set list of categories that describe how your company intends to use that data. Some common categories include personalization, advertising, and site performance.
Expand Down