feat(data-warehouse): add salesforce integration #23212

EDsCODE · 2024-06-24T22:49:52Z

Problem

no salesforce integration

Changes

adjusts the incremental fields api so that if there are no incremental fields, the set up button still works
adds salesforce integration flow
had to modify the onboarding flow slightly because for salesforce, a team needs to give us their subdomain first

TODO:

decide on how to import custom columns too , right now it's all standard columns. Custom columns require you to provide a limit
set client app id and secret in charts

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

How did you test this code?

github-actions · 2024-06-24T23:00:17Z

Size Change: 0 B

Total Size: 1.07 MB

ℹ️ View Unchanged

Filename	Size
`frontend/dist/toolbar.js`	1.07 MB

_{compressed-size-action}

EDsCODE · 2024-06-25T20:42:39Z

posthog/temporal/data_imports/pipelines/salesforce/__init__.py

+                    "q": {
+                        "type": "incremental",
+                        "cursor_path": "SystemModstamp",
+                        "initial_value": "SELECT FIELDS(STANDARD) FROM Account WHERE SystemModstamp > 2000-01-01T00:00:00.000+0000",


Unclear how to do incremental with this rest query format. I tried to do this but initial_value needs to match the cursor_path type i believe and cursor_path in this case is a datetime.

Oddly enough, the data does return but dlt doesn't write it

@Gilbert09

Hey @EDsCODE,

initial_value needs to match the cursor_path type i believe and cursor_path in this case is a datetime.

This is true. We're going to allow data conversion for the incremental params, here's the issue: dlt-hub/verified-sources#507

Please take a look to see if the proposed solution fits. Do we miss anything?

Yep, having a custom function to map the types/parameters would be great.

I think the suggestion in the issue would be enough for my case as we would be receiving a timestamp field and I'd be transforming it into the full query to pass back in

@EDsCODE the PR with this feature has been merged: dlt-hub/verified-sources#515
Please give it a go. We'll document it soon as well.

posthog-bot · 2024-07-11T07:30:54Z

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in another week.

posthog-bot · 2024-07-18T07:31:03Z

This PR was closed due to lack of activity. Feel free to reopen if it's still relevant.

…g into dw-salesforce-integration

posthog-bot · 2024-08-08T22:23:27Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 1, diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-08-08T22:42:08Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

Gilbert09

Rest source stuff mostly looks good - need to figure out refresh tokens and maybe make the UI changes through config

Gilbert09 · 2024-08-12T13:54:58Z

frontend/src/scenes/data-warehouse/new/sourceWizardLogic.tsx

+        addToSalesforceButtonUrl: [
+            (s) => [s.preflight],
+            (preflight) => {
+                return (subdomain: string) => {
+                    const clientId = preflight?.data_warehouse_integrations?.salesforce.client_id
+
+                    if (!clientId) {
+                        return null
+                    }
+
+                    const params = new URLSearchParams()
+                    params.set('client_id', clientId)
+                    params.set('redirect_uri', `${window.location.origin}/data-warehouse/salesforce/redirect`)
+                    params.set('response_type', 'code')
+                    params.set('scope', 'refresh_token api')
+                    params.set('state', subdomain)
+
+                    return `https://${subdomain}.my.salesforce.com/services/oauth2/authorize?${params.toString()}`
+                }
+            },
+        ],


Nice to have: to use Bens new oauth stuff he added - although there is certainly a larger piece of work around how we do credentials, so may not be one for now. e.g. storing S3 details, or Stripe connection details, etc.

frontend/src/scenes/data-warehouse/new/NewSourceWizard.tsx

frontend/src/scenes/data-warehouse/new/sourceWizardLogic.tsx

posthog/temporal/data_imports/pipelines/salesforce/auth.py

posthog/temporal/data_imports/pipelines/salesforce/helpers.py

posthog/temporal/data_imports/workflow_activities/import_data.py

Gilbert09 · 2024-08-12T14:53:29Z

requirements.in

@@ -59,7 +59,7 @@ posthoganalytics==3.5.0
 psycopg2-binary==2.9.7
 PyMySQL==1.1.1
 psycopg[binary]==3.1.18
-pyarrow==15.0.0
+pyarrow==17.0.0


Any breaking changes with 2 new major versions of pyarrow?

Not on dw end. DLT with delta requires it now

@tomasfarias note here incase of batch exports impact

Removed these updates altogether. Realized they weren't needed. Also, note that the DLT version upgrade causes a test to fail. Errors about s3 configuring missing

posthog-bot · 2024-08-13T00:13:10Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (wasn't pushed!)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

…g into dw-salesforce-integration

posthog-bot · 2024-08-13T14:03:31Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-08-13T14:22:17Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-08-13T14:41:04Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

…g into dw-salesforce-integration

posthog-bot · 2024-08-13T16:06:43Z

📸 UI snapshots have been updated

4 snapshot changes in total. 0 added, 4 modified, 0 deleted:

chromium: 0 added, 4 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-08-13T16:25:17Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-08-13T17:43:29Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-08-13T18:43:27Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-08-13T19:01:54Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-08-13T19:20:34Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

This reverts commit 7131400.

add boilerplate

33c6b20

EDsCODE added 3 commits June 25, 2024 11:08

Merge branch 'master' into dw-salesforce-integration

6382aa1

working

d612887

incremental

f3bc292

EDsCODE commented Jun 25, 2024

View reviewed changes

github-actions bot and others added 2 commits June 25, 2024 20:56

Update query snapshots

bd17653

use verified source

71d5473

posthog-bot added the stale label Jul 11, 2024

posthog-bot closed this Jul 18, 2024

EDsCODE reopened this Aug 1, 2024

EDsCODE removed the stale label Aug 1, 2024

EDsCODE and others added 5 commits August 8, 2024 18:03

merge base

10e89c4

Merge branch 'dw-salesforce-integration' of github.com:PostHog/postho…

1ac7d80

…g into dw-salesforce-integration

Update query snapshots

3d9011c

Update UI snapshots for chromium (1)

4a30dcf

Update UI snapshots for chromium (2)

f5bbe03

Update UI snapshots for chromium (1)

bfa8c20

EDsCODE added 4 commits August 9, 2024 10:48

Merge branch 'master' into dw-salesforce-integration

1df1a51

merge main

c94868a

new stub

ead1d27

incremental working

23011d2

EDsCODE marked this pull request as ready for review August 12, 2024 03:48

Gilbert09 approved these changes Aug 12, 2024

View reviewed changes

EDsCODE added 2 commits August 12, 2024 11:56

typing

7f932d0

use convert

5ae3dba

Gilbert09 approved these changes Aug 13, 2024

View reviewed changes

EDsCODE and others added 3 commits August 13, 2024 09:46

merge main

115b154

Merge branch 'dw-salesforce-integration' of github.com:PostHog/postho…

daa46d6

…g into dw-salesforce-integration

Update UI snapshots for chromium (1)

d3e0c0f

Update UI snapshots for chromium (1)

6e89305

Update UI snapshots for chromium (1)

8a6d5e8

EDsCODE and others added 3 commits August 13, 2024 11:46

Merge main

97a8128

Merge branch 'dw-salesforce-integration' of github.com:PostHog/postho…

7d2cd1d

…g into dw-salesforce-integration

Update UI snapshots for chromium (1)

36ea722

Update UI snapshots for chromium (1)

dd60148

EDsCODE and others added 3 commits August 13, 2024 12:52

Merge branch 'master' into dw-salesforce-integration

9e541c4

Merge branch 'master' into dw-salesforce-integration

7dd8274

Update UI snapshots for chromium (1)

5528bfd

Update UI snapshots for chromium (1)

21f2e27

Update UI snapshots for chromium (1)

69a6e35

Update UI snapshots for chromium (1)

5c273cc

EDsCODE merged commit 7131400 into master Aug 13, 2024
93 checks passed

EDsCODE deleted the dw-salesforce-integration branch August 13, 2024 21:30

EDsCODE added a commit that referenced this pull request Aug 14, 2024

Revert "feat(data-warehouse): add salesforce integration (#23212)"

c16a3b8

This reverts commit 7131400.

Gilbert09 mentioned this pull request Sep 5, 2024

One click Salesforce Data Warehouse integration #23208

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(data-warehouse): add salesforce integration #23212

feat(data-warehouse): add salesforce integration #23212

EDsCODE commented Jun 24, 2024 •

edited

Loading

github-actions bot commented Jun 24, 2024 •

edited

Loading

EDsCODE Jun 25, 2024 •

edited

Loading

burnash Jun 26, 2024

EDsCODE Jun 26, 2024

burnash Jul 3, 2024

posthog-bot commented Jul 11, 2024

posthog-bot commented Jul 18, 2024

posthog-bot commented Aug 8, 2024

posthog-bot commented Aug 8, 2024

Gilbert09 left a comment

Gilbert09 Aug 12, 2024

Gilbert09 Aug 12, 2024

EDsCODE Aug 12, 2024

EDsCODE Aug 12, 2024

posthog-bot commented Aug 13, 2024

posthog-bot commented Aug 13, 2024

posthog-bot commented Aug 13, 2024

posthog-bot commented Aug 13, 2024

posthog-bot commented Aug 13, 2024

posthog-bot commented Aug 13, 2024

posthog-bot commented Aug 13, 2024

posthog-bot commented Aug 13, 2024

posthog-bot commented Aug 13, 2024

posthog-bot commented Aug 13, 2024

feat(data-warehouse): add salesforce integration #23212

feat(data-warehouse): add salesforce integration #23212

Conversation

EDsCODE commented Jun 24, 2024 • edited Loading

Problem

Changes

Does this work well for both Cloud and self-hosted?

How did you test this code?

github-actions bot commented Jun 24, 2024 • edited Loading

EDsCODE Jun 25, 2024 • edited Loading

Choose a reason for hiding this comment

burnash Jun 26, 2024

Choose a reason for hiding this comment

EDsCODE Jun 26, 2024

Choose a reason for hiding this comment

burnash Jul 3, 2024

Choose a reason for hiding this comment

posthog-bot commented Jul 11, 2024

posthog-bot commented Jul 18, 2024

posthog-bot commented Aug 8, 2024

📸 UI snapshots have been updated

posthog-bot commented Aug 8, 2024

📸 UI snapshots have been updated

Gilbert09 left a comment

Choose a reason for hiding this comment

Gilbert09 Aug 12, 2024

Choose a reason for hiding this comment

Gilbert09 Aug 12, 2024

Choose a reason for hiding this comment

EDsCODE Aug 12, 2024

Choose a reason for hiding this comment

EDsCODE Aug 12, 2024

Choose a reason for hiding this comment

posthog-bot commented Aug 13, 2024

📸 UI snapshots have been updated

posthog-bot commented Aug 13, 2024

📸 UI snapshots have been updated

posthog-bot commented Aug 13, 2024

📸 UI snapshots have been updated

posthog-bot commented Aug 13, 2024

📸 UI snapshots have been updated

posthog-bot commented Aug 13, 2024

📸 UI snapshots have been updated

posthog-bot commented Aug 13, 2024

📸 UI snapshots have been updated

posthog-bot commented Aug 13, 2024

📸 UI snapshots have been updated

posthog-bot commented Aug 13, 2024

📸 UI snapshots have been updated

posthog-bot commented Aug 13, 2024

📸 UI snapshots have been updated

posthog-bot commented Aug 13, 2024

📸 UI snapshots have been updated

EDsCODE commented Jun 24, 2024 •

edited

Loading

github-actions bot commented Jun 24, 2024 •

edited

Loading

EDsCODE Jun 25, 2024 •

edited

Loading