Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add postgres trigger captures #5165

Merged
merged 55 commits into from
Feb 11, 2025
Merged
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
ab0ad5d
feat: :construction: add postgres trigger config in back
dieriba Jan 28, 2025
c53a316
feat: add capture section for postgres
dieriba Jan 28, 2025
d5ba797
feat: update postgres trigger panel, update capture migration
dieriba Jan 29, 2025
2ee63b9
fix: ci and casing style
dieriba Jan 29, 2025
9826e2d
feat: capture done
dieriba Jan 29, 2025
716685f
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Jan 29, 2025
6e0d953
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Jan 29, 2025
bde5dee
fix: update sqlx
dieriba Jan 29, 2025
0ee9f7d
fix: remove unused
dieriba Jan 29, 2025
f776633
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Jan 29, 2025
943fe48
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Jan 30, 2025
bd99865
feat: handle persisting data
dieriba Feb 1, 2025
e539ad9
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 1, 2025
93063d4
fix: add feature flag
dieriba Feb 1, 2025
cf258fc
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 1, 2025
85db473
feat: capture done
dieriba Feb 1, 2025
eebc043
fix: feature flag
dieriba Feb 1, 2025
19d85cc
feat: capture done
dieriba Feb 3, 2025
995a152
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 3, 2025
d2056ed
nits: add plus into postgres capture label
dieriba Feb 3, 2025
2273a91
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 3, 2025
4a1d935
feat: trigger done
dieriba Feb 3, 2025
4593353
Update PostgresTriggerEditorInner.svelte
dieriba Feb 4, 2025
018a283
Update script_helpers.ts
dieriba Feb 4, 2025
cf6e6af
Merge branch 'main' into feat/postgres-trigger-captures
HugoCasa Feb 4, 2025
c514c7d
nit
HugoCasa Feb 4, 2025
2532492
chore: update to meet pr comment requirement
dieriba Feb 4, 2025
026dabc
refactor:
dieriba Feb 5, 2025
ca93efe
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 5, 2025
09e11ab
fix:
dieriba Feb 5, 2025
2ba31ce
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 5, 2025
77e9644
fix: miss closing curly braces
dieriba Feb 5, 2025
cf0c2dc
fix:
dieriba Feb 5, 2025
61b3a73
fix: ci
dieriba Feb 5, 2025
0b02eb8
Update mod.rs
dieriba Feb 5, 2025
507fe30
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 5, 2025
8d5ca59
polishing postgres trigger
Guilhem-lm Feb 6, 2025
c87f5b9
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 6, 2025
90c9a13
Merge branch 'glm/postgres_trigger_polishing' into feat/postgres-trig…
dieriba Feb 6, 2025
d0759d8
update minor changes on UI
dieriba Feb 6, 2025
f9c4894
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 6, 2025
1d9a691
nits: remove log and update function parameters
dieriba Feb 6, 2025
5ecfba6
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 7, 2025
10fc00d
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 9, 2025
a7b432a
fix: remove wrong name
dieriba Feb 9, 2025
e040751
nits: change icon
dieriba Feb 9, 2025
00036b5
Merge branch 'main' into feat/postgres-trigger-captures
HugoCasa Feb 10, 2025
cd521bc
fix: nit migration
HugoCasa Feb 10, 2025
54e5b85
nits: refacto and remove alert
dieriba Feb 10, 2025
3fb5fd2
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 10, 2025
d2ee988
nits: change variable name
dieriba Feb 10, 2025
c41228a
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 10, 2025
1c9cbc0
update workspaces_export
dieriba Feb 10, 2025
be5427a
fix: ci
dieriba Feb 11, 2025
1da8b15
Merge branch 'main' into feat/postgres-trigger-captures
dieriba Feb 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat: update postgres trigger panel, update capture migration
  • Loading branch information
dieriba committed Jan 29, 2025
commit d5ba797fde67325a6897daf91b96286f6e7e5d8e
2 changes: 1 addition & 1 deletion backend/migrations/20250102145420_more_captures.up.sql
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Add up migration script here
CREATE TYPE TRIGGER_KIND AS ENUM ('webhook', 'http', 'websocket', 'kafka', 'email');
CREATE TYPE TRIGGER_KIND AS ENUM ('webhook', 'http', 'websocket', 'kafka', 'email', 'postgres');
ALTER TABLE capture ADD COLUMN is_flow BOOLEAN NOT NULL DEFAULT TRUE, ADD COLUMN trigger_kind TRIGGER_KIND NOT NULL DEFAULT 'webhook', ADD COLUMN trigger_extra JSONB;
ALTER TABLE capture ALTER COLUMN is_flow DROP DEFAULT, ALTER COLUMN trigger_kind DROP DEFAULT;
ALTER TABLE capture DROP CONSTRAINT capture_pkey;
7 changes: 3 additions & 4 deletions backend/windmill-api/src/capture.rs
Original file line number Diff line number Diff line change
@@ -77,7 +77,7 @@ pub fn workspaced_unauthed_service() -> Router {
}
}

#[derive(sqlx::Type, Serialize, Deserialize)]
#[derive(sqlx::Type, Serialize, Deserialize, Debug)]
#[sqlx(type_name = "TRIGGER_KIND", rename_all = "lowercase")]
#[serde(rename_all = "lowercase")]
pub enum TriggerKind {
@@ -150,7 +150,7 @@ pub struct WebsocketTriggerConfig {
pub url_runnable_args: Option<serde_json::Value>,
}

#[derive(Serialize, Deserialize)]
#[derive(Serialize, Deserialize, Debug)]
#[serde(untagged)]
enum TriggerConfig {
#[cfg(feature = "http_trigger")]
@@ -165,7 +165,7 @@ enum TriggerConfig {
Nats(NatsTriggerConfig),
}

#[derive(Serialize, Deserialize)]
#[derive(Serialize, Deserialize, Debug)]
struct NewCaptureConfig {
trigger_kind: TriggerKind,
path: String,
@@ -212,7 +212,6 @@ async fn set_config(
Json(nc): Json<NewCaptureConfig>,
) -> Result<()> {
let mut tx = user_db.begin(&authed).await?;

sqlx::query!(
"INSERT INTO capture_config
(workspace_id, path, is_flow, trigger_kind, trigger_config, owner, email)
34 changes: 34 additions & 0 deletions backend/windmill-api/src/postgres_triggers/handler.rs
Original file line number Diff line number Diff line change
@@ -112,6 +112,40 @@ pub struct NewPostgresTrigger {
publication: Option<PublicationData>,
}

#[derive(Serialize, Deserialize)]
pub struct TestPostgres {
pub postgres_resource_path: String,
}

pub async fn test_postgres_connection(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Extension(user_db): Extension<UserDB>,
Path(workspace_id): Path<String>,
Json(test_postgres): Json<TestPostgres>,
) -> error::Result<()> {
let connect_f = async {
get_database_connection(
authed,
Some(user_db),
&db,
&test_postgres.postgres_resource_path,
&workspace_id,
)
.await
.map_err(|err| {
error::Error::BadConfig(format!("Error connecting to postgres: {}", err.to_string()))
})
};
tokio::time::timeout(tokio::time::Duration::from_secs(30), connect_f)
.await
.map_err(|_| {
error::Error::BadConfig(format!("Timeout connecting to websocket after 30 seconds"))
})??;

Ok(())
}

pub async fn get_database_connection(
authed: ApiAuthed,
user_db: Option<UserDB>,
4 changes: 3 additions & 1 deletion backend/windmill-api/src/postgres_triggers/mod.rs
Original file line number Diff line number Diff line change
@@ -16,7 +16,8 @@ use handler::{
create_template_script, delete_postgres_trigger, delete_publication, drop_slot_name,
exists_postgres_trigger, get_postgres_trigger, get_publication_info, get_template_script,
is_database_in_logical_level, list_database_publication, list_postgres_triggers,
list_slot_name, set_enabled, update_postgres_trigger, Database, PostgresTrigger,
list_slot_name, set_enabled, test_postgres_connection, update_postgres_trigger, Database,
PostgresTrigger,
};
use windmill_common::{db::UserDB, error::Error, utils::StripPath};
use windmill_queue::PushArgsOwned;
@@ -86,6 +87,7 @@ fn slot_service() -> Router {

pub fn workspaced_service() -> Router {
Router::new()
.route("/test", post(test_postgres_connection))
.route("/create", post(create_postgres_trigger))
.route("/list", get(list_postgres_triggers))
.route("/get/*path", get(get_postgres_trigger))
2 changes: 2 additions & 0 deletions frontend/src/lib/components/triggers.ts
Original file line number Diff line number Diff line change
@@ -65,6 +65,8 @@ export function captureTriggerKindToTriggerKind(kind: CaptureTriggerKind): Trigg
return 'kafka'
case 'nats':
return 'nats'
case 'postgres':
return 'postgres'
default:
throw new Error(`Unknown CaptureTriggerKind: ${kind}`)
}
11 changes: 10 additions & 1 deletion frontend/src/lib/components/triggers/CaptureButton.svelte
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<script lang="ts">
import Popover from '$lib/components/meltComponents/Popover.svelte'
import { Button } from '$lib/components/common'
import { Webhook, Route, Unplug, Mail, Plus } from 'lucide-svelte'
import { Webhook, Route, Unplug, Mail, Plus, Database } from 'lucide-svelte'
import KafkaIcon from '$lib/components/icons/KafkaIcon.svelte'
import { enterpriseLicense } from '$lib/stores'
import { type CaptureTriggerKind } from '$lib/gen'
@@ -82,6 +82,15 @@
<p class="text-xs text-secondary">Websocket</p>
</div>
</button>
<button
class="hover:bg-surface-hover p-2 transition-colors duration-150"
on:click={() => handleClick('postgres')}
>
<div class="flex flex-row items-center gap-2">
<Database size={16} />
<p class="text-xs text-secondary">Postgres</p>
</div>
</button>
<button
class="hover:bg-surface-hover p-2 transition-colors duration-150"
on:click={() => handleClick('email')}
32 changes: 29 additions & 3 deletions frontend/src/lib/components/triggers/CaptureWrapper.svelte
Original file line number Diff line number Diff line change
@@ -14,6 +14,7 @@
import type { CaptureInfo } from './CaptureSection.svelte'
import CaptureTable from './CaptureTable.svelte'
import NatsTriggersConfigSection from './nats/NatsTriggersConfigSection.svelte'
import PostgresEditorConfigSection from './postgres/PostgresEditorConfigSection.svelte'

export let isFlow: boolean
export let path: string
@@ -55,7 +56,10 @@
return acc
}, {})

if ((captureType === 'websocket' || captureType === 'kafka') && captureActive) {
if (
(captureType === 'postgres' || captureType === 'websocket' || captureType === 'kafka') &&
captureActive
) {
const config = captureConfigs[captureType]
if (config && config.error) {
const serverEnabled = getServerEnabled(config)
@@ -123,12 +127,18 @@
$: config = captureConfigs[captureType]

let cloudDisabled =
(captureType === 'websocket' || captureType === 'kafka' || captureType === 'nats') &&
(captureType === 'postgres' ||
captureType === 'websocket' ||
captureType === 'kafka' ||
captureType === 'nats') &&
isCloudHosted()

function updateConnectionInfo(config: CaptureConfig | undefined, captureActive: boolean) {
if (
(captureType === 'websocket' || captureType === 'kafka' || captureType === 'nats') &&
(captureType === 'postgres' ||
captureType === 'websocket' ||
captureType === 'kafka' ||
captureType === 'nats') &&
config &&
captureActive
) {
@@ -183,6 +193,22 @@
}}
on:testWithArgs
/>
{:else if captureType === 'postgres'}
<PostgresEditorConfigSection
bind:postgres_resource_path={args.postgres_resource_path}
bind:relations={args.relations}
bind:transaction_to_track={args.transaction_to_track}
{showCapture}
{captureInfo}
bind:captureTable
on:applyArgs
on:updateSchema
on:addPreprocessor
on:captureToggle={() => {
handleCapture()
}}
on:testWithArgs
/>
{:else if captureType === 'webhook'}
<WebhooksConfigSection
{isFlow}
Original file line number Diff line number Diff line change
@@ -16,7 +16,8 @@
const kindToName: { [key: string]: string } = {
websocket: 'Websocket',
nats: 'NATS server(s)',
kafka: 'Kafka broker(s)'
kafka: 'Kafka broker(s)',
postgres: 'Postgres'
}

let testLoading: boolean = false
@@ -44,8 +45,7 @@
workspace: $workspaceStore!,
requestBody: args as any
})
}
else if (kind === 'postgres') {
} else if (kind === 'postgres') {
promise = PostgresTriggerService.testPostgresConnection({
workspace: $workspaceStore!,
requestBody: args as any
@@ -54,7 +54,6 @@
await promise
sendUserToast(`Successfully connected to ${kindToName[kind]}`)
} catch (err) {

if (!promise?.isCancelled) {
sendUserToast(`Error testing ${kindToName[kind]}: ${err?.body ?? 'Unknown error'}`, true)
}
13 changes: 12 additions & 1 deletion frontend/src/lib/components/triggers/TriggersEditor.svelte
Original file line number Diff line number Diff line change
@@ -140,7 +140,18 @@
</div>
{:else if $selectedTrigger === 'postgres'}
<div class="p-4">
<PostgresTriggersPanel {newItem} path={currentPath} {isFlow} />
<PostgresTriggersPanel
on:applyArgs
on:addPreprocessor
on:updateSchema
on:testWithArgs
{newItem}
path={currentPath}
{isFlow}
{canHavePreprocessor}
{hasPreprocessor}
isEditor={true}
/>
</div>
{:else if $selectedTrigger === 'kafka' || $selectedTrigger === 'nats'}
<div class="p-4 flex flex-col gap-2">
Original file line number Diff line number Diff line change
@@ -28,7 +28,8 @@
webhook: 'Webhook',
kafka: 'New kafka trigger',
email: 'Email trigger',
nats: 'NATS trigger'
nats: 'NATS trigger',
postgres: 'New postgres trigger'
}

const { captureOn } = getContext<TriggerContext>('TriggerContext')
8 changes: 8 additions & 0 deletions frontend/src/lib/components/triggers/TriggersWrapper.svelte
Original file line number Diff line number Diff line change
@@ -8,6 +8,7 @@
import EmailTriggerConfigSection from '../details/EmailTriggerConfigSection.svelte'
import KafkaTriggersConfigSection from './kafka/KafkaTriggersConfigSection.svelte'
import NatsTriggersConfigSection from './nats/NatsTriggersConfigSection.svelte'
import PostgresEditorConfigSection from './postgres/PostgresEditorConfigSection.svelte'

export let triggerType: CaptureTriggerKind = 'webhook'
export let cloudDisabled: boolean = false
@@ -30,6 +31,13 @@
bind:url_runnable_args={args.url_runnable_args}
showCapture={false}
/>
{:else if triggerType === 'postgres'}
<PostgresEditorConfigSection
bind:relations={args.relations}
bind:transaction_to_track={args.transaction_to_track}
bind:postgres_resource_path={args.postgres_resource_path}
showCapture={false}
/>
{:else if triggerType === 'webhook'}
<WebhooksConfigSection
{isFlow}
Original file line number Diff line number Diff line change
@@ -1,19 +1,30 @@
<script lang="ts">
import ResourcePicker from '$lib/components/ResourcePicker.svelte'
import Section from '$lib/components/Section.svelte'
import ToggleButtonGroup from '$lib/components/common/toggleButton-v2/ToggleButtonGroup.svelte'
import ToggleButton from '$lib/components/common/toggleButton-v2/ToggleButton.svelte'
import CaptureSection, { type CaptureInfo } from '../CaptureSection.svelte'
import CaptureTable from '../CaptureTable.svelte'
import TestTriggerConnection from '../TestTriggerConnection.svelte'
import Required from '$lib/components/Required.svelte'
import MultiSelect from 'svelte-multiselect'
import RelationPicker from './RelationPicker.svelte'
import type { Relations } from '$lib/gen'
import { emptyString } from '$lib/utils'

export let url: string | undefined
export let url_runnable_args: Record<string, unknown> | undefined
let transactionType: string[] = ['Insert', 'Update', 'Delete']
let selectedTable: 'all' | 'specific' = 'specific'
export let transaction_to_track: string[] = ['Insert', 'Update', 'Delete']
export let relations: Relations[] = []
export let headless: boolean = false
export let showCapture: boolean = false
export let captureTable: CaptureTable | undefined = undefined
export let captureInfo: CaptureInfo | undefined = undefined
export let isValid: boolean = false
let postgres_resource_path: string = ''
export let postgres_resource_path: string = ''
$: isValid =
!emptyString(postgres_resource_path) &&
transaction_to_track?.length > 0 &&
(selectedTable === 'all' || table_to_track?.length > 0)
$: table_to_track = selectedTable === 'all' ? [] : relations
</script>

<div>
@@ -30,19 +41,46 @@
bind:captureTable
/>
{/if}
<Section label="Postgres" {headless}>
<div class="mb-2">
<ToggleButtonGroup
selected={url?.startsWith('$') ? 'runnable' : 'static'}
on:selected={(ev) => {
url = ev.detail === 'runnable' ? '$script:' : ''
url_runnable_args = {}
}}
>
<ToggleButton value="static" label="Static URL" />
<ToggleButton value="runnable" label="Runnable result as URL" />
</ToggleButtonGroup>
<Section label="Postgres Config" {headless}>
<div class="flex flex-col gap-3">
<div class="mb-2">
<p class="text-xs mb-1 text-tertiary">
Pick a database to connect to <Required required={true} />
</p>
<ResourcePicker bind:value={postgres_resource_path} resourceType={'postgresql'} />
{#if postgres_resource_path}
<TestTriggerConnection kind="postgres" args={{ postgres_resource_path }} />
{/if}
</div>
<Section label="Transactions">
<p class="text-xs mb-3 text-tertiary">
Choose the types of database transactions that should trigger a script or flow. You can
select from <strong>Insert</strong>, <strong>Update</strong>,
<strong>Delete</strong>, or any combination of these operations to define when the trigger
should activate.
</p>
<MultiSelect
ulOptionsClass={'!bg-surface-secondary'}
noMatchingOptionsMsg=""
createOptionMsg={null}
duplicates={false}
bind:value={transaction_to_track}
options={transactionType}
allowUserOptions="append"
bind:selected={transaction_to_track}
/>
</Section>
<Section label="Table Tracking">
<p class="text-xs mb-3 text-tertiary">
Select the tables to track. You can choose to track
<strong>all tables in your database</strong>,
<strong>all tables within a specific schema</strong>,
<strong>specific tables in a schema</strong>, or even
<strong>specific columns of a table</strong>. Additionally, you can apply a
<strong>filter</strong> to retrieve only rows that do not match the specified criteria.
</p>
<RelationPicker bind:selectedTable bind:relations />
</Section>
</div>
<TestTriggerConnection kind="postgres" args={{ postgres_resource_path }} />
</Section>
</div>
Original file line number Diff line number Diff line change
@@ -9,10 +9,10 @@
drawer?.openEdit(ePath, isFlow)
}

export async function openNew(is_flow: boolean, initial_script_path?: string) {
export async function openNew(is_flow: boolean, initial_script_path?: string, defaultValues?: Record<string, any>) {
open = true
await tick()
drawer?.openNew(is_flow, initial_script_path)
drawer?.openNew(is_flow, initial_script_path, defaultValues)
}

let drawer: PostgresTriggerEditorInner
Loading