Skip to content

fix: validate a filter's operator against the column type by evaluating it #2099

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hamodywe wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix: validate a filter's operator against the column type by evaluating it #2099

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6fa7061
fix: validate a filter's operator against the column type by evaluati…
hamodywe Aug 15, 2026
99a159a
Merge branch 'main' into fix/validate-filter-operator-at-subscribe-time
hamodywe Aug 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion lib/realtime/tenants/migrations.ex
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,8 @@ defmodule Realtime.Tenants.Migrations do
{20_260_626_120_000, Migrations.ReAddPostgrestFilterOps},
{20_260_706_120_000, Migrations.GrantCheckEqualityOp5Arg},
{20_260_707_120_000, Migrations.RestrictRealtimeSchema},
{20_260_709_120_000, Migrations.FixApplyRlsFilterRoleLeak}
{20_260_709_120_000, Migrations.FixApplyRlsFilterRoleLeak},
{20_260_815_140_000, Migrations.ValidateFilterOperatorAgainstColumnType}
]

defstruct [:tenant_external_id, :settings, migrations_ran: 0]
Expand Down
130 changes: 130 additions & 0 deletions ...me/tenants/repo/migrations/20260815140000_validate_filter_operator_against_column_type.ex
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
defmodule Realtime.Tenants.Migrations.ValidateFilterOperatorAgainstColumnType do
@moduledoc """
`subscription_check_filters` decided whether an operator applies to a column type by looking it
up in `pg_operator`. That answer is wrong in both directions: `varchar` has no `~~` entry of its
own yet resolves one through `text`, so `like` was rejected on varchar columns; and `bytea` has
`~~` but no `~~*`, so `ilike` was accepted on bytea columns and then raised at WAL time. The
comparison operators were not checked at all, so `eq` on a `json`, `xml` or `point` column
registered fine and raised inside `apply_rls`, aborting the batch for every subscription on the
tenant.

Probe `check_equality_op` itself instead - the same call `apply_rls` makes - so acceptance at
subscribe time and evaluation at WAL time cannot disagree.
"""

use Ecto.Migration

def change do
execute("""
create or replace function realtime.subscription_check_filters()
returns trigger
language plpgsql
as $$
declare
col_names text[] = coalesce(
array_agg(a.attname order by a.attnum),
'{}'::text[]
)
from
pg_catalog.pg_attribute a
where
a.attrelid = new.entity
and a.attnum > 0
and not a.attisdropped
and pg_catalog.has_column_privilege(
(new.claims ->> 'role'),
a.attrelid,
a.attnum,
'SELECT'
);
filter realtime.user_defined_filter;
col_type regtype;
in_val jsonb;
selected_col text;
begin
for filter in select * from unnest(new.filters) loop
if not filter.column_name = any(col_names) then
raise exception 'invalid column for filter %', filter.column_name;
end if;

col_type = (
select atttypid::regtype
from pg_catalog.pg_attribute
where attrelid = new.entity
and attname = filter.column_name
);
if col_type is null then
raise exception 'failed to lookup type for column %', filter.column_name;
end if;

if filter.op = 'in'::realtime.equality_op then
in_val = realtime.cast(filter.value, (col_type::text || '[]')::regtype);
if coalesce(jsonb_array_length(in_val), 0) > 100 then
raise exception 'too many values for `in` filter. Maximum 100';
end if;
elsif filter.op = 'is'::realtime.equality_op then
-- `is` requires a keyword RHS rather than a typed literal
if filter.value not in ('null', 'true', 'false', 'unknown') then
raise exception 'invalid value for is filter: must be null, true, false, or unknown';
end if;
-- IS NULL works for any type, but IS TRUE/FALSE/UNKNOWN require a boolean
-- operand. Reject the non-null keywords on non-boolean columns here so they
-- don't abort apply_rls at WAL time.
if filter.value <> 'null' and col_type <> 'boolean'::regtype then
raise exception 'is % filter requires a boolean column, got %', filter.value, col_type::text;
end if;
else
if filter.op in ('match'::realtime.equality_op, 'imatch'::realtime.equality_op) then
-- validate the regex eagerly so a bad pattern gets its own error rather than
-- surfacing as an unsupported operator below
begin
perform '' ~ filter.value;
exception when others then
raise exception 'invalid regular expression for % filter: %', filter.op::text, sqlerrm;
end;
elsif filter.op not in ('like'::realtime.equality_op, 'ilike'::realtime.equality_op) then
-- eq/neq/lt/lte/gt/gte/isdistinct: value must be coercable to the type
perform realtime.cast(filter.value, col_type);
end if;

-- The operator also has to be applicable to the column type. pg_operator answers
-- that wrongly in both directions - varchar carries no ~~ of its own but resolves
-- one through text, bytea carries ~~ but no ~~* - so evaluate the operator instead
-- of looking it up. This is the exact call apply_rls makes, which is what keeps
-- the two from disagreeing; anything that raises here would otherwise raise at WAL
-- time, where it aborts the batch for every subscription on the tenant.
begin
perform realtime.check_equality_op(filter.op, col_type, filter.value, filter.value, filter.negate);
exception when others then
raise exception 'operator % is not supported on column % of type %: %',
filter.op::text, filter.column_name, col_type::text, sqlerrm;
end;
end if;
end loop;

if new.selected_columns is not null then
for selected_col in select * from unnest(new.selected_columns) loop
if not selected_col = any(col_names) then
raise exception 'invalid column for select %', selected_col;
end if;
end loop;
end if;

-- Apply consistent order to filters so the unique constraint can't be tricked by a
-- different filter order. negate is part of the sort key.
new.filters = coalesce(
array_agg(f order by f.column_name, f.op, f.value, f.negate),
'{}'
) from unnest(new.filters) f;

new.selected_columns = (
select array_agg(c order by c)
from unnest(new.selected_columns) c
);

return new;
end;
$$;
""")
end
end
51 changes: 23 additions & 28 deletions priv/repo/tenant_db_dump_15.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -946,38 +946,32 @@ begin
if filter.value <> 'null' and col_type <> 'boolean'::regtype then
raise exception 'is % filter requires a boolean column, got %', filter.value, col_type::text;
end if;
elsif filter.op in ('like'::realtime.equality_op, 'ilike'::realtime.equality_op) then
-- like/ilike apply the text pattern operator (~~); reject column types that
-- have no such operator instead of failing at WAL time
if not exists (
select 1 from pg_catalog.pg_operator
where oprname = '~~' and oprleft = col_type
) then
raise exception 'operator % requires a text-compatible column type, got %', filter.op::text, col_type::text;
end if;
elsif filter.op in ('match'::realtime.equality_op, 'imatch'::realtime.equality_op) then
-- match/imatch apply the regex operators ~ / ~*; reject column types that have
-- no such operator (e.g. integer) instead of failing at WAL time, mirroring the
-- like/ilike guard above.
if not exists (
select 1 from pg_catalog.pg_operator
where oprname = case when filter.op = 'imatch'::realtime.equality_op then '~*' else '~' end
and oprleft = col_type
and oprright = col_type
and oprresult = 'boolean'::regtype
) then
raise exception 'operator % requires a text-compatible column type, got %', filter.op::text, col_type::text;
else
if filter.op in ('match'::realtime.equality_op, 'imatch'::realtime.equality_op) then
-- validate the regex eagerly so a bad pattern gets its own error rather than
-- surfacing as an unsupported operator below
begin
perform '' ~ filter.value;
exception when others then
raise exception 'invalid regular expression for % filter: %', filter.op::text, sqlerrm;
end;
elsif filter.op not in ('like'::realtime.equality_op, 'ilike'::realtime.equality_op) then
-- eq/neq/lt/lte/gt/gte/isdistinct: value must be coercable to the type
perform realtime.cast(filter.value, col_type);
end if;
-- validate the regex eagerly so a bad pattern is rejected here, not inside
-- apply_rls where it would abort the WAL stream for the entity

-- The operator also has to be applicable to the column type. pg_operator answers
-- that wrongly in both directions - varchar carries no ~~ of its own but resolves
-- one through text, bytea carries ~~ but no ~~* - so evaluate the operator instead
-- of looking it up. This is the exact call apply_rls makes, which is what keeps
-- the two from disagreeing; anything that raises here would otherwise raise at WAL
-- time, where it aborts the batch for every subscription on the tenant.
begin
perform '' ~ filter.value;
perform realtime.check_equality_op(filter.op, col_type, filter.value, filter.value, filter.negate);
exception when others then
raise exception 'invalid regular expression for % filter: %', filter.op::text, sqlerrm;
raise exception 'operator % is not supported on column % of type %: %',
filter.op::text, filter.column_name, col_type::text, sqlerrm;
end;
else
-- eq/neq/lt/lte/gt/gte: value must be coercable to the type
perform realtime.cast(filter.value, col_type);
end if;
end loop;

Expand Down Expand Up @@ -1484,3 +1478,4 @@ INSERT INTO realtime."schema_migrations" (version) VALUES (20260626120000);
INSERT INTO realtime."schema_migrations" (version) VALUES (20260706120000);
INSERT INTO realtime."schema_migrations" (version) VALUES (20260707120000);
INSERT INTO realtime."schema_migrations" (version) VALUES (20260709120000);
INSERT INTO realtime."schema_migrations" (version) VALUES (20260815140000);
51 changes: 23 additions & 28 deletions priv/repo/tenant_db_dump_17.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -947,38 +947,32 @@ begin
if filter.value <> 'null' and col_type <> 'boolean'::regtype then
raise exception 'is % filter requires a boolean column, got %', filter.value, col_type::text;
end if;
elsif filter.op in ('like'::realtime.equality_op, 'ilike'::realtime.equality_op) then
-- like/ilike apply the text pattern operator (~~); reject column types that
-- have no such operator instead of failing at WAL time
if not exists (
select 1 from pg_catalog.pg_operator
where oprname = '~~' and oprleft = col_type
) then
raise exception 'operator % requires a text-compatible column type, got %', filter.op::text, col_type::text;
end if;
elsif filter.op in ('match'::realtime.equality_op, 'imatch'::realtime.equality_op) then
-- match/imatch apply the regex operators ~ / ~*; reject column types that have
-- no such operator (e.g. integer) instead of failing at WAL time, mirroring the
-- like/ilike guard above.
if not exists (
select 1 from pg_catalog.pg_operator
where oprname = case when filter.op = 'imatch'::realtime.equality_op then '~*' else '~' end
and oprleft = col_type
and oprright = col_type
and oprresult = 'boolean'::regtype
) then
raise exception 'operator % requires a text-compatible column type, got %', filter.op::text, col_type::text;
else
if filter.op in ('match'::realtime.equality_op, 'imatch'::realtime.equality_op) then
-- validate the regex eagerly so a bad pattern gets its own error rather than
-- surfacing as an unsupported operator below
begin
perform '' ~ filter.value;
exception when others then
raise exception 'invalid regular expression for % filter: %', filter.op::text, sqlerrm;
end;
elsif filter.op not in ('like'::realtime.equality_op, 'ilike'::realtime.equality_op) then
-- eq/neq/lt/lte/gt/gte/isdistinct: value must be coercable to the type
perform realtime.cast(filter.value, col_type);
end if;
-- validate the regex eagerly so a bad pattern is rejected here, not inside
-- apply_rls where it would abort the WAL stream for the entity

-- The operator also has to be applicable to the column type. pg_operator answers
-- that wrongly in both directions - varchar carries no ~~ of its own but resolves
-- one through text, bytea carries ~~ but no ~~* - so evaluate the operator instead
-- of looking it up. This is the exact call apply_rls makes, which is what keeps
-- the two from disagreeing; anything that raises here would otherwise raise at WAL
-- time, where it aborts the batch for every subscription on the tenant.
begin
perform '' ~ filter.value;
perform realtime.check_equality_op(filter.op, col_type, filter.value, filter.value, filter.negate);
exception when others then
raise exception 'invalid regular expression for % filter: %', filter.op::text, sqlerrm;
raise exception 'operator % is not supported on column % of type %: %',
filter.op::text, filter.column_name, col_type::text, sqlerrm;
end;
else
-- eq/neq/lt/lte/gt/gte: value must be coercable to the type
perform realtime.cast(filter.value, col_type);
end if;
end loop;

Expand Down Expand Up @@ -1485,3 +1479,4 @@ INSERT INTO realtime."schema_migrations" (version) VALUES (20260626120000);
INSERT INTO realtime."schema_migrations" (version) VALUES (20260706120000);
INSERT INTO realtime."schema_migrations" (version) VALUES (20260707120000);
INSERT INTO realtime."schema_migrations" (version) VALUES (20260709120000);
INSERT INTO realtime."schema_migrations" (version) VALUES (20260815140000);
50 changes: 22 additions & 28 deletions priv/repo/tenant_schema/schemas/realtime/functions/subscription_check_filters.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,38 +55,32 @@ begin
if filter.value <> 'null' and col_type <> 'boolean'::regtype then
raise exception 'is % filter requires a boolean column, got %', filter.value, col_type::text;
end if;
elsif filter.op in ('like'::realtime.equality_op, 'ilike'::realtime.equality_op) then
-- like/ilike apply the text pattern operator (~~); reject column types that
-- have no such operator instead of failing at WAL time
if not exists (
select 1 from pg_catalog.pg_operator
where oprname = '~~' and oprleft = col_type
) then
raise exception 'operator % requires a text-compatible column type, got %', filter.op::text, col_type::text;
end if;
elsif filter.op in ('match'::realtime.equality_op, 'imatch'::realtime.equality_op) then
-- match/imatch apply the regex operators ~ / ~*; reject column types that have
-- no such operator (e.g. integer) instead of failing at WAL time, mirroring the
-- like/ilike guard above.
if not exists (
select 1 from pg_catalog.pg_operator
where oprname = case when filter.op = 'imatch'::realtime.equality_op then '~*' else '~' end
and oprleft = col_type
and oprright = col_type
and oprresult = 'boolean'::regtype
) then
raise exception 'operator % requires a text-compatible column type, got %', filter.op::text, col_type::text;
else
if filter.op in ('match'::realtime.equality_op, 'imatch'::realtime.equality_op) then
-- validate the regex eagerly so a bad pattern gets its own error rather than
-- surfacing as an unsupported operator below
begin
perform '' ~ filter.value;
exception when others then
raise exception 'invalid regular expression for % filter: %', filter.op::text, sqlerrm;
end;
elsif filter.op not in ('like'::realtime.equality_op, 'ilike'::realtime.equality_op) then
-- eq/neq/lt/lte/gt/gte/isdistinct: value must be coercable to the type
perform realtime.cast(filter.value, col_type);
end if;
-- validate the regex eagerly so a bad pattern is rejected here, not inside
-- apply_rls where it would abort the WAL stream for the entity

-- The operator also has to be applicable to the column type. pg_operator answers
-- that wrongly in both directions - varchar carries no ~~ of its own but resolves
-- one through text, bytea carries ~~ but no ~~* - so evaluate the operator instead
-- of looking it up. This is the exact call apply_rls makes, which is what keeps
-- the two from disagreeing; anything that raises here would otherwise raise at WAL
-- time, where it aborts the batch for every subscription on the tenant.
begin
perform '' ~ filter.value;
perform realtime.check_equality_op(filter.op, col_type, filter.value, filter.value, filter.negate);
exception when others then
raise exception 'invalid regular expression for % filter: %', filter.op::text, sqlerrm;
raise exception 'operator % is not supported on column % of type %: %',
filter.op::text, filter.column_name, col_type::text, sqlerrm;
end;
else
-- eq/neq/lt/lte/gt/gte: value must be coercable to the type
perform realtime.cast(filter.value, col_type);
end if;
end loop;

Expand Down
Loading