Skip to content

Rename add_index to add_search_config #107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 22, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,10 +191,10 @@ In order to perform searchable operations on encrypted data, you must configure

### Adding an index

Add an index to an encrypted column using the `eql_v2.add_index` function:
Add an index to an encrypted column using the `eql_v2.add_search_config` function:

```sql
SELECT eql_v2.add_index(
SELECT eql_v2.add_search_config(
'table_name', -- Name of the table
'column_name', -- Name of the column
'index_name', -- Index kind ('unique', 'match', 'ore', 'ste_vec')
Expand All @@ -208,7 +208,7 @@ You can read more about the index configuration options [here](docs/reference/IN
**Example (Unique index):**

```sql
SELECT eql_v2.add_index(
SELECT eql_v2.add_search_config(
'users',
'encrypted_email',
'unique',
Expand Down Expand Up @@ -236,7 +236,7 @@ Enable equality search on encrypted data using the `eql_v2.unique` function.
**Index configuration example:**

```sql
SELECT eql_v2.add_index(
SELECT eql_v2.add_search_config(
'users',
'encrypted_email',
'unique',
Expand Down Expand Up @@ -266,7 +266,7 @@ Enables basic full-text search on encrypted data using the `eql_v2.match` functi
**Index configuration example:**

```sql
SELECT eql_v2.add_index(
SELECT eql_v2.add_search_config(
'users',
'encrypted_email',
'match',
Expand Down
8 changes: 4 additions & 4 deletions docs/reference/INDEX.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
# EQL index configuration

The following functions allow you to configure indexes for encrypted columns.
All these functions modify the `cs_configuration_v2` table in your database, and are added during the EQL installation.
All these functions modify the `eql_v2_configuration` table in your database, and are added during the EQL installation.

> **IMPORTANT:** When you modify or add an index, you must re-encrypt data that's already been stored in the database.
> **IMPORTANT:** When you modify or add search configuration index, you must re-encrypt data that's already been stored in the database.
> The CipherStash encryption solution will encrypt the data based on the current state of the configuration.

### Adding an index (`cs_add_index`)
### Configuring search (`eql_v2.add_search_config`)

Add an index to an encrypted column.

```sql
SELECT cs_add_index_v2(
SELECT eql_v2.add_search_config(
'table_name', -- Name of the table
'column_name', -- Name of the column
'index_name', -- Index kind ('unique', 'match', 'ore', 'ste_vec')
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/JSON.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Similar to how you configure indexes for text data, you can configure indexes fo
The only difference is that you need to specify the `cast_as` parameter as `json` or `jsonb`.

```sql
SELECT cs_add_index_v2(
SELECT eql_v2.add_search_config(
'users',
'encrypted_json',
'ste_vec',
Expand Down
6 changes: 3 additions & 3 deletions docs/tutorials/GETTINGSTARTED.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ We now have our database schema setup to store encrypted data.

In this part we will learn about why we need to add indexes and how to add them.

When you install EQL, a table called `cs_configuration_v2` is created in your database.
When you install EQL, a table called `eql_v2_configuration` is created in your database.

Adding indexes updates this table with the details and configuration needed for CipherStash Proxy to know how to encrypt your data, and what types of queries are able to be performed

Expand Down Expand Up @@ -196,7 +196,7 @@ SELECT cs_add_index_v2('users', 'email_encrypted', 'ore', 'text');
CREATE INDEX ON users (cs_ore_64_8_v2(email_encrypted));
```

After adding these indexes, our `cs_configuration_v2` table will look like this:
After adding these indexes, our `eql_v2_configuration` table will look like this:

```bash
id | 1
Expand Down Expand Up @@ -228,7 +228,7 @@ Prerequisites:
- [Database is setup](#setup-your-database)
- [Indexes added](#adding-indexes)

Ensure CipherStash Proxy has the most up to date configuration from the `cs_configuration_v2` table.
Ensure CipherStash Proxy has the most up to date configuration from the `eql_v2_configuration` table.

CipherStash Proxy pings the database every 60 seconds to refresh the configuration but we can force the refresh by running:

Expand Down
52 changes: 26 additions & 26 deletions src/config/config_test.sql
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
--
-- Helper function for assertions
--
DROP FUNCTION IF EXISTS _index_exists(text, text, text, text);
CREATE FUNCTION _index_exists(table_name text, column_name text, index_name text, state text DEFAULT 'pending')
DROP FUNCTION IF EXISTS _search_config_exists(text, text, text, text);
CREATE FUNCTION _search_config_exists(table_name text, column_name text, index_name text, state text DEFAULT 'pending')
RETURNS boolean
LANGUAGE sql STRICT PARALLEL SAFE
BEGIN ATOMIC
Expand All @@ -25,23 +25,23 @@ DO $$
BEGIN

-- Add indexes
PERFORM eql_v2.add_index('users', 'name', 'match');
ASSERT (SELECT _index_exists('users', 'name', 'match'));
PERFORM eql_v2.add_search_config('users', 'name', 'match');
ASSERT (SELECT _search_config_exists('users', 'name', 'match'));

-- Add index with cast
PERFORM eql_v2.add_index('users', 'name', 'unique', 'int');
ASSERT (SELECT _index_exists('users', 'name', 'unique'));
PERFORM eql_v2.add_search_config('users', 'name', 'unique', 'int');
ASSERT (SELECT _search_config_exists('users', 'name', 'unique'));

ASSERT (SELECT EXISTS (SELECT id FROM eql_v2_configuration c
WHERE c.state = 'pending' AND
c.data #> array['tables', 'users', 'name'] ? 'cast_as'));

-- Match index removed
PERFORM eql_v2.remove_index('users', 'name', 'match');
ASSERT NOT (SELECT _index_exists('users', 'name', 'match'));
PERFORM eql_v2.remove_search_config('users', 'name', 'match');
ASSERT NOT (SELECT _search_config_exists('users', 'name', 'match'));

-- All indexes removed, delete the emtpty pending config
PERFORM eql_v2.remove_index('users', 'name', 'unique');
PERFORM eql_v2.remove_search_config('users', 'name', 'unique');
ASSERT (SELECT NOT EXISTS (SELECT FROM eql_v2_configuration c WHERE c.state = 'pending'));

END;
Expand All @@ -60,16 +60,16 @@ DO $$
BEGIN

-- Add indexes
PERFORM eql_v2.add_index('users', 'name', 'match');
ASSERT (SELECT _index_exists('users', 'name', 'match'));
PERFORM eql_v2.add_search_config('users', 'name', 'match');
ASSERT (SELECT _search_config_exists('users', 'name', 'match'));

ASSERT (SELECT EXISTS (SELECT id FROM eql_v2_configuration c
WHERE c.state = 'pending' AND
c.data #> array['tables', 'users', 'name', 'indexes'] ? 'match'));

-- Add index with cast
PERFORM eql_v2.add_index('blah', 'vtha', 'unique', 'int');
ASSERT (SELECT _index_exists('blah', 'vtha', 'unique'));
PERFORM eql_v2.add_search_config('blah', 'vtha', 'unique', 'int');
ASSERT (SELECT _search_config_exists('blah', 'vtha', 'unique'));

ASSERT (SELECT EXISTS (SELECT id FROM eql_v2_configuration c
WHERE c.state = 'pending' AND
Expand All @@ -82,12 +82,12 @@ DO $$


-- Match index removed
PERFORM eql_v2.remove_index('users', 'name', 'match');
ASSERT NOT (SELECT _index_exists('users', 'name', 'match'));
PERFORM eql_v2.remove_search_config('users', 'name', 'match');
ASSERT NOT (SELECT _search_config_exists('users', 'name', 'match'));

-- Match index removed
PERFORM eql_v2.remove_index('blah', 'vtha', 'unique');
ASSERT NOT (SELECT _index_exists('users', 'vtha', 'unique'));
PERFORM eql_v2.remove_search_config('blah', 'vtha', 'unique');
ASSERT NOT (SELECT _search_config_exists('users', 'vtha', 'unique'));

-- All indexes removed, delete the emtpty pending config
ASSERT (SELECT NOT EXISTS (SELECT FROM eql_v2_configuration c WHERE c.state = 'pending'));
Expand All @@ -107,12 +107,12 @@ $$ LANGUAGE plpgsql;

DO $$
BEGIN
PERFORM eql_v2.add_index('users', 'name', 'match');
ASSERT (SELECT _index_exists('users', 'name', 'match'));
PERFORM eql_v2.add_search_config('users', 'name', 'match');
ASSERT (SELECT _search_config_exists('users', 'name', 'match'));

-- Pending configuration contains the path `user/name.match.option`
PERFORM eql_v2.modify_index('users', 'name', 'match', 'int', '{"option": "value"}'::jsonb);
ASSERT (SELECT _index_exists('users', 'name', 'match'));
PERFORM eql_v2.modify_search_config('users', 'name', 'match', 'int', '{"option": "value"}'::jsonb);
ASSERT (SELECT _search_config_exists('users', 'name', 'match'));

ASSERT (SELECT EXISTS (SELECT id FROM eql_v2_configuration c
WHERE c.state = 'pending' AND
Expand All @@ -123,7 +123,7 @@ DO $$
c.data #> array['tables', 'users', 'name'] ? 'cast_as'));

-- All indexes removed, delete the emtpty pending config
PERFORM eql_v2.remove_index('users', 'name', 'match');
PERFORM eql_v2.remove_search_config('users', 'name', 'match');
ASSERT (SELECT NOT EXISTS (SELECT FROM eql_v2_configuration c WHERE c.state = 'pending'));
END;
$$ LANGUAGE plpgsql;
Expand Down Expand Up @@ -160,16 +160,16 @@ INSERT INTO eql_v2_configuration (state, data) VALUES (
-- An encrypting config should exist
DO $$
BEGIN
ASSERT (SELECT _index_exists('users', 'blah', 'match', 'active'));
ASSERT (SELECT _search_config_exists('users', 'blah', 'match', 'active'));

PERFORM eql_v2.add_index('users', 'name', 'match');
PERFORM eql_v2.add_search_config('users', 'name', 'match');

-- index added to name
ASSERT (SELECT _index_exists('users', 'name', 'match' ));
ASSERT (SELECT _search_config_exists('users', 'name', 'match' ));

-- pending is a copy of the active config
-- and the active index still exists
ASSERT (SELECT _index_exists('users', 'blah', 'match'));
ASSERT (SELECT _search_config_exists('users', 'blah', 'match'));

END;
$$ LANGUAGE plpgsql;
Expand Down
104 changes: 8 additions & 96 deletions src/config/functions.sql
Original file line number Diff line number Diff line change
@@ -1,104 +1,16 @@
-- REQUIRE: src/config/types.sql
-- REQUIRE: src/config/functions_private.sql
--
-- Configuration functions
-- Customer-facing configuration functions
-- Depends on private functions for implemenation
--
--


CREATE FUNCTION eql_v2.config_default(config jsonb)
RETURNS jsonb
IMMUTABLE PARALLEL SAFE
AS $$
BEGIN
IF config IS NULL THEN
SELECT jsonb_build_object('v', 1, 'tables', jsonb_build_object()) INTO config;
END IF;
RETURN config;
END;
$$ LANGUAGE plpgsql;



CREATE FUNCTION eql_v2.config_add_table(table_name text, config jsonb)
RETURNS jsonb
IMMUTABLE PARALLEL SAFE
AS $$
DECLARE
tbl jsonb;
BEGIN
IF NOT config #> array['tables'] ? table_name THEN
SELECT jsonb_insert(config, array['tables', table_name], jsonb_build_object()) INTO config;
END IF;
RETURN config;
END;
$$ LANGUAGE plpgsql;


-- Add the column if it doesn't exist

CREATE FUNCTION eql_v2.config_add_column(table_name text, column_name text, config jsonb)
RETURNS jsonb
IMMUTABLE PARALLEL SAFE
AS $$
DECLARE
col jsonb;
BEGIN
IF NOT config #> array['tables', table_name] ? column_name THEN
SELECT jsonb_build_object('indexes', jsonb_build_object()) into col;
SELECT jsonb_set(config, array['tables', table_name, column_name], col) INTO config;
END IF;
RETURN config;
END;
$$ LANGUAGE plpgsql;


-- Set the cast

CREATE FUNCTION eql_v2.config_add_cast(table_name text, column_name text, cast_as text, config jsonb)
RETURNS jsonb
IMMUTABLE PARALLEL SAFE
AS $$
BEGIN
SELECT jsonb_set(config, array['tables', table_name, column_name, 'cast_as'], to_jsonb(cast_as)) INTO config;
RETURN config;
END;
$$ LANGUAGE plpgsql;


-- Add the column if it doesn't exist

CREATE FUNCTION eql_v2.config_add_index(table_name text, column_name text, index_name text, opts jsonb, config jsonb)
RETURNS jsonb
IMMUTABLE PARALLEL SAFE
AS $$
BEGIN
SELECT jsonb_insert(config, array['tables', table_name, column_name, 'indexes', index_name], opts) INTO config;
RETURN config;
END;
$$ LANGUAGE plpgsql;


--
-- Default options for match index
--

CREATE FUNCTION eql_v2.config_match_default()
RETURNS jsonb
LANGUAGE sql STRICT PARALLEL SAFE
BEGIN ATOMIC
SELECT jsonb_build_object(
'k', 6,
'm', 2048,
'include_original', true,
'tokenizer', json_build_object('kind', 'ngram', 'token_length', 3),
'token_filters', json_build_array(json_build_object('kind', 'downcase')));
END;

--
-- Adds an index term to the configuration
--

CREATE FUNCTION eql_v2.add_index(table_name text, column_name text, index_name text, cast_as text DEFAULT 'text', opts jsonb DEFAULT '{}')
CREATE FUNCTION eql_v2.add_search_config(table_name text, column_name text, index_name text, cast_as text DEFAULT 'text', opts jsonb DEFAULT '{}')
RETURNS jsonb

AS $$
Expand Down Expand Up @@ -149,7 +61,7 @@ $$ LANGUAGE plpgsql;



CREATE FUNCTION eql_v2.remove_index(table_name text, column_name text, index_name text)
CREATE FUNCTION eql_v2.remove_search_config(table_name text, column_name text, index_name text)
RETURNS jsonb
AS $$
DECLARE
Expand Down Expand Up @@ -209,12 +121,12 @@ $$ LANGUAGE plpgsql;



CREATE FUNCTION eql_v2.modify_index(table_name text, column_name text, index_name text, cast_as text DEFAULT 'text', opts jsonb DEFAULT '{}')
CREATE FUNCTION eql_v2.modify_search_config(table_name text, column_name text, index_name text, cast_as text DEFAULT 'text', opts jsonb DEFAULT '{}')
RETURNS jsonb
AS $$
BEGIN
PERFORM eql_v2.remove_index(table_name, column_name, index_name);
RETURN eql_v2.add_index(table_name, column_name, index_name, cast_as, opts);
PERFORM eql_v2.remove_search_config(table_name, column_name, index_name);
RETURN eql_v2.add_search_config(table_name, column_name, index_name, cast_as, opts);
END;
$$ LANGUAGE plpgsql;

Expand Down
Loading