Skip to content

Commit

Permalink
feat: Propagate all env vars with DATACONTRACT_SNOWFLAKE_ prefix (#339)
Browse files Browse the repository at this point in the history
According to Soda documentation:
https://docs.soda.io/soda/connect-snowflake.html
All parameters from the official
Snowflake Python Connector library
are supported.
Thus, we only need to propagate them
from environmental variables
to the underlying Soda configuration.

Co-authored-by: Taras Slipets <taras.slipets@flixbus.com>
  • Loading branch information
tarys and Taras Slipets authored Jul 21, 2024
1 parent 579028a commit 3b3419a
Show file tree
Hide file tree
Showing 2 changed files with 33 additions and 13 deletions.
33 changes: 25 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -588,14 +588,31 @@ models:
```

#### Environment Variables

| Environment Variable | Example | Description |
|------------------------------------|--------------------|-----------------------------------------------------|
| `DATACONTRACT_SNOWFLAKE_USERNAME` | `datacontract` | Username |
| `DATACONTRACT_SNOWFLAKE_PASSWORD` | `mysecretpassword` | Password |
| `DATACONTRACT_SNOWFLAKE_ROLE` | `DATAVALIDATION` | The snowflake role to use. |
| `DATACONTRACT_SNOWFLAKE_WAREHOUSE` | `COMPUTE_WH` | The Snowflake Warehouse to use executing the tests. |

All [parameters supported by Soda](https://docs.soda.io/soda/connect-snowflake.html), uppercased and prepended by `DATACONTRACT_SNOWFLAKE_` prefix.
For example:

| Soda parameter | Environment Variable |
|----------------------|---------------------------------------------|
| `username` | `DATACONTRACT_SNOWFLAKE_USERNAME` |
| `password` | `DATACONTRACT_SNOWFLAKE_PASSWORD` |
| `warehouse` | `DATACONTRACT_SNOWFLAKE_WAREHOUSE` |
| `role` | `DATACONTRACT_SNOWFLAKE_ROLE` |
| `connection_timeout` | `DATACONTRACT_SNOWFLAKE_CONNECTION_TIMEOUT` |

Beware, that parameters:
* `account`
* `database`
* `schema`

are obtained from the `servers` section of the YAML-file.
E.g. from the example above:
```yaml
servers:
snowflake:
account: abcdefg-xn12345
database: ORDER_DB
schema: ORDERS_PII_V2
```


### Kafka
Expand Down
13 changes: 8 additions & 5 deletions datacontract/engines/soda/connections/snowflake.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,20 @@


def to_snowflake_soda_configuration(server):
prefix = "DATACONTRACT_SNOWFLAKE_"
snowflake_soda_params = {k.replace(prefix, "").lower(): v for k, v in os.environ.items() if k.startswith(prefix)}

# backward compatibility
if "connection_timeout" not in snowflake_soda_params:
snowflake_soda_params["connection_timeout"] = "5" # minutes

soda_configuration = {
f"data_source {server.type}": {
"type": "snowflake",
"username": os.getenv("DATACONTRACT_SNOWFLAKE_USERNAME"),
"password": os.getenv("DATACONTRACT_SNOWFLAKE_PASSWORD"),
"role": os.getenv("DATACONTRACT_SNOWFLAKE_ROLE"),
"account": server.account,
"database": server.database,
"schema": server.schema_,
"warehouse": os.getenv("DATACONTRACT_SNOWFLAKE_WAREHOUSE"),
"connection_timeout": 5, # minutes
**snowflake_soda_params,
}
}
soda_configuration_str = yaml.dump(soda_configuration)
Expand Down

0 comments on commit 3b3419a

Please sign in to comment.