Data-diff --dbt doesn't work with profile targets that use env vars

[jnoynaert]

Describe the bug

Given a profiles.yml that contains something like:

my_dwh:
  target: "{{ env_var('DBT_SNOWFLAKE_TARGET', 'dev') }}"
  outputs:
    dev:
      user: "{{ env_var('DBT_SNOWFLAKE_USER') }}"
      password: "{{ env_var('DBT_SNOWFLAKE_PASSWORD') }}"
      database: "dev_{{ env_var('DBT_SNOWFLAKE_DB') }}"
      schema: none
      warehouse: "{{ env_var('DBT_SNOWFLAKE_WAREHOUSE', 'dev_xsmall') }}"
      role: "{{ env_var('DBT_SNOWFLAKE_ROLE', 'developer') }}"

then at

data-diff/data_diff/dbt.py

Line 305 in 29bfa73

credentials = profile_outputs.get("outputs").get(profile_target)

"{{ env_var('DBT_SNOWFLAKE_TARGET', 'dev') }}" gets used to index into the profile dict

Resulting in something like the below on the next call:

> data-diff --dbt --dbt-profiles-dir '.' --debug
Found 1 successful model runs from the last dbt command.
[21:29:23] ERROR - 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "/usr/local/bin/data-diff", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/data_diff/__main__.py", line 262, in main
    dbt_diff(
  File "/usr/local/lib/python3.10/site-packages/data_diff/dbt.py", line 55, in dbt_diff
    dbt_parser.set_connection()
  File "/usr/local/lib/python3.10/site-packages/data_diff/dbt.py", line 306, in set_connection
    conn_type = credentials.get("type").lower()
AttributeError: 'NoneType' object has no attribute 'get'

The way to get around this is to hardcode all values in profiles.yml.

Additionally, even when specifying the db for a schema-separated project, you still have to specify the prod_schema key in the dbt project config or else you get an error from

data-diff/data_diff/dbt.py

Lines 57 to 58 in 29bfa73

	if config_prod_database is None or config_prod_schema is None:
	raise ValueError("Expected a value for prod_database: or prod_schema: under \nvars:\n data_diff: ")

Describe the environment

> data-diff --version
v0.4.0

docker / Debian-bullseye / Python 3.10

[dlawin]

I opened a separate issue for the schema bug here: #391

[dlawin]

Thanks for reporting this!

Edit: I didn't read it correctly the first time. Using the example above, we are only rendering the fields under the specified target:

      user: "{{ env_var('DBT_SNOWFLAKE_USER') }}"
      password: "{{ env_var('DBT_SNOWFLAKE_PASSWORD') }}"
      database: "dev_{{ env_var('DBT_SNOWFLAKE_DB') }}"
      schema: none
      warehouse: "{{ env_var('DBT_SNOWFLAKE_WAREHOUSE', 'dev_xsmall') }}"
      role: "{{ env_var('DBT_SNOWFLAKE_ROLE', 'developer') }}"

We should consider rendering the whole profiles.yml file before parsing for values.

Or at least the values under my_dwh:

data-diff/data_diff/dbt.py

Line 309 in 91e1d98

rendered_credentials = ProfileRenderer().render_data(credentials)