Should the project ignore the DBT_PROFILES_DIR environmental variable?

[leoebfolsom]

Hi Doug, this is awesome. I just got it up and running in 5(!) minutes! Thanks so much for setting it up.

I did run into one hiccup, which I think might have to do with my own lack of understanding of some of the recent changes to how dbt handles profiles.

I initially received this error:

% dbt run -s simple_model --target prod
17:53:26  Encountered an error while reading the project:
17:53:26    ERROR: Runtime Error
  Could not find profile named 'diffle_shop'
17:53:27  Encountered an error:
Runtime Error
  Could not run dbt

This occurred (I think) because I've saved an environmental variable in my local ~/.zshrc file that informs dbt of the location of my profiles.yml file.

To that effect, the following line exists in my .zshrc:

export DBT_PROFILES_DIR=~/.dbt

I included that line in my .zshrc because in (most) other contexts, I don't want dbt to automatically use the profiles.yml file in the project itself.

As a workaround to get this project up and running, I copy/pasted the contents of the project's profiles.yml into my local ~/.dbt/profiles.yml. But ideally, I would use the project's profiles.yml, because of course now the profile is not source controlled--not to mention, that's a manual step that is not documented/intuitive.

Could something be included in the README's recommended commands so that even a user who had the profile path configured as a DBT_PROFILES_DIR environmental variable (like I did) receive the profile from the project's profiles.yml? Or is it a bad/outdated practice to set DBT_PROFILES_DIR in .zshrc?

[leoebfolsom]

I did a bit of learning (aka I asked @dlawin):

there are 3 possible ways to set the profiles dir with this precedence:
--profiles-dir flag > DBT_PROFILES_DIR > “default directory”

so I think the following update to the Setup section would address what I've raised:

Establish the baseline production table:

dbt run -s simple_model --target prod --profiles-dir .

Simulate a change to the table during development:

dbt run -s simple_model --profiles-dir .

[leoebfolsom]

Feel free to close/ignore if this is going to address it, assuming there aren't unforeseen delays/complications to merging it. 🤠 Thank you for coming on this journey with me.

[dbeatty10]

I just got it up and running in 5(!) minutes! Thanks so much for setting it up.

🎉 Love hearing that @leoebfolsom ! Being able to get up and running as fast and easy as possible was the goal, so I'm delighted to hear that.

The work y'all are doing with data-diff is really cool, so excited to see people get introduced to it.

Also, thanks for reaching out and letting me know the barrier you ran into.

Making the instructions as helpful as possible

Funny story: in this recent commit: df32ccd, I just un-did the commands you are proposing. This was an attempt to make the command as short and sweet as possible.

I really prefer commands to be as brief as humanly possible, but it also bums me out when people have to spend time troubleshooting. All things be equal, I think good instructions should "just work, no matter what".

datafold/data-diff#415 won't address it because it will still favor DBT_PROFILES_DIR over any profiles.yml in the current working directory. (I just updated the description of that PR to be more clear -- thanks for pointing it out!)

Options

There's a couple options we could take here:

1. Revert df32ccd so that it "just works" for everyone
   • accepting that the command will be more verbose
2. Leave df32ccd as-is and add a caveat that users that have DBT_PROFILES_DIR set will need to either a) add another connection to their ~/.dbt/profiles.yml or b) use the --profiles-dir flag
   • accepting that there will be more text within the README, and it won't be as simple as copy-paste for some people

I'm leaning towards Option 1.

[leoebfolsom]

Well, I think there's a fundamental tension between as brief as humanly possible and just work, no matter what 😇 and when I must choose between brevity and functionality, I pretty much always choose functionality. But I recognize that's a preference (that functionality can come at the cost of people being scared by long commands or explanations) and there are inevitably tradeoffs!

I prefer Option 1, but I would be happy with either 1 or 2--as long as something is in there to address the hiccup. I think the additional flag doesn't compromise brevity too much; anyone using dbt-core is going to at least have a sense for what that flag is doing (we've all had to fiddle with profiles.yml before); and it's not going to change the experience very much at all for anyone who is (justifiably!) looking to simply copy/paste the commands.

Thank you for your quick reply and again for all your work on this!