Conversation
| sqlmesh_path / "config.yml", | ||
| sqlmesh_path / "config.yaml", | ||
| ] | ||
| personal_paths = [sqlmesh_path / name for name in YAML_CONFIG_FILENAMES] |
There was a problem hiding this comment.
I noticed that python config wasnt previously allowed in personal paths, so that's why there is both YAML_CONFIG_FILENAMES and ALL_CONFIG_FILENAMES
There was a problem hiding this comment.
There is a reason for this. We can't correctly merge configs if the one in the home folder is a python config.
| table = exp.to_table(table_name) | ||
| if len(table.parts) == 3 and "." in table.name: | ||
| self.execute(exp.select("*").from_(table).limit(0)) | ||
| if not self._query_job: |
There was a problem hiding this comment.
This is potentially covered by #5153 but I wanted to unblock the build on this PR
| # todo: this may format the file differently, is that acceptable? | ||
| with project_yaml_path.open("w") as f: | ||
| dump(loaded_project_file, f) | ||
| from sqlmesh.cli.project_init import init_example_project, ProjectTemplate |
There was a problem hiding this comment.
Shouldn't all these imports be under the check of the file existence? Otherwise doesn't this defeat the purpose?
There was a problem hiding this comment.
How do you mean? The sqlmesh module has already been imported by the time init_project_if_required() is called, so these imports are "free"
erindru
force-pushed
the
erin/load-config-dbt
branch
from
18ce1a9 to
9db95a1
Compare
2 participants
Second attempt at #5132
This PR enables SQLMesh to load DBT projects using just yaml config, rather than forcing users to create a
config.pyand use python config.The following changes have been made:
sqlmesh.yamlis now a valid config file name. This is because it gets written next to the existingdbt_project.ymlfile so I wanted it to be clear that this config is relevant to SQLMesh only, which the existingconfig.yamlname is not clear aboutsqlmesh init -t dbtnow produces a small yaml file instead of a python config fileload_configsto return aConfigconfigured with theDbtLoaderif adbt_project.ymlfile is detected. This essentially transparently automates what the users had to do manually beforesqlmesh_dbtCLI automatically runssqlmesh init -t dbtif it doesn't detect an existing sqlmesh config file, and then proceeds normally. This enables a user to get started with dbt on the SQLMesh engine with zero manual stepsThe existing config mechanism is preserved - that is, the python base config is overlayed with all the yaml configs to produce a final config
Note: if this approach is considered acceptable, i'll update the docs in a followup PR