diff --git a/website/docs/reference/global-configs.md b/website/docs/reference/global-configs.md index 4c15d0b54a2..5e130093bf5 100644 --- a/website/docs/reference/global-configs.md +++ b/website/docs/reference/global-configs.md @@ -3,7 +3,7 @@ title: "Global Configs" id: "global-configs" --- -### About Global Configs +## About Global Configs Global configs enable you to fine-tune how dbt runs projects on your machine—whether your personal laptop, an orchestration tool running remotely, or (in some cases) dbt Cloud. They differ from [project configs](reference/dbt_project.yml) and [resource configs](reference/configs-and-properties), which tell dbt _what_ to run. @@ -13,7 +13,7 @@ These configs are "global" because they are available for all dbt commands, and Starting in v1.0, you can set global configs in three places. When all three are set, command line flags take precedence, then environment variables, and last profile configs. -#### Command line flags +## Command line flags Command line (CLI) flags immediately follow `dbt` and precede your subcommand. When set, CLI flags override environment variables and profile configs. @@ -73,7 +73,7 @@ $ dbt --no-version-check run -#### Environment variables +## Environment variables Environment variables contain a `DBT_` prefix @@ -88,7 +88,7 @@ $ dbt run -#### Profile (or user) configurations +## Profile (or user) configurations You can set profile (or user) configurations in the `config:` block of `profiles.yml`. You would use the profile config to set defaults for all projects running on your local machine. @@ -103,46 +103,54 @@ config: -## Failing fast + -Supply the `-x` or `--fail-fast` flag to `dbt run` to make dbt exit immediately if a single resource fails to build. If other models are in-progress when the first model fails, then dbt will terminate the connections for these still-running models. +### Cache database objects for selected resource -For example, you can select four models to run, but if a failure occurs in the first model, the failure will prevent other models from running: +:::caution Experimental config flag +This should be considered an _unstable_ config flag because it is experimental and subject to change. We reserve the right to make breaking changes to this config flag. +::: + +At the start of runs, dbt caches metadata about all the objects in all the schemas where it might materialize resources (such as models). By default, dbt caches all schemas related to the project. When this config is enabled, dbt will only cache schemas related to selected resources for the current run. This can offer significant speed improvements when running a small subset of a large project. + +For example, to improve speed and performance while focused on developing Salesforce models, which are materialized into their own dedicated schema, you would select those models and pass the `cache-selected-only` flag: ```text -$ dbt -x run --threads 1 -Running with dbt=1.0.0 -Found 4 models, 1 test, 1 snapshot, 2 analyses, 143 macros, 0 operations, 1 seed file, 0 sources -14:47:39 | Concurrency: 1 threads (target='dev') -14:47:39 | -14:47:39 | 1 of 4 START table model test_schema.model_1........... [RUN] -14:47:40 | 1 of 4 ERROR creating table model test_schema.model_1.. [ERROR in 0.06s] -14:47:40 | 2 of 4 START view model test_schema.model_2............ [RUN] -14:47:40 | CANCEL query model.debug.model_2....................... [CANCEL] -14:47:40 | 2 of 4 ERROR creating view model test_schema.model_2... [ERROR in 0.05s] +$ dbt --cache-selected-only run --select salesforce -Database Error in model model_1 (models/model_1.sql) - division by zero - compiled SQL at target/run/debug/models/model_1.sql +``` + +The previous example enables you to start working with your salesforce models, and dbt will only cache those models instead of the entire project. + +To set the config in your `profile.yml`: + + +```yaml + +config: + cache_selected_only: true -Encountered an error: -FailFast Error in model model_1 (models/model_1.sql) - Failing early due to test failure or runtime error ``` -## Checking version compatibility + + + + +### Checking version compatibility Projects are recommended to set [dbt version requirements](require-dbt-version), especially if they use features that are newer, or which may break in future versions of dbt Core. By default, if you run a project with an incompatible dbt version, dbt will raise an error. You can use the `VERSION_CHECK` config to disable this check and suppress the error message: + ``` $ dbt --no-version-check run Running with dbt=1.0.0 Found 13 models, 2 tests, 1 archives, 0 analyses, 204 macros, 2 operations.... ``` -## Debug-level logging +### Debug-level logging + The `DEBUG` config redirects dbt's debug logs to standard out. The has the effect of showing debug-level log information in the terminal in addition to the `logs/dbt.log` file. This output is verbose. The `--debug` flag is also available via shorthand as `-d`. @@ -155,9 +163,52 @@ $ dbt --debug run ``` + + +### Experimental parser + +With the `USE_EXPERIMENTAL_PARSER` config, you can opt into the latest and greatest experimental version of the static parser, which is still being sampled for 100% correctness. See [the docs on parsing](parsing#experimental-parser) for more details. + + + +```yaml + +config: + use_experimental_parser: true + +``` + -## Log Formatting +### Failing fast + +Supply the `-x` or `--fail-fast` flag to `dbt run` to make dbt exit immediately if a single resource fails to build. If other models are in-progress when the first model fails, then dbt will terminate the connections for these still-running models. + +For example, you can select four models to run, but if a failure occurs in the first model, the failure will prevent other models from running: + +```text +$ dbt -x run --threads 1 +Running with dbt=1.0.0 +Found 4 models, 1 test, 1 snapshot, 2 analyses, 143 macros, 0 operations, 1 seed file, 0 sources + +14:47:39 | Concurrency: 1 threads (target='dev') +14:47:39 | +14:47:39 | 1 of 4 START table model test_schema.model_1........... [RUN] +14:47:40 | 1 of 4 ERROR creating table model test_schema.model_1.. [ERROR in 0.06s] +14:47:40 | 2 of 4 START view model test_schema.model_2............ [RUN] +14:47:40 | CANCEL query model.debug.model_2....................... [CANCEL] +14:47:40 | 2 of 4 ERROR creating view model test_schema.model_2... [ERROR in 0.05s] + +Database Error in model model_1 (models/model_1.sql) + division by zero + compiled SQL at target/run/debug/models/model_1.sql + +Encountered an error: +FailFast Error in model model_1 (models/model_1.sql) + Failing early due to test failure or runtime error +``` + +### Log Formatting The `LOG_FORMAT` config specifies how dbt's logs should be formatted. If the value of this config is `json`, dbt will output fully structured logs in JSON format; otherwise, it will output text-formatted logs that are sparser for the CLI and more detailed in `logs/dbt.log`. @@ -171,6 +222,7 @@ $ dbt --log-format json run :::tip Tip: verbose structured logs Use `json` formatting value in conjunction with the `DEBUG` config to produce rich log information which can be piped into monitoring tools for analysis: + ```text $ dbt --debug --log-format json run ``` @@ -179,61 +231,64 @@ See [structured logging](events-logging#structured-logging) for more details. ::: - -## Writing JSON artifacts - -The `WRITE_JSON` config determines whether dbt writes JSON artifacts (eg. `manifest.json`, `run_results.json`) to the `target/` directory. JSON serialization can be slow, and turning this flag off _might_ make invocations of dbt faster. Alternatively, you might disable this config if you want to perform a dbt operation and avoid overwriting artifacts from a previous run step. - - +### Partial Parsing -```text -$ dbt --no-write-json run -``` +The `PARTIAL_PARSE` config can turn partial parsing on or off in your project. See [the docs on parsing](parsing#partial-parsing) for more details. - + -## Strict +```yaml -As of v1.0, the `-S` or `--strict` flag has been deprecated. +config: + partial_parse: true -## Warnings as Errors +``` -Turning on the `WARN_ERROR` config will convert dbt warnings into errors. Any time dbt would normally warn, it will instead raise an error. Examples include `--select` criteria that selects no resources, deprecations, configurations with no associated models, invalid test configurations, or tests and freshness checks that are configured to return warnings. + ```text -$ dbt --warn-error run -... +dbt --no-partial-parse run ``` -## Partial Parsing -The `PARTIAL_PARSE` config can turn partial parsing on or off in your project. See [the docs on parsing](parsing#partial-parsing) for more details. +### Printer width + +By default, dbt will print out lines padded to 80 characters wide. You can change this setting by adding the following to your `profiles.yml` file: ```yaml - config: - partial_parse: true - + printer_width: 120 ``` - +### Send anonymous usage stats -```text -$ dbt --no-partial-parse run +We want to build the best version of dbt possible, and a crucial part of that is understanding how users work with dbt. To this end, we've added some simple event tracking to dbt (using Snowplow). We do not track credentials, model contents or model names (we consider these private, and frankly none of our business). + +Usage statistics are fired when dbt is invoked and when models are run. These events contain basic platform information (OS + python version). You can see all the event definitions in [`tracking.py`](https://github.com/dbt-labs/dbt-core/blob/HEAD/core/dbt/tracking.py). + +By default this is turned on – you can opt out of event tracking at any time by adding the following to your `profiles.yml` file: + +```yaml +config: + send_anonymous_usage_stats: False ``` - +You can also use the DO_NOT_TRACK environmental variable to enable or disable sending anonymous data. For more information, see [Environmental variables](/dbt-cloud/using-dbt-cloud/cloud-environment-variables). + +`DO_NOT_TRACK=1` is the same as `DBT_SEND_ANONYMOUS_USAGE_STATS=False` +`DO_NOT_TRACK=0` is the same as `DBT_SEND_ANONYMOUS_USAGE_STATS=True` + +### Static parser -## Static parser The `STATIC_PARSER` config can enable or disable use of the static parser. See [the docs on parsing](parsing#static-parser) for more details. @@ -247,21 +302,57 @@ config: -## Experimental parser -With the `USE_EXPERIMENTAL_PARSER` config, you can opt into the latest and greatest experimental version of the static parser, which is still being sampled for 100% correctness. See [the docs on parsing](parsing#experimental-parser) for more details. +### Strict + +As of v1.0, the `-S` or `--strict` flag has been deprecated. + + + +### Suppress non-error logs in output + +By default, dbt shows all logs in standard out (stdout). You can use the `QUIET` config to show only error logs in stdout. Logs will still include the output of anything passed to the `print()` macro. For example, you might suppress all but error logs to more easily find and debug a jinja error. ```yaml - config: - use_experimental_parser: true + quiet: true +``` + + + +Supply the `-q` or `--quiet` flag to `dbt run` to show only error logs and suppress non-error logs. + +```text +$ dbt --quiet run +... ``` +### Suppress `print()` messages in stdout + +By default, dbt includes `print()` messages in standard out (stdout). You can use the `NO_PRINT` config to prevent these messages from showing up in stdout. + + + +```yaml +config: + no_print: true +``` + -## Use colors +Supply `--no-print` flag to `dbt run` to suppress `print()` messages from showing in stdout. + +```text +$ dbt --no-print run +... + +``` + + + +### Use colors By default, dbt will colorize the output it prints in your terminal. You can turn this off by adding the following to your `profiles.yml` file: @@ -272,36 +363,34 @@ config: use_colors: False ``` -``` +```text $ dbt --use-colors run $ dbt --no-use-colors run ``` +### Warnings as Errors -## Printer width - -By default, dbt will print out lines padded to 80 characters wide. You can change this setting by adding the following to your `profiles.yml` file: +Turning on the `WARN_ERROR` config will convert dbt warnings into errors. Any time dbt would normally warn, it will instead raise an error. Examples include `--select` criteria that selects no resources, deprecations, configurations with no associated models, invalid test configurations, or tests and freshness checks that are configured to return warnings. - + -```yaml -config: - printer_width: 120 +```text +$ dbt --warn-error run +... ``` -## Send anonymous usage stats +### Writing JSON artifacts -We want to build the best version of dbt possible, and a crucial part of that is understanding how users work with dbt. To this end, we've added some simple event tracking to dbt (using Snowplow). We do not track credentials, model contents or model names (we consider these private, and frankly none of our business). +The `WRITE_JSON` config determines whether dbt writes JSON artifacts (eg. `manifest.json`, `run_results.json`) to the `target/` directory. JSON serialization can be slow, and turning this flag off _might_ make invocations of dbt faster. Alternatively, you might disable this config if you want to perform a dbt operation and avoid overwriting artifacts from a previous run step. -Usage statistics are fired when dbt is invoked and when models are run. These events contain basic platform information (OS + python version). You can see all the event definitions in [`tracking.py`](https://github.com/dbt-labs/dbt-core/blob/HEAD/core/dbt/tracking.py). + -By default this is turned on – you can opt out of event tracking at any time by adding the following to your `profiles.yml` file: -```yaml -config: - send_anonymous_usage_stats: False +```text +dbt --no-write-json run ``` +