Skip to content

Commit

Permalink
Update global configs for v 1.1 (#1319)
Browse files Browse the repository at this point in the history 
Co-authored-by: runleonarun <leona@dbtlabs.com>
Co-authored-by: Jeremy Cohen <jeremy@dbtlabs.com>
  • Loading branch information
@runleonarun @jtcohen6
3 people authored Apr 14, 2022
1 parent 22dd097 commit bd5f272
Showing 1 changed file with 161 additions and 72 deletions.
233 changes: 161 additions & 72 deletions website/docs/reference/global-configs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: "Global Configs"
id: "global-configs"
---

### About Global Configs
## About Global Configs

Global configs enable you to fine-tune how dbt runs projects on your machine—whether your personal laptop, an orchestration tool running remotely, or (in some cases) dbt Cloud. They differ from [project configs](reference/dbt_project.yml) and [resource configs](reference/configs-and-properties), which tell dbt _what_ to run.

Expand All @@ -13,7 +13,7 @@ These configs are "global" because they are available for all dbt commands, and

Starting in v1.0, you can set global configs in three places. When all three are set, command line flags take precedence, then environment variables, and last profile configs.

#### Command line flags
## Command line flags

Command line (CLI) flags immediately follow `dbt` and precede your subcommand. When set, CLI flags override environment variables and profile configs.

Expand Down Expand Up @@ -73,7 +73,7 @@ $ dbt --no-version-check run

</File>

#### Environment variables
## Environment variables

Environment variables contain a `DBT_` prefix

Expand All @@ -88,7 +88,7 @@ $ dbt run

</File>

#### Profile (or user) configurations
## Profile (or user) configurations

You can set profile (or user) configurations in the `config:` block of `profiles.yml`. You would use the profile config to set defaults for all projects running on your local machine.

Expand All @@ -103,46 +103,54 @@ config:

</File>

## Failing fast
<VersionBlock firstVersion="1.1">

Supply the `-x` or `--fail-fast` flag to `dbt run` to make dbt exit immediately if a single resource fails to build. If other models are in-progress when the first model fails, then dbt will terminate the connections for these still-running models.
### Cache database objects for selected resource

For example, you can select four models to run, but if a failure occurs in the first model, the failure will prevent other models from running:
:::caution Experimental config flag
This should be considered an _unstable_ config flag because it is experimental and subject to change. We reserve the right to make breaking changes to this config flag.
:::

At the start of runs, dbt caches metadata about all the objects in all the schemas where it might materialize resources (such as models). By default, dbt caches all schemas related to the project. When this config is enabled, dbt will only cache schemas related to selected resources for the current run. This can offer significant speed improvements when running a small subset of a large project.

For example, to improve speed and performance while focused on developing Salesforce models, which are materialized into their own dedicated schema, you would select those models and pass the `cache-selected-only` flag:

```text
$ dbt -x run --threads 1
Running with dbt=1.0.0
Found 4 models, 1 test, 1 snapshot, 2 analyses, 143 macros, 0 operations, 1 seed file, 0 sources
14:47:39 | Concurrency: 1 threads (target='dev')
14:47:39 |
14:47:39 | 1 of 4 START table model test_schema.model_1........... [RUN]
14:47:40 | 1 of 4 ERROR creating table model test_schema.model_1.. [ERROR in 0.06s]
14:47:40 | 2 of 4 START view model test_schema.model_2............ [RUN]
14:47:40 | CANCEL query model.debug.model_2....................... [CANCEL]
14:47:40 | 2 of 4 ERROR creating view model test_schema.model_2... [ERROR in 0.05s]
$ dbt --cache-selected-only run --select salesforce
Database Error in model model_1 (models/model_1.sql)
division by zero
compiled SQL at target/run/debug/models/model_1.sql
```

The previous example enables you to start working with your salesforce models, and dbt will only cache those models instead of the entire project.

To set the config in your `profile.yml`:
<File name='profiles.yml'>

```yaml

config:
cache_selected_only: true

Encountered an error:
FailFast Error in model model_1 (models/model_1.sql)
Failing early due to test failure or runtime error
```

## Checking version compatibility
</File>

</VersionBlock>

### Checking version compatibility

Projects are recommended to set [dbt version requirements](require-dbt-version), especially if they use features that are newer, or which may break in future versions of dbt Core. By default, if you run a project with an incompatible dbt version, dbt will raise an error.

You can use the `VERSION_CHECK` config to disable this check and suppress the error message:

```
$ dbt --no-version-check run
Running with dbt=1.0.0
Found 13 models, 2 tests, 1 archives, 0 analyses, 204 macros, 2 operations....
```

## Debug-level logging
### Debug-level logging

The `DEBUG` config redirects dbt's debug logs to standard out. The has the effect of showing debug-level log information in the terminal in addition to the `logs/dbt.log` file. This output is verbose.

The `--debug` flag is also available via shorthand as `-d`.
Expand All @@ -155,9 +163,52 @@ $ dbt --debug run
```

</File>

### Experimental parser

With the `USE_EXPERIMENTAL_PARSER` config, you can opt into the latest and greatest experimental version of the static parser, which is still being sampled for 100% correctness. See [the docs on parsing](parsing#experimental-parser) for more details.

<File name='profiles.yml'>

```yaml

config:
use_experimental_parser: true

```

</File>

## Log Formatting
### Failing fast

Supply the `-x` or `--fail-fast` flag to `dbt run` to make dbt exit immediately if a single resource fails to build. If other models are in-progress when the first model fails, then dbt will terminate the connections for these still-running models.

For example, you can select four models to run, but if a failure occurs in the first model, the failure will prevent other models from running:

```text
$ dbt -x run --threads 1
Running with dbt=1.0.0
Found 4 models, 1 test, 1 snapshot, 2 analyses, 143 macros, 0 operations, 1 seed file, 0 sources
14:47:39 | Concurrency: 1 threads (target='dev')
14:47:39 |
14:47:39 | 1 of 4 START table model test_schema.model_1........... [RUN]
14:47:40 | 1 of 4 ERROR creating table model test_schema.model_1.. [ERROR in 0.06s]
14:47:40 | 2 of 4 START view model test_schema.model_2............ [RUN]
14:47:40 | CANCEL query model.debug.model_2....................... [CANCEL]
14:47:40 | 2 of 4 ERROR creating view model test_schema.model_2... [ERROR in 0.05s]
Database Error in model model_1 (models/model_1.sql)
division by zero
compiled SQL at target/run/debug/models/model_1.sql
Encountered an error:
FailFast Error in model model_1 (models/model_1.sql)
Failing early due to test failure or runtime error
```

### Log Formatting

The `LOG_FORMAT` config specifies how dbt's logs should be formatted. If the value of this config is `json`, dbt will output fully structured logs in JSON format; otherwise, it will output text-formatted logs that are sparser for the CLI and more detailed in `logs/dbt.log`.

Expand All @@ -171,6 +222,7 @@ $ dbt --log-format json run
:::tip Tip: verbose structured logs

Use `json` formatting value in conjunction with the `DEBUG` config to produce rich log information which can be piped into monitoring tools for analysis:

```text
$ dbt --debug --log-format json run
```
Expand All @@ -179,61 +231,64 @@ See [structured logging](events-logging#structured-logging) for more details.

:::


</File>

## Writing JSON artifacts

The `WRITE_JSON` config determines whether dbt writes JSON artifacts (eg. `manifest.json`, `run_results.json`) to the `target/` directory. JSON serialization can be slow, and turning this flag off _might_ make invocations of dbt faster. Alternatively, you might disable this config if you want to perform a dbt operation and avoid overwriting artifacts from a previous run step.

<File name='Usage'>
### Partial Parsing

```text
$ dbt --no-write-json run
```
The `PARTIAL_PARSE` config can turn partial parsing on or off in your project. See [the docs on parsing](parsing#partial-parsing) for more details.

</File>
<File name='profiles.yml'>

## Strict
```yaml

As of v1.0, the `-S` or `--strict` flag has been deprecated.
config:
partial_parse: true

## Warnings as Errors
```

Turning on the `WARN_ERROR` config will convert dbt warnings into errors. Any time dbt would normally warn, it will instead raise an error. Examples include `--select` criteria that selects no resources, deprecations, configurations with no associated models, invalid test configurations, or tests and freshness checks that are configured to return warnings.
</File>

<File name='Usage'>

```text
$ dbt --warn-error run
...
dbt --no-partial-parse run
```

</File>

## Partial Parsing
The `PARTIAL_PARSE` config can turn partial parsing on or off in your project. See [the docs on parsing](parsing#partial-parsing) for more details.
### Printer width

By default, dbt will print out lines padded to 80 characters wide. You can change this setting by adding the following to your `profiles.yml` file:

<File name='profiles.yml'>

```yaml

config:
partial_parse: true

printer_width: 120
```
</File>
<File name='Usage'>
### Send anonymous usage stats
```text
$ dbt --no-partial-parse run
We want to build the best version of dbt possible, and a crucial part of that is understanding how users work with dbt. To this end, we've added some simple event tracking to dbt (using Snowplow). We do not track credentials, model contents or model names (we consider these private, and frankly none of our business).
Usage statistics are fired when dbt is invoked and when models are run. These events contain basic platform information (OS + python version). You can see all the event definitions in [`tracking.py`](https://github.com/dbt-labs/dbt-core/blob/HEAD/core/dbt/tracking.py).

By default this is turned on – you can opt out of event tracking at any time by adding the following to your `profiles.yml` file:

```yaml
config:
send_anonymous_usage_stats: False
```

</File>
You can also use the DO_NOT_TRACK environmental variable to enable or disable sending anonymous data. For more information, see [Environmental variables](/dbt-cloud/using-dbt-cloud/cloud-environment-variables).

`DO_NOT_TRACK=1` is the same as `DBT_SEND_ANONYMOUS_USAGE_STATS=False`
`DO_NOT_TRACK=0` is the same as `DBT_SEND_ANONYMOUS_USAGE_STATS=True`

### Static parser

## Static parser
The `STATIC_PARSER` config can enable or disable use of the static parser. See [the docs on parsing](parsing#static-parser) for more details.

<File name='profiles.yml'>
Expand All @@ -247,21 +302,57 @@ config:

</File>

## Experimental parser
With the `USE_EXPERIMENTAL_PARSER` config, you can opt into the latest and greatest experimental version of the static parser, which is still being sampled for 100% correctness. See [the docs on parsing](parsing#experimental-parser) for more details.
### Strict

As of v1.0, the `-S` or `--strict` flag has been deprecated.

<VersionBlock firstVersion="1.1">

### Suppress non-error logs in output

By default, dbt shows all logs in standard out (stdout). You can use the `QUIET` config to show only error logs in stdout. Logs will still include the output of anything passed to the `print()` macro. For example, you might suppress all but error logs to more easily find and debug a jinja error.

<File name='profiles.yml'>

```yaml

config:
use_experimental_parser: true
quiet: true
```

</File>

Supply the `-q` or `--quiet` flag to `dbt run` to show only error logs and suppress non-error logs.

```text
$ dbt --quiet run
...
```

### Suppress `print()` messages in stdout

By default, dbt includes `print()` messages in standard out (stdout). You can use the `NO_PRINT` config to prevent these messages from showing up in stdout.

<File name='profiles.yml'>

```yaml
config:
no_print: true
```

</File>

## Use colors
Supply `--no-print` flag to `dbt run` to suppress `print()` messages from showing in stdout.

```text
$ dbt --no-print run
...
```

</VersionBlock>

### Use colors

By default, dbt will colorize the output it prints in your terminal. You can turn this off by adding the following to your `profiles.yml` file:

Expand All @@ -272,36 +363,34 @@ config:
use_colors: False
```

```
```text
$ dbt --use-colors run
$ dbt --no-use-colors run
```

</File>

### Warnings as Errors

## Printer width

By default, dbt will print out lines padded to 80 characters wide. You can change this setting by adding the following to your `profiles.yml` file:
Turning on the `WARN_ERROR` config will convert dbt warnings into errors. Any time dbt would normally warn, it will instead raise an error. Examples include `--select` criteria that selects no resources, deprecations, configurations with no associated models, invalid test configurations, or tests and freshness checks that are configured to return warnings.

<File name='profiles.yml'>
<File name='Usage'>

```yaml
config:
printer_width: 120
```text
$ dbt --warn-error run
...
```

</File>

## Send anonymous usage stats
### Writing JSON artifacts

We want to build the best version of dbt possible, and a crucial part of that is understanding how users work with dbt. To this end, we've added some simple event tracking to dbt (using Snowplow). We do not track credentials, model contents or model names (we consider these private, and frankly none of our business).
The `WRITE_JSON` config determines whether dbt writes JSON artifacts (eg. `manifest.json`, `run_results.json`) to the `target/` directory. JSON serialization can be slow, and turning this flag off _might_ make invocations of dbt faster. Alternatively, you might disable this config if you want to perform a dbt operation and avoid overwriting artifacts from a previous run step.

Usage statistics are fired when dbt is invoked and when models are run. These events contain basic platform information (OS + python version). You can see all the event definitions in [`tracking.py`](https://github.com/dbt-labs/dbt-core/blob/HEAD/core/dbt/tracking.py).
<File name='Usage'>

By default this is turned on – you can opt out of event tracking at any time by adding the following to your `profiles.yml` file:
```yaml
config:
send_anonymous_usage_stats: False
```text
dbt --no-write-json run
```

</File>

0 comments on commit bd5f272

Please sign in to comment.