Quickstart updates and changes to overall help flow. (#222)

* Quickstart updates and changes to overall help flow.

* Additions to guides, more TOC/flow updates, some editing.

* Updated per PR feedback.

Author: technically-tracy

README.md

@@ -1,31 +1,33 @@
-# Overview
+## What is SQLMesh?
 
 SQLMesh is a next-generation SQL transformation platform. It provides you with powerful automation for versioning, backfilling, deployment, and testing — allowing you to focus on simply writing SQL.
 
 SQLMesh is able to achieve all of this with minimal setup; there are no additional services or dependencies required to get started using SQLMesh other than a connection to your existing data warehouse or engine.
 
 ## Why SQLMesh?
 
-One of the main advantages over other transformation frameworks is that SQLMesh does not categorize incrementality as an "advanced" use case that should be avoided unless absolutely necessary. While other  frameworks default to full refresh compute, the default for SQLMesh is to optimize for incremental compute, i.e. computing one day or hour at a time. This allows SQLMesh to be faster and more scalable than other frameworks, allowing you to take advantage of the cost and time savings of incrementality.
+One of the main advantages over other transformation frameworks is that SQLMesh does not categorize incrementality as an "advanced" use case that should be avoided unless absolutely necessary. While other frameworks default to full refresh compute, the default for SQLMesh is to optimize for incremental compute, i.e. computing one day or hour at a time. This allows SQLMesh to be faster and more scalable than other frameworks, allowing you to take advantage of the cost and time savings of incrementality.
 
-SQLMesh also automates away complexity, so configuring models is no longer tricky due to complex macros that require understanding of the context for execution. Writing your data pipelines incrementally with SQLMesh not only saves you money and time, but keeps your systems maintainable, reliable, and accessible to all of your data practicioners.
+SQLMesh also automates away complexity, so configuring models is no longer tricky due to complex macros that require understanding of the context for execution. Writing your data pipelines incrementally with SQLMesh not only saves you money and time, but keeps your systems maintainable, reliable, and accessible to all of your data practictioners.
 
 ### Reduced cost
-As discussed above, incremental compute is significantly cheaper than full refresh compute.
+Incremental compute is significantly cheaper than full refresh compute.
 
-For example, if you have one year of history but only receive new data on a daily basis, only processing that new data is ~365x cheaper than reprocessing one year each day. As your data grows, it's possible that refreshing your tables may take longer than a day, which means you would never be able to catch up!
+For example, if you have one year of history but only receive new data on a daily basis, just processing that new data is ~365x cheaper than reprocessing one year each day. As your data grows, it's possible that refreshing your tables may take longer than a day, which means you would never be able to catch up!
 
 In addition, you may not be able to refresh particular tables all at once; they may need to be batched into smaller intervals. The cost of your data pipelines compound as more dependent pipelines are created. Therefore, writing your data pipelines incrementally as much as possible can result in exponential savings.
 
 ### Increased efficiency
 SQLMesh safely reuses physical tables across isolated environments. Some databases, such as Snowflake, have [zero-copy cloning](https://docs.snowflake.com/en/user-guide/tables-storage-considerations.html#label-cloning-tables) — but this is a manual process, and not widely supported.
 
-SQLMesh is able to automatically reuse tables regardless of which data warehouse or engine you're using. This is achieved by storing fingerprints of your models and by employing [views](https://en.wikipedia.org/wiki/View_(SQL)) like pointers to physical locations. Therefore, spinning up a new development environment is fast and cheap; only models with incompatible changes need to be materialized, once again saving time and money.
+SQLMesh is able to automatically reuse tables regardless of which data warehouse or engine you're using. This is achieved by storing fingerprints of your models and by employing [views](https://en.wikipedia.org/wiki/View_(SQL)) like pointers to physical locations. Therefore, spinning up a new development environment is fast and cheap; only models with incompatible changes need to be materialized, saving time and money.
 
 ### Automation for everyone
-Creating maintainable and scalable data pipelines is extremely difficult, and is a task usually reserved for data engineers. As your data grows, the need for incremental compute becomes mandatory due to the cost and time constaints.
+Creating maintainable and scalable data pipelines is extremely difficult, and a task usually reserved for data engineers. As your data grows, the need for incremental compute becomes mandatory due to the cost and time constaints.
 
-Incremental models have inherent state of which partitions have been computed. This makes managing the consistency and accuracy challenging (leaving no data leakages or gaps). Although a seasoned engineer may have the expertise or tooling to operate one of these tables, an analyst would not. In these organizations, analysts would either need to file a ticket and wait on data engineering resources, or bypass core data models by running their own custom jobs, which inevitably leads to an ungoverned data mess. SQLMesh democratizes the ability to write safe and scalable data pipelines to all data practitioners, regardless of technical ability.
+Incremental models have inherent state of which partitions have been computed. This makes managing the consistency and accuracy challenging (leaving no data leakages or gaps). 
+
+Although a seasoned engineer may have the expertise or tooling to operate one of these tables, an analyst would not. In these organizations, analysts would either need to file a ticket and wait on data engineering resources, or bypass core data models by running their own custom jobs, which inevitably leads to an ungoverned data mess. SQLMesh democratizes the ability to write safe and scalable data pipelines to all data practitioners, regardless of technical ability.
 
 ### Complexity made simple
 As more and more models and users depend on core tables, the complexity of making changes increases. You must ensure that all downstream data consumers are compatible and updated with any new changes.
@@ -35,7 +37,9 @@ Propagating a change throughout a complex graph of dependencies is difficult to
 ### Collaboration and integration
 SQLMesh allows for data pipelines to be a collaborative experience. It both empowers less technical data users to contribute and enables them to collaborate with others who may be more familiar with data engineering. Development can be done in a fully isolated environment that can be accessed and validated by others.
 
-SQLMesh provides information about changes and how they may affect your downstream consumers. This transparency, along with the ability to categorize changes, makes it more feasible for a less technically savvy user to make updates to core data pipelines. By integrating with our Continuous Integration/Continuous Delivery (CI/CD) flows, you can require approval for any changes before going to production, ensuring that the relevant data owners or experts can review and validate the changes.
+SQLMesh provides information about changes and how they may affect your downstream consumers. This transparency, along with the ability to categorize changes, makes it more feasible for a less technically savvy user to make updates to core data pipelines. 
+
+By integrating with our Continuous Integration/Continuous Delivery (CI/CD) flows, you can require approval for any changes before going to production, ensuring that the relevant data owners or experts can review and validate the changes.
 
 ### Testing and reliability
 SQLMesh supports both audits and tests. Although unit tests has been commonplace in the world of software engineering, they are relatively unknown in the data world. SQLMesh's data unit tests allow for stability and reliability, as data pipeline owners can ensure that changes to models don't change underlying logic. These tests can run quickly in CI, or locally without having to create full scale tables.

docs/api/overview.md

@@ -1,11 +1,11 @@
-# Overview
+# API
 
-SQLMesh can be used with a [cli](cli.md), [notebook](notebook.md), or directly through [Python](python.md). Each interface aims to have parity in both functionality and arguments.
+SQLMesh can be used with a [CLI](cli.md), [notebook](notebook.md), or directly through [Python](python.md). Each interface aims to have parity in both functionality and arguments. The following is a list of available commands.
 
 ## plan
 Plan is the main command of SQLMesh. It allows you to interactively create a migration plan, understand the downstream impact, and apply it. All changes to models and environments are materialized through plan.
 
-Read more about [plan](/concepts/plans).
+Read more about [plans](/concepts/plans).
 
 ## evaluate
 Evaluate a model or snapshot (running its query against a DB/Engine). This method is used to test or iterate on models without side effects.
@@ -19,14 +19,18 @@ Given a SQL query, fetches a pandas dataframe.
 ## test
 Runs all tests.
 
+Read more about [testing](/guides/tests).
+
 ## audit
 Runs all audits.
 
+Read more about [auditing](/guides/audits).
+
 ## format
 Formats all SQL model files in place.
 
 ## diff
-Shows the diff between the local model and a model in an evironment.
+Shows the diff between the local model and a model in an environment.
 
 ## dag
 Shows the [DAG](../glossary.md).

docs/api/python.md

@@ -1 +1,3 @@
 # Python
+
+## TODO

docs/concepts/audits.md

@@ -1,8 +1,10 @@
-# Audits
-Audits are one of the tools SQLMesh provides to validate your data. Along with tests, audits are a great way to ensure the quality of your data and to build trust in your data across your organization. A comprehensive suite of audits can identify data issues upstream, whether they are from your vendors or other teams. Audits also empower your data engineers and analysts to work with confidence by catching problems early as they work on new features or make updates to your models.
+# Auditing
+Audits are one of the tools SQLMesh provides to validate your models. Along with [tests](/concepts/tests), they are a great way to ensure the quality of your data and to build trust in it across your organization.  
 
-## What exactly are audits?
-Audits are SQL queries that should not return any rows. In other words, they query for bad data, so returned rows would indicate something is wrong. In its simplest form, an audit is defined with the custom AUDIT expression along with a query as in the following example:
+A comprehensive suite of audits can identify data issues upstream, whether they are from your vendors or other teams. Audits also empower your data engineers and analysts to work with confidence by catching problems early as they work on new features or make updates to your models. 
+
+## Example audit
+In SQLMesh, audits are defined in `.sql` files in an `audit` directory in your SQLMesh project. Multiple audits can be defined in a single file, so you can organize them to your liking. Audits are SQL queries that should not return any rows; in other words, they query for bad data, so returned rows indicates that something is wrong. In its simplest form, an audit is defined with the custom AUDIT expression along with a query, as in the following example:
 
 ```sql
 AUDIT (
@@ -15,15 +17,16 @@ WHERE ds BETWEEN @start_ds AND @end_ds AND
    price IS NULL
 ```
 
-In the example above, we defined an audit named `assert_item_price_is_not_null` on the model `sushi.items` to ensure that every sushi item has a price. If the query is in a different dialect than the rest of your project, you can specify it here as we did in the example, and SQLGlot will automatically understand how to execute the query. While the query can technically be on any model or even multiple models, the model specified in the audit definition tells SQLMesh when to run the audit during your pipeline's execution. If the query returns any records, it means there may be an issue that requires your attention.
+In the example, we defined an audit named `assert_item_price_is_not_null` on the model `sushi.items`, ensuring that every sushi item has a price. 
 
-Audits are defined in `.sql` files in an `audit` directory in your SQLMesh project. Multiple audits can be defined in a single file, so you can organize them to your liking.
+**Note:** If the query is in a different dialect than the rest of your project, you can specify it here as we did in the example, and SQLGlot will automatically understand how to execute the query. 
 
-## Running audits
+While the query can technically be on any model or even multiple models, the model specified in the audit definition tells SQLMesh when to run the audit during your pipeline's execution. If the query returns any records, it means there is a potential issue requiring your attention.
 
+## Run an audit
 ### The CLI audit command
 
-You can execute audits with the `sqlmesh audit` command, as in the following example:
+You can execute audits with the `sqlmesh audit` command as follows:
 ```
 % sqlmesh --path project audit -start 2022-01-01 -end 2022-01-02
 Found 1 audit(s).
@@ -41,10 +44,8 @@ Done.
 When you apply a plan, SQLMesh will automatically run each model's audits. By default, SQLMesh will halt the pipeline when an audit fails in order to prevent potentially invalid data from propagating further downstream. This behvavior can be changed for individual audits. Refer to [Non-blocking audits](#non-blocking-audits).
 
 ## Advanced usage
-
 ### Skipping audits
-
-Audits can be skipped by setting the skip argument to true as in the following example:
+Audits can be skipped by setting the `skip` argument to `true` as in the following example:
 
 ```sql
 AUDIT (
@@ -58,8 +59,7 @@ WHERE ds BETWEEN @start_ds AND @end_ds AND
 ```
 
 ### Non-blocking audits
-
-By default, audits that fail will stop the execution of the pipeline in order to prevent bad data from propagating further downstream. An audit can be configured to notify you when it fails without blocking the execution of the pipeline, as in the following example:
+By default, audits that fail will stop the execution of the pipeline in order to prevent bad data from propagating further. An audit can be configured to notify you when it fails without blocking the execution of the pipeline, as in the following example:
 
 ```sql
 AUDIT (

docs/concepts/configs.md

@@ -1,9 +1,9 @@
 # Configs
-Configs define settings for things like engines (eg. Snowflake or Spark), schedulers (eg. Airflow or Dagster), and the SQL dialect. The config file is defined in config.py in the root directory of your SQLMesh project.
+Configs define settings for things like engines (such as Snowflake or Spark), schedulers (such as Airflow or Dagster), and the SQL dialect. The config file is defined in config.py in the root directory of your SQLMesh project.
 
 ## Settings
 ### connections
-A dictionary of supported connection and their configurations. The key represents a unique connection name. If there is only one connection, its configuration can be provided directly omitting the dictionary.
+A dictionary of supported connection and their configurations. The key represents a unique connection name. If there is only one connection, its configuration can be provided directly, omitting the dictionary.
 
 ```python
 import duckdb
@@ -17,8 +17,7 @@ Config(
 ```
 
 ### scheduler
-Identifies which scheduler backend to use. The scheduler backend is used for both storing metadata and executing [plans](/concepts/plans). By default, the `BuiltinSchedulerBackend` is used which uses the existing SQL engine to store metadata and has a simple scheduler. The `AirflowSchedulerBackend` should be used if you want to integrate with Airflow.
-
+Identifies which scheduler backend to use. The scheduler backend is used both for storing metadata and executing [plans](/concepts/plans). By default, the `BuiltinSchedulerBackend` is used, which uses the existing SQL engine to store metadata and has a simple scheduler. The `AirflowSchedulerBackend` should be used if you want to integrate with Airflow.
 
 ```python
 from sqlmesh.core.config import AirflowSchedulerConfig, Config
@@ -27,16 +26,16 @@ Config(scheduler=AirflowSchedulerConfig())
 ```
 
 ### notification_targets
-Notification targets are used to receive logging or updates as SQLMesh processes things. Notification targets can be used to implement things like integration with Github or Slack.
+Used to receive logging or updates as SQLMesh processes things. Notification targets can be used to implement things such as integration with Github or Slack.
 
 ### dialect
 The default sql dialect of model queries. Default: same as engine dialect. The dialect is used if a [model](/concepts/models) does not define a dialect. Note that this dialect only specifies what the model is written as. At runtime, model queries will be transpiled to the correct engine dialect.
 
 ### physical_schema
-The default schema used to store materialized tables. By default this will store all physical tables managed by SQLMesh in the `sqlmesh` schema/db in your warehouse.
+The default schema used to store materialized tables. By default, this will store all physical tables managed by SQLMesh in the `sqlmesh` schema/db in your warehouse.
 
 ### snapshot_ttl
-Duration before unpromoted snapshots are removed. This is defined as a string with the default be `in 1 week`. Other [relative strings](https://dateparser.readthedocs.io/en/latest/) can be used liked `in 30 days`.
+Duration before unpromoted snapshots are removed. This is defined as a string with the default `in 1 week`. Other [relative strings](https://dateparser.readthedocs.io/en/latest/) can be used, such as `in 30 days`.
 
 ### time_column_format
 The default format to use for all model time columns. Defaults to %Y-%m-%d.
@@ -47,9 +46,8 @@ This time format uses python format codes. https://docs.python.org/3/library/dat
 A list of users that can be used for approvals/notifications.
 
 ## Precedence
-
 You can configure your project in multiple places, and SQLMesh will prioritize configurations according to
-the following order. From least to greatest precedence:
+the following order, from least to greatest precedence:
 
 - A Config object defined in a config.py file at the root of your project:
 
@@ -90,11 +88,10 @@ local_config = Config(
 ... )
 ```
 
-## Using Config
-
-The most common way to configure your SQLMesh project is with a `config.py` module at the root of the
+## Using Config objects
+The most common way to configure your SQLMesh project is with a `config.py` module at the root of your
 project. A SQLMesh Context will automatically look for Config objects there. You can have multiple
-Config objects defined, and then tell Context which one to use. For example, you can have different
+Config objects defined and tell Context which one to use. For example, you can have different
 Configs for local and production environments, Airflow, and Model tests.
 
 Example config.py:
@@ -127,4 +124,3 @@ To use a Config, pass in its variable name to Context.
 
 ```
 
-For more information about the Config class and its parameters, see `sqlmesh.core.config.Config`.