Restructuring and editing docs. (#171)

* Restructuring, rewriting, editing docs.

* Added more content; editing.

* Style changes for tobiko theme.

* Updated per PR review.

* Indigo -> deep purple.

Author: technically-tracy

README.md

@@ -1,5 +1,6 @@
-# About SQLMesh
-SQLMesh is a next-generation SQL transformation platform. It provides you with powerful automation for versioning, backfilling, deployment, and testing -- allowing you to focus on simply writing SQL.
+# Overview
+
+SQLMesh is a next-generation SQL transformation platform. It provides you with powerful automation for versioning, backfilling, deployment, and testing — allowing you to focus on simply writing SQL.
 
 SQLMesh is able to achieve all of this with minimal setup; there are no additional services or dependencies required to get started using SQLMesh other than a connection to your existing data warehouse or engine.
 
@@ -9,42 +10,32 @@ One of the main advantages over other transformation frameworks is that SQLMesh
 
 SQLMesh also automates away complexity, so configuring models is no longer tricky due to complex macros that require understanding of the context for execution. Writing your data pipelines incrementally with SQLMesh not only saves you money and time, but keeps your systems maintainable, reliable, and accessible to all of your data practicioners.
 
-## Reduced cost
+### Reduced cost
 As discussed above, incremental compute is significantly cheaper than full refresh compute.
 
 For example, if you have one year of history but only receive new data on a daily basis, only processing that new data is ~365x cheaper than reprocessing one year each day. As your data grows, it's possible that refreshing your tables may take longer than a day, which means you would never be able to catch up!
 
 In addition, you may not be able to refresh particular tables all at once; they may need to be batched into smaller intervals. The cost of your data pipelines compound as more dependent pipelines are created. Therefore, writing your data pipelines incrementally as much as possible can result in exponential savings.
 
-## Increased efficiency
-SQLMesh safely reuses physical tables across isolated environments. Some databases, such as Snowflake, have [zero-copy cloning](https://docs.snowflake.com/en/user-guide/tables-storage-considerations.html#label-cloning-tables) -- but this is a manual process, and not widely supported.
+### Increased efficiency
+SQLMesh safely reuses physical tables across isolated environments. Some databases, such as Snowflake, have [zero-copy cloning](https://docs.snowflake.com/en/user-guide/tables-storage-considerations.html#label-cloning-tables) — but this is a manual process, and not widely supported.
 
 SQLMesh is able to automatically reuse tables regardless of which data warehouse or engine you're using. This is achieved by storing fingerprints of your models and by employing [views](https://en.wikipedia.org/wiki/View_(SQL)) like pointers to physical locations. Therefore, spinning up a new development environment is fast and cheap; only models with incompatible changes need to be materialized, once again saving time and money.
 
-## Automation for everyone
+### Automation for everyone
 Creating maintainable and scalable data pipelines is extremely difficult, and is a task usually reserved for data engineers. As your data grows, the need for incremental compute becomes mandatory due to the cost and time constaints.
 
 Incremental models have inherent state of which partitions have been computed. This makes managing the consistency and accuracy challenging (leaving no data leakages or gaps). Although a seasoned engineer may have the expertise or tooling to operate one of these tables, an analyst would not. In these organizations, analysts would either need to file a ticket and wait on data engineering resources, or bypass core data models by running their own custom jobs, which inevitably leads to an ungoverned data mess. SQLMesh democratizes the ability to write safe and scalable data pipelines to all data practitioners, regardless of technical ability.
 
-## Complexity made simple
+### Complexity made simple
 As more and more models and users depend on core tables, the complexity of making changes increases. You must ensure that all downstream data consumers are compatible and updated with any new changes.
 
 Propagating a change throughout a complex graph of dependencies is difficult to communicate, and also challenging to do accurately. The introduction of other schedulers such as [Airflow](https://airflow.apache.org/) adds even more complexity. SQLMesh seamlessly integrates directly with your existing scheduler so that your entire data pipeline, including jobs outside of SQLMesh, will be unified and robust.
 
-## Collaboration and integration
+### Collaboration and integration
 SQLMesh allows for data pipelines to be a collaborative experience. It both empowers less technical data users to contribute and enables them to collaborate with others who may be more familiar with data engineering. Development can be done in a fully isolated environment that can be accessed and validated by others.
 
 SQLMesh provides information about changes and how they may affect your downstream consumers. This transparency, along with the ability to categorize changes, makes it more feasible for a less technically savvy user to make updates to core data pipelines. By integrating with our Continuous Integration/Continuous Delivery (CI/CD) flows, you can require approval for any changes before going to production, ensuring that the relevant data owners or experts can review and validate the changes.
 
-## Testing and reliability
-SQLMesh supports both [audits](#audits) and [tests](#tests). Although unit tests has been commonplace in the world of software engineering, they are relatively unknown in the data world. SQLMesh's data unit tests allow for stability and reliability, as data pipeline owners can ensure that changes to models don't change underlying logic. These tests can run quickly in CI, or locally without having to create full scale tables.
-
-Ready to jump in? Refer to `sqlmesh.docs.getting_started`.
-
-## Community
-
-We'd love to help guide you along your data journey. Follow the links below to connect with us:
-
-* Join the [tobiko Slack community](https://join.slack.com/t/tobiko-data/shared_invite/zt-1je7o3xhd-C7~GuZTj0a8xz_uQbTJjHg) to ask questions, or just to say hi!
-* File an issue on our [GitHub](https://github.com/TobikoData/sqlmesh/issues/new).
-* Send us an email at [hello@tobikodata.com](hello@tobikodata.com) with your questions or feedback.
+### Testing and reliability
+SQLMesh supports both audits and tests. Although unit tests has been commonplace in the world of software engineering, they are relatively unknown in the data world. SQLMesh's data unit tests allow for stability and reliability, as data pipeline owners can ensure that changes to models don't change underlying logic. These tests can run quickly in CI, or locally without having to create full scale tables.

docs/api/cli.md

@@ -31,20 +31,20 @@ Usage: sqlmesh plan [OPTIONS] [ENVIRONMENT]
   Plan a migration of the current context's models with the given environment.
 
 Options:
-  -s, --start TEXT          The start datetime of the interval this command
-                            will be applied for.
-  -e, --end TEXT            The end datetime of the interval this command will
-                            be applied for.
-  -f, --from TEXT           The environment to base the plan on instead of
+  -s, --start TEXT          The start datetime of the interval for which this
+                            command will be applied.
+  -e, --end TEXT            The end datetime of the interval for which this 
+                            command will be applied.
+  -f, --from TEXT           The environment to base the plan on rather than
                             local files.
   --skip-tests TEXT         Skip tests prior to generating the plan if they
                             are defined.
   -r, --restate-model TEXT  Restate data for specified models and models
                             downstream from the one specified. For production
-                            environment all related model versions will have
-                            their intervals wiped but only the current
+                            environment, all related model versions will have
+                            their intervals wiped, but only the current
                             versions will be backfilled. For development
-                            enviornment only the current model versions will
+                            environment, only the current model versions will
                             be affected.
   --no-gaps                 Ensure that new snapshots have no data gaps when
                             comparing to existing snapshots for matching
@@ -53,7 +53,7 @@ Options:
   --forward-only            Create a plan for forward-only changes.
   --no-prompts              Disable interactive prompts for the backfill time
                             range. Please note that if this flag is set and
-                            there are uncategorized changes the plan creation
+                            there are uncategorized changes, plan creation
                             will fail.
   --auto-apply              Automatically apply the new plan after creation.
   --help                    Show this message and exit.
@@ -66,13 +66,13 @@ Usage: sqlmesh evaluate [OPTIONS] MODEL
   Evaluate a model and return a dataframe with a default limit of 1000.
 
 Options:
-  -s, --start TEXT   The start datetime of the interval this command will be
-                     applied for.
-  -e, --end TEXT     The end datetime of the interval this command will be
-                     applied for.
-  -l, --latest TEXT  The latest time used for non incremental datasets
+  -s, --start TEXT   The start datetime of the interval for which this 
+                     command will be applied.
+  -e, --end TEXT     The end datetime of the interval for which this 
+                     command will be applied.
+  -l, --latest TEXT  The latest time used for non-incremental datasets
                      (defaults to now).
-  --limit INTEGER    The number of rows which the query should be limited to.
+  --limit INTEGER    The number of rows the query should be limited to.
   --help             Show this message and exit.
 ```
 
@@ -83,11 +83,11 @@ Usage: sqlmesh render [OPTIONS] MODEL
   Renders a model's query, optionally expanding referenced models.
 
 Options:
-  -s, --start TEXT   The start datetime of the interval this command will be
-                     applied for.
-  -e, --end TEXT     The end datetime of the interval this command will be
-                     applied for.
-  -l, --latest TEXT  The latest time used for non incremental datasets
+  -s, --start TEXT   The start datetime of the interval for which this 
+                     command will be applied.
+  -e, --end TEXT     The end datetime of the interval for which this 
+                     command will be applied.
+  -l, --latest TEXT  The latest time used for non-incremental datasets
                      (defaults to now).
   --expand TEXT      Whether or not to expand materialized models (defaults to
                      False). If True, all referenced models are expanded as
@@ -126,11 +126,11 @@ Usage: sqlmesh audit [OPTIONS]
 
 Options:
   --model TEXT       A model to audit. Multiple models can be audited.
-  -s, --start TEXT   The start datetime of the interval this command will be
-                     applied for.
-  -e, --end TEXT     The end datetime of the interval this command will be
-                     applied for.
-  -l, --latest TEXT  The latest time used for non incremental datasets
+  -s, --start TEXT   The start datetime of the interval for which this 
+                     command will be applied.
+  -e, --end TEXT     The end datetime of the interval for which this 
+                     command will be applied.
+  -l, --latest TEXT  The latest time used for non-incremental datasets
                      (defaults to now).
   --help             Show this message and exit.
 ```
@@ -165,6 +165,6 @@ Usage: sqlmesh dag [OPTIONS]
   graphviz package.
 
 Options:
-  --file TEXT  The file to write the dag image to.
+  --file TEXT  The file to which the dag image should be written.
   --help       Show this message and exit.
 ```

docs/api/notebook.md

@@ -1,6 +1,6 @@
 # Notebook
 
-SQLMesh supports JupyterLabs and Databricks Notebooks. Magics are loaded automatically and use the variable `context`
+SQLMesh supports JupyterLabs and Databricks Notebooks. Magics are loaded automatically and use the variable `context`.
 
 ```python
 from sqlmesh import Context
@@ -15,25 +15,25 @@ context = Context(path="example")
         [--skip-backfill] [--forward-only] [--no-prompts] [--auto-apply]
         [environment]
 
-Goes through a set of prompts to both establish a plan and apply it
+Iterates through a set of prompts to both establish a plan and apply it.
 
 positional arguments:
-  environment           The environment to run the plan against
+  environment           The environment to run the plan against.
 
 options:
   --start START, -s START
                         Start date to backfill.
   --end END, -e END     End date to backfill.
   --from FROM_, -f FROM_
-                        The environment to base the plan on instead of local
+                        The environment to base the plan on rather than local
                         files.
   --skip-tests, -t      Skip the unit tests defined for the model.
   --restate-model <[RESTATE_MODEL ...]>, -r <[RESTATE_MODEL ...]>
                         Restate data for specified models (and models
                         downstream from the one specified). For production
-                        environment all related model versions will have their
-                        intervals wiped but only the current versions will be
-                        backfilled. For development enviornment only the
+                        environment, all related model versions will have their
+                        intervals wiped, but only the current versions will be
+                        backfilled. For development environment, only the
                         current model versions will be affected.
   --no-gaps, -g         Ensure that new snapshots have no data gaps when
                         comparing to existing snapshots for matching models in
@@ -42,7 +42,7 @@ options:
   --forward-only        Create a plan for forward-only changes.
   --no-prompts          Disables interactive prompts for the backfill time
                         range. Please note that if this flag is set and there
-                        are uncategorized changes the plan creation will fail.
+                        are uncategorized changes, plan creation will fail.
   --auto-apply          Automatically applies the new plan after creation.
 ```
 
@@ -52,7 +52,7 @@ options:
 %evaluate [--start START] [--end END] [--latest LATEST] [--limit LIMIT]
             model
 
-Evaluate a model query and fetches a dataframe.
+Evaluate a model query and fetch a dataframe.
 
 positional arguments:
   model                 The model.
@@ -63,8 +63,8 @@ options:
   --end END, -e END     End date to render.
   --latest LATEST, -l LATEST
                         Latest date to render.
-  --limit LIMIT         The number of rows which the query should be limited
-                        to.
+  --limit LIMIT         The number of rows for which which the query 
+                        should be limited.
 ```
 
 ## render
@@ -77,21 +77,22 @@ TODO
 Fetches a dataframe from sql, optionally storing it in a variable.
 
 positional arguments:
-  df_var  An optional variable name to the store the resulting dataframe in.
+  df_var  An optional variable name to store the resulting dataframe.
 ```
 
 ## test
 ```
   %test [--ls] model [test_name]
 
-Allow the user to list tests for a model, output a specific test and then write their changes back
+Allow the user to list tests for a model, output a specific test, and 
+then write their changes back.
 
 positional arguments:
   model      The model.
-  test_name  The test name to display
+  test_name  The test name to display.
 
 options:
-  --ls       List tests associated with a model
+  --ls       List tests associated with a model.
 ```
 
 ## audit
@@ -107,5 +108,5 @@ TODO
 ```
  %dag
 
-Displays the dag
+Displays the dag.
 ```

docs/api/overview.md

@@ -1,9 +1,9 @@
 # Overview
 
-SQLMesh can be used with a [cli](cli.md), [notebook](notebook.md), or directly through [Python](python.md). Each interface tries to have equivalent functionality and arguments.
+SQLMesh can be used with a [cli](cli.md), [notebook](notebook.md), or directly through [Python](python.md). Each interface aims to have parity in both functionality and arguments.
 
 ## plan
-Plan is the main command of SQLMesh. It allows you to interactively create a migration plan, understand the downstream impact, and apply it. All changes to models and environments will be materialized through plan.
+Plan is the main command of SQLMesh. It allows you to interactively create a migration plan, understand the downstream impact, and apply it. All changes to models and environments are materialized through plan.
 
 Read more about [plan](/concepts/plans).
 
@@ -29,4 +29,4 @@ Formats all SQL model files in place.
 Shows the diff between the local model and a model in an evironment.
 
 ## dag
-Shows the dag.
+Shows the [DAG](../glossary.md).

docs/community.md

@@ -0,0 +1,8 @@
+# Join our community
+
+We'd love to help guide you along your data journey. Connect with us in the following ways:
+
+* Join the [tobiko Slack community](https://join.slack.com/t/tobiko-data/shared_invite/zt-1je7o3xhd-C7~GuZTj0a8xz_uQbTJjHg) to ask questions, or just to say hi!
+* File an issue on our [GitHub](https://github.com/TobikoData/sqlmesh/issues/new).
+* Send us an email at [hello@tobikodata.com](hello@tobikodata.com) with your questions or feedback.
+