Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/examples/incremental_time_full_walkthrough.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Incremental by Time Range: Full Walkthrough
# Incremental by Time Range
Comment thread
sungchun12 marked this conversation as resolved.

SQLMesh incremental models are a powerful feature that come in many flavors and configurations so you can fine tune your query performance and scheduled runs **exactly** how you want with a plethora of guardrails.

Expand Down Expand Up @@ -100,7 +100,7 @@ We have data like the below that gets ingested into our data warehouse on a dail

??? "Code to load the data into BigQuery"

If you want to follow along, here are BigQuery SQL queries to make it easier for you! Just run it directly in the query console. Feel free to adjust for your data warehouse.
If you want to follow along, here are BigQuery SQL queries to make it easier for you! Just run them directly in the query console. Feel free to adjust for your data warehouse.

```sql
-- Create the product_usage table with appropriate schema
Expand Down Expand Up @@ -619,11 +619,11 @@ INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
```

![image.png](../guides/ui/node_level_audit_trail.png)
![image.png](./incremental_time/node_level_audit_trail.png)

When I click on a column in `demo.incrementals_demo`, it will trace the column to the source!

![image.png](../guides/ui/column_level_audit_trail.png)
![image.png](./incremental_time/column_level_audit_trail.png)

Now, typically, I will promote these changes to production using SQLMesh’s open source GitHub CICD bot as shown in [this demo pull request](https://github.com/TobikoData/tobiko-cloud-demo/pull/4), but to keep this guide simpler, let’s run `sqlmesh plan` directly.

Expand Down
41 changes: 41 additions & 0 deletions docs/examples/overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Overview

Realistic examples are a fantastic way to understand SQLMesh better.

They allow you to tinker with a project's code and data, issuing different SQLMesh commands to see what happens.

You can reset the examples at any time, so if things get turned around you can just start over!

This page links to a few different types of examples:

- **Walkthroughs** pose a specific story or task, and you follow along as we work through the story
- Walkthroughs **do not** require running code, although the code is available if you would like to
- Different walkthroughs use different SQL engines, so if you want to run the code you might need to update it for your SQL engine
- **Projects** are self-contained SQLMesh projects and datasets
- Projects generally use DuckDB so you can run them locally without installing or accessing a separate SQL engine

!!! tip

If you haven't tried out SQLMesh before, we recommending working through the [SQLMesh Quickstart](../quick_start.md) before trying these examples!

## Walkthroughs

We're proud to present our very first walkthrough, an in-depth description of incremental by time models:

- [Incremental by time: full walkthrough](./incremental_time_full_walkthrough.md) (BigQuery SQL engine)

## Projects

SQLMesh example projects are stored in the [sqlmesh-examples Github repository](https://github.com/TobikoData/sqlmesh-examples). The repository's front page includes additional information about how to download the files and set up the projects.

The two most comprehensive example projects use the SQLMesh `sushi` data, based on a fictional sushi restaurant. ("Tobiko" is the Japanese word for flying fish roe, commonly used in sushi.)

The `sushi` data is described in an [overview notebook](https://github.com/TobikoData/sqlmesh-examples/blob/main/001_sushi/sushi-overview.ipynb) in the repository.

The example repository include two versions of the `sushi` project, at different levels of complexity:

- The [`simple` project](https://github.com/TobikoData/sqlmesh-examples/tree/main/001_sushi/1_simple) contains four `VIEW` and one `SEED` model
- The `VIEW` model kind refreshes every run, making it easy to reason about SQLMesh's behavior
- The [`moderate` project](https://github.com/TobikoData/sqlmesh-examples/tree/main/001_sushi/2_moderate) contains five `INCREMENTAL_BY_TIME_RANGE`, one `FULL`, one `VIEW`, and one `SEED` model
- The incremental models allow you to observe how and when new data is transformed by SQLMesh
- Some models, like `customer_revenue_lifetime`, demonstrate more advanced incremental queries like customer lifetime value calculation
4 changes: 3 additions & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,9 @@ nav:
- concepts/architecture/serialization.md
- concepts/glossary.md
- Examples:
- examples/incremental_time_full_walkthrough.md
- examples/overview.md
- Walkthroughs:
- examples/incremental_time_full_walkthrough.md
- Integrations:
- "Overview": integrations/overview.md
- Tools:
Expand Down