Skip to content

Commit 64801d8

Browse files
treyspsungchun12
andauthored
Docs: add examples section overview page (#3388)
Co-authored-by: Sung Won Chung <sungwonchung3@gmail.com>
1 parent 016ab98 commit 64801d8

5 files changed

Lines changed: 49 additions & 6 deletions

File tree

File renamed without changes.
File renamed without changes.

docs/examples/incremental_time_full_walkthrough.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Incremental by Time Range: Full Walkthrough
1+
# Incremental by Time Range
22

33
SQLMesh incremental models are a powerful feature that come in many flavors and configurations so you can fine tune your query performance and scheduled runs **exactly** how you want with a plethora of guardrails.
44

@@ -100,7 +100,7 @@ We have data like the below that gets ingested into our data warehouse on a dail
100100

101101
??? "Code to load the data into BigQuery"
102102

103-
If you want to follow along, here are BigQuery SQL queries to make it easier for you! Just run it directly in the query console. Feel free to adjust for your data warehouse.
103+
If you want to follow along, here are BigQuery SQL queries to make it easier for you! Just run them directly in the query console. Feel free to adjust for your data warehouse.
104104

105105
```sql
106106
-- Create the product_usage table with appropriate schema
@@ -619,11 +619,11 @@ INFO: Application startup complete.
619619
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
620620
```
621621
622-
![image.png](../guides/ui/node_level_audit_trail.png)
622+
![image.png](./incremental_time/node_level_audit_trail.png)
623623
624624
When I click on a column in `demo.incrementals_demo`, it will trace the column to the source!
625625
626-
![image.png](../guides/ui/column_level_audit_trail.png)
626+
![image.png](./incremental_time/column_level_audit_trail.png)
627627
628628
Now, typically, I will promote these changes to production using SQLMesh’s open source GitHub CICD bot as shown in [this demo pull request](https://github.com/TobikoData/tobiko-cloud-demo/pull/4), but to keep this guide simpler, let’s run `sqlmesh plan` directly.
629629
@@ -1213,7 +1213,7 @@ Now that I’ve done all this great work, how do I get this promoted into produc
12131213
12141214
Typically, I will open a pull request combined with the [SQLMesh GitHub CI/CD bot](../integrations/github.md) as I mentioned earlier in this guide. But to keep it simple, I’ll run `sqlmesh plan` as I did above.
12151215
1216-
This time because it it’s promoting a forward-only dev model into prod, it’s a virtual update to the SQL definition.
1216+
This time because it’s promoting a forward-only dev model into prod, it’s a virtual update to the SQL definition.
12171217
12181218
We run a bunch of metadata queries to version tables. More queries (read: 15/15 in the progress bar) are run in this forward-only model promotion to track schema evolution, if it appears, between the old and new schema.
12191219

docs/examples/overview.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Overview
2+
3+
Realistic examples are a fantastic way to understand SQLMesh better.
4+
5+
They allow you to tinker with a project's code and data, issuing different SQLMesh commands to see what happens.
6+
7+
You can reset the examples at any time, so if things get turned around you can just start over!
8+
9+
This page links to a few different types of examples:
10+
11+
- **Walkthroughs** pose a specific story or task, and you follow along as we work through the story
12+
- Walkthroughs **do not** require running code, although the code is available if you would like to
13+
- Different walkthroughs use different SQL engines, so if you want to run the code you might need to update it for your SQL engine
14+
- **Projects** are self-contained SQLMesh projects and datasets
15+
- Projects generally use DuckDB so you can run them locally without installing or accessing a separate SQL engine
16+
17+
!!! tip
18+
19+
If you haven't tried out SQLMesh before, we recommending working through the [SQLMesh Quickstart](../quick_start.md) before trying these examples!
20+
21+
## Walkthroughs
22+
23+
We're proud to present our very first walkthrough, an in-depth description of incremental by time range models:
24+
25+
- [Incremental by Time Range: Full Walkthrough](./incremental_time_full_walkthrough.md) (BigQuery SQL engine)
26+
27+
## Projects
28+
29+
SQLMesh example projects are stored in the [sqlmesh-examples Github repository](https://github.com/TobikoData/sqlmesh-examples). The repository's front page includes additional information about how to download the files and set up the projects.
30+
31+
The two most comprehensive example projects use the SQLMesh `sushi` data, based on a fictional sushi restaurant. ("Tobiko" is the Japanese word for flying fish roe, commonly used in sushi.)
32+
33+
The `sushi` data is described in an [overview notebook](https://github.com/TobikoData/sqlmesh-examples/blob/main/001_sushi/sushi-overview.ipynb) in the repository.
34+
35+
The example repository include two versions of the `sushi` project, at different levels of complexity:
36+
37+
- The [`simple` project](https://github.com/TobikoData/sqlmesh-examples/tree/main/001_sushi/1_simple) contains four `VIEW` and one `SEED` model
38+
- The `VIEW` model kind refreshes every run, making it easy to reason about SQLMesh's behavior
39+
- The [`moderate` project](https://github.com/TobikoData/sqlmesh-examples/tree/main/001_sushi/2_moderate) contains five `INCREMENTAL_BY_TIME_RANGE`, one `FULL`, one `VIEW`, and one `SEED` model
40+
- The incremental models allow you to observe how and when new data is transformed by SQLMesh
41+
- Some models, like `customer_revenue_lifetime`, demonstrate more advanced incremental queries like customer lifetime value calculation

mkdocs.yml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,9 @@ nav:
6060
- concepts/architecture/serialization.md
6161
- concepts/glossary.md
6262
- Examples:
63-
- examples/incremental_time_full_walkthrough.md
63+
- examples/overview.md
64+
- Walkthroughs:
65+
- examples/incremental_time_full_walkthrough.md
6466
- Integrations:
6567
- "Overview": integrations/overview.md
6668
- Tools:

0 commit comments

Comments
 (0)