Feat/3154 convert script preprocess docs to python and add destination capabilities section to destination pages

.github/workflows/build_docs.yml

@@ -0,0 +1,36 @@
+name: docs | build docs
+
+on:
+  workflow_call:
+  workflow_dispatch:
+
+jobs:
+  build_docs:
+    name: docs | build docs
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Check out
+        uses: actions/checkout@master
+
+      - uses: pnpm/action-setup@v2
+        with:
+          version: 9.13.2
+
+      - uses: actions/setup-node@v5
+        with:
+          node-version: '22'
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install node dependencies
+        run: cd docs/website && npm install
+
+      - name: Install python dependencies
+        run: cd docs/website && pip install -r requirements.txt
+
+      - name: Build docs
+        run: cd docs/website && npm run build:cloudflare

.github/workflows/main.yml

@@ -30,6 +30,11 @@ jobs:
   test_docs_snippets:
     name: test snippets in docs
     uses: ./.github/workflows/test_docs_snippets.yml
+
+  # NOTE: we build docs the same way as on cloudflare, so we can catch problems early
+  build_docs:
+    name: build docs
+    uses: ./.github/workflows/build_docs.yml
 
   lint:
     name: lint on all python versions

docs/website/docs/dlt-ecosystem/destinations/athena.md

@@ -8,6 +8,8 @@ keywords: [aws, athena, glue catalog]
 
 The Athena destination stores data as Parquet files in S3 buckets and creates [external tables in AWS Athena](https://docs.aws.amazon.com/athena/latest/ug/creating-tables.html). You can then query those tables with Athena SQL commands, which will scan the entire folder of Parquet files and return the results. This destination works very similarly to other SQL-based destinations, with the exception that the merge write disposition is not supported at this time. The `dlt` metadata will be stored in the same bucket as the Parquet files, but as iceberg tables. Athena also supports writing individual data tables as Iceberg tables, so they may be manipulated later. A common use case would be to strip GDPR data from them.
 
+<!--@@@DLT_DESTINATION_CAPABILITIES athena-->
+
 ## Install dlt with Athena
 **To install the dlt library with Athena dependencies:**
 ```sh

docs/website/docs/dlt-ecosystem/destinations/bigquery.md

@@ -13,6 +13,7 @@ keywords: [bigquery, destination, data warehouse]
 ```sh
 pip install "dlt[bigquery]"
 ```
+<!--@@@DLT_DESTINATION_CAPABILITIES bigquery-->
 
 ## Setup guide
 
@@ -228,8 +229,8 @@ BigQuery supports the following [column hints](../../general-usage/schema#tables
 
 :::warning
 **Deprecation Notice:**
-Per-column `cluster` hints are deprecated and will be removed in a future release.  
-**To migrate, use the `cluster` argument of the `bigquery_adapter` instead.**  
+Per-column `cluster` hints are deprecated and will be removed in a future release.
+**To migrate, use the `cluster` argument of the `bigquery_adapter` instead.**
 See the [example below](#use-an-adapter-to-apply-hints-to-a-resource) for how to specify clustering columns with the adapter.
 :::
 

docs/website/docs/dlt-ecosystem/destinations/clickhouse.md

@@ -14,6 +14,8 @@ keywords: [ clickhouse, destination, data warehouse ]
 pip install "dlt[clickhouse]"
 ```
 
+<!--@@@DLT_DESTINATION_CAPABILITIES clickhouse-->
+
 ## Setup guide
 
 ### 1. Initialize the dlt project

docs/website/docs/dlt-ecosystem/destinations/databricks.md

@@ -26,6 +26,8 @@ Databricks supports both **Delta** (default) and **Apache Iceberg** table format
 pip install "dlt[databricks]"
 ```
 
+<!--@@@DLT_DESTINATION_CAPABILITIES databricks-->
+
 ## Set up your Databricks workspace
 
 To use the Databricks destination, you need:

docs/website/docs/dlt-ecosystem/destinations/destination.md

@@ -19,6 +19,8 @@ To install `dlt` without additional dependencies:
 pip install dlt
 ```
 
+<!--@@@DLT_DESTINATION_CAPABILITIES destination-->
+
 ## Set up a destination function for your pipeline
 
 The custom destination decorator differs from other destinations in that you do not need to provide connection credentials, but rather you provide a function that gets called for all items loaded during a pipeline run or load operation. With the `@dlt.destination`, you can convert any function that takes two arguments into a `dlt` destination.

docs/website/docs/dlt-ecosystem/destinations/dremio.md

@@ -12,6 +12,8 @@ keywords: [dremio, iceberg, aws, glue catalog]
 pip install "dlt[dremio,s3]"
 ```
 
+<!--@@@DLT_DESTINATION_CAPABILITIES dremio-->
+
 ## Setup guide
 ### 1. Initialize the dlt project
 

docs/website/docs/dlt-ecosystem/destinations/duckdb.md

@@ -12,6 +12,8 @@ keywords: [duckdb, destination, data warehouse]
 pip install "dlt[duckdb]"
 ```
 
+<!--@@@DLT_DESTINATION_CAPABILITIES duckdb-->
+
 ## Setup guide
 
 **1. Initialize a project with a pipeline that loads to DuckDB by running:**
@@ -280,7 +282,7 @@ dest_ = dlt.destinations.duckdb(
     DuckDbCredentials("duck.db", extensions=["spatial"], local_config={"errors_as_json": True})
 )
 ```
-Code above install **spatial** (`dlt` only loads extension) and passes duckdb credentials to the destination constructor. Database file is **duck.db**, logging and error messages as `json` are enabled. 
+Code above install **spatial** (`dlt` only loads extension) and passes duckdb credentials to the destination constructor. Database file is **duck.db**, logging and error messages as `json` are enabled.
 
 ## Data access after loading
 After loading, it is available in **read/write** mode via `with pipeline.sql_client() as con:`, which is a wrapper over `DuckDBPyConnection`. See [duckdb docs](https://duckdb.org/docs/api/python/overview#persistent-storage) for details. If you want to **read** data, use [pipeline.dataset()](../../general-usage/dataset-access/dataset) instead of `sql_client`.

docs/website/docs/dlt-ecosystem/destinations/ducklake.md

@@ -14,6 +14,8 @@ In order to use ducklake you must provide the following infrastructure:
 
 If you are looking for a managed ducklake infra, check the [Motherduck Ducklake support](motherduck.md#ducklake-setup). `dlt` is also able to set-up a local ducklake with `sqlite` as catalog fully automatically.
 
+<!--@@@DLT_DESTINATION_CAPABILITIES ducklake-->
+
 ## Quick start
 
 - Install dlt with DuckDB dependencies:
@@ -152,7 +154,7 @@ destination = dlt.destinations.ducklake(credentials=credentials)
 import dlt
 from dlt.sources.credentials import ConnectionStringCredentials
 
-# set catalog name using connection string credentials    
+# set catalog name using connection string credentials
 catalog_credentials = ConnectionStringCredentials()
 # use duckdb with the default name
 catalog_credentials.drivername = "duckdb"
@@ -215,7 +217,7 @@ with pipeline.sql_client() as client:
 All write dispositions are supported. `upsert` is supported on **duckdb 1.4.x** (without hard deletes for now)
 
 ## Data loading
-By default, Parquet files and the `COPY` command are used to move local files to the remote storage, 
+By default, Parquet files and the `COPY` command are used to move local files to the remote storage,
 
 The **INSERT** format is also supported and will execute large INSERT queries directly into the remote database. This method is significantly slower and may exceed the maximum query size, so it is not advised.
 

docs/website/docs/dlt-ecosystem/destinations/filesystem.md

@@ -25,6 +25,8 @@ pip install s3fs
 so pip does not fail on backtracking.
 :::
 
+<!--@@@DLT_DESTINATION_CAPABILITIES filesystem-->
+
 ## Initialize the dlt project
 
 Let's start by initializing a new dlt project as follows:

docs/website/docs/dlt-ecosystem/destinations/lancedb.md

@@ -9,6 +9,9 @@ keywords: [ lancedb, vector database, destination, dlt ]
 [LanceDB](https://lancedb.com/) is an open-source, high-performance vector database. It allows you to store data objects and perform similarity searches over them.
 This destination helps you load data into LanceDB from [dlt resources](../../general-usage/resource.md).
 
+
+<!--@@@DLT_DESTINATION_CAPABILITIES lancedb-->
+
 ## Setup guide
 
 ### Choose a model provider
@@ -209,7 +212,7 @@ If you plan to use `merge` write disposition, remember to [enable load ids](../v
 
 ## Access loaded data
 
-You can access the data that got loaded in many ways. You can create lancedb client yourself, pass it to `dlt` pipeline 
+You can access the data that got loaded in many ways. You can create lancedb client yourself, pass it to `dlt` pipeline
 for loading and then use it for querying:
 ```py
 import dlt

docs/website/docs/dlt-ecosystem/destinations/motherduck.md

@@ -21,6 +21,9 @@ workers=3
 or export the **LOAD__WORKERS=3** env variable. See more in [performance](../../reference/performance.md)
 :::
 
+
+<!--@@@DLT_DESTINATION_CAPABILITIES motherduck-->
+
 ## Setup guide
 
 **1. Initialize a project with a pipeline that loads to MotherDuck by running**
@@ -69,7 +72,7 @@ python3 chess_pipeline.py
 ```
 
 ### DuckLake setup
-DuckLake can be used to manage and persist your MotherDuck databases on external object storage like S3. This is especially useful if you want more control over where your data is stored or if you’re integrating with your own cloud infrastructure. 
+DuckLake can be used to manage and persist your MotherDuck databases on external object storage like S3. This is especially useful if you want more control over where your data is stored or if you’re integrating with your own cloud infrastructure.
 The steps below show how to set up a DuckLake-managed database backed by S3.
 
 **1. Create the S3-Backed DuckLake Database**

docs/website/docs/dlt-ecosystem/destinations/mssql.md

@@ -12,6 +12,8 @@ keywords: [mssql, sqlserver, destination, data warehouse]
 pip install "dlt[mssql]"
 ```
 
+<!--@@@DLT_DESTINATION_CAPABILITIES mssql-->
+
 ## Setup guide
 
 ### Prerequisites

docs/website/docs/dlt-ecosystem/destinations/postgres.md

@@ -12,6 +12,8 @@ keywords: [postgres, destination, data warehouse]
 pip install "dlt[postgres]"
 ```
 
+<!--@@@DLT_DESTINATION_CAPABILITIES postgres-->
+
 ## Setup guide
 
 **1. Initialize a project with a pipeline that loads to Postgres by running:**

docs/website/docs/dlt-ecosystem/destinations/qdrant.md

@@ -9,6 +9,9 @@ keywords: [qdrant, vector database, destination, dlt]
 [Qdrant](https://qdrant.tech/) is an open-source, high-performance vector search engine/database. It deploys as an API service, providing a search for the nearest high-dimensional vectors.
 This destination helps you load data into Qdrant from [dlt resources](../../general-usage/resource.md).
 
+
+<!--@@@DLT_DESTINATION_CAPABILITIES qdrant-->
+
 ## Setup guide
 
 1. To use Qdrant as a destination, make sure `dlt` is installed with the `qdrant` extra:

docs/website/docs/dlt-ecosystem/destinations/redshift.md

@@ -12,6 +12,8 @@ keywords: [redshift, destination, data warehouse]
 pip install "dlt[redshift]"
 ```
 
+<!--@@@DLT_DESTINATION_CAPABILITIES redshift-->
+
 ## Setup guide
 ### 1. Initialize the dlt project
 

docs/website/docs/dlt-ecosystem/destinations/snowflake.md

@@ -6,12 +6,15 @@ keywords: [Snowflake, destination, data warehouse]
 
 # Snowflake
 
+
 ## Install `dlt` with Snowflake
 **To install the `dlt` library with Snowflake dependencies, run:**
 ```sh
 pip install "dlt[snowflake]"
 ```
 
+<!--@@@DLT_DESTINATION_CAPABILITIES snowflake-->
+
 ## Setup guide
 
 **1. Initialize a project with a pipeline that loads to Snowflake by running:**

docs/website/docs/dlt-ecosystem/destinations/sqlalchemy.md

@@ -26,6 +26,8 @@ pip install mysqlclient
 
 Refer to the [SQLAlchemy documentation on dialects](https://docs.sqlalchemy.org/en/20/dialects/) for information about client libraries required for supported databases.
 
+<!--@@@DLT_DESTINATION_CAPABILITIES sqlalchemy-->
+
 ### Create a pipeline
 
 **1. Initialize a project with a pipeline that loads to MS SQL by running:**
@@ -140,7 +142,7 @@ Please report issues with particular dialects. We'll try to make them work.
 
 ### Trino limitations
 * Trino dialect does not case fold identifiers. Use `snake_case` naming convention only.
-* Trino does not support merge/scd2 write disposition (or you somehow create PRIMARY KEYs on engine tables) 
+* Trino does not support merge/scd2 write disposition (or you somehow create PRIMARY KEYs on engine tables)
 * We convert JSON and BINARY types are cast to STRING (dialect seems to have a conversion bug)
 * Trino does not support PRIMARY/UNIQUE constraints
 

docs/website/docs/dlt-ecosystem/destinations/synapse.md

@@ -12,6 +12,8 @@ keywords: [synapse, destination, data warehouse]
 pip install "dlt[synapse]"
 ```
 
+<!--@@@DLT_DESTINATION_CAPABILITIES synapse-->
+
 ## Setup guide
 
 ### Prerequisites

docs/website/docs/dlt-ecosystem/destinations/weaviate.md

@@ -9,6 +9,9 @@ keywords: [weaviate, vector database, destination, dlt]
 [Weaviate](https://weaviate.io/) is an open-source vector database. It allows you to store data objects and perform similarity searches over them.
 This destination helps you load data into Weaviate from [dlt resources](../../general-usage/resource.md).
 
+
+<!--@@@DLT_DESTINATION_CAPABILITIES weaviate-->
+
 ## Setup guide
 
 1. To use Weaviate as a destination, make sure dlt is installed with the 'weaviate' extra:

docs/website/docs/walkthroughs/create-new-destination.md

@@ -183,7 +183,7 @@ We can quickly repurpose existing GitHub source and `secrets.toml` already prese
 ```py
 import dlt
 
-from github import github_repo_events
+from github import github_repo_events # type: ignore[attr-defined]
 from presto import presto  # importing destination factory
 
 def load_airflow_events() -> None: