Merge branch 'master' into feat/stats-for-dbt

Author: dlawin

README.md

@@ -1,93 +1,118 @@
-<p align="center">
-    <img alt="Datafold" src="https://user-images.githubusercontent.com/1799931/196497110-d3de1113-a97f-4322-b531-026d859b867a.png" width="50%" />
+<p align="left">
+    <a href="https://datafold.com/"><img alt="Datafold" src="https://user-images.githubusercontent.com/1799931/196497110-d3de1113-a97f-4322-b531-026d859b867a.png" width="30%" /></a>
 </p>
 
-<h1 align="center">
-data-diff
+<h1 align="left">
+data-diff: compare datasets fast, within or across SQL databases
 </h1>
 
-<h2 align="center">
-Develop dbt models faster by testing as you code.
-</h2>
-<h4 align="center">
-See how every change to dbt code affects the data produced in the modified model and downstream.
-</h4>
 <br>
 
-## What is `data-diff`?
+# How it works
 
-data-diff is an open source package that you can use to see the impact of your dbt code changes on your dbt models as you code.
+When comparing the data, `data-diff` utilizes the resources of the underlying databases as much as possible. It has two primary modes of comparison:
 
-<div align="center">
+## joindiff
+- Recommended for comparing data within the same database
+- Uses the outer join operation to diff the rows as efficiently as possible within the same database
+- Fully relies on the underlying database engine for computation
+- Requires both datasets to be queryable with a single SQL query
+- Time complexity approximates JOIN operation and is largely independent of the number of differences in the dataset
+  
+## hashdiff
+- Recommended for comparing datasets across different databases
+- Can also be helpful in diffing very large tables with few expected differences within the same database
+- Employs a divide-and-conquer algorithm based on hashing and binary search
+- Can diff data across distinct database engines, e.g., PostgreSQL <> Snowflake
+- Time complexity approximates COUNT(*) operation when there are few differences
+- Performance degrades when datasets have a large number of differences
 
-![development_testing_gif](https://user-images.githubusercontent.com/1799931/236354286-d1d044cf-2168-4128-8a21-8c8ca7fd494c.gif)
+More information about the algorithm and performance considerations can be found [here](https://github.com/datafold/data-diff/blob/master/docs/technical-explanation.md)
 
-</div>
+# Get started
 
-<br>
-
-:eyes: **Watch 4-min demo video [here](https://www.loom.com/share/ad3df969ba6b4298939efb2fbcc14cde)**
-
-## Getting Started
+Install `data-diff` with specific database adapters, e.g.:
 
-**Install `data-diff`**
-
-Install `data-diff` with the command that is specific to the database you use with dbt.
-
-### Snowflake
 ```
-pip install data-diff 'data-diff[snowflake,dbt]' -U
+pip install data-diff 'data-diff[postgresql,snowflake]' -U
 ```
 
-### BigQuery
+Run `data-diff` with connection URIs. In the following example, we compare tables between PostgreSQL and Snowflake using hashdiff algorithm:
 ```
-pip install data-diff 'data-diff[dbt]' google-cloud-bigquery -U
+data-diff \
+  postgresql://<username>:'<password>'@localhost:5432/<database> \
+  <table> \
+  "snowflake://<username>:<password>@<password>/<DATABASE>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<ROLE>" \
+  <TABLE> \
+  -k <primary key column> \
+  -c <columns to compare> \
+  -w <filter condition>
 ```
 
-### Redshift
-```
-pip install data-diff 'data-diff[redshift,dbt]' -U
-```
+Check out [documentation](https://docs.datafold.com/reference/open_source/cli) for the full command reference.
 
-### Postgres
-```
-pip install data-diff 'data-diff[postgres,dbt]' -U
-```
 
-### Databricks
-```
-pip install data-diff 'data-diff[databricks,dbt]' -U
-```
+# Use cases
 
-### DuckDB
-```
-pip install data-diff 'data-diff[duckdb,dbt]' -U
-```
+## Data Migration & Replication Testing
+Compare source to target and check for discrepancies when moving data between systems:
+- Migrating to a new data warehouse (e.g., Oracle > Snowflake)
+- Converting SQL to a new transformation framework (e.g., stored procedures > dbt)
+- Continuously replicating data from an OLTP DB to OLAP DWH (e.g., MySQL > Redshift)
 
-**Update a few lines in your `dbt_project.yml`**.
-```
-#dbt_project.yml
-vars:
-  data_diff:
-    prod_database: my_database
-    prod_schema: my_default_schema
-```
 
-**Run your first data diff!**
+## Data Development Testing
+Test SQL code and preview changes by comparing development/staging environment data to production:
+1. Make a change to some SQL code
+2. Run the SQL code to create a new dataset
+3. Compare the dataset with its production version or another iteration
+
+  <p align="left">
+  <img alt="dbt" src="https://seeklogo.com/images/D/dbt-logo-E4B0ED72A2-seeklogo.com.png" width="10%" />
+  </p>
+  
+`data-diff` integrates with dbt Core and dbt Cloud to seamlessly compare local development to production datasets. 
+
+:eyes: **Watch [4-min demo video](https://www.loom.com/share/ad3df969ba6b4298939efb2fbcc14cde)**
+
+**[Get started with data-diff & dbt](https://docs.datafold.com/development_testing/open_source)**
+
+Reach out on the dbt Slack in [#tools-datafold](https://getdbt.slack.com/archives/C03D25A92UU) for advice and support
+
+# Supported databases
 
-```
-dbt run && data-diff --dbt
-```
 
-We recommend you get started by walking through [our simple setup instructions](https://docs.datafold.com/development_testing/open_source) which contain examples and details.
+| Database      | Status | Connection string |
+|---------------|-------------------------------------------------------------------------------------------------------------------------------------|--------|
+| PostgreSQL >=10 |  💚    | `postgresql://<user>:<password>@<host>:5432/<database>`                                                                        |
+| MySQL         |  💚    | `mysql://<user>:<password>@<hostname>:5432/<database>`                                                                              |
+| Snowflake     |  💚    | `"snowflake://<user>[:<password>]@<account>/<database>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<role>[&authenticator=externalbrowser]"` |
+| BigQuery      |  💚    | `bigquery://<project>/<dataset>`                                                                                                    |
+| Redshift      |  💚    | `redshift://<username>:<password>@<hostname>:5439/<database>`                                                                       |
+| Oracle        |  💛    | `oracle://<username>:<password>@<hostname>/database`                                                                                |
+| Presto        |  💛    | `presto://<username>:<password>@<hostname>:8080/<database>`                                                                         |
+| Databricks    |  💛    | `databricks://<http_path>:<access_token>@<server_hostname>/<catalog>/<schema>`                                                      |
+| Trino         |  💛    | `trino://<username>:<password>@<hostname>:8080/<database>`                                                                          |
+| Clickhouse    |  💛    | `clickhouse://<username>:<password>@<hostname>:9000/<database>`                                                                     |
+| Vertica       |  💛    | `vertica://<username>:<password>@<hostname>:5433/<database>`                                                                        |
+| DuckDB        |  💛    |                                                                                                                                     |
+| ElasticSearch |  📝    |                                                                                                                                     |
+| Planetscale   |  📝    |                                                                                                                                     |
+| Pinot         |  📝    |                                                                                                                                     |
+| Druid         |  📝    |                                                                                                                                     |
+| Kafka         |  📝    |                                                                                                                                     |
+| SQLite        |  📝    |                                                                                                                                     |
 
-Please reach out on the dbt Slack in [#tools-datafold](https://getdbt.slack.com/archives/C03D25A92UU) if you have any trouble whatsoever getting started!
+* 💚: Implemented and thoroughly tested.
+* 💛: Implemented, but not thoroughly tested yet.
+* ⏳: Implementation in progress.
+* 📝: Implementation planned. Contributions welcome.
 
-<br><br>
+Your database not listed here?
 
-### Diffing between databases
+- Contribute a [new database adapter](https://github.com/datafold/data-diff/blob/master/docs/new-database-driver-guide.rst) – we accept pull requests!
+- [Get in touch](https://www.datafold.com/demo) about enterprise support and adding new adapters and features
 
-Check out our [documentation](https://docs.datafold.com/reference/open_source/cli) if you're looking to compare data across databases (for example, between Postgres and Snowflake).
 
 <br>
 

data_diff/__main__.py

@@ -33,7 +33,7 @@
     "-": "red",
 }
 
-set_entrypoint_name("CLI")
+set_entrypoint_name(os.getenv("DATAFOLD_TRIGGERED_BY", "CLI"))
 
 
 def _get_log_handlers(is_dbt: Optional[bool] = False) -> Dict[str, logging.Handler]:
@@ -48,8 +48,8 @@ def _get_log_handlers(is_dbt: Optional[bool] = False) -> Dict[str, logging.Handl
     rich_handler.setLevel(logging.WARN)
     handlers["rich_handler"] = rich_handler
 
-    # only use log_status_handler in a terminal
-    if rich_handler.console.is_terminal and is_dbt:
+    # only use log_status_handler in an interactive terminal session
+    if rich_handler.console.is_interactive and is_dbt:
         log_status_handler = LogStatusHandler()
         log_status_handler.setFormatter(logging.Formatter(log_format_status, datefmt=date_format))
         log_status_handler.setLevel(logging.DEBUG)
@@ -318,6 +318,7 @@ def main(conf, run, **kw):
                 state=state,
                 where_flag=kw["where"],
                 stats_flag=kw["stats"],
+                columns_flag=kw["columns"],
             )
         else:
             return _data_diff(

data_diff/cloud/datafold_api.py

@@ -7,7 +7,7 @@
 import pydantic
 import requests
 
-from data_diff.errors import DataDiffDatasourceIdNotFoundError
+from data_diff.errors import DataDiffCloudDiffFailed, DataDiffCloudDiffTimedOut, DataDiffDatasourceIdNotFoundError
 
 from ..utils import getLogger
 
@@ -248,8 +248,8 @@ def create_data_diff(self, payload: TCloudApiDataDiff) -> int:
     def poll_data_diff_results(self, diff_id: int) -> TCloudApiDataDiffSummaryResult:
         summary_results = None
         start_time = time.monotonic()
-        sleep_interval = 5  # starts at 5 sec
-        max_sleep_interval = 30
+        sleep_interval = 3
+        max_sleep_interval = 20
         max_wait_time = 300
 
         diff_url = f"{self.host}/datadiffs/{diff_id}/overview"
@@ -260,13 +260,15 @@ def poll_data_diff_results(self, diff_id: int) -> TCloudApiDataDiffSummaryResult
             if response_json["status"] == "success":
                 summary_results = response_json
             elif response_json["status"] == "failed":
-                raise Exception(f"Diff failed: {str(response_json)}")
+                raise DataDiffCloudDiffFailed(f"Diff failed: {str(response_json)}")
 
             if time.monotonic() - start_time > max_wait_time:
-                raise Exception(f"Timed out waiting for diff results. Please, go to the UI for details: {diff_url}")
+                raise DataDiffCloudDiffTimedOut(
+                    f"Timed out waiting for diff results. Please, go to the UI for details: {diff_url}"
+                )
 
             time.sleep(sleep_interval)
-            sleep_interval = min(sleep_interval * 2, max_sleep_interval)
+            sleep_interval = min(sleep_interval + 1, max_sleep_interval)
 
         return TCloudApiDataDiffSummaryResult.from_orm(summary_results)
 

data_diff/dbt.py

@@ -23,6 +23,7 @@
 from .format import jsonify, jsonify_error
 from .tracking import (
     bool_ask_for_email,
+    bool_notify_about_extension,
     create_email_signup_event_json,
     set_entrypoint_name,
     set_dbt_user_id,
@@ -48,6 +49,7 @@
 
 logger = getLogger(__name__)
 CLOUD_DOC_URL = "https://docs.datafold.com/development_testing/cloud"
+EXTENSION_INSTALL_URL = "https://get.datafold.com/datafold-vs-code-install"
 
 
 class TDiffVars(pydantic.BaseModel):
@@ -73,10 +75,11 @@ def dbt_diff(
     log_status_handler: Optional[LogStatusHandler] = None,
     where_flag: Optional[str] = None,
     stats_flag: bool = False,
+    columns_flag: Optional[Tuple[str]] = None,
 ) -> None:
     print_version_info()
     diff_threads = []
-    set_entrypoint_name("CLI-dbt")
+    set_entrypoint_name(os.getenv("DATAFOLD_TRIGGERED_BY", "CLI-dbt"))
     dbt_parser = DbtParser(profiles_dir_override, project_dir_override, state)
     models = dbt_parser.get_models(dbt_selection)
     config = dbt_parser.get_datadiff_config()
@@ -112,7 +115,7 @@ def dbt_diff(
             if log_status_handler:
                 log_status_handler.set_prefix(f"Diffing {model.alias} \n")
 
-            diff_vars = _get_diff_vars(dbt_parser, config, model, where_flag, stats_flag)
+            diff_vars = _get_diff_vars(dbt_parser, config, model, where_flag, stats_flag, columns_flag)
 
             # we won't always have a prod path when using state
             # when the model DNE in prod manifest, skip the model diff
@@ -156,32 +159,36 @@ def dbt_diff(
             for thread in diff_threads:
                 thread.join()
 
+    _extension_notification()
+
 
 def _get_diff_vars(
     dbt_parser: "DbtParser",
     config: TDatadiffConfig,
     model,
     where_flag: Optional[str] = None,
     stats_flag: bool = False,
+    columns_flag: Optional[Tuple[str]] = None,
 ) -> TDiffVars:
+    cli_columns = list(columns_flag) if columns_flag else []
     dev_database = model.database
     dev_schema = model.schema_
-
+    dev_alias = prod_alias = model.alias
     primary_keys = dbt_parser.get_pk_from_model(model, dbt_parser.unique_columns, "primary-key")
 
     # prod path is constructed via configuration or the prod manifest via --state
     if dbt_parser.prod_manifest_obj:
-        prod_database, prod_schema = _get_prod_path_from_manifest(model, dbt_parser.prod_manifest_obj)
+        prod_database, prod_schema, prod_alias = _get_prod_path_from_manifest(model, dbt_parser.prod_manifest_obj)
     else:
         prod_database, prod_schema = _get_prod_path_from_config(config, model, dev_database, dev_schema)
 
     if dbt_parser.requires_upper:
-        dev_qualified_list = [x.upper() for x in [dev_database, dev_schema, model.alias] if x]
-        prod_qualified_list = [x.upper() for x in [prod_database, prod_schema, model.alias] if x]
+        dev_qualified_list = [x.upper() for x in [dev_database, dev_schema, dev_alias] if x]
+        prod_qualified_list = [x.upper() for x in [prod_database, prod_schema, prod_alias] if x]
         primary_keys = [x.upper() for x in primary_keys]
     else:
-        dev_qualified_list = [x for x in [dev_database, dev_schema, model.alias] if x]
-        prod_qualified_list = [x for x in [prod_database, prod_schema, model.alias] if x]
+        dev_qualified_list = [x for x in [dev_database, dev_schema, dev_alias] if x]
+        prod_qualified_list = [x for x in [prod_database, prod_schema, prod_alias] if x]
 
     datadiff_model_config = dbt_parser.get_datadiff_model_config(model.meta)
 
@@ -192,10 +199,10 @@ def _get_diff_vars(
         primary_keys=primary_keys,
         connection=dbt_parser.connection,
         threads=dbt_parser.threads,
-        # --where takes precedence over any model level config
+        # cli flags take precedence over any model level config
         where_filter=where_flag or datadiff_model_config.where_filter,
-        include_columns=datadiff_model_config.include_columns,
-        exclude_columns=datadiff_model_config.exclude_columns,
+        include_columns=cli_columns or datadiff_model_config.include_columns,
+        exclude_columns=[] if cli_columns else datadiff_model_config.exclude_columns,
         stats_flag=stats_flag,
     )
 
@@ -229,14 +236,16 @@ def _get_prod_path_from_config(config, model, dev_database, dev_schema) -> Tuple
     return prod_database, prod_schema
 
 
-def _get_prod_path_from_manifest(model, prod_manifest) -> Union[Tuple[str, str], Tuple[None, None]]:
+def _get_prod_path_from_manifest(model, prod_manifest) -> Union[Tuple[str, str, str], Tuple[None, None, None]]:
     prod_database = None
     prod_schema = None
+    prod_alias = None
     prod_model = prod_manifest.nodes.get(model.unique_id, None)
     if prod_model:
         prod_database = prod_model.database
         prod_schema = prod_model.schema_
-    return prod_database, prod_schema
+        prod_alias = prod_model.alias
+    return prod_database, prod_schema, prod_alias
 
 
 def _local_diff(diff_vars: TDiffVars, json_output: bool = False) -> None:
@@ -517,3 +526,10 @@ def _email_signup() -> None:
         if email:
             event_json = create_email_signup_event_json(email)
             run_as_daemon(send_event_json, event_json)
+
+
+def _extension_notification() -> None:
+    if bool_notify_about_extension():
+        rich.print(
+            f"\n\nHaving a good time diffing? :heart_eyes-emoji:\nMake sure to check out the free [bold]Datafold VS Code extension[/bold] for more a more seamless diff experience:\n{EXTENSION_INSTALL_URL}"
+        )