Merge branch 'master' into fix-key-columns-handling

Author: dlawin

README.md

@@ -1,93 +1,118 @@
-<p align="center">
-    <img alt="Datafold" src="https://user-images.githubusercontent.com/1799931/196497110-d3de1113-a97f-4322-b531-026d859b867a.png" width="50%" />
+<p align="left">
+    <a href="https://datafold.com/"><img alt="Datafold" src="https://user-images.githubusercontent.com/1799931/196497110-d3de1113-a97f-4322-b531-026d859b867a.png" width="30%" /></a>
 </p>
 
-<h1 align="center">
-data-diff
+<h1 align="left">
+data-diff: compare datasets fast, within or across SQL databases
 </h1>
 
-<h2 align="center">
-Develop dbt models faster by testing as you code.
-</h2>
-<h4 align="center">
-See how every change to dbt code affects the data produced in the modified model and downstream.
-</h4>
 <br>
 
-## What is `data-diff`?
+# How it works
 
-data-diff is an open source package that you can use to see the impact of your dbt code changes on your dbt models as you code.
+When comparing the data, `data-diff` utilizes the resources of the underlying databases as much as possible. It has two primary modes of comparison:
 
-<div align="center">
+## joindiff
+- Recommended for comparing data within the same database
+- Uses the outer join operation to diff the rows as efficiently as possible within the same database
+- Fully relies on the underlying database engine for computation
+- Requires both datasets to be queryable with a single SQL query
+- Time complexity approximates JOIN operation and is largely independent of the number of differences in the dataset
+  
+## hashdiff
+- Recommended for comparing datasets across different databases
+- Can also be helpful in diffing very large tables with few expected differences within the same database
+- Employs a divide-and-conquer algorithm based on hashing and binary search
+- Can diff data across distinct database engines, e.g., PostgreSQL <> Snowflake
+- Time complexity approximates COUNT(*) operation when there are few differences
+- Performance degrades when datasets have a large number of differences
 
-![development_testing_gif](https://user-images.githubusercontent.com/1799931/236354286-d1d044cf-2168-4128-8a21-8c8ca7fd494c.gif)
+More information about the algorithm and performance considerations can be found [here](https://github.com/datafold/data-diff/blob/master/docs/technical-explanation.md)
 
-</div>
+# Get started
 
-<br>
-
-:eyes: **Watch 4-min demo video [here](https://www.loom.com/share/ad3df969ba6b4298939efb2fbcc14cde)**
-
-## Getting Started
+Install `data-diff` with specific database adapters, e.g.:
 
-**Install `data-diff`**
-
-Install `data-diff` with the command that is specific to the database you use with dbt.
-
-### Snowflake
 ```
-pip install data-diff 'data-diff[snowflake,dbt]' -U
+pip install data-diff 'data-diff[postgresql,snowflake]' -U
 ```
 
-### BigQuery
+Run `data-diff` with connection URIs. In the following example, we compare tables between PostgreSQL and Snowflake using hashdiff algorithm:
 ```
-pip install data-diff 'data-diff[dbt]' google-cloud-bigquery -U
+data-diff \
+  postgresql://<username>:'<password>'@localhost:5432/<database> \
+  <table> \
+  "snowflake://<username>:<password>@<password>/<DATABASE>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<ROLE>" \
+  <TABLE> \
+  -k <primary key column> \
+  -c <columns to compare> \
+  -w <filter condition>
 ```
 
-### Redshift
-```
-pip install data-diff 'data-diff[redshift,dbt]' -U
-```
+Check out [documentation](https://docs.datafold.com/reference/open_source/cli) for the full command reference.
 
-### Postgres
-```
-pip install data-diff 'data-diff[postgres,dbt]' -U
-```
 
-### Databricks
-```
-pip install data-diff 'data-diff[databricks,dbt]' -U
-```
+# Use cases
 
-### DuckDB
-```
-pip install data-diff 'data-diff[duckdb,dbt]' -U
-```
+## Data Migration & Replication Testing
+Compare source to target and check for discrepancies when moving data between systems:
+- Migrating to a new data warehouse (e.g., Oracle > Snowflake)
+- Converting SQL to a new transformation framework (e.g., stored procedures > dbt)
+- Continuously replicating data from an OLTP DB to OLAP DWH (e.g., MySQL > Redshift)
 
-**Update a few lines in your `dbt_project.yml`**.
-```
-#dbt_project.yml
-vars:
-  data_diff:
-    prod_database: my_database
-    prod_schema: my_default_schema
-```
 
-**Run your first data diff!**
+## Data Development Testing
+Test SQL code and preview changes by comparing development/staging environment data to production:
+1. Make a change to some SQL code
+2. Run the SQL code to create a new dataset
+3. Compare the dataset with its production version or another iteration
+
+  <p align="left">
+  <img alt="dbt" src="https://seeklogo.com/images/D/dbt-logo-E4B0ED72A2-seeklogo.com.png" width="10%" />
+  </p>
+  
+`data-diff` integrates with dbt Core and dbt Cloud to seamlessly compare local development to production datasets. 
+
+:eyes: **Watch [4-min demo video](https://www.loom.com/share/ad3df969ba6b4298939efb2fbcc14cde)**
+
+**[Get started with data-diff & dbt](https://docs.datafold.com/development_testing/open_source)**
+
+Reach out on the dbt Slack in [#tools-datafold](https://getdbt.slack.com/archives/C03D25A92UU) for advice and support
+
+# Supported databases
 
-```
-dbt run && data-diff --dbt
-```
 
-We recommend you get started by walking through [our simple setup instructions](https://docs.datafold.com/development_testing/open_source) which contain examples and details.
+| Database      | Status | Connection string |
+|---------------|-------------------------------------------------------------------------------------------------------------------------------------|--------|
+| PostgreSQL >=10 |  💚    | `postgresql://<user>:<password>@<host>:5432/<database>`                                                                        |
+| MySQL         |  💚    | `mysql://<user>:<password>@<hostname>:5432/<database>`                                                                              |
+| Snowflake     |  💚    | `"snowflake://<user>[:<password>]@<account>/<database>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<role>[&authenticator=externalbrowser]"` |
+| BigQuery      |  💚    | `bigquery://<project>/<dataset>`                                                                                                    |
+| Redshift      |  💚    | `redshift://<username>:<password>@<hostname>:5439/<database>`                                                                       |
+| Oracle        |  💛    | `oracle://<username>:<password>@<hostname>/database`                                                                                |
+| Presto        |  💛    | `presto://<username>:<password>@<hostname>:8080/<database>`                                                                         |
+| Databricks    |  💛    | `databricks://<http_path>:<access_token>@<server_hostname>/<catalog>/<schema>`                                                      |
+| Trino         |  💛    | `trino://<username>:<password>@<hostname>:8080/<database>`                                                                          |
+| Clickhouse    |  💛    | `clickhouse://<username>:<password>@<hostname>:9000/<database>`                                                                     |
+| Vertica       |  💛    | `vertica://<username>:<password>@<hostname>:5433/<database>`                                                                        |
+| DuckDB        |  💛    |                                                                                                                                     |
+| ElasticSearch |  📝    |                                                                                                                                     |
+| Planetscale   |  📝    |                                                                                                                                     |
+| Pinot         |  📝    |                                                                                                                                     |
+| Druid         |  📝    |                                                                                                                                     |
+| Kafka         |  📝    |                                                                                                                                     |
+| SQLite        |  📝    |                                                                                                                                     |
 
-Please reach out on the dbt Slack in [#tools-datafold](https://getdbt.slack.com/archives/C03D25A92UU) if you have any trouble whatsoever getting started!
+* 💚: Implemented and thoroughly tested.
+* 💛: Implemented, but not thoroughly tested yet.
+* ⏳: Implementation in progress.
+* 📝: Implementation planned. Contributions welcome.
 
-<br><br>
+Your database not listed here?
 
-### Diffing between databases
+- Contribute a [new database adapter](https://github.com/datafold/data-diff/blob/master/docs/new-database-driver-guide.rst) – we accept pull requests!
+- [Get in touch](https://www.datafold.com/demo) about enterprise support and adding new adapters and features
 
-Check out our [documentation](https://docs.datafold.com/reference/open_source/cli) if you're looking to compare data across databases (for example, between Postgres and Snowflake).
 
 <br>
 

data_diff/__main__.py

@@ -48,8 +48,8 @@ def _get_log_handlers(is_dbt: Optional[bool] = False) -> Dict[str, logging.Handl
     rich_handler.setLevel(logging.WARN)
     handlers["rich_handler"] = rich_handler
 
-    # only use log_status_handler in a terminal
-    if rich_handler.console.is_terminal and is_dbt:
+    # only use log_status_handler in an interactive terminal session
+    if rich_handler.console.is_interactive and is_dbt:
         log_status_handler = LogStatusHandler()
         log_status_handler.setFormatter(logging.Formatter(log_format_status, datefmt=date_format))
         log_status_handler.setLevel(logging.DEBUG)

data_diff/dbt.py

@@ -23,6 +23,7 @@
 from .format import jsonify, jsonify_error
 from .tracking import (
     bool_ask_for_email,
+    bool_notify_about_extension,
     create_email_signup_event_json,
     set_entrypoint_name,
     set_dbt_user_id,
@@ -48,6 +49,7 @@
 
 logger = getLogger(__name__)
 CLOUD_DOC_URL = "https://docs.datafold.com/development_testing/cloud"
+EXTENSION_INSTALL_URL = "https://get.datafold.com/datafold-vs-code-install"
 
 
 class TDiffVars(pydantic.BaseModel):
@@ -155,6 +157,8 @@ def dbt_diff(
             for thread in diff_threads:
                 thread.join()
 
+    _extension_notification()
+
 
 def _get_diff_vars(
     dbt_parser: "DbtParser",
@@ -517,3 +521,10 @@ def _email_signup() -> None:
         if email:
             event_json = create_email_signup_event_json(email)
             run_as_daemon(send_event_json, event_json)
+
+
+def _extension_notification() -> None:
+    if bool_notify_about_extension():
+        rich.print(
+            f"\n\nHaving a good time diffing? :heart_eyes-emoji:\nMake sure to check out the free [bold]Datafold VS Code extension[/bold] for more a more seamless diff experience:\n{EXTENSION_INSTALL_URL}"
+        )

data_diff/dbt_parser.py

@@ -62,7 +62,7 @@ def try_set_dbt_flags():
 PROJECT_FILE = "dbt_project.yml"
 PROFILES_FILE = "profiles.yml"
 LOWER_DBT_V = "1.0.0"
-UPPER_DBT_V = "1.6.0"
+UPPER_DBT_V = "1.7.0"
 
 
 # https://github.com/dbt-labs/dbt-core/blob/c952d44ec5c2506995fbad75320acbae49125d3d/core/dbt/cli/resolvers.py#L6

data_diff/tracking.py

@@ -11,6 +11,7 @@
 import urllib.request
 from uuid import uuid4
 import toml
+from rich import get_console
 
 from .version import __version__
 
@@ -61,6 +62,17 @@ def bool_ask_for_email() -> bool:
     return False
 
 
+def bool_notify_about_extension() -> bool:
+    profile = _load_profile()
+    console = get_console()
+    if "notified_about_extension" not in profile and console.is_interactive:
+        profile["notified_about_extension"] = ""
+        with open(DEFAULT_PROFILE, "w") as conf:
+            toml.dump(profile, conf)
+        return True
+    return False
+
+
 g_tracking_enabled = True
 g_anonymous_id = None
 

data_diff/version.py

@@ -1 +1 @@
-__version__ = "0.7.14"
+__version__ = "0.8.1"

docs/supported-databases.md

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "data-diff"
-version = "0.7.14"
+version = "0.8.1"
 description = "Command-line tool and Python library to efficiently diff rows across two different databases."
 authors = ["Datafold <data-diff@datafold.com>"]
 license = "MIT"
@@ -37,7 +37,7 @@ trino = {version="^0.314.0", optional=true}
 presto-python-client = {version="*", optional=true}
 clickhouse-driver = {version="*", optional=true}
 duckdb = {version="^0.7.0", optional=true}
-dbt-artifacts-parser = {version="^0.3.0"}
+dbt-artifacts-parser = {version="^0.4.0"}
 dbt-core = {version="^1.0.0"}
 keyring = "*"
 tabulate = "^0.9.0"
@@ -59,7 +59,7 @@ presto-python-client = "*"
 clickhouse-driver = "*"
 vertica-python = "*"
 duckdb = "^0.7.0"
-dbt-artifacts-parser = "^0.3.0"
+dbt-artifacts-parser = "^0.4.0"
 dbt-core = "^1.0.0"
 # google-cloud-bigquery = "*"
 # databricks-sql-connector = "*"