Skip to content
This repository was archived by the owner on May 17, 2024. It is now read-only.

Embed sqeleton with git history preserved #485

Merged
merged 116 commits into from
Apr 20, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
a5c3531
Initial commit
williebsweet Nov 22, 2022
362de4a
README WIP
erezsh Dec 15, 2022
b91b622
Copied initial code from data-diff at b23509d4dd439e3fb0af3949638bc2d…
erezsh Dec 15, 2022
ad08db8
Added tests
erezsh Dec 15, 2022
3e348f3
poetry.lock
erezsh Dec 15, 2022
4534fff
Added docker-compose conf
erezsh Dec 15, 2022
3388b8d
Added github CI tests + issue templates
erezsh Dec 15, 2022
ba4bb30
Merge pull request #2 from datafold/dev
erezsh Dec 15, 2022
c8cf3e5
Added main with repl command
erezsh Dec 15, 2022
47385e4
Merge pull request #3 from datafold/dev
erezsh Dec 15, 2022
9ca5403
Edit type to postgresql database
Dec 29, 2022
7ffd57b
Edit type in mysql database
Dec 29, 2022
10a1645
Edit date type
Dec 29, 2022
d5ccbdf
Edit date type
Dec 29, 2022
df8ab6d
Version bump
erezsh Dec 15, 2022
30cb696
main: Fix entrypoint name
erezsh Dec 15, 2022
fa026ee
Add utils
erezsh Dec 19, 2022
466a222
Added mixins RandomSample and Regex (WIP)
erezsh Dec 20, 2022
fb56ae7
code with params; fixes
erezsh Dec 22, 2022
e80e4c6
Update README; Add insert.returning
erezsh Dec 29, 2022
aa437de
Changed group-by syntax to .group_by().agg()
erezsh Dec 29, 2022
3c91aa2
Fix nested join; improve docs & tests
erezsh Dec 30, 2022
0e0f8b9
Added docstrings; small refactoring
erezsh Dec 30, 2022
649439a
Ran black
erezsh Dec 30, 2022
9904911
Oops: Adding bound_exprs.py
erezsh Dec 30, 2022
95faa32
Added docs/intro.md (WIP)
erezsh Dec 30, 2022
924b1d9
More docstrings
erezsh Dec 30, 2022
1322dcd
Added time-travel mixin & API
erezsh Jan 2, 2023
df88633
Added an easy API for loading mixins
erezsh Jan 3, 2023
bf560b4
Added test_utils.py
erezsh Jan 3, 2023
e28c78d
More work on docs/intro.md (WIP)
erezsh Jan 3, 2023
337770c
Fix docstrings
erezsh Jan 3, 2023
3bd7fa8
Merge pull request #8 from datafold/dev
erezsh Jan 4, 2023
412f7dd
Added initial docs
erezsh Jan 4, 2023
520f44f
Merge pull request #9 from datafold/add_docs
erezsh Jan 4, 2023
90abba2
Docs
erezsh Jan 4, 2023
cdbfa5c
Get column names from cursor
erezsh Jan 4, 2023
6d95ad2
More intro
erezsh Jan 4, 2023
a47deac
Small fixes to help URIs and error reporting
erezsh Jan 4, 2023
08e6ccf
Added a connection editor written with 'textual'.
erezsh Jan 5, 2023
cd0783d
Added support for toml:// uri
erezsh Jan 6, 2023
2191dfe
pyproject
erezsh Jan 6, 2023
f01265d
Poetry.lock
erezsh Jan 6, 2023
af89433
Poetry.lock
erezsh Jan 6, 2023
c20385f
Merge pull request #11 from datafold/conn_editor
erezsh Jan 6, 2023
7619df9
Oops
erezsh Jan 12, 2023
406de9a
Bugfix in Redshift for is_distinct_from
erezsh Jan 12, 2023
6b35f10
Merge branch 'Edit_mysql_database_type' of https://github.com/titou41…
erezsh Jan 12, 2023
669a1d5
Fixup for mysql PR
erezsh Jan 12, 2023
3502a14
Merge branch 'Edit_type_database_postgresql' of https://github.com/ti…
erezsh Jan 12, 2023
685ef30
Merge branch 'titou41-Edit_mysql_database_type' into titou41-Edit_typ…
erezsh Jan 12, 2023
e975755
Fixup for postgresql PR
erezsh Jan 12, 2023
50b300c
db.query(..., list) converts to list (instead of returning a QueryRes…
erezsh Jan 12, 2023
bcb7f3e
Merge pull request #12 from datafold/dev
erezsh Jan 12, 2023
e1551a1
Version bump to 0.0.3
erezsh Jan 12, 2023
b4073db
Bugfix for String_UUID
erezsh Jan 12, 2023
ce66eb1
Small fix to boolean
erezsh Jan 12, 2023
7f54e07
Version bump to 0.0.4
erezsh Jan 12, 2023
cffc0af
Bugfix: Boolean for clickhouse
erezsh Jan 12, 2023
ffc4afe
Version bump to 0.0.5
erezsh Jan 12, 2023
ad15c0c
Fixes for BigQuery, REPL
erezsh Jan 16, 2023
ce6dd61
override normalize_table_path to support 3 part id
dlawin Jan 27, 2023
6c43fcf
use db values
dlawin Jan 30, 2023
f069cce
lint
dlawin Feb 2, 2023
66d451c
databricks support 3 part id
dlawin Feb 2, 2023
0117d8b
Improved documentation
erezsh Feb 2, 2023
7e65475
Merge pull request #17 from datafold/dev
erezsh Feb 2, 2023
f415f5a
Merge pull request #13 from datafold/redshift_3_part_id
erezsh Feb 6, 2023
2fe17a1
Merge pull request #16 from datafold/databricks_3_part_id
erezsh Feb 6, 2023
53f804a
Support connections to DuckDB for `data-diff --dbt` (#1)
dbeatty10 Feb 11, 2023
ca3ec72
optimization hint support for oracle and mysql
RoderickJDunn Feb 13, 2023
5092994
unit tests
RoderickJDunn Feb 13, 2023
4f77c64
Fixed unit test
RoderickJDunn Feb 13, 2023
beba986
Fixed NotImplimentedError for other DBs
RoderickJDunn Feb 13, 2023
21f2537
Base optimizer_hints throws NotImplementedError
RoderickJDunn Feb 13, 2023
7457caa
Using mixin for optimizer_hints support
RoderickJDunn Feb 13, 2023
04ab11b
squash add pg 3 part id support
dlawin Feb 23, 2023
446c8e9
Use the same `select_table_schema` and `_normalize_table_path` as Red…
dbeatty10 Feb 24, 2023
d75cb4b
Use the same `select_table_schema` and `_normalize_table_path` as Red…
dbeatty10 Feb 24, 2023
62a2605
Small improvement to tests
erezsh Feb 24, 2023
6c94e17
Fix spelling error
erezsh Feb 24, 2023
201c4e9
Ran black
erezsh Feb 24, 2023
40159a3
Merge pull request #18 from datafold/postgres_3_part_id
erezsh Feb 24, 2023
b593284
Merge pull request #23 from datafold/RoderickJDunn-optimizer-hints-v1
erezsh Feb 24, 2023
5855132
Align with implementation for Postgres
dbeatty10 Feb 24, 2023
ec13b5f
Merge branch 'datafold:master' into master
dbeatty10 Feb 24, 2023
8f30705
Merge branch 'master' into dbeatty10/duckdb-information-schema
dbeatty10 Feb 24, 2023
9d99ceb
Test 3-part fully-qualified names (FQN) for DuckDB
dbeatty10 Feb 24, 2023
062d5cb
Bump to latest version of DuckDB
dbeatty10 Feb 24, 2023
f51e036
poetry lock --no-update
dlawin Feb 24, 2023
e5e51e3
Merge pull request #24 from dbeatty10/dbeatty10/duckdb-information-sc…
erezsh Feb 27, 2023
9e9e276
rebuild poetry.lock
erezsh Mar 2, 2023
2e6c735
Merge pull request #25 from datafold/new_lock
erezsh Mar 2, 2023
41ac680
Version bump (0.0.6)
erezsh Mar 2, 2023
1c736ad
postgres override select_table_unique_columns
dlawin Mar 2, 2023
793ab39
Merge pull request #26 from datafold/postgres_select_table_unique_col…
williebsweet Mar 2, 2023
3815892
Version bump (0.0.7)
erezsh Mar 3, 2023
e5b34f3
Convert timezone
bjoernhaeuser Feb 15, 2023
cda5180
Added infrastructure to support PR #20 (TimestampTZ repr)
erezsh Mar 10, 2023
44cf693
Fixes for redshift and presto
erezsh Mar 10, 2023
04018a1
Fix for presto
erezsh Mar 10, 2023
11e5dca
Merge pull request #30 from datafold/bjoernhaeuser-patch-1
erezsh Mar 10, 2023
ea8ab20
Add snowflake date mapping
bjoernhaeuser Mar 10, 2023
6c53273
Merge pull request #31 from bjoernhaeuser/snowflake-date
erezsh Mar 10, 2023
c4b4b68
snowflake: add support for key password
dlawin Mar 31, 2023
8c5e98c
Merge pull request #34 from datafold/data_diff_issue_442
williebsweet Mar 31, 2023
10159ef
Various changes -
erezsh Apr 2, 2023
417d8f4
Merge pull request #35 from datafold/apr1
erezsh Apr 2, 2023
a5322ca
Version bump (0.0.8)
Apr 3, 2023
c185f50
Add support for smallint to Postgres
MattDelac Apr 4, 2023
693f5b2
Merge pull request #37 from MattDelac/support_smallint_for_redshift
erezsh Apr 4, 2023
6e87b89
add tests to reproduce the issue
vvkh Apr 5, 2023
d0bd81e
fix FLOAT type conversion in presto
vvkh Apr 5, 2023
c3688b2
Merge pull request #38 from datafold/fix-type-conversion-in-presto
vvkh Apr 6, 2023
efcb5b5
Embed sqeleton into data-diff for synchronous changes & releases (wit…
Apr 20, 2023
c87d93c
Shift sqeleton deep inside of data-diff
Apr 20, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion data_diff/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from typing import Sequence, Tuple, Iterator, Optional, Union

from sqeleton.abcs import DbTime, DbPath
from data_diff.sqeleton.abcs import DbTime, DbPath

from .tracking import disable_tracking
from .databases import connect
Expand Down
4 changes: 2 additions & 2 deletions data_diff/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
import rich
import click

from sqeleton.schema import create_schema
from sqeleton.queries.api import current_timestamp
from data_diff.sqeleton.schema import create_schema
from data_diff.sqeleton.queries.api import current_timestamp

from .dbt import dbt_diff
from .utils import eval_name_template, remove_password_from_url, safezip, match_like
Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.databases import MD5_HEXDIGITS, CHECKSUM_HEXDIGITS, QueryError, ConnectError
from data_diff.sqeleton.databases import MD5_HEXDIGITS, CHECKSUM_HEXDIGITS, QueryError, ConnectError

from .postgresql import PostgreSQL
from .mysql import MySQL
Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/_connect.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import logging

from sqeleton.databases import Connect
from data_diff.sqeleton.databases import Connect

from .postgresql import PostgreSQL
from .mysql import MySQL
Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/base.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.abcs.mixins import AbstractMixin_MD5, AbstractMixin_NormalizeValue
from data_diff.sqeleton.abcs.mixins import AbstractMixin_MD5, AbstractMixin_NormalizeValue


class DatadiffDialect(AbstractMixin_MD5, AbstractMixin_NormalizeValue):
Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/bigquery.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.databases import bigquery
from data_diff.sqeleton.databases import bigquery
from .base import DatadiffDialect


Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/clickhouse.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.databases import clickhouse
from data_diff.sqeleton.databases import clickhouse
from .base import DatadiffDialect


Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/databricks.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.databases import databricks
from data_diff.sqeleton.databases import databricks
from .base import DatadiffDialect


Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/duckdb.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.databases import duckdb
from data_diff.sqeleton.databases import duckdb
from .base import DatadiffDialect


Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/mysql.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.databases import mysql
from data_diff.sqeleton.databases import mysql
from .base import DatadiffDialect


Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/oracle.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.databases import oracle
from data_diff.sqeleton.databases import oracle
from .base import DatadiffDialect


Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/postgresql.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.databases import postgresql as pg
from data_diff.sqeleton.databases import postgresql as pg
from .base import DatadiffDialect


Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/presto.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.databases import presto
from data_diff.sqeleton.databases import presto
from .base import DatadiffDialect


Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/redshift.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.databases import redshift
from data_diff.sqeleton.databases import redshift
from .base import DatadiffDialect


Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/snowflake.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.databases import snowflake
from data_diff.sqeleton.databases import snowflake
from .base import DatadiffDialect


Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/trino.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.databases import trino
from data_diff.sqeleton.databases import trino
from .base import DatadiffDialect


Expand Down
2 changes: 1 addition & 1 deletion data_diff/databases/vertica.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from sqeleton.databases import vertica
from data_diff.sqeleton.databases import vertica
from .base import DatadiffDialect


Expand Down
2 changes: 1 addition & 1 deletion data_diff/diff_tables.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
from .thread_utils import ThreadedYielder
from .table_segment import TableSegment, create_mesh_from_points
from .tracking import create_end_event_json, create_start_event_json, send_event_json, is_tracking_enabled
from sqeleton.abcs import IKey
from data_diff.sqeleton.abcs import IKey

logger = getLogger(__name__)

Expand Down
2 changes: 1 addition & 1 deletion data_diff/hashdiff_tables.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

from runtype import dataclass

from sqeleton.abcs import ColType_UUID, NumericType, PrecisionType, StringType, Boolean
from data_diff.sqeleton.abcs import ColType_UUID, NumericType, PrecisionType, StringType, Boolean

from .info_tree import InfoTree
from .utils import safezip
Expand Down
10 changes: 5 additions & 5 deletions data_diff/joindiff_tables.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@

from runtype import dataclass

from sqeleton.databases import Database, MySQL, BigQuery, Presto, Oracle, Snowflake, DbPath
from sqeleton.abcs import NumericType
from sqeleton.queries import (
from data_diff.sqeleton.databases import Database, MySQL, BigQuery, Presto, Oracle, Snowflake, DbPath
from data_diff.sqeleton.abcs import NumericType
from data_diff.sqeleton.queries import (
table,
sum_,
min_,
Expand All @@ -27,8 +27,8 @@
this,
Compiler,
)
from sqeleton.queries.ast_classes import Concat, Count, Expr, Random, TablePath, Code, ITable
from sqeleton.queries.extras import NormalizeAsString
from data_diff.sqeleton.queries.ast_classes import Concat, Count, Expr, Random, TablePath, Code, ITable
from data_diff.sqeleton.queries.extras import NormalizeAsString

from .info_tree import InfoTree

Expand Down
4 changes: 2 additions & 2 deletions data_diff/query_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

from contextlib import suppress

from sqeleton.databases import DbPath, QueryError, Oracle
from sqeleton.queries import table, commit, Expr
from data_diff.sqeleton.databases import DbPath, QueryError, Oracle
from data_diff.sqeleton.queries import table, commit, Expr


def _drop_table_oracle(name: DbPath):
Expand Down
2 changes: 2 additions & 0 deletions data_diff/sqeleton/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
from .databases import connect
from .queries import table, this, SKIP, code
17 changes: 17 additions & 0 deletions data_diff/sqeleton/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import click
from .repl import repl as repl_main


@click.group(no_args_is_help=True)
def main():
pass


@main.command(no_args_is_help=True)
@click.argument("database", required=True)
def repl(database):
return repl_main(database)


if __name__ == "__main__":
main()
14 changes: 14 additions & 0 deletions data_diff/sqeleton/abcs/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
from .database_types import (
AbstractDatabase,
AbstractDialect,
DbKey,
DbPath,
DbTime,
IKey,
ColType_UUID,
NumericType,
PrecisionType,
StringType,
Boolean,
)
from .compiler import AbstractCompiler, Compilable
15 changes: 15 additions & 0 deletions data_diff/sqeleton/abcs/compiler.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
from typing import Any, Dict
from abc import ABC, abstractmethod


class AbstractCompiler(ABC):
@abstractmethod
def compile(self, elem: Any, params: Dict[str, Any] = None) -> str:
...


class Compilable(ABC):
# TODO generic syntax, so we can write Compilable[T] for expressions returning a value of type T
@abstractmethod
def compile(self, c: AbstractCompiler) -> str:
...
Loading