Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update DuckDB connector for destinations V2 #46904

Open
wants to merge 76 commits into
base: master
Choose a base branch
from
Open

Update DuckDB connector for destinations V2 #46904

wants to merge 76 commits into from
+5,472 −0

Commits on Oct 11, 2024

  1. import raw classes from pyairbyte to the cdk

    @aaronsteers
    aaronsteers committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    62eca39 View commit details
    Browse the repository at this point in the history

  2. use local cdk and run poetry lock

    @aaronsteers
    aaronsteers committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    cd7818b View commit details
    Browse the repository at this point in the history

  3. imported duckdb-specific processor classes into destination codebase

    @aaronsteers
    aaronsteers committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    53a5f26 View commit details
    Browse the repository at this point in the history

  4. add venv to gitignore

    @guenp
    guenp committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    2eb8a47 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2024

  1. move tests to tests folder

    @guenp
    guenp committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    bd996af View commit details
    Browse the repository at this point in the history

  2. add sqlalchemy dependency

    @guenp
    guenp committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    656cea5 View commit details
    Browse the repository at this point in the history

  3. formatting

    @guenp
    guenp committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    bb5f606 View commit details
    Browse the repository at this point in the history

  4. basic poc

    @guenp
    guenp committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    d53f682 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. use faker

    @guenp
    guenp committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    e81d78b View commit details
    Browse the repository at this point in the history

  2. remove unused files

    @guenp
    guenp committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    4ef8875 View commit details
    Browse the repository at this point in the history

  3. wip

    @guenp
    guenp committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    d9b271c View commit details
    Browse the repository at this point in the history

  4. fix issues where values are not null

    @guenp
    guenp committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    c51869e View commit details
    Browse the repository at this point in the history

  5. remove json_schema

    @guenp
    guenp committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    ac755c2 View commit details
    Browse the repository at this point in the history

  6. remove processors

    @guenp
    guenp committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    5be0c27 View commit details
    Browse the repository at this point in the history

  7. mypy fixes

    @guenp
    guenp committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    1a1d326 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'master' into guenp/destination-duckdb-v2

    @guenp
    guenp committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    d0f360d View commit details
    Browse the repository at this point in the history

  9. formatting

    @guenp
    guenp committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    3c1f687 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. address review comments

    @guenp
    guenp committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    0342e12 View commit details
    Browse the repository at this point in the history

  2. add sql_processor.py

    @guenp
    guenp committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    b4ea42a View commit details
    Browse the repository at this point in the history

  3. add sql_processor ctd

    @guenp
    guenp committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    69d7c51 View commit details
    Browse the repository at this point in the history

  4. add sql_processor ctd

    @guenp
    guenp committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    4246134 View commit details
    Browse the repository at this point in the history

  5. rename PyAirbyte -> Airbyte

    @guenp
    guenp committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    47f2cea View commit details
    Browse the repository at this point in the history

  6. refactor imports

    @guenp
    guenp committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    dd7dbc6 View commit details
    Browse the repository at this point in the history

  7. use processor to create table

    @guenp
    guenp committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    4dd5656 View commit details
    Browse the repository at this point in the history

  8. rename to _create_table_if_not_exists, add _ensure_compatible_table_s… 

    …chema
    @guenp
    guenp committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    853928a View commit details
    Browse the repository at this point in the history

  9. add MD sql processor

    @guenp
    guenp committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    9f9edd5 View commit details
    Browse the repository at this point in the history

  10. replace direct sql queries with processor

    @guenp
    guenp committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    573ae91 View commit details
    Browse the repository at this point in the history

  11. add _write_with_executemany and _write_from_pa_table

    @guenp
    guenp committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    3d8e3d9 View commit details
    Browse the repository at this point in the history

  12. move _safe_write to sql processor

    @guenp
    guenp committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    47f972a View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. add write_stream_data_from_buffer, flush_buffer

    @guenp
    guenp committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    52e5e03 View commit details
    Browse the repository at this point in the history

  2. drop _safe_write

    @guenp
    guenp committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    6389541 View commit details
    Browse the repository at this point in the history

  3. get primary keys

    @guenp
    guenp committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    fcf6413 View commit details
    Browse the repository at this point in the history

  4. make test work with primary key

    @guenp
    guenp committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    ddfed60 View commit details
    Browse the repository at this point in the history

  5. add UT for upsert

    @guenp
    guenp committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    7b2ed8a View commit details
    Browse the repository at this point in the history

  6. add deduping and insert from a temp table

    @guenp
    guenp committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    5ef5c19 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'master' into guenp/destination-duckdb-v2

    @guenp
    guenp committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    b222bd9 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. move files to _util and shared

    @guenp
    guenp committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    064e112 View commit details
    Browse the repository at this point in the history

  2. create _processors and move files

    @guenp
    guenp committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    0fbe5a9 View commit details
    Browse the repository at this point in the history

  3. remove duplicate method, use processor instead

    @guenp
    guenp committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    eded011 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into guenp/destination-duckdb-v2

    @guenp
    guenp committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    64572ef View commit details
    Browse the repository at this point in the history

  5. formatting

    @guenp
    guenp committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    6fbf0b8 View commit details
    Browse the repository at this point in the history

  6. address mypy errors

    @guenp
    guenp committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    093657f View commit details
    Browse the repository at this point in the history

  7. mypy errors

    @guenp
    guenp committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    df2dee8 View commit details
    Browse the repository at this point in the history

  8. foo.model_dump_json() -> json.dumps(asdict(foo))

    @guenp
    guenp committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    3d541a0 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2024

  1. address mypy errors

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    fced79f View commit details
    Browse the repository at this point in the history

  2. mypy error

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    50e2298 View commit details
    Browse the repository at this point in the history

  3. formatting

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    d82b031 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into guenp/destination-duckdb-v2

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    dc0245a View commit details
    Browse the repository at this point in the history

  5. run formatter

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    8138100 View commit details
    Browse the repository at this point in the history

  6. prune: remove unused methods on sql_processor.py, remove unused modules

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    65509ca View commit details
    Browse the repository at this point in the history

  7. prune: remove unused classes and modules

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    e1afab5 View commit details
    Browse the repository at this point in the history

  8. prune: rm unused utils, unused classes and methods

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    20d75fd View commit details
    Browse the repository at this point in the history

  9. formatting

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    6a6819b View commit details
    Browse the repository at this point in the history

  10. rm unused imports

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    472695e View commit details
    Browse the repository at this point in the history

  11. remove unused dependencies

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    12aae08 View commit details
    Browse the repository at this point in the history

  12. ignore mypy error

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    dfbc510 View commit details
    Browse the repository at this point in the history

  13. run formatter

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    b10dd7d View commit details
    Browse the repository at this point in the history

  14. formatting

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    fac8139 View commit details
    Browse the repository at this point in the history

  15. Merge branch 'master' into guenp/destination-duckdb-v2

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    1b685c8 View commit details
    Browse the repository at this point in the history

  16. formatting

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    e347e2d View commit details
    Browse the repository at this point in the history

  17. run isort

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    59211c8 View commit details
    Browse the repository at this point in the history

  18. reverse change to gitignore

    @guenp
    guenp committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    7691d7a View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. Merge branch 'master' into guenp/destination-duckdb-v2

    @guenp
    guenp committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    678202b View commit details
    Browse the repository at this point in the history

  2. address flake8 errors

    @guenp
    guenp committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    a25460f View commit details
    Browse the repository at this point in the history

  3. add destination-motherduck, revert destination-duckdb back to prior v… 

    …ersion
    @guenp
    guenp committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    c858167 View commit details
    Browse the repository at this point in the history

  4. update imports

    @guenp
    guenp committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    7d9f629 View commit details
    Browse the repository at this point in the history

  5. add docs

    @guenp
    guenp committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    f04e10a View commit details
    Browse the repository at this point in the history

  6. move tests back to destination-motherduck folder, skip sqlalchemy 2.0… 

    ….36 for now until duckdb_engine is fixed
    @guenp
    guenp committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    18e3dde View commit details
    Browse the repository at this point in the history

  7. formatting

    @guenp
    guenp committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    ad946f5 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'master' into guenp/destination-duckdb-v2

    @guenp
    guenp committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    0a0df91 View commit details
    Browse the repository at this point in the history

  9. isort

    @guenp
    guenp committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    0620e77 View commit details
    Browse the repository at this point in the history

  10. formatting

    @guenp
    guenp committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    71bdd6b View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Merge branch 'master' into guenp/destination-duckdb-v2

    @guenp
    guenp authored Oct 22, 2024
    Configuration menu
    Copy the full SHA
    5dfd75a View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into guenp/destination-duckdb-v2

    @guenp
    guenp authored Oct 22, 2024
    Configuration menu
    Copy the full SHA
    6b04986 View commit details
    Browse the repository at this point in the history

  3. remove unused import

    @guenp
    guenp committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    7680c63 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into guenp/destination-duckdb-v2

    @guenp
    guenp committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    a8dc586 View commit details
    Browse the repository at this point in the history