Update DuckDB connector for destinations V2

[guenp]

What

Enable Destinations V2 support for the DuckDB connector

How

• Migrated sql_processor.py and related files from the pyairbyte repo to airbyte-cdk/python/airbyte_cdk/sql
• Added new _get_sql_column_definitions method to DestinationDuckDb based on pyairbyte's SqlProcessorBase._get_sql_column_definitions
• Add SQLAlchemy and rich dependencies
• Moved all tests into tests/ folder to make it easier to set up VS Code

Review guide

Copied SqlProcessor and related files into /sql directory:

1. airbyte-cdk/python/airbyte_cdk/sql/init.py
2. airbyte-cdk/python/airbyte_cdk/sql/_processors/init.py
3. airbyte-cdk/python/airbyte_cdk/sql/_processors/duckdb.py
4. airbyte-cdk/python/airbyte_cdk/sql/_processors/motherduck.py
5. airbyte-cdk/python/airbyte_cdk/sql/_util/init.py
6. airbyte-cdk/python/airbyte_cdk/sql/_util/hashing.py
7. airbyte-cdk/python/airbyte_cdk/sql/_util/name_normalizers.py
8. airbyte-cdk/python/airbyte_cdk/sql/constants.py
9. airbyte-cdk/python/airbyte_cdk/sql/exceptions.py
10. airbyte-cdk/python/airbyte_cdk/sql/secrets.py
11. airbyte-cdk/python/airbyte_cdk/sql/shared/init.py
12. airbyte-cdk/python/airbyte_cdk/sql/shared/catalog_providers.py
13. airbyte-cdk/python/airbyte_cdk/sql/shared/sql_processor.py
14. airbyte-cdk/python/airbyte_cdk/sql/types.py

(Update 10/21/24: we moved all changes from destination-duckdb to destination-motherduck)

Update destination to use SqlProcessor to support Destinations V2/Typing and deduping
15. airbyte-integrations/connectors/destination-motherduck/destination_motherduck/destination.py
(just a mypy typing update)
17. airbyte-integrations/connectors/destination-motherduck/destination_motherduck/run.py

Updated dependencies:
18. airbyte-integrations/connectors/destination-motherduck/poetry.lock
19. airbyte-integrations/connectors/destination-motherduck/pyproject.toml

Moved tests to /tests folder:
20. airbyte-integrations/connectors/destination-motherduck/tests/integration_tests/config.json
21. airbyte-integrations/connectors/destination-motherduck/tests/integration_tests/configured_catalog.json
22. airbyte-integrations/connectors/destination-motherduck/tests/integration_tests/invalid_config.json
23. airbyte-integrations/connectors/destination-motherduck/tests/integration_tests/messages.jsonl
24. airbyte-integrations/connectors/destination-motherduck/tests/integration_tests/spec.json
25. airbyte-integrations/connectors/destination-motherduck/tests/unit_tests/destination_unit_tests.py
26. airbyte-integrations/connectors/destination-motherduck/tests/unit_tests/unit_test.py

Added test for duplicate message w/ upsert strategy
27. airbyte-integrations/connectors/destination-motherduck/tests/integration_tests/integration_test.py

User Impact

Airbyte will no longer create tables with column names _airbyte_ab_id, _airbyte_emitted_at, _airbyte_data, but instead _airbyte_raw_id, _airbyte_extracted_at, _airbyte_meta plus any columns defined in the JSON schema.

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ aaronsteers
❌ guenp
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[vercel]

@guenp is attempting to deploy a commit to the Airbyte Growth Team on Vercel.

A member of the Team first needs to authorize it.