add motherduck support for duckdb plugin

[dansola]

Why are the changes needed?

We would like to integrate motherduck support with the current duckdb plugin such that a DuckDBQuery can query the motherduck data warehouse and/or in memory files at the same time.

What changes were proposed in this pull request?

Simply add a connection to motherduck and authenticate via a secret. duckdb handles the rest.

How was this patch tested?

Tested on a motherduck free trial with a combinarion query like:

"""
WITH db1 AS (
    SELECT
        SUM(a) AS sum_a,
    FROM mydf
),
db2 AS (
    SELECT
        MEAN(trip_time) AS mean_tt
    FROM sample_data.nyc.rideshare
)
SELECT
    db1.sum_a + db2.mean_tt AS total_sum,
FROM db1, db2;
"""

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[codecov]

Codecov Report

Attention: Patch coverage is 88.23529% with 6 lines in your changes missing coverage. Please review.

Project coverage is 83.05%. Comparing base (556dad2) to head (d2d4098).
Report is 39 commits behind head on master.

Files with missing lines	Patch %	Lines
...ins/flytekit-duckdb/flytekitplugins/duckdb/task.py	82.35%	6 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2680       +/-   ##
===========================================
- Coverage   94.08%   83.05%   -11.04%     
===========================================
  Files          32      333      +301     
  Lines        1842    26287    +24445     
  Branches        0     4071     +4071     
===========================================
+ Hits         1733    21832    +20099     
- Misses        109     3756     +3647     
- Partials        0      699      +699     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[thomasjpfan]

There are linting issues: https://github.com/flyteorg/flytekit/actions/runs/10374544320/job/28722185574?pr=2680

Can you run make fmt? (Also you can configure pre-commit to automatically lint your files.

[samhita-alla]

could you add some test cases?

[thomasjpfan]

Using an enum restricts someone from passing a callable. Lets say someone wants to add a new keyword to duckdb.connect for motherduck:

def custom_connect_motherduck(token: str):
    """Connect to MotherDuck."""
    return duckdb.connect("md:", config={"motherduck_token": token, "another_config": "hello"})

They should be a to pass it into provider:

DuckDBQuery(..., provider=custom_connect_motherduck)

If you prefer an enum, I think we also open it up to callables:

    provider: Union[DuckDBProvider, Callable]

---

In the above callable API, I propose making the input just be a string. To pass in the secret the user API becomes:

secret = Secret(key="my-key", group="my-group")

DuckDBQuery(..., connect_secret=secret)

then in `DuckDBQuery._connect_to_duckdb:

def _connect_to_duckdb(self):
    connect_token = None
    if self._connect_secret:
        connect_token = current_context().secrets.get(
            group=self._connect_secret.group,
            key=self._connect_secret.key,
            group_version=self._connect_secret.group_version,
        )
    if isinstance(self.provider, DuckDBProvider):
        return self.provider.value(connect_token)
    else:  # callable
        return self.provider(connect_token)

This way the callable does not need to be responsible for handling current_context.

[dansola]

Will do!! Thank you.

I will also use secret_requests from PythonAutoContainerTask and check for it in the constructor rather than adding a new secret argument.