Filesystem: Properly forward omniload reader hints to Polars `read_xyz` kwargs

[amotl]

Problem

In omniload.source.filesystem.adapter:resource_for_reader, we are currently using a very basic implementation to forward omniload reader hints to Polars read_xyz kwargs, which does not support casting ingress parameters (all str) to their designated target types as defined by Polars' function signature. In this spirit, the procedure currently just barely works for string argument types.

reader = reader.bind(**ref.hints)

Because no type conversion takes place, it is currently not possible to convey int, float, bool, nor any container types like list or dict.

Solution

Short-term, in order to forward all kwargs to polars.read_excel properly, let's either supply a custom mapper tailored to the problem at hand, or, more generic, introspect the target read_excel function to learn about relevant types at runtime to be able to automatically cast them to their expected types, when possible. For list and dict types, let's use compact JSON as serialized representation (no whitespace), when possible.

Outlook

In the long run, this could be another candidate for a more thorough approach using Pydantic.

Originally posted by @amotl in #123 (comment)

[amotl]

@coderabbitai: Could you suggest how to convert all values in a Dict[str, str] to match their designated value types as defined by the target Polars function signature at runtime?

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
fd adapter.py -p omniload/source/filesystem
fd -p '*filesystem*'

Length of output: 714

---

🏁 Script executed:

#!/bin/bash
cat -n omniload/source/filesystem/adapter.py

Length of output: 7874

---

🏁 Script executed:

#!/bin/bash
fd readers.py -p omniload/source/filesystem
sed -n '1,200p' omniload/source/filesystem/format/readers.py 2>/dev/null | head -150

Length of output: 6995

---

Reply

@amotl Since read_excel(items, **kwargs) has no annotations of its own, introspect the actual polars.read_excel signature at runtime and cast against its type hints. Here's a generic helper:

import inspect
import json
import typing
from typing import Any, Callable, Dict


def cast_kwargs_to_signature(func: Callable, hints: Dict[str, str]) -> Dict[str, Any]:
    """Cast string hint values to the types declared by `func`'s signature."""
    sig = inspect.signature(func)
    type_hints = typing.get_type_hints(func)
    casted: Dict[str, Any] = {}
    for key, value in hints.items():
        if key not in sig.parameters:
            casted[key] = value  # unknown kwarg, pass through
            continue
        casted[key] = _cast_value(value, type_hints.get(key, str))
    return casted


def _cast_value(value: str, target_type: Any) -> Any:
    origin = typing.get_origin(target_type)

    # Unwrap Optional[...] / Union[...]
    if origin is typing.Union:
        args = [a for a in typing.get_args(target_type) if a is not type(None)]
        if len(args) == 1:
            return _cast_value(value, args[0])
        for arg in args:  # ambiguous union, try each candidate
            try:
                return _cast_value(value, arg)
            except (ValueError, TypeError, json.JSONDecodeError):
                continue
        return value

    if target_type is bool:
        return value.strip().lower() in ("true", "1", "yes", "on")
    if target_type is int:
        return int(value)
    if target_type is float:
        return float(value)
    if target_type is str:
        return value
    if target_type in (list, dict) or origin in (list, dict):
        return json.loads(value)

    # Fallback for anything else (e.g. Sequence[...], Literal[...]): try JSON,
    # else leave as string.
    try:
        return json.loads(value)
    except (json.JSONDecodeError, TypeError):
        return value

Usage in resource_for_reader, introspecting the real Polars function instead of the wrapper (since read_excel(items, **kwargs) itself carries no annotations):

import polars as pl

elif ref.reader_name == "read_excel":
    reader = reader.bind(**cast_kwargs_to_signature(pl.read_excel, ref.hints))

Notes:

• list/dict values are parsed via json.loads, so hints must be supplied as compact JSON strings (e.g. '["a","b"]', '{"a":1}'), matching the compact-JSON serialization mentioned in the issue.
• Unknown kwargs (not in pl.read_excel's signature) pass through unchanged, keeping forward-compat with future Polars kwargs.
• For unresolvable unions (e.g. Sequence[int] | Sequence[str]), it falls back to JSON-decoding and finally the raw string — reasonably safe but worth a follow-up if you want stricter validation (Pydantic, as noted in the issue's "Outlook").

[amotl]

@coderabbitai: What about using the @convert_types decorator as suggested in converting the type according to type hints in python?

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🤖 web_search executed:

converting the type of function's arguments in python at runtime convert_types decorator blog.samrid.me

Length of output: 7709

---

@amotl The @convert_types decorator pattern doesn't quite fit here, though the core idea (get_type_hints + per-arg conversion) is the same one I used.

Why the decorator doesn't map directly:
@convert_types wraps a function you call directly — it binds *args/**kwargs to that function's own signature via inspect.signature(fn).bind() and converts each bound argument in place. But in resource_for_reader, we never call pl.read_excel(**hints) ourselves — we build a dict of kwargs and hand it to reader.bind(**hints), which stores them for later use. There's no function call site to decorate; we'd have to fabricate a stand-in function with pl.read_excel's signature just to get the decorator's binding machinery to run, which is more indirection than just introspecting the signature directly (as in the earlier snippet).

The naive type_conversion in that blog post also has gaps you'd hit immediately with Polars kwargs:

• bool("false") → True (calling bool(arg) on a non-empty string is always truthy) — needs explicit string parsing.
• list("[1,2]") / dict("{}") split into characters instead of parsing JSON — needs json.loads for container types, per the issue's "compact JSON" requirement.
• Optional[int]("1") / Union[...] annotations raise TypeError since Union isn't callable — needs unwrapping before conversion.

So I'd keep the plain function (cast_kwargs_to_signature from before) rather than the decorator — same get_type_hints idea, but it fixes those three cases and works against pl.read_excel's real signature without needing a call-site to decorate. If you'd like, I can fold this into a `` for omniload/source/filesystem/adapter.py.

(っ◔◡◔)っ 🥕

---

💡 For code changes across multiple files, use @coderabbitai plan in an issue comment