About
GH-106 opened on this:
it would probably be worth centralizing the format-to-reader mapping so new readers can be
registered once
Format-name resolution is centralized now (GH-126, GH-160, GH-286): FORMAT_TO_READER is the one
table and reader_for_format the one lookup. Reader construction is not. Adding a format at
02a60bcb means editing three files:
dlt_filesystem/source/format/readers.py, the reader function.dlt_filesystem/source/format/registry.py, the BASE_FILE_FORMATS or
ITERABLE_FILE_FORMATS entry.dlt_filesystem/source/adapter.py, an import in the reader import block and a
filesystem_resource | dlt.transformer(name="read_x", max_table_nesting=0)(read_x) line in
the tuple readers() returns.
Step 3 restates by hand what FORMAT_TO_READER already knows, and the two have drifted:
read_csv_duckdb is imported, wired as a transformer and exposed on ReadersSource, but has no
FORMAT_TO_READER entry, so reader_for_format can never return it and no extension or
#format hint reaches it. It is reachable only through the Python readers() API, and nothing
tests or documents it.
The registry maps a format to a reader-function name, so the tuple can be built from it, which
would leave step 2 as the only registration.
Question
@amotl, worth building the transformer tuple from the registry, or is the explicit list
deliberate? It does keep every reader statically imported and greppable, which a generated tuple
gives up. And is read_csv_duckdb meant to be reachable by format hint, or is it Python-API only
on purpose?
About
GH-106 opened on this:
Format-name resolution is centralized now (GH-126, GH-160, GH-286):
FORMAT_TO_READERis the onetable and
reader_for_formatthe one lookup. Reader construction is not. Adding a format at02a60bcbmeans editing three files:dlt_filesystem/source/format/readers.py, the reader function.dlt_filesystem/source/format/registry.py, theBASE_FILE_FORMATSorITERABLE_FILE_FORMATSentry.dlt_filesystem/source/adapter.py, an import in the reader import block and afilesystem_resource | dlt.transformer(name="read_x", max_table_nesting=0)(read_x)line inthe tuple
readers()returns.Step 3 restates by hand what
FORMAT_TO_READERalready knows, and the two have drifted:read_csv_duckdbis imported, wired as a transformer and exposed onReadersSource, but has noFORMAT_TO_READERentry, soreader_for_formatcan never return it and no extension or#formathint reaches it. It is reachable only through the Pythonreaders()API, and nothingtests or documents it.
The registry maps a format to a reader-function name, so the tuple can be built from it, which
would leave step 2 as the only registration.
Question
@amotl, worth building the transformer tuple from the registry, or is the explicit list
deliberate? It does keep every reader statically imported and greppable, which a generated tuple
gives up. And is
read_csv_duckdbmeant to be reachable by format hint, or is it Python-API onlyon purpose?