Skip to content

support custom backends, row loaders etc. via custom TableLoader in sql_database #2736

@rudolfix

Description

@rudolfix

Background
We provide several ways to customize sql_database via callback that let you ie. to rewrite the queries. Here we are allowing for a deep customization by subclassing TableLoader that is responsible for execution of the query over already reflected SqlAlchemy table. Overriding this class will allow:

  • implementing additional backends (ie. adbc fast arrow readers or ibis backend)
  • implementing custom row readers ie. with pagination (Add pagination to sql_table #2432)
  • implementing own retry mechanisms
  • very deep customization of generated queries

Implementation

    • design an internal interface that users are supposed to override in subclasses
    • test the interface so we do not break users' code by chaning it
    • allow to: easy add new backends and override row loading
    • provide two examples in tests and docs: paginated loading + adbc backend for postgres / snowflake (we may merge it into OSS if works well)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    Planned

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions