Skip to content

DBMS-DataSources should use memoization for queries with :params #97

@schuderer

Description

@schuderer

Description

Replace caching (independent from arguments) with memoization (dependent on arguments).

The expires config setting of OracleDataSource just blindly caches the result. This limits its use case.

It does not take into consideration that the optional arg_dict/params parameter can contain query parameters, and that successive calls to get_dataframes therefore should actually not return the same data.

It is probably the same for the examples' RecordsDataSource and ImpalaDatasource (cf #90).

Probably also a good opportunity to refactor and clean up the caching/memoization code in the base classes and existing DataSource implementations (e.g. a decorator wrapping lru_cache in a non-memory-leaking way, see e.g. https://stackoverflow.com/questions/33672412/python-functools-lru-cache-with-class-methods-release-object)

If #96 is ever tackled, this issue should be taken into consideration there, as well.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions