Skip to content

Latest commit

 

History

History
53 lines (38 loc) · 1.38 KB

README.md

File metadata and controls

53 lines (38 loc) · 1.38 KB
Raw

Viadot

Build

Documentation: https://dyvenia.github.io/viadot/

Source Code: https://github.com/dyvenia/viadot

A simple data ingestion library to guide data flows from some places to other places.

Getting Data from a Source

Viadot supports several API and RDBMS sources, private and public. Currently, we support the UK Carbon Intensity public API and base the examples on it.

from viadot.sources.uk_carbon_intensity import UKCarbonIntensity
ukci = UKCarbonIntensity()
ukci.query("/intensity")
ukci.to_df()

The above code pulls data from the API to a pandas DataFrame.

Loading Data to a Source

For creating SQlite database and uploading table with data (pandas DataFrame as input) use Insert class.

from viadot.tasks.sqlite_tasks import Insert
insert = Insert()
insert.run(table_name=TABLE_NAME, dtypes=dtypes, db_path=database_path, df=df, if_exists="replace")

Running tests

run.sh
docker exec -it viadot_testing bash
cd tests/ && pytest .

Running flows locally

run.sh
FLOW_NAME=supermetrics_to_azure_sql; python -m viadot.flows.$FLOW_NAME