forked from dyvenia/viadot
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Alessio Civitillo
committed
May 8, 2021
1 parent
773d36a
commit b78cc37
Showing
10 changed files
with
61 additions
and
2,028 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
# Viadot | ||
A simple data ingestion library to guide data flows from some places to other places | ||
|
||
## Getting Data from a Source | ||
|
||
viadot supports few sources. For instance, the UK Carbon Intensity API does not require credentials. | ||
|
||
```python | ||
from viadot.sources.uk_carbon_intensity import UKCarbonIntensity | ||
ukci = UKCarbonIntensity() | ||
ukci.query("/intensity") | ||
ukci.to_df() | ||
``` | ||
|
||
The above code pulls the UK Carbon Insentity data from the external API to the local Pandas dataframe (df). | ||
|
||
## Loading Data to a Source | ||
|
||
TODO | ||
|
||
## Running tests | ||
``` | ||
run.sh | ||
docker exec -it viadot_testing bash | ||
cd tests/ && pytest . | ||
``` | ||
|
||
## Running flows locally | ||
``` | ||
run.sh | ||
poetry shell | ||
FLOW_NAME=supermetrics_to_azure_sql; python -m viadot.flows.$FLOW_NAME | ||
``` | ||
|
||
## Uploading pkg to PyPi | ||
|
||
Generate the `requirements.txt` file from poetry. | ||
|
||
```bash | ||
poetry export -f requirements.txt --output requirements.txt --with-credentials --dev | ||
``` | ||
|
||
And then publish with poetry. | ||
|
||
```bash | ||
poetry update | ||
poetry publish --build | ||
``` |
Oops, something went wrong.