Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[KED-1876] Allow GBQTableDataSet to optionally accept a sql query to load data #443

Merged
merged 10 commits into from
Jul 29, 2020
Merged

[KED-1876] Allow GBQTableDataSet to optionally accept a sql query to load data #443

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
21e214f
Added notes to Release.md
ajb7 Jul 21, 2020
5960660
Modified GBQTableDataSet to accept custom queries
ajb7 Jul 21, 2020
f45e71a
Replaced DataSetError with default query
ajb7 Jul 22, 2020
e0908a0
Converted query parameter to str before passing it to read_gbq
ajb7 Jul 22, 2020
d6f56b1
Added exception handling
ajb7 Jul 22, 2020
34f9f20
Merge branch 'master' into feature/modified-gbqtabledataset-to-accept…
ajb7 Jul 23, 2020
5acee52
fixed gbqdataset and added test case for query argument
ajb7 Jul 23, 2020
767e158
Merge branch 'master' of https://github.com/quantumblacklabs/kedro in…
ajb7 Jul 23, 2020
e7c5275
Merge branch 'feature/modified-gbqtabledataset-to-accept-custom-queri…
ajb7 Jul 23, 2020
6257bbe
Merge branch 'master' into feature/modified-gbqtabledataset-to-accept…
mzjp2 Jul 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion RELEASE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,12 @@
## Major features and improvements

## Bug fixes and other changes
* Modified `GBQTableDataSet` to load customized results using customized queries from Google Big Query tables.

## Breaking changes to the API

## Thanks for supporting contributions

[Ajay Bisht](https://github.com/ajb7)

# Release 0.16.3

Expand Down
2 changes: 1 addition & 1 deletion kedro/extras/datasets/pandas/gbq_dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,8 @@ def _describe(self) -> Dict[str, Any]:

def _load(self) -> pd.DataFrame:
sql = "select * from {}.{}".format(self._dataset, self._table_name) # nosec
self._load_args.setdefault("query", sql)
return pd.read_gbq(
sql,
project_id=self._project_id,
credentials=self._credentials,
**self._load_args
Expand Down
17 changes: 16 additions & 1 deletion tests/extras/datasets/pandas/test_gbq_dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,10 +150,25 @@ def test_save_load_data(self, gbq_dataset, dummy_dataframe, mocker):
table_id, project_id=PROJECT, credentials=None, progress_bar=False
)
mocked_read_gbq.assert_called_once_with(
sql, project_id=PROJECT, credentials=None
project_id=PROJECT, credentials=None, query=sql
)
assert_frame_equal(dummy_dataframe, loaded_data)

@pytest.mark.parametrize("load_args", [{"query": "Select 1"}], indirect=True)
def test_read_gbq_with_query(self, gbq_dataset, dummy_dataframe, mocker, load_args):
"""Test loading data set with query in the argument."""
mocked_read_gbq = mocker.patch(
"kedro.extras.datasets.pandas.gbq_dataset.pd.read_gbq"
)
mocked_read_gbq.return_value = dummy_dataframe
loaded_data = gbq_dataset.load()

mocked_read_gbq.assert_called_once_with(
project_id=PROJECT, credentials=None, query=load_args["query"]
Copy link
Contributor

@mzjp2 mzjp2 Jul 23, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’d check query=“Select 1” explicitly here but up to you!

)

assert_frame_equal(dummy_dataframe, loaded_data)

@pytest.mark.parametrize(
"dataset,table_name",
[
Expand Down