Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(bigquery): write pandas datetime[ns] columns to BigQuery TIMESTAMP columns #10028

Merged
merged 8 commits into from
Jan 11, 2020
Merged

fix(bigquery): write pandas datetime[ns] columns to BigQuery TIMESTAMP columns #10028

merged 8 commits into from
Jan 11, 2020

Conversation

tswast
Copy link
Contributor

@tswast tswast commented Dec 30, 2019 •
edited
Loading

Also:

  • Enable TIMESTAMP and DATETIME unit tests for _pandas_helpers.
  • Add more data types to load dataframe sample.

Fixes #9996 🦕

@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Dec 30, 2019
@tswast tswast mentioned this pull request Dec 30, 2019
Closed
@tswast tswast changed the title doc(bigquery): add more data types to load dataframe sample fix(bigquery): interpret datetime columns from pandas dataframe as nanoseconds Jan 2, 2020
@tswast tswast force-pushed the issue9996-load_table_from_dataframe-sample-tests branch from e10f755 to ee48df8 Compare January 2, 2020 20:32
@tswast
fix(bigquery): write pandas datetime[ns] columns to BigQuery TIMESTAM…
6965558 
…P columns

Also:

* Enable TIMESTAMP and DATETIME unit tests for `_pandas_helpers`.
* Add more data types to load dataframe sample.
@tswast tswast force-pushed the issue9996-load_table_from_dataframe-sample-tests branch from ee48df8 to 6965558 Compare January 2, 2020 23:13
@tswast tswast changed the title fix(bigquery): interpret datetime columns from pandas dataframe as nanoseconds fix(bigquery): write pandas datetime[ns] columns to BigQuery TIMESTAMP columns Jan 2, 2020
@tswast tswast marked this pull request as ready for review January 2, 2020 23:18
@tswast tswast requested review from a team and plamut January 2, 2020 23:18
@tswast
update client tests
36fee78
@tswast
doc: show timezone conversions for timestamp columns
4fda7f5 
Pandas doesn't automatically convert datetime objects to UTC time, so
show how to do this in the code sample.
@tswast
Copy link
Contributor Author

tswast commented Jan 3, 2020

Failure is for the docs session and appears to be just a temporary flake with the gax docs.

sphinx.errors.SphinxWarning: failed to reach any of the inventories with the following issues:
intersphinx inventory 'https://gax-python.readthedocs.org/en/latest/objects.inv' not fetchable due to <class 'requests.exceptions.ConnectionError'>: HTTPSConnectionPool(host='gax-python.readthedocs.org', port=443): Max retries exceeded with url: /en/latest/objects.inv (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fdef467d050>: Failed to establish a new connection: [Errno 110] Connection timed out'))

@plamut plamut added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 9, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 9, 2020
plamut
plamut suggested changes Jan 9, 2020
View reviewed changes
Copy link
Contributor

@plamut plamut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to fix an undefined name that causes the snippets and lint checks to fail. Seems good otherwise.

bigquery/samples/tests/test_load_table_dataframe.py Show resolved Hide resolved
@tswast tswast requested a review from plamut January 10, 2020 15:38
plamut
plamut approved these changes Jan 11, 2020
View reviewed changes
Copy link
Contributor

@plamut plamut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@plamut plamut merged commit d897d56 into googleapis:master Jan 11, 2020
@tswast tswast deleted the issue9996-load_table_from_dataframe-sample-tests branch January 13, 2020 14:29
This was referenced Jan 29, 2020
Merged
Merged
Merged
Merged
Merged
This was referenced Jan 29, 2020
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@plamut plamut plamut approved these changes

Assignees
No one assigned
Labels
cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

BigQuery: DATETIME columns invalid when uploaded with load_table_from_dataframe
4 participants
@tswast @plamut @googlebot @yoshi-kokoro