Skip to content

Loading AVRO files from local filesystem.#3427

Merged
tseaver merged 4 commits intogoogleapis:masterfrom
tswast:bq-3416
May 16, 2017
Merged

Loading AVRO files from local filesystem.#3427
tseaver merged 4 commits intogoogleapis:masterfrom
tswast:bq-3416

Conversation

@tswast
Copy link
Contributor

@tswast tswast commented May 16, 2017

Adds an integration test for loading AVRO files.

Don't set the schema if none is provided (patch).

Closes #3416.

tswast added 3 commits May 16, 2017 13:01
@tswast
Add test avro data.
c27100c 
Generated with `avro-tools`

    $ avro-tools fromjson \
          --schema-file tests/data/colors.avroschema \
          tests/data/colors.json > tests/data/colors.avro

See:
http://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/
@tswast
Add test to reproduce googleapis#3416.
dcb05b3
@tswast
Don't set schema in load job if none is provided.
2b1f149
@tswast tswast requested a review from tseaver May 16, 2017 21:00
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label May 16, 2017
@tseaver
Copy link
Contributor

tseaver commented May 16, 2017

@tswast Can you push your branch to this repository, so that the system test actually gets run by the CI before we merge?

@dhermes
Copy link
Contributor

dhermes commented May 16, 2017

@tswast If you don't have push privileges one of us can do it

@tseaver
Copy link
Contributor

tseaver commented May 16, 2017

Also, @tweeter0830 reported that the patch did not fix the issue with uploading AVRO data.

@tswast
Copy link
Contributor Author

tswast commented May 16, 2017

I saw that, but I was not able to reproduce the issue after the patch.

@tseaver tseaver added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. api: bigquery Issues related to the BigQuery API. labels May 16, 2017
@tswast
Copy link
Contributor Author

tswast commented May 16, 2017

Hmm, I see that this change made the Avro load test pass, but now the CSV load test fails.
https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/1920

@tseaver
Copy link
Contributor

tseaver commented May 16, 2017

@tswast That is weird, as the testcase clearly sets the schema on the table: it should be passing the schema along.

@tswast
Copy link
Contributor Author

tswast commented May 16, 2017

We missed the "fields" property of the schema object in the patch. Seems to be running fine now. https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/1922

@tseaver
Copy link
Contributor

tseaver commented May 16, 2017

I've pushed the branch here (as tswast-bq-3146) to get the system tests running under CI before merging.

@tseaver
Copy link
Contributor

tseaver commented May 16, 2017

BQ system tests are green on Circle CI.

@tseaver tseaver merged commit 4877315 into googleapis:master May 16, 2017
@tswast tswast deleted the bq-3416 branch May 16, 2017 23:14
@tseaver tseaver mentioned this pull request Jun 26, 2017
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tseaver tseaver Awaiting requested review from tseaver

Assignees

No one assigned

Labels

api: bigquery Issues related to the BigQuery API. cla: yes This human has signed the Contributor License Agreement. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tswast @tseaver @dhermes @googlebot