BigQuery: error when using created table ref as query destination

[bspeice]

Environment details

BigQuery API, Linux, Python 2.7

Steps to reproduce

I'm getting a 400 error when using a newly-created table ref as a destination:

Code example

from google.cloud import bigquery
client = biquery.Client()
export_ref = client.create_table('project_name.dataset.table')
config = bigquery.QueryJobConfig()
config.destination = export_ref
client.query('SELECT MY_FIELDS', job_config=config)

Stack trace

Traceback (most recent call last):
  File "bq_export.py", line 112, in <module>
    main(args.table_ref[0])
  File "bq_export.py", line 102, in main
    export_block(client, table_ref, block_id, block[0], block[1])
  File "bq_export.py", line 50, in export_block
    q = client.query(query, job_config=q_config)
  File "/home/user/.local/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 1439, in query
    query_job._begin(retry=retry)
  File "/home/user/.local/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 553, in _begin
    retry, method="POST", path=path, data=self.to_api_repr()
  File "/home/user/.local/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 382, in _call_api
    return call()
  File "/home/user/.local/lib/python2.7/site-packages/google/api_core/retry.py", line 270, in retry_wrapped_func
    on_error=on_error,
  File "/home/user/.local/lib/python2.7/site-packages/google/api_core/retry.py", line 179, in retry_target
    return target()
  File "/home/user/.local/lib/python2.7/site-packages/google/cloud/_http.py", line 319, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://www.googleapis.com/bigquery/v2/projects/my_bucket/jobs: Required parameter is missing

What appears to be happening when I enable httplib logging is that the table ref internal representation includes a bunch of extra fields. However, the error message is incredibly confusing, as it says something required is missing, when the docs mention very clear that all parameters are optional.

[shollyman]

create_table() returns the Table, not a TableReference. To confirm, does it work when you change the setting of the destination to export_ref.reference?

Perhaps we should extract the TableReference when the destination is passed a Table.

[tseaver]

@shollyman That seems sensible, given that Client.create_table itself is willing to take any of a Table, a TableReference, or a string for its table argument. We've explicitly avoided doing client-side validation for the various job configuration fields, but that conversion would be a different thing.

Likely we'd want to do the same for any job / job configuration arguments currently expecting DatasetReference(s) or TableReference(s):

• LoadJob.destination
• CopyJob.sources
• CopyJob.destination
• ExtractJob.source
• QueryJobConfig.default_dataset
• QueryJobConfig.destination
• QueryJobConfig.ddl_target_table

[tswast]

Thanks for the exhaustive list @tseaver

The specific request of query destination was fixed in #7560 It looks like the others were tackled as part of the client methods update in #6164