Spanner Batch Query Sample #1402

kurtisvg · 2018-03-14T20:08:32Z

No description provided.

theacodes · 2018-03-14T20:36:34Z

spanner/cloud-client/batch_sample.py

+"""
+
+import argparse
+from multiprocessing.pool import ThreadPool


I'd suggest you use concurrent.futures which is a preferred over directly using multiprocessing.pool.

Fixed in 172ee2c.

theacodes · 2018-03-14T20:36:48Z

spanner/cloud-client/batch_sample.py

+def run_batch_query(instance_id, database_id):
+    """Runs an example batch query."""
+
+    # CREATE TABLE Singers (


Why is this here?

It's the definition of the table the sample expects to work on.

Can you leave a comment that explains that?

kurtisvg · 2018-03-15T03:39:55Z

Updated for 1.2.0. I think the only reason it is failing travis is because the version isn't on PyPI yet.

theacodes · 2018-03-15T15:53:11Z

@kurtisvg it should be now.

theacodes · 2018-03-15T15:53:42Z

spanner/cloud-client/batch_sample.py

+    with concurrent.futures.ThreadPoolExecutor() as executor:
+        futures = [executor.submit(process, snapshot, p) for p in partitions]
+
+        for future in futures:


Use as_completed, it's a wonderful thing.

theacodes · 2018-03-15T16:15:40Z

spanner/cloud-client/requirements.txt

@@ -1 +1,2 @@
-google-cloud-spanner==1.1.0
+google-cloud-spanner==1.2.0
+futures==3.1.0


DPEbot will bump this back up.

Just do

futures; python_version < 3

theacodes · 2018-03-15T17:32:12Z

spanner/cloud-client/batch_sample.py

+            print(u'SingerId: {}, AlbumId: {}, AlbumTitle: {}'.format(*row))
+            row_ct += 1
+        return time.time(), row_ct
+    except Exception as e:


I would just let the exception bubble up. The executor will print the whole stack trace, this will just print the error message.

…samples#1402)

* Add spanner samples [(#804)](GoogleCloudPlatform/python-docs-samples#804) * Update snippets.py [(#815)](GoogleCloudPlatform/python-docs-samples#815) * Changed read_write minimum amount to 300,000 as per bug [(#818)](GoogleCloudPlatform/python-docs-samples#818) * Remove cloud config fixture [(#887)](GoogleCloudPlatform/python-docs-samples#887) * Remove cloud config fixture * Fix client secrets * Fix bigtable instance * Fix reference to our testing tools * Auto-update dependencies. [(#914)](GoogleCloudPlatform/python-docs-samples#914) * Auto-update dependencies. * xfail the error reporting test * Fix lint * Re-generate all readmes * Auto-update dependencies. [(#922)](GoogleCloudPlatform/python-docs-samples#922) * Auto-update dependencies. * Fix pubsub iam samples * Update spanner sample to use bind parameters [(#928)](GoogleCloudPlatform/python-docs-samples#928) * Fix default arguments * Fix README rst links [(#962)](GoogleCloudPlatform/python-docs-samples#962) * Fix README rst links * Update all READMEs * Auto-update dependencies. [(#992)](GoogleCloudPlatform/python-docs-samples#992) * Auto-update dependencies. [(#1004)](GoogleCloudPlatform/python-docs-samples#1004) * Auto-update dependencies. * Fix natural language samples * Fix pubsub iam samples * Fix language samples * Fix bigquery samples * Swap the album titles to be consistent with other samples [(#1035)](GoogleCloudPlatform/python-docs-samples#1035) * Auto-update dependencies. [(#1055)](GoogleCloudPlatform/python-docs-samples#1055) * Auto-update dependencies. * Explicitly use latest bigtable client Change-Id: Id71e9e768f020730e4ca9514a0d7ebaa794e7d9e * Revert language update for now Change-Id: I8867f154e9a5aae00d0047c9caf880e5e8f50c53 * Remove pdb. smh Change-Id: I5ff905fadc026eebbcd45512d4e76e003e3b2b43 * Pass multi_use=True to spanner read-only transaction [(#1063)](GoogleCloudPlatform/python-docs-samples#1063) Change-Id: Ied9d9f519edd572d79dc95d2812c1b98f5a92794 * fix typo Change-Id: I887507fa33ea30f5859707063326934e5c11208f * Auto-update dependencies. [(#1093)](GoogleCloudPlatform/python-docs-samples#1093) * Auto-update dependencies. * Fix storage notification poll sample Change-Id: I6afbc79d15e050531555e4c8e51066996717a0f3 * Fix spanner samples Change-Id: I40069222c60d57e8f3d3878167591af9130895cb * Drop coverage because it's not useful Change-Id: Iae399a7083d7866c3c7b9162d0de244fbff8b522 * Try again to fix flaky logging test Change-Id: I6225c074701970c17c426677ef1935bb6d7e36b4 * Add spanner stale data sample [(#1107)](GoogleCloudPlatform/python-docs-samples#1107) * Update all generated readme auth instructions [(#1121)](GoogleCloudPlatform/python-docs-samples#1121) Change-Id: I03b5eaef8b17ac3dc3c0339fd2c7447bd3e11bd2 * Added Link to Python Setup Guide [(#1158)](GoogleCloudPlatform/python-docs-samples#1158) * Update Readme.rst to add Python setup guide As requested in b/64770713. This sample is linked in documentation https://cloud.google.com/bigtable/docs/scaling, and it would make more sense to update the guide here than in the documentation. * Update README.rst * Update README.rst * Update README.rst * Update README.rst * Update README.rst * Update install_deps.tmpl.rst * Updated readmegen scripts and re-generated related README files * Fixed the lint error * Auto-update dependencies. [(#1138)](GoogleCloudPlatform/python-docs-samples#1138) * Auto-update dependencies. [(#1186)](GoogleCloudPlatform/python-docs-samples#1186) * Fixed failed tests on Kokoro (Spanner + Translate) [(#1192)](GoogleCloudPlatform/python-docs-samples#1192) * Fixed failed tests on Kokoro (Spanner + Translate) * Update quickstart_test.py * Bump spanner stale read from 10 to 15 seconds. [(#1207)](GoogleCloudPlatform/python-docs-samples#1207) At the request of the spanner team. * Added "Open in Cloud Shell" buttons to README files [(#1254)](GoogleCloudPlatform/python-docs-samples#1254) * Auto-update dependencies. [(#1316)](GoogleCloudPlatform/python-docs-samples#1316) * Auto-update dependencies. [(#1354)](GoogleCloudPlatform/python-docs-samples#1354) * Add Spanner region tags. [(#1376)](GoogleCloudPlatform/python-docs-samples#1376) * Auto-update dependencies. [(#1377)](GoogleCloudPlatform/python-docs-samples#1377) * Auto-update dependencies. * Update requirements.txt * Spanner Batch Query Sample [(#1402)](GoogleCloudPlatform/python-docs-samples#1402) * Auto-update dependencies. [(#1406)](GoogleCloudPlatform/python-docs-samples#1406) * Spanner Commit Timestamp Sample [(#1425)](GoogleCloudPlatform/python-docs-samples#1425) * Regenerate the README files and fix the Open in Cloud Shell link for some samples [(#1441)](GoogleCloudPlatform/python-docs-samples#1441) * Update READMEs to fix numbering and add git clone [(#1464)](GoogleCloudPlatform/python-docs-samples#1464) * Adding Spanner STRUCT param samples [(#1519)](GoogleCloudPlatform/python-docs-samples#1519) * Adding Spanner STRUCT param samples * Fix python Cloud Spanner tests. [(#1548)](GoogleCloudPlatform/python-docs-samples#1548) * Fix python Cloud Spanner tests. * Lint. * Cleanup spanner tests. [(#1633)](GoogleCloudPlatform/python-docs-samples#1633) * Cleanup spanner tests. * Update requirements. * Added Spanner DML/PDML samples. [(#1742)](GoogleCloudPlatform/python-docs-samples#1742) * Added Spanner DML/PDML samples. * Fixed lint issues and bumped version. * Update method name to match action. [(#1836)](GoogleCloudPlatform/python-docs-samples#1836) * Auto-update dependencies. [(#1846)](GoogleCloudPlatform/python-docs-samples#1846) ACK, merging. * Add sample to delete data. [(#1872)](GoogleCloudPlatform/python-docs-samples#1872) * Update snippets.py * Update snippets_test.py * Auto-update dependencies. [(#1980)](GoogleCloudPlatform/python-docs-samples#1980) * Auto-update dependencies. * Update requirements.txt * Update requirements.txt * Add Cloud Spanner Batch DML sample [(#2068)](GoogleCloudPlatform/python-docs-samples#2068) * Add Cloud Spanner Batch DML sample * Fix test. * Lint. * More Lint. * Add queryWithParameter to Cloud Spanner sample. [(#2153)](GoogleCloudPlatform/python-docs-samples#2153) * Add queryWithParameter to Cloud Spanner sample. * Lint. * Update to fix test. * Deflake bigtable and spanner tests. [(#2224)](GoogleCloudPlatform/python-docs-samples#2224) * Spanner doesn't actually promise the order of the results, so make the assertion work regardless of ordering. * Bigtable might need some more time to scale, so retry the assertion up to 10 times. * Improve and fix Cloud Spanner samples that transfer marketing budget [(#2198)](GoogleCloudPlatform/python-docs-samples#2198) The samples that transfer part of an album's marketing budget had some issues: + `read_write_transaction`: Compared `second_album_budget` with an arbitrary integer, rather than explicitly checking against `transfer_amount`. + `write_with_dml_transaction`: Moved money from album 1 to album 2, even though `read_write_transaction` was the other way around. Also retrieved album 1's budget where it should have retrieved album 2's budget. This change fixes those issues and updates the tests accordingly. * Add Datatypes examples to Spanner sample. [(#2251)](GoogleCloudPlatform/python-docs-samples#2251) * Add Datatypes examples to Spanner sample. * Lint. * Lint. * Fix test. * Add bulk loading Python Sample [(#2295)](GoogleCloudPlatform/python-docs-samples#2295) Adds the following functionality: * Create bulk_load_csv * Delete bulk_load_csv * Create schema.ddl * Fix Spanner `BOOL` example after upstream typo fix [(#2356)](GoogleCloudPlatform/python-docs-samples#2356) Summary: This code used to be correct, when the Spanner Python API had a typo in the parameter name, but that typo was fixed in an upstream pull request: <googleapis/google-cloud-python#7295> Test Plan: Running `git grep BOOE` now returns no results. wchargin-branch: bool-not-booe * Updates to spanner version with BOOL correctly spelled. [(#2392)](GoogleCloudPlatform/python-docs-samples#2392) * Adds updates for samples profiler ... vision [(#2439)](GoogleCloudPlatform/python-docs-samples#2439) * update filenames to match the CSV files [(#2535)](GoogleCloudPlatform/python-docs-samples#2535) fixes GoogleCloudPlatform/python-docs-samples#2532 * Auto-update dependencies. [(#2005)](GoogleCloudPlatform/python-docs-samples#2005) * Auto-update dependencies. * Revert update of appengine/flexible/datastore. * revert update of appengine/flexible/scipy * revert update of bigquery/bqml * revert update of bigquery/cloud-client * revert update of bigquery/datalab-migration * revert update of bigtable/quickstart * revert update of compute/api * revert update of container_registry/container_analysis * revert update of dataflow/run_template * revert update of datastore/cloud-ndb * revert update of dialogflow/cloud-client * revert update of dlp * revert update of functions/imagemagick * revert update of functions/ocr/app * revert update of healthcare/api-client/fhir * revert update of iam/api-client * revert update of iot/api-client/gcs_file_to_device * revert update of iot/api-client/mqtt_example * revert update of language/automl * revert update of run/image-processing * revert update of vision/automl * revert update testing/requirements.txt * revert update of vision/cloud-client/detect * revert update of vision/cloud-client/product_search * revert update of jobs/v2/api_client * revert update of jobs/v3/api_client * revert update of opencensus * revert update of translate/cloud-client * revert update to speech/cloud-client Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> Co-authored-by: Doug Mahugh <dmahugh@gmail.com> * Delete spanner/cloud-client/bulk_load_csv. [(#2721)](GoogleCloudPlatform/python-docs-samples#2721) This has been moved to https://github.com/cloudspannerecosystem/sampledb. Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> * spanner: add query options versioning samples [(#3093)](GoogleCloudPlatform/python-docs-samples#3093) * spanner: add query versioning samples * update test assertions * Fixed the region tags. * Removed extra whitespace. * update required spanner version Co-authored-by: larkee <larkee@users.noreply.github.com> Co-authored-by: skuruppu <skuruppu@google.com> * spanner: Add Cloud Spanner Backup samples [(#3101)](GoogleCloudPlatform/python-docs-samples#3101) * add backup samples * update required spanner version * fix lint errors * run backup samples tests against a new instance * fix lint * wait for instance creation to complete * Apply suggestions from code review Co-Authored-By: skuruppu <skuruppu@google.com> * add list_backups test * fix lint * add missing newline character in assert * update samples to be consistent with other languages * lint fix * add pagination sample * reorder tests Co-authored-by: larkee <larkee@users.noreply.github.com> Co-authored-by: skuruppu <skuruppu@google.com> * Simplify noxfile setup. [(#2806)](GoogleCloudPlatform/python-docs-samples#2806) * chore(deps): update dependency requests to v2.23.0 * Simplify noxfile and add version control. * Configure appengine/standard to only test Python 2.7. * Update Kokokro configs to match noxfile. * Add requirements-test to each folder. * Remove Py2 versions from everything execept appengine/standard. * Remove conftest.py. * Remove appengine/standard/conftest.py * Remove 'no-sucess-flaky-report' from pytest.ini. * Add GAE SDK back to appengine/standard tests. * Fix typo. * Roll pytest to python 2 version. * Add a bunch of testing requirements. * Remove typo. * Add appengine lib directory back in. * Add some additional requirements. * Fix issue with flake8 args. * Even more requirements. * Readd appengine conftest.py. * Add a few more requirements. * Even more Appengine requirements. * Add webtest for appengine/standard/mailgun. * Add some additional requirements. * Add workaround for issue with mailjet-rest. * Add responses for appengine/standard/mailjet. Co-authored-by: Renovate Bot <bot@renovateapp.com> * Update dependency google-cloud-spanner to v1.15.1 [(#3377)](GoogleCloudPlatform/python-docs-samples#3377) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-spanner](https://togithub.com/googleapis/python-spanner) | patch | `==1.15.0` -> `==1.15.1` | --- ### Release Notes <details> <summary>googleapis/python-spanner</summary> ### [`v1.15.1`](https://togithub.com/googleapis/python-spanner/blob/master/CHANGELOG.md#1151-httpswwwgithubcomgoogleapispython-spannercomparev1150v1151-2020-04-08) [Compare Source](https://togithub.com/googleapis/python-spanner/compare/v1.15.0...v1.15.1) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * [spanner] fix: bump the timeout for instance creation [(#3468)](GoogleCloudPlatform/python-docs-samples#3468) fixes #3466 * [spanner] fix: set timeout for polling on operations [(#3488)](GoogleCloudPlatform/python-docs-samples#3488) * [spanner] fix: set timeout for polling on operations fixes #3471 * bump the deadline to 1200 for backup and restore * bumped the deadline to 120 * fix: use DELETE FROM for consistency [(#3498)](GoogleCloudPlatform/python-docs-samples#3498) Being consistent with [docs](https://cloud.google.com/spanner/docs/dml-syntax#delete_examples). Co-authored-by: gcf-merge-on-green[bot] <60162190+gcf-merge-on-green[bot]@users.noreply.github.com> Co-authored-by: Takashi Matsuo <tmatsuo@google.com> * chore: pin new release [(#3688)](GoogleCloudPlatform/python-docs-samples#3688) Co-authored-by: larkee <larkee@users.noreply.github.com> * chore: some lint fixes [(#3749)](GoogleCloudPlatform/python-docs-samples#3749) * chore(deps): update dependency google-cloud-spanner to v1.17.0 [(#3885)](GoogleCloudPlatform/python-docs-samples#3885) * Improve Spanner delete_data sample coverage [(#3922)](GoogleCloudPlatform/python-docs-samples#3922) * improve Spanner delete_data sample coverage * specify end to stay consistent with other languages Co-authored-by: larkee <larkee@users.noreply.github.com> * chore(deps): update dependency google-cloud-spanner to v1.17.1 [(#4160)](GoogleCloudPlatform/python-docs-samples#4160) Co-authored-by: Takashi Matsuo <tmatsuo@google.com> * fix(spanner): use uuid for unique id [(#4198)](GoogleCloudPlatform/python-docs-samples#4198) fixes #4197 (possibly) * feat(spanner): add sample for create instance [(#4230)](GoogleCloudPlatform/python-docs-samples#4230) Co-authored-by: larkee <larkee@users.noreply.github.com> Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * chore(deps): update dependency pytest to v5.4.3 [(#4279)](GoogleCloudPlatform/python-docs-samples#4279) * chore(deps): update dependency pytest to v5.4.3 * specify pytest for python 2 in appengine Co-authored-by: Leah Cole <coleleah@google.com> * chore(deps): update dependency mock to v4 [(#4287)](GoogleCloudPlatform/python-docs-samples#4287) * chore(deps): update dependency mock to v4 * specify mock version for appengine python 2 Co-authored-by: Leah Cole <coleleah@google.com> * test(spanner): add sleep to fix flaky test [(#4289)](GoogleCloudPlatform/python-docs-samples#4289) Co-authored-by: larkee <larkee@users.noreply.github.com> * chore: update templates * fix lint Co-authored-by: Jon Wayne Parrott <jonwayne@google.com> Co-authored-by: Jason Morton <jason.morton@gmail.com> Co-authored-by: Ryan Matsumoto <ryanmats@google.com> Co-authored-by: DPE bot <dpebot@google.com> Co-authored-by: Bill Prin <waprin@gmail.com> Co-authored-by: michaelawyu <chenyumic@google.com> Co-authored-by: Jeffrey Rennie <rennie@google.com> Co-authored-by: Jason Dobry <jmdobry@users.noreply.github.com> Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> Co-authored-by: Frank Natividad <frankyn@users.noreply.github.com> Co-authored-by: Jisha Abubaker <jishaa@google.com> Co-authored-by: Jonathan Simon <jsimonweb@gmail.com> Co-authored-by: Robin Reynolds-Haertle <RobinRH@users.noreply.github.com> Co-authored-by: Thea Flowers <theaflowers@google.com> Co-authored-by: Jeff Williams <jefesaurus@google.com> Co-authored-by: Oluwatoni Oshikanlu <39501338+tonioshikanlu@users.noreply.github.com> Co-authored-by: William Chargin <wchargin@gmail.com> Co-authored-by: Gus Class <gguuss@gmail.com> Co-authored-by: Mark <mmmarklu@gmail.com> Co-authored-by: Doug Mahugh <dmahugh@gmail.com> Co-authored-by: Leonhard Gruenschloss <leonhard.gruenschloss@gmail.com> Co-authored-by: larkee <larkee@users.noreply.github.com> Co-authored-by: skuruppu <skuruppu@google.com> Co-authored-by: Renovate Bot <bot@renovateapp.com> Co-authored-by: Takashi Matsuo <tmatsuo@google.com> Co-authored-by: gcf-merge-on-green[bot] <60162190+gcf-merge-on-green[bot]@users.noreply.github.com> Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Co-authored-by: Leah Cole <coleleah@google.com> Co-authored-by: root <root@google.com>

kurtisvg added 2 commits March 14, 2018 13:07

Add intial sample.

0137d22

Clean up sample.

7c3c881

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Mar 14, 2018

kurtisvg requested a review from jsimonweb March 14, 2018 20:08

Update license and address style check.

cdd7da1

kurtisvg requested a review from theacodes March 14, 2018 20:41

jsimonweb approved these changes Mar 14, 2018

View reviewed changes

theacodes suggested changes Mar 14, 2018

View reviewed changes

Replace multiproccessing with concurrent.futures.

172ee2c

kurtisvg force-pushed the spanner-batch branch from 1578a08 to 172ee2c Compare March 14, 2018 21:16

kurtisvg added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Mar 14, 2018

Update to 1.2.0 and fix threadpool.

bc1ecdb

kurtisvg requested a review from theacodes March 15, 2018 03:37

theacodes reviewed Mar 15, 2018

View reviewed changes

kurtisvg added 2 commits March 15, 2018 09:07

Address feedback.

707ec8d

Downgrade futures to 3.1.0 for python3.

092cc88

theacodes reviewed Mar 15, 2018

View reviewed changes

kurtisvg force-pushed the spanner-batch branch from a80c348 to 7eeb080 Compare March 15, 2018 16:27

Futures requirement only for python2.

c3b862a

kurtisvg force-pushed the spanner-batch branch from 7eeb080 to c3b862a Compare March 15, 2018 16:32

kurtisvg removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Mar 15, 2018

theacodes reviewed Mar 15, 2018

View reviewed changes

Remove try/catch.

7bb0eb5

theacodes approved these changes Mar 15, 2018

View reviewed changes

theacodes merged commit f2a5a72 into master Mar 15, 2018

theacodes deleted the spanner-batch branch March 15, 2018 17:52

larkee pushed a commit to larkee/python-spanner that referenced this pull request Jul 23, 2020

Spanner Batch Query Sample [(#1402)](GoogleCloudPlatform/python-docs-…

9c217e3

…samples#1402)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spanner Batch Query Sample #1402

Spanner Batch Query Sample #1402

kurtisvg commented Mar 14, 2018

theacodes Mar 14, 2018

kurtisvg Mar 14, 2018

theacodes Mar 14, 2018

kurtisvg Mar 14, 2018

theacodes Mar 15, 2018

kurtisvg commented Mar 15, 2018

theacodes commented Mar 15, 2018

theacodes Mar 15, 2018

theacodes Mar 15, 2018

theacodes Mar 15, 2018

kurtisvg Mar 15, 2018

Spanner Batch Query Sample #1402

Spanner Batch Query Sample #1402

Conversation

kurtisvg commented Mar 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kurtisvg commented Mar 15, 2018

theacodes commented Mar 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment