Repository navigation
[BEAM-10917] Add support for BigQuery Read API in Python BEAM - #15602
Conversation
Codecov Report
@@ Coverage Diff @@
## master #15602 +/- ##
==========================================
- Coverage 83.81% 83.46% -0.36%
==========================================
Files 444 445 +1
Lines 60464 61074 +610
==========================================
+ Hits 50676 50973 +297
- Misses 9788 10101 +313
Continue to review full report at Codecov.
|
8c90ff6 to
1121998
Compare
1121998 to
24bf089
Compare
|
R: @pabloem |
| """ | ||
| class Method(object): | ||
| EXPORT = 'EXPORT' # This is currently the default. | ||
| DIRECT_READ = 'DIRECT_READ' |
There was a problem hiding this comment.
nit: I believe this api are known as "BigQuery Storage Read API". Would it be better to specify method as STORAGE_API_READ or something similar not to confuse the user?
There was a problem hiding this comment.
I used DIRECT_READ to keep it the same as the Java SDK.
@kmjung: What do you think?
There was a problem hiding this comment.
I'm not sure that I have a sense of what Beam values here (consistency across languages vs. consistency with external naming). I guess my slight preference would be for the former, which would mean carrying over the DIRECT_READ name. @satybald @pabloem @chamikaramj what do you think?
There was a problem hiding this comment.
I also like keeping consistency with Java
satybald
left a comment
satybald
left a comment
There was a problem hiding this comment.
Left couple comments but overall approach LGTM. Thank you for support of BQ Stroage API 💯
|
Run Python 3.8 PostCommit |
|
Run PythonDocs PreCommit |
|
Run Python 3.8 PostCommit |
1 similar comment
|
Run Python 3.8 PostCommit |
|
Run Python 3.8 PostCommit |
|
Run Python 3.8 PostCommit |
|
storage api tests are passing: https://ci-beam.apache.org/job/beam_PostCommit_Python38_PR/331/testReport/apache_beam.io.gcp.bigquery_read_it_test/ we can rernu to get a passing run |
|
Run Python 3.8 PostCommit |
|
Run Python 3.8 PostCommit |
|
ok thisLGTM. We only have one failure related to the DebeziumIO test which is failing elsewhere. I'll merge this |
…Query Read API in Python BEAM * Enable reading from BigQuery using BQ Storage Read API. * Add error handling for temp dataset deletion and some lint fixes. * Adding some EXPORT source code that was mistakenly removed. * Remove table labelling and other minor fixes. * Adding the "method" of reading to display data. * Fix for failing pre-commit tests. * Adding query priority to ReadAPI source. * Doc fix. * Adding retries to Read API source query execution. * Use unique labels for temp datasets and remove retries for query execution in source. * Updating CHANGES.md.
#40430) _CustomBigQueryStorageSource._setup_temporary_dataset logged the dataset labels at warning level, right before passing them to create_temporary_dataset. The "###" prefix appears in no other logging call under sdks/python/apache_beam, and str() is redundant with %s, both of which point at a debug print rather than an intended log line. The labels come from the user, so the value carries no information the caller does not already have. The line dates back to #15602 and no test refers to it. _LOGGER is still used at nine call sites in the module.
4 participants
The proposal detailing these changes can be found here.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunnercompliance status (on master branch)Examples testing status on various runners
Post-Commit SDK/Transform Integration Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.