Skip to content

Commit 6e73c25

Browse files
tswastsharoonthomasnayaknishantgcf-owl-bot[bot]kiraksi
authored
feat: support SQLAlchemy 2.0, raise minimum required version to 1.4.x (#1053)
* feat!: Support SQLAlchemy 2.0, drop support for 1.x This patch makes this library backward incompatible for versions of SQLAlchemy < 2.0. Fixes #510 * constraints updated * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fixing README.rst * fixing README.rst * upping sqlalchemy version in constraints-3.8.txt * adding 2.0 version restrictions to owlbot.py * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix for * Updated some compliance tests for sqla2 and bq * Addressed snippet errors * revert bad commit * More compliance tests checking * reformatted with black * Changed more compliance tests, updated requirements for testing * Fixed attributeerror in failing sample test * Fixed geolography test failing issue * Minor tweaks to tests and code * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Fixed small error in compliance tests, added pip freeze for owl bot testing * Fixed some failing compliance tests by reformatting * Added UuidTest to compliance tests * Moved back sqlalchemy constraints to 1.4 * Update testing/constraints-3.8.txt * Fixed minimum version of sqlalchemy for 1.4 backwards compatibility * Bumping support for sqlalchemy 1.4.16 for sample tests * Bump setup.py sqlalchemy to 1.4.16 * Updated compliance sqlalchemy to 1.4.16 * Fixed broken code in last merged main, as we need to avoid duplicate entries and potential implicit joins due to identical table names * modified tests for join order variation in 1.4 vs 2.0 * typo * Modified one compliance StringTest thats been flaky * Updated docs * minor fixes to noxfile and README * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * cleaned up code from review, removed unnecessary code and files * Update tests/sqlalchemy_dialect_compliance/test_dialect_compliance.py * create development release 1.11.0.dev0 branch * chore: cleanup compliance tests for sqlalchemy migration (#1013) * chore: remove code for sqlalchemy before 1_4 * reformatted with black: * Removed sqlalchemy compliance tests from versions before 1.4 * removed code in base.py for sqlalchemy < 1.4 * fix coverage issues in base.py * temporarily commented out code lines not passing coverage for testing purposes * replaced functions previously removed for not passing cover * testing removing functions for coverage * add no cover tag to untested code and clean up commented out functions * fix lint issues * black * chore: cleanup compliance file tests after migration * lint * fixed small import error --------- Co-authored-by: Sharoon Thomas <sharoon.thomas@fulfil.io> * Revert "chore: cleanup compliance tests for sqlalchemy migration" (#1015) * chore: fix coverage tests sqlalchemy 2.0 migration (#987) * chore: remove code for sqlalchemy before 1_4 * reformatted with black: * Removed sqlalchemy compliance tests from versions before 1.4 * removed code in base.py for sqlalchemy < 1.4 * fix coverage issues in base.py * temporarily commented out code lines not passing coverage for testing purposes * replaced functions previously removed for not passing cover * testing removing functions for coverage * add no cover tag to untested code and clean up commented out functions * fix lint issues * black * Readded deleted tests and renamed them from deprecated names * black --------- Co-authored-by: Sharoon Thomas <sharoon.thomas@fulfil.io> * chore: sqlalchemy test compliance suite cleanup (#1018) * chore: remove code for sqlalchemy before 1_4 * reformatted with black: * Removed sqlalchemy compliance tests from versions before 1.4 * removed code in base.py for sqlalchemy < 1.4 * fix coverage issues in base.py * temporarily commented out code lines not passing coverage for testing purposes * replaced functions previously removed for not passing cover * testing removing functions for coverage * add no cover tag to untested code and clean up commented out functions * fix lint issues * black * Readded deleted tests and renamed them from deprecated names * black * chore: sqlalchemy test compliance suite cleanup code * black * black --------- Co-authored-by: Sharoon Thomas <sharoon.thomas@fulfil.io> * create development release 1.11.0.dev1 branch * feat: grouping sets, rollup and cube compatibility * create development release 1.11.0.dev2 * test commit to run kokooro tests * removed unnecessary clause function changes, edited tests * test basic implementation of group_by_clause and visit_label * fixed render label as label assignment * added test case * reformat logic * test commit * create development build 1.11.0.dev3 * chore: add more grouping sets/rollup/cube tests (#1029) * chore: add more tests for grouping functions fix * reformatted tests * update changelog * revert changelog * remove note * don't install prerelease in compliance session * sync owlbot * Update tests/sqlalchemy_dialect_compliance/test_dialect_compliance.py * make results order not matter * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Sharoon Thomas <sharoon.thomas@fulfil.io> Co-authored-by: Nishant Nayak <nishantnayak@google.com> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: kiraksi <kirnendra@google.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com> Co-authored-by: Chalmer Lowe <chalmer.lowe@gmail.com> Co-authored-by: Kira <kira.kaur.sidhu@gmail.com>
1 parent ec10fae commit 6e73c25

22 files changed

+1005
-579
lines changed

‎packages/sqlalchemy-bigquery/AUTHORS‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Maksym Voitko
1919
Maxim Zudilov (mxmzdlv)
2020
Maxime Beauchemin (mistercrunch)
2121
Romain Rigaux
22+
Sharoon Thomas (sharoonthomas)
2223
Sumedh Sakdeo
2324
Tim Swast (tswast)
2425
Vince Broz

‎packages/sqlalchemy-bigquery/README.rst‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,6 @@ In order to use this library, you first need to go through the following steps:
3434
.. _Enable the BigQuery Storage API.: https://console.cloud.google.com/apis/library/bigquery.googleapis.com
3535
.. _Setup Authentication.: https://googleapis.dev/python/google-api-core/latest/auth.html
3636

37-
.. note::
38-
This library is only compatible with SQLAlchemy versions < 2.0.0
3937

4038
Installation
4139
------------
@@ -108,7 +106,8 @@ SQLAlchemy
108106
from sqlalchemy.schema import *
109107
engine = create_engine('bigquery://project')
110108
table = Table('dataset.table', MetaData(bind=engine), autoload=True)
111-
print(select([func.count('*')], from_obj=table).scalar())
109+
print(select([func.count('*')], from_obj=table().scalar())
110+
112111
113112
Project
114113
^^^^^^^
@@ -281,7 +280,7 @@ If you need additional control, you can supply a BigQuery client of your own:
281280
282281
engine = create_engine(
283282
'bigquery://some-project/some-dataset?user_supplied_client=True',
284-
connect_args={'client': custom_bq_client},
283+
connect_args={'client': custom_bq_client},
285284
)
286285
287286

‎packages/sqlalchemy-bigquery/noxfile.py‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -368,8 +368,6 @@ def compliance(session):
368368
if not os.path.exists(system_test_folder_path):
369369
session.skip("Compliance tests were not found")
370370

371-
session.install("--pre", "grpcio")
372-
session.install("--pre", "--no-deps", "--upgrade", "sqlalchemy<2.0.0")
373371
session.install(
374372
"mock",
375373
"pytest",
@@ -543,7 +541,7 @@ def prerelease_deps(session):
543541

544542
prerel_deps = [
545543
"protobuf",
546-
"sqlalchemy<2.0.0",
544+
"sqlalchemy",
547545
# dependency of grpc
548546
"six",
549547
"googleapis-common-protos",

‎packages/sqlalchemy-bigquery/owlbot.py‎

Lines changed: 34 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -42,14 +42,17 @@
4242
system_test_extras=extras,
4343
system_test_extras_by_python=extras_by_python,
4444
)
45-
s.move(templated_files, excludes=[
46-
# sqlalchemy-bigquery was originally licensed MIT
47-
"LICENSE",
48-
"docs/multiprocessing.rst",
49-
# exclude gh actions as credentials are needed for tests
50-
".github/workflows",
51-
"README.rst",
52-
])
45+
s.move(
46+
templated_files,
47+
excludes=[
48+
# sqlalchemy-bigquery was originally licensed MIT
49+
"LICENSE",
50+
"docs/multiprocessing.rst",
51+
# exclude gh actions as credentials are needed for tests
52+
".github/workflows",
53+
"README.rst",
54+
],
55+
)
5356

5457
# ----------------------------------------------------------------------------
5558
# Fixup files
@@ -59,7 +62,7 @@
5962
[".coveragerc"],
6063
"google/cloud/__init__.py",
6164
"sqlalchemy_bigquery/requirements.py",
62-
)
65+
)
6366

6467
s.replace(
6568
["noxfile.py"],
@@ -75,49 +78,43 @@
7578

7679

7780
s.replace(
78-
["noxfile.py"], "--cov=google", "--cov=sqlalchemy_bigquery",
81+
["noxfile.py"],
82+
"--cov=google",
83+
"--cov=sqlalchemy_bigquery",
7984
)
8085

8186

8287
s.replace(
83-
["noxfile.py"],
88+
["noxfile.py"],
8489
"\+ SYSTEM_TEST_EXTRAS",
8590
"",
8691
)
8792

8893

8994
s.replace(
9095
["noxfile.py"],
91-
'''"protobuf",
92-
# dependency of grpc''',
93-
'''"protobuf",
94-
"sqlalchemy<2.0.0",
95-
# dependency of grpc''',
96+
""""protobuf",
97+
# dependency of grpc""",
98+
""""protobuf",
99+
"sqlalchemy",
100+
# dependency of grpc""",
96101
)
97102

98103

99104
s.replace(
100105
["noxfile.py"],
101106
r"def default\(session\)",
102-
"def default(session, install_extras=True)",
107+
"def default(session, install_extras=True)",
103108
)
104109

105110

106-
107-
108111
def place_before(path, text, *before_text, escape=None):
109112
replacement = "\n".join(before_text) + "\n" + text
110113
if escape:
111114
for c in escape:
112-
text = text.replace(c, '\\' + c)
115+
text = text.replace(c, "\\" + c)
113116
s.replace([path], text, replacement)
114117

115-
place_before(
116-
"noxfile.py",
117-
"SYSTEM_TEST_PYTHON_VERSIONS=",
118-
"",
119-
"# We're using two Python versions to test with sqlalchemy 1.3 and 1.4.",
120-
)
121118

122119
place_before(
123120
"noxfile.py",
@@ -126,15 +123,15 @@ def place_before(path, text, *before_text, escape=None):
126123
)
127124

128125

129-
install_logic = '''
126+
install_logic = """
130127
if install_extras and session.python in ["3.11", "3.12"]:
131128
install_target = ".[geography,alembic,tests,bqstorage]"
132129
elif install_extras:
133130
install_target = ".[all]"
134131
else:
135132
install_target = "."
136133
session.install("-e", install_target, "-c", constraints_path)
137-
'''
134+
"""
138135

139136
place_before(
140137
"noxfile.py",
@@ -162,8 +159,6 @@ def compliance(session):
162159
if not os.path.exists(system_test_folder_path):
163160
session.skip("Compliance tests were not found")
164161
165-
session.install("--pre", "grpcio")
166-
session.install("--pre", "--no-deps", "--upgrade", "sqlalchemy<2.0.0")
167162
session.install(
168163
"mock",
169164
"pytest",
@@ -206,12 +201,11 @@ def compliance(session):
206201
'''
207202

208203
place_before(
209-
"noxfile.py",
210-
"@nox.session(python=DEFAULT_PYTHON_VERSION)\n"
211-
"def cover(session):",
212-
compliance,
213-
escape="()",
214-
)
204+
"noxfile.py",
205+
"@nox.session(python=DEFAULT_PYTHON_VERSION)\n" "def cover(session):",
206+
compliance,
207+
escape="()",
208+
)
215209

216210
s.replace(["noxfile.py"], '"alabaster"', '"alabaster", "geoalchemy2", "shapely"')
217211

@@ -267,11 +261,10 @@ def system_noextras(session):
267261

268262
place_before(
269263
"noxfile.py",
270-
"@nox.session(python=SYSTEM_TEST_PYTHON_VERSIONS[-1])\n"
271-
"def compliance(session):",
264+
"@nox.session(python=SYSTEM_TEST_PYTHON_VERSIONS[-1])\n" "def compliance(session):",
272265
system_noextras,
273266
escape="()[]",
274-
)
267+
)
275268

276269

277270
# Add DB config for SQLAlchemy dialect test suite.
@@ -288,7 +281,7 @@ def system_noextras(session):
288281
[tool:pytest]
289282
addopts= --tb native -v -r fxX -p no:warnings
290283
python_files=tests/*test_*.py
291-
"""
284+
""",
292285
)
293286

294287
# ----------------------------------------------------------------------------
@@ -299,7 +292,7 @@ def system_noextras(session):
299292
python.py_samples(skip_readmes=True)
300293

301294
s.replace(
302-
["./samples/snippets/noxfile.py"],
295+
["./samples/snippets/noxfile.py"],
303296
"""session.install\("-e", _get_repo_root\(\)\)""",
304297
"""session.install("-e", _get_repo_root())
305298
else:

‎packages/sqlalchemy-bigquery/setup.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,9 +99,9 @@ def readme():
9999
# Until this issue is closed
100100
# https://github.com/googleapis/google-cloud-python/issues/10566
101101
"google-auth>=1.25.0,<3.0.0dev", # Work around pip wack.
102-
"google-cloud-bigquery>=2.25.2,<4.0.0dev",
102+
"google-cloud-bigquery>=3.3.6,<4.0.0dev",
103103
"packaging",
104-
"sqlalchemy>=1.2.0,<2.0.0dev",
104+
"sqlalchemy>=1.4.16,<3.0.0dev",
105105
],
106106
extras_require=extras,
107107
python_requires=">=3.8, <3.13",

‎packages/sqlalchemy-bigquery/sqlalchemy_bigquery/_struct.py‎

Lines changed: 12 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -17,20 +17,14 @@
1717
# IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
1818
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
1919

20-
import packaging.version
2120
import sqlalchemy.sql.default_comparator
2221
import sqlalchemy.sql.sqltypes
2322
import sqlalchemy.types
2423

2524
from . import base
2625

27-
sqlalchemy_1_4_or_more = packaging.version.parse(
28-
sqlalchemy.__version__
29-
) >= packaging.version.parse("1.4")
30-
31-
if sqlalchemy_1_4_or_more:
32-
import sqlalchemy.sql.coercions
33-
import sqlalchemy.sql.roles
26+
import sqlalchemy.sql.coercions
27+
import sqlalchemy.sql.roles
3428

3529

3630
def _get_subtype_col_spec(type_):
@@ -103,34 +97,20 @@ def _setup_getitem(self, name):
10397
def __getattr__(self, name):
10498
if name.lower() in self.expr.type._STRUCT_byname:
10599
return self[name]
100+
else:
101+
raise AttributeError(name)
106102

107103
comparator_factory = Comparator
108104

109105

110-
# In the implementations of _field_index below, we're stealing from
111-
# the JSON type implementation, but the code to steal changed in
112-
# 1.4. :/
113-
114-
if sqlalchemy_1_4_or_more:
115-
116-
def _field_index(self, name, operator):
117-
return sqlalchemy.sql.coercions.expect(
118-
sqlalchemy.sql.roles.BinaryElementRole,
119-
name,
120-
expr=self.expr,
121-
operator=operator,
122-
bindparam_type=sqlalchemy.types.String(),
123-
)
124-
125-
else:
126-
127-
def _field_index(self, name, operator):
128-
return sqlalchemy.sql.default_comparator._check_literal(
129-
self.expr,
130-
operator,
131-
name,
132-
bindparam_type=sqlalchemy.types.String(),
133-
)
106+
def _field_index(self, name, operator):
107+
return sqlalchemy.sql.coercions.expect(
108+
sqlalchemy.sql.roles.BinaryElementRole,
109+
name,
110+
expr=self.expr,
111+
operator=operator,
112+
bindparam_type=sqlalchemy.types.String(),
113+
)
134114

135115

136116
def struct_getitem_op(a, b):

0 commit comments

Comments
 (0)