Skip to content

[SPARK-19239][PySpark] Check parameters whether equals None when specify the column in jdbc API #16599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

[SPARK-19239][PySpark] Check parameters whether equals None when specify the column in jdbc API #16599

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
94c44ba
[SPARK-19239][PySparK] Check the lowerBound and upperBound whether eq…
Jan 16, 2017
43602b5
update the None equal comparison
Jan 17, 2017
f2dad9c
Update numPartitions check to keep API consistent
Jan 17, 2017
e9a1bca
Update the document for numPartition
Jan 17, 2017
d851382
Fix the python style
Jan 17, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions python/pyspark/sql/readwriter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -399,7 +399,8 @@ def jdbc(self, url, table, column=None, lowerBound=None, upperBound=None, numPar
accessible via JDBC URL ``url`` and connection ``properties``.

Partitions of the table will be retrieved in parallel if either ``column`` or
``predicates`` is specified.
``predicates`` is specified. ``lowerBound`, ``upperBound`` and ``numPartitions``
is needed when ``column`` is specified.

If both ``column`` and ``predicates`` are specified, ``column`` will be used.

Expand Down Expand Up @@ -429,8 +430,10 @@ def jdbc(self, url, table, column=None, lowerBound=None, upperBound=None, numPar
for k in properties:
jprop.setProperty(k, properties[k])
if column is not None:
if numPartitions is None:
numPartitions = self._spark._sc.defaultParallelism
assert lowerBound is not None, "lowerBound can not be None when ``column`` is specified"
assert upperBound is not None, "upperBound can not be None when ``column`` is specified"
assert numPartitions is not None, \
"numPartitions can not be None when ``column`` is specified"
return self._df(self._jreader.jdbc(url, table, column, int(lowerBound), int(upperBound),
int(numPartitions), jprop))
if predicates is not None:
Expand Down