Skip to content

Commit 2ec19d5

Browse files
DjvuLeecmonkey
DjvuLee
authored andcommitted
[SPARK-19239][PYSPARK] Check parameters whether equals None when specify the column in jdbc API
## What changes were proposed in this pull request? The `jdbc` API do not check the `lowerBound` and `upperBound` when we specified the ``column``, and just throw the following exception: >```int() argument must be a string or a number, not 'NoneType'``` If we check the parameter, we can give a more friendly suggestion. ## How was this patch tested? Test using the pyspark shell, without the lowerBound and upperBound parameters. Author: DjvuLee <lihu@bytedance.com> Closes apache#16599 from djvulee/pysparkFix.
1 parent 18d93d8 commit 2ec19d5

File tree

1 file changed

+6
-3
lines changed

1 file changed

+6
-3
lines changed

python/pyspark/sql/readwriter.py

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -399,7 +399,8 @@ def jdbc(self, url, table, column=None, lowerBound=None, upperBound=None, numPar
399399
accessible via JDBC URL ``url`` and connection ``properties``.
400400
401401
Partitions of the table will be retrieved in parallel if either ``column`` or
402-
``predicates`` is specified.
402+
``predicates`` is specified. ``lowerBound`, ``upperBound`` and ``numPartitions``
403+
is needed when ``column`` is specified.
403404
404405
If both ``column`` and ``predicates`` are specified, ``column`` will be used.
405406
@@ -429,8 +430,10 @@ def jdbc(self, url, table, column=None, lowerBound=None, upperBound=None, numPar
429430
for k in properties:
430431
jprop.setProperty(k, properties[k])
431432
if column is not None:
432-
if numPartitions is None:
433-
numPartitions = self._spark._sc.defaultParallelism
433+
assert lowerBound is not None, "lowerBound can not be None when ``column`` is specified"
434+
assert upperBound is not None, "upperBound can not be None when ``column`` is specified"
435+
assert numPartitions is not None, \
436+
"numPartitions can not be None when ``column`` is specified"
434437
return self._df(self._jreader.jdbc(url, table, column, int(lowerBound), int(upperBound),
435438
int(numPartitions), jprop))
436439
if predicates is not None:

0 commit comments

Comments
 (0)