Skip to content

Commit 1be1443

Browse files
beliefercloud-fan
beliefer
authored and
cloud-fan
committed
[SPARK-39479][SQL] DS V2 supports push down math functions(non ANSI)
### What changes were proposed in this pull request? #36140 makes DS V2 supports push down math functions are claimed by ANSI standard. Spark have a lot common used math functions are not claimed by ANSI standard. https://github.com/apache/spark/blob/2f8613f22c0750c00cf1dcfb2f31c431d8dc1be7/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L388 The mainstream databases support these functions show below. | Function name | PostgreSQL | ClickHouse | H2 | MySQL | Oracle | Redshift | Presto | Teradata | Snowflake | DB2 | Vertica | Exasol | SqlServer | Yellowbrick | Impala | Mariadb | Druid | Pig | Singlestore | ElasticSearch | SQLite | Influxdata | Sybase | | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | | `SIN` | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | | `SINH` | Yes | Yes | Yes | No | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | No | No | Yes | No | Yes | Yes | Yes | No | | `COS` | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | | `COSH` | Yes | Yes | Yes | No | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | No | No | Yes | No | Yes | Yes | Yes | No | | `TAN` | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | | `TANH` | Yes | No | Yes | No | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | No | No | Yes | No | No | No | Yes | No | | `COT` | Yes | No | Yes | Yes | No | Yes | No | No | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | No | Yes | Yes | No | No | Yes | | `ASIN` | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | | `ASINH` | Yes | Yes | No | No | No | No | No | Yes | Yes | No | No | No | No | No | No | No | No | No | No | No | Yes | Yes | No | | `ACOS` | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | | `ACOSH` | Yes | Yes | No | No | No | No | No | Yes | Yes | No | No | No | No | No | No | No | No | No | No | No | Yes | Yes | No | | `ATAN` | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | | `ATAN2` | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | | `ATANH` | Yes | Yes | No | No | No | No | No | Yes | Yes | Yes | No | No | No | No | No | No | No | No | No | No | Yes | Yes | No | | `LOG` | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | | `LOG10` | Yes | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | | `LOG2` | No | Yes | No | Yes | No | No | Yes | Yes | No | No | No | Yes | No | No | Yes | Yes | No | No | Yes | No | Yes | Yes | No | | `CBRT` | Yes | Yes | No | No | No | Yes | Yes | No | Yes | No | Yes | No | No | Yes | No | No | No | Yes | No | Yes | No | Yes | No | | `DEGREES` | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | No | Yes | | `RADIANS` | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | No | Yes | | `ROUND` | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | | `SIGN` | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | No | No | Yes | ### Why are the changes needed? DS V2 supports push down math functions supported by mainstream databases. ### Does this PR introduce _any_ user-facing change? 'No'. New feature. ### How was this patch tested? New tests. Closes #36877 from beliefer/SPARK-39479. Authored-by: Jiaan Geng <beliefer@163.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
1 parent b88cabb commit 1be1443

File tree

5 files changed

+299
-37
lines changed

5 files changed

+299
-37
lines changed

‎sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/GeneralScalarExpression.java

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,24 @@
124124
* <li>Since version: 3.4.0</li>
125125
* </ul>
126126
* </li>
127+
* <li>Name: <code>LOG</code>
128+
* <ul>
129+
* <li>SQL semantic: <code>LOG(base, expr)</code></li>
130+
* <li>Since version: 3.4.0</li>
131+
* </ul>
132+
* </li>
133+
* <li>Name: <code>LOG10</code>
134+
* <ul>
135+
* <li>SQL semantic: <code>LOG10(expr)</code></li>
136+
* <li>Since version: 3.4.0</li>
137+
* </ul>
138+
* </li>
139+
* <li>Name: <code>LOG2</code>
140+
* <ul>
141+
* <li>SQL semantic: <code>LOG2(expr)</code></li>
142+
* <li>Since version: 3.4.0</li>
143+
* </ul>
144+
* </li>
127145
* <li>Name: <code>LN</code>
128146
* <ul>
129147
* <li>SQL semantic: <code>LN(expr)</code></li>
@@ -160,6 +178,120 @@
160178
* <li>Since version: 3.3.0</li>
161179
* </ul>
162180
* </li>
181+
* <li>Name: <code>ROUND</code>
182+
* <ul>
183+
* <li>SQL semantic: <code>ROUND(expr, [scale])</code></li>
184+
* <li>Since version: 3.4.0</li>
185+
* </ul>
186+
* </li>
187+
* <li>Name: <code>SIN</code>
188+
* <ul>
189+
* <li>SQL semantic: <code>SIN(expr)</code></li>
190+
* <li>Since version: 3.4.0</li>
191+
* </ul>
192+
* </li>
193+
* <li>Name: <code>SINH</code>
194+
* <ul>
195+
* <li>SQL semantic: <code>SINH(expr)</code></li>
196+
* <li>Since version: 3.4.0</li>
197+
* </ul>
198+
* </li>
199+
* <li>Name: <code>COS</code>
200+
* <ul>
201+
* <li>SQL semantic: <code>COS(expr)</code></li>
202+
* <li>Since version: 3.4.0</li>
203+
* </ul>
204+
* </li>
205+
* <li>Name: <code>COSH</code>
206+
* <ul>
207+
* <li>SQL semantic: <code>COSH(expr)</code></li>
208+
* <li>Since version: 3.4.0</li>
209+
* </ul>
210+
* </li>
211+
* <li>Name: <code>TAN</code>
212+
* <ul>
213+
* <li>SQL semantic: <code>TAN(expr)</code></li>
214+
* <li>Since version: 3.4.0</li>
215+
* </ul>
216+
* </li>
217+
* <li>Name: <code>TANH</code>
218+
* <ul>
219+
* <li>SQL semantic: <code>TANH(expr)</code></li>
220+
* <li>Since version: 3.4.0</li>
221+
* </ul>
222+
* </li>
223+
* <li>Name: <code>COT</code>
224+
* <ul>
225+
* <li>SQL semantic: <code>COT(expr)</code></li>
226+
* <li>Since version: 3.4.0</li>
227+
* </ul>
228+
* </li>
229+
* <li>Name: <code>ASIN</code>
230+
* <ul>
231+
* <li>SQL semantic: <code>ASIN(expr)</code></li>
232+
* <li>Since version: 3.4.0</li>
233+
* </ul>
234+
* </li>
235+
* <li>Name: <code>ASINH</code>
236+
* <ul>
237+
* <li>SQL semantic: <code>ASINH(expr)</code></li>
238+
* <li>Since version: 3.4.0</li>
239+
* </ul>
240+
* </li>
241+
* <li>Name: <code>ACOS</code>
242+
* <ul>
243+
* <li>SQL semantic: <code>ACOS(expr)</code></li>
244+
* <li>Since version: 3.4.0</li>
245+
* </ul>
246+
* </li>
247+
* <li>Name: <code>ACOSH</code>
248+
* <ul>
249+
* <li>SQL semantic: <code>ACOSH(expr)</code></li>
250+
* <li>Since version: 3.4.0</li>
251+
* </ul>
252+
* </li>
253+
* <li>Name: <code>ATAN</code>
254+
* <ul>
255+
* <li>SQL semantic: <code>ATAN(expr)</code></li>
256+
* <li>Since version: 3.4.0</li>
257+
* </ul>
258+
* </li>
259+
* <li>Name: <code>ATANH</code>
260+
* <ul>
261+
* <li>SQL semantic: <code>ATANH(expr)</code></li>
262+
* <li>Since version: 3.4.0</li>
263+
* </ul>
264+
* </li>
265+
* <li>Name: <code>ATAN2</code>
266+
* <ul>
267+
* <li>SQL semantic: <code>ATAN2(exprY, exprX)</code></li>
268+
* <li>Since version: 3.4.0</li>
269+
* </ul>
270+
* </li>
271+
* <li>Name: <code>CBRT</code>
272+
* <ul>
273+
* <li>SQL semantic: <code>CBRT(expr)</code></li>
274+
* <li>Since version: 3.4.0</li>
275+
* </ul>
276+
* </li>
277+
* <li>Name: <code>DEGREES</code>
278+
* <ul>
279+
* <li>SQL semantic: <code>DEGREES(expr)</code></li>
280+
* <li>Since version: 3.4.0</li>
281+
* </ul>
282+
* </li>
283+
* <li>Name: <code>RADIANS</code>
284+
* <ul>
285+
* <li>SQL semantic: <code>RADIANS(expr)</code></li>
286+
* <li>Since version: 3.4.0</li>
287+
* </ul>
288+
* </li>
289+
* <li>Name: <code>SIGN</code>
290+
* <ul>
291+
* <li>SQL semantic: <code>SIGN(expr)</code></li>
292+
* <li>Since version: 3.4.0</li>
293+
* </ul>
294+
* </li>
163295
* <li>Name: <code>WIDTH_BUCKET</code>
164296
* <ul>
165297
* <li>SQL semantic: <code>WIDTH_BUCKET(expr)</code></li>

‎sql/catalyst/src/main/java/org/apache/spark/sql/connector/util/V2ExpressionSQLBuilder.java

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,12 +100,34 @@ public String build(Expression expr) {
100100
case "GREATEST":
101101
case "LEAST":
102102
case "RAND":
103+
case "LOG":
104+
case "LOG10":
105+
case "LOG2":
103106
case "LN":
104107
case "EXP":
105108
case "POWER":
106109
case "SQRT":
107110
case "FLOOR":
108111
case "CEIL":
112+
case "ROUND":
113+
case "SIN":
114+
case "SINH":
115+
case "COS":
116+
case "COSH":
117+
case "TAN":
118+
case "TANH":
119+
case "COT":
120+
case "ASIN":
121+
case "ASINH":
122+
case "ACOS":
123+
case "ACOSH":
124+
case "ATAN":
125+
case "ATANH":
126+
case "ATAN2":
127+
case "CBRT":
128+
case "DEGREES":
129+
case "RADIANS":
130+
case "SIGN":
109131
case "WIDTH_BUCKET":
110132
case "SUBSTRING":
111133
case "UPPER":

‎sql/core/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,18 @@ class V2ExpressionBuilder(e: Expression, isPredicate: Boolean = false) {
120120
generateExpression(child)
121121
.map(v => new GeneralScalarExpression("RAND", Array[V2Expression](v)))
122122
}
123+
case log: Logarithm =>
124+
val l = generateExpression(log.left)
125+
val r = generateExpression(log.right)
126+
if (l.isDefined && r.isDefined) {
127+
Some(new GeneralScalarExpression("LOG", Array[V2Expression](l.get, r.get)))
128+
} else {
129+
None
130+
}
131+
case Log10(child) => generateExpression(child)
132+
.map(v => new GeneralScalarExpression("LOG10", Array[V2Expression](v)))
133+
case Log2(child) => generateExpression(child)
134+
.map(v => new GeneralScalarExpression("LOG2", Array[V2Expression](v)))
123135
case Log(child) => generateExpression(child)
124136
.map(v => new GeneralScalarExpression("LN", Array[V2Expression](v)))
125137
case Exp(child) => generateExpression(child)
@@ -138,6 +150,56 @@ class V2ExpressionBuilder(e: Expression, isPredicate: Boolean = false) {
138150
.map(v => new GeneralScalarExpression("FLOOR", Array[V2Expression](v)))
139151
case Ceil(child) => generateExpression(child)
140152
.map(v => new GeneralScalarExpression("CEIL", Array[V2Expression](v)))
153+
case round: Round =>
154+
val l = generateExpression(round.left)
155+
val r = generateExpression(round.right)
156+
if (l.isDefined && r.isDefined) {
157+
Some(new GeneralScalarExpression("ROUND", Array[V2Expression](l.get, r.get)))
158+
} else {
159+
None
160+
}
161+
case Sin(child) => generateExpression(child)
162+
.map(v => new GeneralScalarExpression("SIN", Array[V2Expression](v)))
163+
case Sinh(child) => generateExpression(child)
164+
.map(v => new GeneralScalarExpression("SINH", Array[V2Expression](v)))
165+
case Cos(child) => generateExpression(child)
166+
.map(v => new GeneralScalarExpression("COS", Array[V2Expression](v)))
167+
case Cosh(child) => generateExpression(child)
168+
.map(v => new GeneralScalarExpression("COSH", Array[V2Expression](v)))
169+
case Tan(child) => generateExpression(child)
170+
.map(v => new GeneralScalarExpression("TAN", Array[V2Expression](v)))
171+
case Tanh(child) => generateExpression(child)
172+
.map(v => new GeneralScalarExpression("TANH", Array[V2Expression](v)))
173+
case Cot(child) => generateExpression(child)
174+
.map(v => new GeneralScalarExpression("COT", Array[V2Expression](v)))
175+
case Asin(child) => generateExpression(child)
176+
.map(v => new GeneralScalarExpression("ASIN", Array[V2Expression](v)))
177+
case Asinh(child) => generateExpression(child)
178+
.map(v => new GeneralScalarExpression("ASINH", Array[V2Expression](v)))
179+
case Acos(child) => generateExpression(child)
180+
.map(v => new GeneralScalarExpression("ACOS", Array[V2Expression](v)))
181+
case Acosh(child) => generateExpression(child)
182+
.map(v => new GeneralScalarExpression("ACOSH", Array[V2Expression](v)))
183+
case Atan(child) => generateExpression(child)
184+
.map(v => new GeneralScalarExpression("ATAN", Array[V2Expression](v)))
185+
case Atanh(child) => generateExpression(child)
186+
.map(v => new GeneralScalarExpression("ATANH", Array[V2Expression](v)))
187+
case atan2: Atan2 =>
188+
val l = generateExpression(atan2.left)
189+
val r = generateExpression(atan2.right)
190+
if (l.isDefined && r.isDefined) {
191+
Some(new GeneralScalarExpression("ATAN2", Array[V2Expression](l.get, r.get)))
192+
} else {
193+
None
194+
}
195+
case Cbrt(child) => generateExpression(child)
196+
.map(v => new GeneralScalarExpression("CBRT", Array[V2Expression](v)))
197+
case ToDegrees(child) => generateExpression(child)
198+
.map(v => new GeneralScalarExpression("DEGREES", Array[V2Expression](v)))
199+
case ToRadians(child) => generateExpression(child)
200+
.map(v => new GeneralScalarExpression("RADIANS", Array[V2Expression](v)))
201+
case Signum(child) => generateExpression(child)
202+
.map(v => new GeneralScalarExpression("SIGN", Array[V2Expression](v)))
141203
case wb: WidthBucket =>
142204
val childrenExpressions = wb.children.flatMap(generateExpression(_))
143205
if (childrenExpressions.length == wb.children.length) {

‎sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,10 @@ private[sql] object H2Dialect extends JdbcDialect {
3535
url.toLowerCase(Locale.ROOT).startsWith("jdbc:h2")
3636

3737
private val supportedFunctions =
38-
Set("ABS", "COALESCE", "GREATEST", "LEAST", "RAND", "LN", "EXP", "POWER", "SQRT", "FLOOR",
39-
"CEIL", "SUBSTRING", "UPPER", "LOWER", "TRANSLATE", "TRIM")
38+
Set("ABS", "COALESCE", "GREATEST", "LEAST", "RAND", "LOG", "LOG10", "LN", "EXP",
39+
"POWER", "SQRT", "FLOOR", "CEIL", "ROUND", "SIN", "SINH", "COS", "COSH", "TAN",
40+
"TANH", "COT", "ASIN", "ACOS", "ATAN", "ATAN2", "DEGREES", "RADIANS", "SIGN",
41+
"PI", "SUBSTRING", "UPPER", "LOWER", "TRANSLATE", "TRIM")
4042

4143
override def isSupportedFunction(funcName: String): Boolean =
4244
supportedFunctions.contains(funcName)

0 commit comments

Comments
 (0)