Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python UDF in Ingestion being used for feature validation #1234

Merged
merged 24 commits into from
Dec 22, 2020
Merged

Python UDF in Ingestion being used for feature validation #1234

Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
0e68ff9
first draft
pyalex Dec 11, 2020
b9ed047
restore udf in ingestion job
pyalex Dec 14, 2020
063d8fc
it tests for pandas udf
pyalex Dec 15, 2020
903e743
e2e test
pyalex Dec 16, 2020
2b86ea7
skip udf build
pyalex Dec 16, 2020
0ba6a4b
$skipITs
pyalex Dec 16, 2020
13ed5ab
$skipITs
pyalex Dec 16, 2020
366605f
add reporting to scalatest
pyalex Dec 17, 2020
bb72032
platform specific libs path
pyalex Dec 17, 2020
afbf8c2
continue on error
pyalex Dec 17, 2020
b5efcd6
build pylibs for all platforms
pyalex Dec 17, 2020
5a908e3
build pylibs for all platforms
pyalex Dec 17, 2020
65a2b7b
pull python path from env
pyalex Dec 17, 2020
e56dd9e
freeze dataproc python version
pyalex Dec 17, 2020
fb18c92
use python from config
pyalex Dec 17, 2020
6b67e63
add pyarrow to package
pyalex Dec 17, 2020
0fd3c3d
drop dataclasses to support 3.6
pyalex Dec 17, 2020
2d126c6
support python 3.6
pyalex Dec 17, 2020
8d4e476
set spark.yarn.isPython=true for emr
pyalex Dec 18, 2020
bf63033
pass isPython through spark-submit conf
pyalex Dec 18, 2020
54fcb6e
some cleanup
pyalex Dec 18, 2020
c348481
lint-java
pyalex Dec 18, 2020
a6d272f
Update infra/scripts/build-ingestion-py-dependencies.sh
pyalex Dec 21, 2020
4aa0233
revert batch sources fix
pyalex Dec 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
lint-java 
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
  • Loading branch information
@pyalex
pyalex committed Dec 18, 2020
commit c348481fe333a37d775d65f6ce6186b0b386d83c
3 changes: 2 additions & 1 deletion spark/ingestion/src/main/scala/org/apache/spark/api/python/DynamicPythonFunction.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ import collection.JavaConverters._
object DynamicPythonFunction {
private val conf = SparkEnv.get.conf

val pythonExec = conf.get(PYSPARK_DRIVER_PYTHON)
val pythonExec = conf
.get(PYSPARK_DRIVER_PYTHON)
.orElse(conf.get(PYSPARK_PYTHON))
.orElse(sys.env.get("PYSPARK_DRIVER_PYTHON"))
.orElse(sys.env.get("PYSPARK_PYTHON"))
Expand Down