Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUGFIX] fix reusing a sparkSession using the force_reuse_spark_context keyword #3245

Closed
wants to merge 31 commits into from
Closed

[BUGFIX] fix reusing a sparkSession using the force_reuse_spark_context keyword #3245

Changes from 2 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
7201b77
[MAINTENANCE] Adds spark_config and force_reuse_spark_context fields…
Jul 28, 2021
e16462d
Merge branch 'develop' into maintenance/reuse-spark-context-emr
pasmavie Jul 28, 2021
a4e4ff8
Merge branch 'develop' into maintenance/reuse-spark-context-emr
pasmavie Aug 6, 2021
d2c80fa
Adds changes bullet point to docs_rtd/changelog.rst
pasmavie Aug 6, 2021
e40de89
Merge remote-tracking branch 'origin/develop' into maintenance/reuse-…
mbakunze Aug 13, 2021
1ce2ca1
Merge branch 'develop' into maintenance/reuse-spark-context-emr
mbakunze Aug 13, 2021
01e5e51
remove spark_config from DatasourceConfigSchema
mbakunze Aug 13, 2021
851cd04
allow None for force_reuse_spark_context
mbakunze Aug 13, 2021
9ec3735
fix linting
mbakunze Aug 13, 2021
c170c9b
remove missing entry for force_reuse_spark_context
mbakunze Aug 13, 2021
c5a6c66
adding back spark_config
mbakunze Aug 14, 2021
d5ca8d5
Merge branch 'develop' into maintenance/reuse-spark-context-emr
mbakunze Aug 15, 2021
10aca2a
adding unittests for `force_reuse_spark_context`
mbakunze Aug 16, 2021
44cd676
Merge remote-tracking branch 'origin/maintenance/reuse-spark-context-…
mbakunze Aug 16, 2021
1adde14
Merge branch 'develop' into maintenance/reuse-spark-context-emr
mbakunze Aug 16, 2021
cb39f1b
adding unittest for `spark_config` pass through
mbakunze Aug 16, 2021
bbc4ec0
Merge remote-tracking branch 'origin/maintenance/reuse-spark-context-…
mbakunze Aug 16, 2021
cfb8c98
adding unittest for `spark_config` pass through + run black
mbakunze Aug 16, 2021
82abefe
add docstrings to unittests to clarify intent
mbakunze Aug 16, 2021
d336103
add isort:skip for spark imports
mbakunze Aug 16, 2021
3ae3805
rename test to be more meaningful
mbakunze Aug 16, 2021
16f1c71
fix TypeError in unittest
mbakunze Aug 16, 2021
0a4ff1d
adding a non-default config in order to hopefully see the Py4jError i…
mbakunze Aug 16, 2021
6bc4ac7
remove explicit P4jError test
mbakunze Aug 16, 2021
014140c
Merge branch 'develop' into maintenance/reuse-spark-context-emr
mbakunze Aug 17, 2021
e0a128e
add explicit test for sparkdf kwargs pass-through
mbakunze Aug 17, 2021
c634a28
Merge remote-tracking branch 'origin/maintenance/reuse-spark-context-…
mbakunze Aug 17, 2021
ff93e0b
update changelog entry to reflect this is a BUGFIX
mbakunze Aug 17, 2021
3ca3fb1
fix linting
mbakunze Aug 17, 2021
2daf0eb
Merge branch 'develop' into maintenance/reuse-spark-context-emr
mbakunze Aug 19, 2021
1ac14a7
Merge branch 'develop' into maintenance/reuse-spark-context-emr
mbakunze Aug 19, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

These merge commits were added into this branch cleanly.

There are no new changes to show.