-
Notifications
You must be signed in to change notification settings - Fork 28.6k
[MINOR][DOCS] Fix invalid documentation for StreamingQueryManager Class #24547
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
When following the example for using `spark.streams().awaitAnyTermination()` a valid pyspark code will output the following error: ```Traceback (most recent call last): File "pyspark_app.py", line 182, in <module> spark.streams().awaitAnyTermination() TypeError: 'StreamingQueryManager' object is not callable``` This changes the documentation line to properly call the method under the StreamingQueryManager Class https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.streaming.StreamingQueryManager
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you see any other instances like this?
Test build #4778 has finished for PR 24547 at commit
|
@@ -2554,11 +2554,11 @@ spark.streams().awaitAnyTermination(); // block until any one of them terminat | |||
{% highlight python %} | |||
spark = ... # spark session |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Honestly, I think we should make this as a self-contained example .. that's maybe the root cause that we have not-working examples. We can do it separately later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I quickly double checked. Let's fix sparkSession.streams()
in the doc -> sparkSession.streams
. Looks good otherwise
@srowen for sure, There are more places in the spark docs that have not-working examples, that I have encountered, personally. I believe @HyukjinKwon is right, as I was searching for pyspark knowledge resources, databricks saved me many times, with a full working example, somewhat self-contained (enough to learn from the example and implement on my code without actually 'running' the example) I'm currently in the process of building a spark cluster for data science infrastructure and data lake, As a first time user starting out with spark, which looks like (and is) a mature program, This is just the first time I actually make a PR about something that I found was wrong in official docs, I'll try to be more alert, and post more PR's as I re-encounter invalid docs.. |
Merged to master, branch-2.4 and branch-2.3. |
## What changes were proposed in this pull request? When following the example for using `spark.streams().awaitAnyTermination()` a valid pyspark code will output the following error: ``` Traceback (most recent call last): File "pyspark_app.py", line 182, in <module> spark.streams().awaitAnyTermination() TypeError: 'StreamingQueryManager' object is not callable ``` Docs URL: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#managing-streaming-queries This changes the documentation line to properly call the method under the StreamingQueryManager Class https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.streaming.StreamingQueryManager ## How was this patch tested? After changing the syntax, error no longer occurs and pyspark application works This is only docs change Closes #24547 from asaf400/patch-1. Authored-by: Asaf Levy <asaf400@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit 09422f5) Signed-off-by: HyukjinKwon <gurwls223@apache.org>
## What changes were proposed in this pull request? When following the example for using `spark.streams().awaitAnyTermination()` a valid pyspark code will output the following error: ``` Traceback (most recent call last): File "pyspark_app.py", line 182, in <module> spark.streams().awaitAnyTermination() TypeError: 'StreamingQueryManager' object is not callable ``` Docs URL: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#managing-streaming-queries This changes the documentation line to properly call the method under the StreamingQueryManager Class https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.streaming.StreamingQueryManager ## How was this patch tested? After changing the syntax, error no longer occurs and pyspark application works This is only docs change Closes #24547 from asaf400/patch-1. Authored-by: Asaf Levy <asaf400@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit 09422f5) Signed-off-by: HyukjinKwon <gurwls223@apache.org>
## What changes were proposed in this pull request? When following the example for using `spark.streams().awaitAnyTermination()` a valid pyspark code will output the following error: ``` Traceback (most recent call last): File "pyspark_app.py", line 182, in <module> spark.streams().awaitAnyTermination() TypeError: 'StreamingQueryManager' object is not callable ``` Docs URL: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#managing-streaming-queries This changes the documentation line to properly call the method under the StreamingQueryManager Class https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.streaming.StreamingQueryManager ## How was this patch tested? After changing the syntax, error no longer occurs and pyspark application works This is only docs change Closes apache#24547 from asaf400/patch-1. Authored-by: Asaf Levy <asaf400@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit 09422f5) Signed-off-by: HyukjinKwon <gurwls223@apache.org>
## What changes were proposed in this pull request? When following the example for using `spark.streams().awaitAnyTermination()` a valid pyspark code will output the following error: ``` Traceback (most recent call last): File "pyspark_app.py", line 182, in <module> spark.streams().awaitAnyTermination() TypeError: 'StreamingQueryManager' object is not callable ``` Docs URL: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#managing-streaming-queries This changes the documentation line to properly call the method under the StreamingQueryManager Class https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.streaming.StreamingQueryManager ## How was this patch tested? After changing the syntax, error no longer occurs and pyspark application works This is only docs change Closes apache#24547 from asaf400/patch-1. Authored-by: Asaf Levy <asaf400@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit 09422f5) Signed-off-by: HyukjinKwon <gurwls223@apache.org>
What changes were proposed in this pull request?
When following the example for using
spark.streams().awaitAnyTermination()
a valid pyspark code will output the following error:
Docs URL: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#managing-streaming-queries
This changes the documentation line to properly call the method under the StreamingQueryManager Class
https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.streaming.StreamingQueryManager
How was this patch tested?
After changing the syntax, error no longer occurs and pyspark application works
This is only docs change