[SPARK-14287] isStreaming method for Dataset #12080
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #54585 has finished for PR 12080 at commit
|
|
Test build #54589 has finished for PR 12080 at commit
|
|
LGTM |
| @@ -449,6 +450,17 @@ class Dataset[T] private[sql]( | |||
| def isLocal: Boolean = logicalPlan.isInstanceOf[LocalRelation] | |||
|
|
|||
| /** | |||
| * Returns true if the underlying query will be executed continuously as new data comes in. | |||
| * Methods that return bounded values, e.g. [[count()]], [[collect()]] will throw | |||
| * an exception if a Dataset is streaming. | |||
There was a problem hiding this comment.
How about?
Returns true if this [[Dataset]] contains one or more sources that continuously return data as it arrives.
A [[Dataset]] that reads data from a streaming source must be executed as a [[ContinuousQuery]] using
the `startStream()` method in [[DataFrameWriter]]. Methods that return a single answer, (e.g., `count()` or
`collect()`) will throw an [[AnalysisException]] when there is a streaming source present.
|
Implementation LGTM. |
|
@marmbrus Addressed your comment |
|
Test build #54875 has finished for PR 12080 at commit
|
|
Test build #54896 has finished for PR 12080 at commit
|
|
test this please |
|
@tdas here is another failure |
|
Test build #54907 has finished for PR 12080 at commit
|
|
Thanks, merging to master! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With the addition of StreamExecution (ContinuousQuery) to Datasets, data will become unbounded. With unbounded data, the execution of some methods and operations will not make sense, e.g.
Dataset.count().A simple API is required to check whether the data in a Dataset is bounded or unbounded. This will allow users to check whether their Dataset is in streaming mode or not. ML algorithms may check if the data is unbounded and throw an exception for example.
The implementation of this method is simple, however naming it is the challenge. Some possible names for this method are:
I've gone with
isStreamingfor now. We can change it before Spark 2.0 if we decide to come up with a different name. For that reason I've marked it as@Experimental