[SPARK-10809] [MLlib] Single-document topicDistributions method for LocalLDAModel #9484
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #45087 has finished for PR 9484 at commit
|
| * @return (document ID, topic mixture distribution for document) | ||
| */ | ||
| @Since("1.6.0") | ||
| def topicDistributions(document: (Long, Vector)): (Long, Vector) = { |
There was a problem hiding this comment.
Can you please remove the doc ID? It's not necessary for a single doc, and removing it will make this more Java-friendly.
|
Test build #45202 has finished for PR 9484 at commit
|
asfgit
pushed a commit
that referenced
this pull request
Nov 11, 2015
This adds LDA to spark.ml, the Pipelines API. It follows the design doc in the JIRA: [https://issues.apache.org/jira/browse/SPARK-5565], with one major change: * I eliminated doc IDs. These are not necessary with DataFrames since the user can add an ID column as needed. Note: This will conflict with [#9484], but I'll try to merge [#9484] first and then rebase this PR. CC: hhbyyh feynmanliang If you have a chance to make a pass, that'd be really helpful--thanks! Now that I'm done traveling & this PR is almost ready, I'll see about reviewing other PRs critical for 1.6. CC: mengxr Author: Joseph K. Bradley <joseph@databricks.com> Closes #9513 from jkbradley/lda-pipelines. (cherry picked from commit e281b87) Signed-off-by: Joseph K. Bradley <joseph@databricks.com>
asfgit
pushed a commit
that referenced
this pull request
Nov 11, 2015
This adds LDA to spark.ml, the Pipelines API. It follows the design doc in the JIRA: [https://issues.apache.org/jira/browse/SPARK-5565], with one major change: * I eliminated doc IDs. These are not necessary with DataFrames since the user can add an ID column as needed. Note: This will conflict with [#9484], but I'll try to merge [#9484] first and then rebase this PR. CC: hhbyyh feynmanliang If you have a chance to make a pass, that'd be really helpful--thanks! Now that I'm done traveling & this PR is almost ready, I'll see about reviewing other PRs critical for 1.6. CC: mengxr Author: Joseph K. Bradley <joseph@databricks.com> Closes #9513 from jkbradley/lda-pipelines.
kiszk
pushed a commit
to kiszk/spark-gpu
that referenced
this pull request
Dec 26, 2015
This adds LDA to spark.ml, the Pipelines API. It follows the design doc in the JIRA: [https://issues.apache.org/jira/browse/SPARK-5565], with one major change: * I eliminated doc IDs. These are not necessary with DataFrames since the user can add an ID column as needed. Note: This will conflict with [apache/spark#9484], but I'll try to merge [apache/spark#9484] first and then rebase this PR. CC: hhbyyh feynmanliang If you have a chance to make a pass, that'd be really helpful--thanks! Now that I'm done traveling & this PR is almost ready, I'll see about reviewing other PRs critical for 1.6. CC: mengxr Author: Joseph K. Bradley <joseph@databricks.com> Closes #9513 from jkbradley/lda-pipelines.
| * to [[topicDistributions(documents: RDD[(Long, Vector)])]] to avoid overhead. | ||
| * | ||
| * @param document document to predict topic mixture distributions for | ||
| * @return (document ID, topic mixture distribution for document) |
|
@hhbyyh Sorry again for the delay, but we can get this merged now |
|
@jkbradley It's quite all right. Thanks for reviewing. Update sent. |
|
Test build #48895 has finished for PR 9484 at commit
|
| * literature). Returns a vector of zeros for an empty document. | ||
| * | ||
| * Note this means to allow quick query for single document. For batch documents, please refer | ||
| * to [[topicDistributions(documents: RDD[(Long, Vector)])]] to avoid overhead. |
There was a problem hiding this comment.
The Scala doc for this line is not generated correctly. Can you try removing the argument and just writing [[topicDistributions]] instead?
|
Sorry for the late response. Update sent |
|
Jenkins, retest this please. |
|
Test build #49109 has finished for PR 9484 at commit
|
|
Getting many TimeoutException. |
|
Test build #49124 has finished for PR 9484 at commit
|
|
LGTM |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
jira: https://issues.apache.org/jira/browse/SPARK-10809
We could provide a single-document topicDistributions method for LocalLDAModel to allow for quick queries which avoid RDD operations. Currently, the user must use an RDD of documents.
add some missing assert too.