Skip to content

[SPARK-6793][SPARK-5567][SPARK-8936][MLlib]Perplexity, prediction, and hyperparameter optimization in OnlineLDAOptimizer #7507

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 19 commits into from
Closed

[SPARK-6793][SPARK-5567][SPARK-8936][MLlib]Perplexity, prediction, and hyperparameter optimization in OnlineLDAOptimizer #7507

wants to merge 19 commits into from

Conversation

feynmanliang
Copy link
Contributor

Adds logPerplexity and topicDistribution methods to LocalLDAModel as well as {get,set}OptimizeAlphato OnlineLDAOptimizer. DRY out variationalTopicInference and bound code.

@feynmanliang feynmanliang changed the title [WIP][SPARK-5564][SPARK-5567][SPARK-8936]Perplexity, prediction, and hyperparameter optimization in OnlineLDAOptimizer [WIP][SPARK-6793][SPARK-5567][SPARK-8936]Perplexity, prediction, and hyperparameter optimization in OnlineLDAOptimizer Jul 19, 2015
@feynmanliang feynmanliang changed the title [WIP][SPARK-6793][SPARK-5567][SPARK-8936]Perplexity, prediction, and hyperparameter optimization in OnlineLDAOptimizer [WIP][SPARK-6793][SPARK-5567][SPARK-8936][MLlib]Perplexity, prediction, and hyperparameter optimization in OnlineLDAOptimizer Jul 19, 2015
@SparkQA
Copy link

SparkQA commented Jul 19, 2015

Test build #37764 has finished for PR 7507 at commit e481484.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Feynman Liang added 4 commits July 19, 2015 00:21
Style fix
7d1d3e2
Merge branch 'master' into SPARK-5567-LDA-predict
6bcd5ba 
* master: (133 commits)
  [SPARK-8536] [MLLIB] Generalize OnlineLDAOptimizer to asymmetric document-topic Dirichlet priors
  [SPARK-4366] [SQL] [Follow-up] Fix SqlParser compiling warning.
  [SPARK-9224] [MLLIB] OnlineLDA Performance Improvements
  [SPARK-9024] Unsafe HashJoin/HashOuterJoin/HashSemiJoin
  [SPARK-9165] [SQL] codegen for CreateArray, CreateStruct and CreateNamedStruct
  [SPARK-9082] [SQL] Filter using non-deterministic expressions should not be pushed down
  [SPARK-9254] [BUILD] [HOTFIX] sbt-launch-lib.bash should support HTTP/HTTPS redirection
  [SPARK-4233] [SPARK-4367] [SPARK-3947] [SPARK-3056] [SQL] Aggregation Improvement
  [SPARK-9232] [SQL] Duplicate code in JSONRelation
  [SPARK-9121] [SPARKR] Get rid of the warnings about `no visible global function definition` in SparkR
  [SPARK-9154][SQL] Rename formatString to format_string.
  [SPARK-9154] [SQL] codegen StringFormat
  [SPARK-9206] [SQL] Fix HiveContext classloading for GCS connector.
  [SPARK-8906][SQL] Move all internal data source classes into execution.datasources.
  [SPARK-8357] Fix unsafe memory leak on empty inputs in GeneratedAggregate
  Revert "[SPARK-9154] [SQL] codegen StringFormat"
  [SPARK-5989] [MLLIB] Model save/load for LDA
  [SPARK-9154] [SQL] codegen StringFormat
  [SPARK-5423] [CORE] Register a TaskCompletionListener to make sure release all resources
  [SPARK-4598] [WEBUI] Task table pagination for the Stage page
  ...
Integrate asymmetric alpha and model save-load
648a006
Remove merge files
01953e2
@feynmanliang feynmanliang changed the title [WIP][SPARK-6793][SPARK-5567][SPARK-8936][MLlib]Perplexity, prediction, and hyperparameter optimization in OnlineLDAOptimizer [SPARK-6793][SPARK-5567][SPARK-8936][MLlib]Perplexity, prediction, and hyperparameter optimization in OnlineLDAOptimizer Jul 23, 2015
@SparkQA
Copy link

SparkQA commented Jul 23, 2015

Test build #38143 has finished for PR 7507 at commit 5eb2430.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jkbradley
Copy link
Member

Are these several JIRAs tied together, or would it be easy to separate out the PRs?

@jkbradley jkbradley mentioned this pull request Jul 24, 2015
Closed
@feynmanliang
Copy link
Contributor Author

Can probably separate them out; both perplexity and predict depend on
pulling variationalInference out into companion object. I'll split into
multiple PRs after the hackathon.

On Fri, Jul 24, 2015 at 3:00 PM jkbradley notifications@github.com wrote:

Are these several JIRAs tied together, or would it be easy to separate out
the PRs?

—
Reply to this email directly or view it on GitHub
#7507 (comment).

@SparkQA
Copy link

SparkQA commented Jul 27, 2015

Test build #38583 has finished for PR 7507 at commit 0c81c45.

  • This patch fails Scala style tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jul 27, 2015

Test build #38601 has finished for PR 7507 at commit 46fca97.

  • This patch fails Python style tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@feynmanliang
Copy link
Contributor Author

Closing this and splitting into smaller PRs

@feynmanliang feynmanliang deleted the SPARK-5567-LDA-predict branch July 27, 2015 23:26
@feynmanliang feynmanliang mentioned this pull request Aug 26, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@feynmanliang @SparkQA @jkbradley