[SPARK-11551][DOC][Example]Replace example code in ml-features.md using include_example #10002

somideshmukh · 2015-11-26T12:07:22Z

Made new patch contaning only markdown examples moved to exmaple/folder.
Ony three java code were not shfted since they were contaning compliation error ,these classes are
1)StandardScale 2)NormalizerExample 3)VectorIndexer

…ng include_example

AmplabJenkins · 2015-11-26T12:12:12Z

Can one of the admins verify this patch?

yinxusen · 2015-11-27T08:24:35Z

Hi @somideshmukh, since it's lots of files in this pull request, I send a pull request to your repo and you can see it here: somideshmukh#1

All of my modifications are there, you can check the code and merge it if it looks good to you. Then we can check the correctness of those newly added files.

yinxusen · 2015-11-27T08:28:53Z

You can see the files changed in that pull request. I make changes in your original code. The main issues there are:

The mismatch between file name and app name
The indention in code is not very correct
You forget to add // $example on$ and // $example off$ pairs in the code
The reorganize of imports

I also add Python files that you missed and 3 Java codes plus with 1 Scala code.

yinxusen · 2015-11-27T08:31:14Z

Besides, there is no need to create a new pull request every time you changing your code. You can modify your previous code and push the changes in current branch.

review result

yinxusen · 2015-11-28T12:37:26Z

docs/ml-features.md

-  for (String ngram : ngrams) System.out.print(ngram + " --- ");
-  System.out.println();
-}
+{% include_example java/org/apache/spark/examples/ml/JavaNGramExample.java %}
 {% endhighlight %}


Pls remove this line

yinxusen · 2015-11-28T12:55:45Z

@somideshmukh This is really a bunch of code. I check some errors as described above for my carelessness and sorry for that. And after fixing those errors, I can get it run in my computer. I'll also check the example codes one by one to make sure every code example is runnable without errors.

yinxusen · 2015-11-28T12:57:05Z

I think the conflict is due to some changes in ml-features.md, but we can leave it for now. After we fix all errors we can fix the conflict.

yinxusen · 2015-11-28T14:16:45Z

examples/src/main/java/org/apache/spark/examples/ml/JavaVectorIndexerExample.java

+    VectorIndexerModel indexerModel = indexer.fit(data);
+
+    Map<Integer, Map<Double, Integer>> categoryMaps = indexerModel.javaCategoryMaps();
+    System.out.print("Chose " + categoryMaps.size() + "categorical features:");


add a white space here " categorical features:", otherwise the output is like this Chose 351categorical features: ...

somideshmukh · 2015-12-01T10:00:43Z

I have made the changes you have mentioned in the given Java and Scala files and push the changes in Branch "SomilBranch1.33". I haven't modified Python files since I haven't work on that files and I donot have python knowledge.

somideshmukh · 2015-12-01T10:01:59Z

Pls check it and let me know whether you have got the changes.

srowen · 2015-12-01T10:02:23Z

@somideshmukh the changes are here as you can see, but your branch still needs a rebase.

yinxusen · 2015-12-01T10:13:49Z

@somideshmukh I'll help you change the python issues, then I will send another PR to you soon.

yinxusen · 2015-12-01T15:12:56Z

@somideshmukh I have already fix all issues and merged with master. You can accept my pull request here somideshmukh#2. Even though it looks very large, it can be small when you merging it. I believe we can merge the PR after that.

merge with master

yinxusen · 2015-12-02T07:38:42Z

@mengxr LGTM

yinxusen · 2015-12-08T03:16:24Z

@mengxr Shall we merge this PR? I find there are also some PRs modify ml-features.md, so I think it's better to merge it before new conflicts emerging.

mengxr · 2015-12-08T07:23:53Z

docs/ml-features.md

-DataFrame dctDf = dct.transform(df);
-dctDf.select("featuresDCT").show(3);
-{% endhighlight %}
+{% include_example java/org/apache/spark/examples/ml/JavaDCTExample.java %}}


minor: remove extra } at the end. You can submit a follow-up PR to fix it.

yinxusen · 2015-12-08T07:26:21Z

@mengxr Yes, I'll submit a follow-up pr to refine it.

mengxr · 2015-12-08T07:27:34Z

Merged into master and branch-1.6. Thanks!

…ing include_example Made new patch contaning only markdown examples moved to exmaple/folder. Ony three java code were not shfted since they were contaning compliation error ,these classes are 1)StandardScale 2)NormalizerExample 3)VectorIndexer Author: Xusen Yin <yinxusen@gmail.com> Author: somideshmukh <somilde@us.ibm.com> Closes #10002 from somideshmukh/SomilBranch1.33. (cherry picked from commit 78209b0) Signed-off-by: Xiangrui Meng <meng@databricks.com>

JoshRosen · 2015-12-08T10:24:52Z

Looks like this broke the Python style tests:

PEP8 checks failed.
./examples/src/main/python/ml/binarizer_example.py:41:4: E114 indentation is not a multiple of four (comment)
./examples/src/main/python/ml/onehot_encoder_example.py:39:1: W293 blank line contains whitespace
./examples/src/main/python/ml/pca_example.py:33:9: E128 continuation line under-indented for visual indent
./examples/src/main/python/ml/pca_example.py:35:41: E231 missing whitespace after ','
./examples/src/main/python/ml/polynomial_expansion_example.py:34:9: E128 continuation line under-indented for visual indent
./examples/src/main/python/ml/polynomial_expansion_example.py:35:9: E128 continuation line under-indented for visual indent

I guess that Jenkins never tested this PR?

liancheng · 2015-12-08T10:29:40Z

Reverting this one to bring back Spark builds.

yinxusen · 2015-12-08T10:53:17Z

Sorry for that, I will help fixing them.

yinxusen · 2015-12-08T11:11:57Z

@somideshmukh I give you another PR to fix the Python style issue: somideshmukh#3. After merging it, we can call jenkins testing once here.

This reverts PR apache#10002, commit 78209b0. The original PR wasn't tested on Jenkins before being merged. Author: Cheng Lian <lian@databricks.com> Closes apache#10200 from liancheng/revert-pr-10002.

This reverts PR #10002, commit 78209b0. The original PR wasn't tested on Jenkins before being merged. Author: Cheng Lian <lian@databricks.com> Closes #10200 from liancheng/revert-pr-10002. (cherry picked from commit da2012a) Signed-off-by: Cheng Lian <lian@databricks.com>

liancheng · 2015-12-08T11:23:28Z

Hit some network issue, just reverted this PR from master and branch-1.6 in #10200.

yinxusen · 2015-12-09T06:25:25Z

@somideshmukh Do you still have time work on this? Pleas let me know ASAP thanks.

liancheng · 2015-12-09T06:32:27Z

@yinxusen Maybe you can fork this PR branch and get it merged. You may add a note in your PR description to remind committers to attribute your PR to @somideshmukh. We can specify primary author while merging a PR using our merge script.

yinxusen · 2015-12-09T06:34:00Z

Sure, I will do it

somideshmukh · 2015-12-09T06:35:47Z

Please let me know what needs to be done,I have not worked on python code.

yinxusen · 2015-12-09T06:49:23Z

@somideshmukh Never mind, I add a new one on behalf of you: #10219 @liancheng if it looks good to you, please help me merge it after the Jenkins test. Thanks!

liancheng · 2015-12-09T06:59:12Z

@yinxusen Well, I'm probably too ignorant to review #10219 :) I'd ping @mengxr to sign it off.

somideshmukh added 2 commits November 26, 2015 14:43

[SPARK-11551][DOC][Example]Replace example code in ml-features.md usi…

d14f55d

…ng include_example

[SPARK-11551][DOC][Example]Replace example code in ml-features.md usi…

12b1cf3

…ng include_example

yinxusen added 6 commits November 27, 2015 13:06

fix java code issues

87e673e

fix scala issues

0e19113

add java vectorindexer, standardscaler, normalizer

f6a975e

add jsc stop

dd1d2c1

fix scala issues

3d1efc3

add python examples

c23bab4

yinxusen mentioned this pull request Nov 27, 2015

[SPARK-11551][DOC][Example]Replace example code in ml-features.md using include_example #9735

Closed

Merge pull request #1 from yinxusen/SomilBranch1.33

c143d4b

review result

yinxusen reviewed Nov 28, 2015
View reviewed changes

yinxusen added 2 commits December 1, 2015 22:58

fix minor issues

8a0d883

merge with master

bed2192

Merge pull request #2 from yinxusen/SomilBranch1.33

15121bd

merge with master

mengxr reviewed Dec 8, 2015
View reviewed changes

asfgit closed this in 78209b0 Dec 8, 2015

liancheng mentioned this pull request Dec 8, 2015

[SPARK-11551][DOC][EXAMPLE] Revert PR #10002 #10200

Closed

yinxusen mentioned this pull request Dec 9, 2015

[SPARK-12199][DOC] Follow-up: Refine example code in ml-features.md #10193

Closed

[SPARK-11551][DOC][Example]Replace example code in ml-features.md using include_example #10002

[SPARK-11551][DOC][Example]Replace example code in ml-features.md using include_example #10002

Uh oh!

Conversation

somideshmukh commented Nov 26, 2015

Uh oh!

AmplabJenkins commented Nov 26, 2015

Uh oh!

yinxusen commented Nov 27, 2015

Uh oh!

yinxusen commented Nov 27, 2015

Uh oh!

yinxusen commented Nov 27, 2015

Uh oh!

yinxusen Nov 28, 2015

Choose a reason for hiding this comment

Uh oh!

yinxusen commented Nov 28, 2015

Uh oh!

yinxusen commented Nov 28, 2015

Uh oh!

yinxusen Nov 28, 2015

Choose a reason for hiding this comment

Uh oh!

somideshmukh commented Dec 1, 2015

Uh oh!

somideshmukh commented Dec 1, 2015

Uh oh!

srowen commented Dec 1, 2015

Uh oh!

yinxusen commented Dec 1, 2015

Uh oh!

yinxusen commented Dec 1, 2015

Uh oh!

yinxusen commented Dec 2, 2015

Uh oh!

yinxusen commented Dec 8, 2015

Uh oh!

mengxr Dec 8, 2015

Choose a reason for hiding this comment

Uh oh!

yinxusen commented Dec 8, 2015

Uh oh!

mengxr commented Dec 8, 2015

Uh oh!

JoshRosen commented Dec 8, 2015

Uh oh!

liancheng commented Dec 8, 2015

Uh oh!

yinxusen commented Dec 8, 2015

Uh oh!

yinxusen commented Dec 8, 2015

Uh oh!

liancheng commented Dec 8, 2015

Uh oh!

yinxusen commented Dec 9, 2015

Uh oh!

liancheng commented Dec 9, 2015

Uh oh!

yinxusen commented Dec 9, 2015

Uh oh!

somideshmukh commented Dec 9, 2015

Uh oh!

yinxusen commented Dec 9, 2015

Uh oh!

liancheng commented Dec 9, 2015

Uh oh!

Uh oh!