-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-1199][REPL] Remove VALId and use the original import style for defined classes. #1179
[SPARK-1199][REPL] Remove VALId and use the original import style for defined classes. #1179
Conversation
Merged build triggered. |
Merged build started. |
Merged build finished. |
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16022/ |
Merged build triggered. |
Merged build started. |
Merged build finished. All automated tests passed. |
All automated tests passed. |
@@ -182,15 +182,26 @@ trait SparkImports { | |||
// ambiguity errors will not be generated. Also, quote | |||
// the name of the variable, so that we don't need to | |||
// handle quoting keywords separately. | |||
case x: ClassHandler => | |||
// I am trying to guess if the import is a defined class | |||
// This is an ugly hack, I am not 100% sure of the consequences. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think is ugly about this? Is this the original way the REPL used to include classes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This portion of code is just about importing, the ugly part is that there should not be special casing for different things ideally. This can lead to situations where we don't know what next needs a special case. Thus it is hard for us to pitch this to change to original scala REPL.
The actual ugly part is next case statement
where we put Val id = something and then import it. This was the actual reason for this SPARK-1199 issue. But unfortunately we can not avoid it, because otherwise the remote executors try to pull in classes which they don't need to. But since we are living with that code for a while it should ideally be okay.
Yes importing without the intermediate val creation was the originally followed style, which we changed to solve the problem described above.
Suppose we only used code.append("import " + objName + ".INSTANCE" + req.accessPath + ".
" + imv + "\n")
this to import and no other way, then it is much better than what we are doing and there will be no need for migration of REPL for next upgrade.
…a class or not. and also support " // show"
Merged build triggered. |
Merged build started. |
Merged build finished. All automated tests passed. |
All automated tests passed. |
… defined classes. This is an alternate solution to #1176. Author: Prashant Sharma <prashant.s@imaginea.com> Closes #1179 from ScrapCodes/SPARK-1199/repl-fix-second-approach and squashes the following commits: 820b34b [Prashant Sharma] Here we generate two kinds of import wrappers based on whether it is a class or not. (cherry picked from commit d434150) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
… defined classes. This is an alternate solution to apache#1176. Author: Prashant Sharma <prashant.s@imaginea.com> Closes apache#1179 from ScrapCodes/SPARK-1199/repl-fix-second-approach and squashes the following commits: 820b34b [Prashant Sharma] Here we generate two kinds of import wrappers based on whether it is a class or not.
This is an alternate solution to #1176.