Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Integrate IPython setup on windows #211

Closed

Conversation

davidanthoff
Copy link
Contributor

I tested this on a brand new windows system with nothing but julia installed:

Pkg.clone("https://github.com/davidanthoff/IJulia.jl.git","IJulia")
Pkg.checkout("IJulia", "integratedipysetup")
Pkg.build("IJulia")

using IJulia
notebook()

works on that system.

This PR lacks proper error handling etc, consider it more a proof of concept.

@vtjnash
Copy link
Member

vtjnash commented Aug 7, 2014

I think this approach makes a lot of sense. I was working last night on doing something very similar to this through WinRPM. I'm still working through some of the details (on how to organize the files).

I think you may want to also use virtualenv (I believe it'll make upgrading easier, since you can more easily just delete the virtualenv folder and leave the rest of the python installation -- but since this is a local private python installation anyways, that may not matter).

@@ -1,2 +1,4 @@
*.pyc
/build/
deps/downloads
deps/usr
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should start with / – this makes them absolute relative to the directory the .gitignore file is in.

@davidanthoff
Copy link
Contributor Author

My take on the private vs user-installed python is this:

I think for IJulia it should always be a private install of python and ipython that should be tightly controlled by IJulia's build.jl, i.e. no one should install anything into that private python, it should just be hidden from users, i.e. users don't even have to know that it is there. From a users points of view it really is an implementation detail that IJulia happens to depend on python. The major benefit of that approach is that it should be much more robust, and hopefully issues like #86 will be a thing of the past.

For things like PyCall and PyPlot etc I think we should never provide a python installation, i.e. I think those should always rely on a user-installed python, with whatever packages the user wants to have etc.

From my point of view users that want to use IJulia should not even need to know that python is involved, but if someone wants to use a bridge to python it seems quite logical that he/she has to get a python first.

The main question is if that setup actually works: a private python for IJulia in combination with a user-installed python for PyCall. I think that should work, but we should make sure I'm right about that.

@vtjnash
Copy link
Member

vtjnash commented Aug 7, 2014

If calling using PyCall inside of the IPython REPL gets the "system" python (eg for integration with PyPlot ) I think the situation you describe is ideal (with a private ijulia python install)

@davidanthoff
Copy link
Contributor Author

I tried this for PyPlot, and it seems to work: IPython notebook based on the private install, but then it uses my anaconda python for pyplot when I use pyplot from a notebook. I haven't used pycall, maybe someone more familiar with it could try that out?

@davidanthoff
Copy link
Contributor Author

I also played around with some code that makes the separation between IPython and IJulia even more strong: instead of storing the julia profile in the default .ipython folder, we could use the --ipython-dir option and store the julia profile in a separate location, probably ideally somewhere in the IJulia package folder. I did not include that code in this PR because it would probably make sense to do that for all platforms, not just windows, if we decide to go that route.

What does get a bit more complicated is the command to launch IJulia. With the new option to use notebook() from within Julia that complexity could be completely hidden from the user, though.

@davidanthoff
Copy link
Contributor Author

Actually, the question of how to start IJulia is a bit more complicated in any case if we were to go with a private python install. When a system python is used, the entry point for users is the system's ipython, which might be on the PATH etc. With the private python install this gets tricky. We certainly don't want to put the private python on the PATH. I see two options:

  1. Have notebook() from within julia be the only user entry point. Easy and fool proof...
  2. Create a ijulia.bat maybe in deps\scripts that starts the private ipython install with the correct parameters (either --profile=julia or --ipython-dir=<juliaprofiledir>) and then add only that deps\scripts directory to the PATH. That way the only thing pulled into the general path would be the ijulia.bat command.

Somehow options 2 strikes me as a bit of a hack, or rather it seems that if that is the goal, really there should be a standard julia way to add commands to the PATH, like the scripts directory in python. But that seems a whole separate discussion... So maybe just stick with option 1... Not sure though.

@Carreau
Copy link
Contributor

Carreau commented Aug 7, 2014

IMHO Don't over-engineer that now. With IPython kernel spec and the IPython/Jupyter split you will soon not need to specify a profile dir anymore.

Envoyé de mon iPhone

Le 7 août 2014 à 20:09, David Anthoff notifications@github.com a écrit :

Actually, the question of how to start IJulia is a bit more complicated in any case if we were to go with a private python install. When a system python is used, the entry point for users is the system's ipython, which might be on the PATH etc. With the private python install this gets tricky. We certainly don't want to put the private python on the PATH. I see two options:

  1. Have notebook() from within julia be the only user entry point. Easy and fool proof...
  2. Create a ijulia.bat maybe in deps\scripts that starts the private ipython install with the correct parameters (either --profile=julia or --ipython-dir=) and then add only that deps\scripts directory to the PATH. That way the only thing pulled into the general path would be the ijulia.bat command.

Somehow options 2 strikes me as a bit of a hack, or rather it seems that if that is the goal, really there should be a standard julia way to add commands to the PATH, like the scripts directory in python. But that seems a whole separate discussion... So maybe just stick with option 1... Not sure though.


Reply to this email directly or view it on GitHub.

@StefanKarpinski
Copy link
Member

I was going to mention Jupyter as well. My understanding is that in the future a single instance of Jupyter will be able to open Julia, Python, R, etc. notebooks. With that design it makes more sense not to separate the configurations.

@davidanthoff
Copy link
Contributor Author

Hm, just looked at the whole kernel spec story and that seems to move in a very different direction than this PR... In particular, it looks like a whole lot of work to make one central IPython/Jupyter install work with different language kernels. It looks to me like that will do nothing to solve the problem we are trying to solve in this PR: to make Pkg.add("IJulia") be all users have to do to get a fully working IJulia.

I think from an IJulia point of view a system where there is as little integration into a system IPython/Jupyter as possible is much preferred because it requires less setup steps from users to get running, and there are way fewer integration points that can go wrong... I guess I just would prefer that from a user's point of view IJulia is a self contained package with no apparent dependencies on anything else on my system. It might happen to share code with IPython under the hood, but is there really a need for users to know about that?

@Carreau
Copy link
Contributor

Carreau commented Aug 7, 2014

Not saying the contrary, I'm just saying the the separate profile stuff is too complicated as in a few month you will not want to start Ipython/Jupyter with a different profile.

The IJulia installing IPython/Jupyter if not present make a lot of sens.

Envoyé de mon iPhone

Le 7 août 2014 à 21:05, David Anthoff notifications@github.com a écrit :

Hm, just looked at the whole kernel spec story and that seems to move in a very different direction than this PR... In particular, it looks like a whole lot of work to make one central IPython/Jupyter install work with different language kernels. It looks to me like that will do nothing to solve the problem we are trying to solve in this PR: to make Pkg.add("IJulia") be all users have to do to get a fully working IJulia.

I think from an IJulia point of view a system where there is as little integration into a system IPython/Jupyter as possible is much preferred because it requires less setup steps from users to get running, and there are way fewer integration points that can go wrong... I guess I just would prefer that from a user's point of view IJulia is a self contained package with no apparent dependencies on anything else on my system. It might happen to share code with IPython under the hood, but is there really a need for users to know about that?


Reply to this email directly or view it on GitHub.

@davidanthoff
Copy link
Contributor Author

The PR here is a VERY different model from IJulia "installing IPython/Jupyter if not present". The PR here will ALWAYS install a private python and ipython, hidden from everything else on the system, only for IJulia use. If there is an IPython already on the system, this PR will completely ignore it and have no integration with that. The python/ipython installed by this PR is not meant to be the new "default system/user ipython". It is a very different model from where Jupyter seems to be heading...

@davidanthoff
Copy link
Contributor Author

Will the new Jupyter still have an option like --ipython-dir so that one can use it without it ever interacting with the normal locations for kernel registration?

I think maybe a good model for the Jupyter time frame would be this:

  • IJulia installs a private python/ipython with a private kernel spec in the IJulia profile folder, i.e. everything is completely separate from any other Jupyter setup on the system. Calling notebook() from within julia will run that IJulia instance. It will have no support for other kernels etc, it is just a plain IJulia notebook, where users don't even know that this is based on IPython, that there might be other languages etc.
  • IF build.jl detects a system wide jupyter installation, it will also create a kernel spec in the normal jupyter kernel registration places, so that julia appears in the selection of kernels if one starts jupyter etc.

@stevengj
Copy link
Member

stevengj commented Aug 7, 2014

@davidanthoff, that makes sense to me.

@davidanthoff
Copy link
Contributor Author

I need some help (@StefanKarpinski maybe?). This line

run(`msiexec /passive /quiet /a $pythonmsifilename TARGETDIR="$pyinstalldir"`)

doesn't work when pyinstalldir contains spaces. What I want is that the following line gets run:

msiexec /passive /quiet /a somename.msi TARGETDIR="C:\path with space\py"

Instead, it looks like the whole TARGETDIR="C:\path with space\py" thing gets quotes, like this:

msiexec /passive /quiet /a somename.msi 'TARGETDIR=C:\path with space\py'

I read the running external programs section but couldn't figure out how to achieve that.

@StefanKarpinski
Copy link
Member

In UNIX shell quoting, those are the same thing. I'm not super clear on how things get executed on Windows, but if it's through a system call, then it's likely similar. @Keno, @vtjnash?

@davidanthoff
Copy link
Contributor Author

It seems msiexec.exe does treat them differently... People trying to call msiexec.exe from Powershell seem to have the same problem...

@Keno
Copy link
Member

Keno commented Aug 7, 2014

Try

run(`msiexec /passive /quiet /a $pythonmsifilename TARGETDIR=$(string('"',pyinstalldir,'"'))`)

@StefanKarpinski
Copy link
Member

Ah, so msiexec needs the quotes in the argument? In that case quoting the quotes out to work:

julia> pythonmsifilename = "somename.msi"
"somename.msi"

julia> pyinstalldir = "C:\\path with space\\py"
"C:\\path with space\\py"

julia> `msiexec /passive /quiet /a $pythonmsifilename TARGETDIR=\"$pyinstalldir\"`
`msiexec /passive /quiet /a somename.msi 'TARGETDIR="C:\path with space\py"'`

@davidanthoff
Copy link
Contributor Author

Tried all of these, no luck. What I can get is this:

`msiexec /passive /quiet /a 'downloads\python-2.7.8.msi' 'TARGETDIR="C:\Users\anthoff\.julia\v0.3\IJulia\deps\usr\python 27"'`

The problem probably is that there are still single quotes around the whole TARGETDIR stuff, i.e. it should be like this instead:

`msiexec /passive /quiet /a 'downloads\python-2.7.8.msi' TARGETDIR="C:\Users\anthoff\.julia\v0.3\IJulia\deps\usr\python 27"`

But not really sure because I don't actually know what these single ' actually mean/do.

@Keno
Copy link
Member

Keno commented Aug 7, 2014

The single quotes don't get passed to the operating system, they just denote which arguments are a unit.

@Keno
Copy link
Member

Keno commented Aug 7, 2014

Alright, next attempt. Try this:

run(`msiexec /passive /quiet /a $pythonmsifilename TARGETDIR=$(split(string('"',pyinstalldir,'"'),' ')`)

@vtjnash
Copy link
Member

vtjnash commented Aug 7, 2014

The single quotes don't get passed to the operating system, they just denote which arguments are a unit.

except that this is windows, so they do get passed out to the operating system (although they will turn into double quotes " and other quotes will be properly escaped)

@davidanthoff
Copy link
Contributor Author

@Keno, tried that (had to add another )), didn't work.

Given what @vtjnash just wrote, we need to get rid of the surrounding quotes around argument units for this to work.

@Keno
Copy link
Member

Keno commented Aug 7, 2014

@davidanthoff I made a mistake in that invocation, hold on

@vtjnash
Copy link
Member

vtjnash commented Aug 7, 2014

libuv quotes the arguments under the assumption that the recipient program will use the correct windows API function to unquote them. perhaps the authors of msiexec are unfamiliar with windows? anyways, there's a flag we should expose in the libuv api that lets you turn this off

@Keno
Copy link
Member

Keno commented Aug 7, 2014

@davidanthoff
Alright, last one before we should look at that flag:

args = split(string('"',pyinstalldir,'"'),' ')
run(`msiexec /passive /quiet /a $pythonmsifilename TARGETDIR=$(args[1]) $(args[2:end])`)

@davidanthoff
Copy link
Contributor Author

@tkelman This is 32 bit Python, and that should work with both 32 and 64 bit Julia.

@davidanthoff
Copy link
Contributor Author

@tkelman I tried the download_cmd command from BinDeps, but I can't get it to work. I'm using

download_cmd("https://sourceforge.net/projects/minimalportablepython/files/python-3.4.2.zip", "python.zip")

But that doesn't download anything. Any pointers what I should be doing instead?

@tkelman
Copy link
Contributor

tkelman commented Feb 14, 2015

BinDeps.download_cmd should return a command object - most likely curl -o python.zip -L https://sourceforge.net/projects/minimalportablepython/files/python-3.4.2.zip with the current version of BinDeps, so you'll need to call run(BinDeps.download_cmd(...)) on it.

@tkelman
Copy link
Contributor

tkelman commented Feb 14, 2015

It definitely looks like the download from Base, which uses URLDownloadToFileW on Windows, just isn't following redirects. Might be worth raising an issue with base Julia to figure out how to make it do that.

@davidanthoff
Copy link
Contributor Author

Alright, it now downloads the python binaries from sourceforge, thanks for the help! I think this is ready for testing by others.

else
eprintln("Found IPython version $ipyvers ... ok.")
end
if @windows? true : false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stylistically I think @windows_only begin would look better here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@tkelman
Copy link
Contributor

tkelman commented Feb 15, 2015

It looks like this works on 32 bit Julia, I'll try 64 bit later. Whatever is happening between Successfully installed pip-6.0.8 setuptools-12.1 and the first [ProfileCreate] really badly needs some feedback to the user that it's doing something though, it takes a while and looks completely dead. Is -qqq the absolute quietest setting for pip? Can we change that to something that isn't completely silent?

And this probably doesn't need to download, extract, and set up python again if you call Pkg.build("IJulia") a second time and the desired version has already been downloaded, extracted, and set up properly... Would be good if there were an early success detection, "looks like things are already set up and working here" with perhaps a tag file or something that you could manually delete if you needed to force a complete reinstall.

pythonzipfilename = normpath(downloadsdir, "python-3.4.2.zip")
getpipfilename = normpath(downloadsdir,"get-pip.py")
run(download_cmd("https://sourceforge.net/projects/minimalportablepython/files/python-3.4.2.zip", "$pythonzipfilename"))
download("https://bootstrap.pypa.io/get-pip.py", "$getpipfilename")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need this separately? don't the latest versions of python have ensurepip included?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I now use ensurepip

@davidanthoff
Copy link
Contributor Author

Is -qqq the absolute quietest setting for pip? Can we change that to something that isn't completely silent?

Done

And this probably doesn't need to download, extract, and set up python again if you call Pkg.build("IJulia") a second time and the desired version has already been downloaded, extracted, and set up properly... Would be good if there were an early success detection, "looks like things are already set up and working here" with perhaps a tag file or something that you could manually delete if you needed to force a complete reinstall.

I added support for this. Essentially if a previous build.jl call was successful, then the next call will only do upgrade commands for pip and ipython for the existing private python installation. I still recreate the private ipython profile that lives in usr/.ijulia on every build.jl call. This is mainly to avoid another update mechanism we might need, namely one where the profile is updated.

using BinDeps

if ispath(downloadsdir)
run(`cmd /C RD "$downloadsdir" /S /Q`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could these be rm(downloadsdir, recursive=true) instead of cmd /C RD ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ha, I didn't find the recursive option when I originally looked. Now changed.

@tkelman
Copy link
Contributor

tkelman commented Feb 16, 2015

This is looking really good. I'll make a point to test this in combination with Pkg.checkout("BinDeps") since I just merged that PR that makes the default download_cmd use powershell on Windows for better results when behind proxies.

eprintln("Found IPython version $ipyvers ... ok.")
end
@windows_only begin
ipython_version_to_install = "2.4.1"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question is still whether we should hardcode the IPython version that is installed or just always get the latest. Right now it is hard coded. I'm worried that if we just get the latest version instead things might break. E.g., as soon as 3.0 is released, it would be used, and who knows whether we might need to adjust things. So I would vote to keep things as they are with a hardcoded version, but would like to hear other input as well. It does mean we would have to keep this line updated.

@tkelman
Copy link
Contributor

tkelman commented Feb 16, 2015

Markupsafe and tornado are invoking MSVC when they install on my machine. I hope they would continue to function correctly when MSVC is not installed...

@davidanthoff
Copy link
Contributor Author

I've tested this on a fresh Windows XP install throughout that certainly has no MSVC installed, and things seem to work. For tornado I know that it will try to compile things for speed, but if that fails for any reason it will still work. I assume (hope) Markupsafe follows a similar strategy.

@tkelman
Copy link
Contributor

tkelman commented Feb 17, 2015

Cool, I guess if the performance would suffer noticeably from not having the C extensions compiled, we can bug (or ask nicely, whatever) the developers of those packages to put up compiled binary wheels and that would resolve the performance difference for users who don't have MSVC installed?

@davidanthoff
Copy link
Contributor Author

Yes, I think that would work. I tried to test on XP again yesterday, but the julia ZMQ package failed to build. I'm traveling this week, but will try again on Fri.

@tkelman
Copy link
Contributor

tkelman commented Feb 17, 2015

the julia ZMQ package failed to build

Presumably that's the same underlying issue as JuliaInterop/ZMQ.jl#69 and should go away after a restart of Julia. I wish people would write down error messages...

@tkelman
Copy link
Contributor

tkelman commented Feb 20, 2015

I think all this might need is a rebase and an eventual squash? The ZMQ install issue is fixed on nightlies, no restarting Julia necessary there.

@davidanthoff
Copy link
Contributor Author

There was a lot of change to build.jl on master this week so I very much doubt I can just merge master right now, will probably require a bit of work to sort that out.

@stevengj
Copy link
Member

Closing in this since the Conda alternative has been merged.

@stevengj stevengj closed this Sep 24, 2015
@davidanthoff
Copy link
Contributor Author

Excellent news!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants