Choosing from a list of possible notebook kernels is confusing compared to Jupyter

[greazer]

In Jupyter classic and lab, the list of kernels presented to run within represent those runtimes that are registered with kernelspec.json files as well as one special/default "Python3 (ipykernel)". When a user chooses this last one, the kernel used represents whatever the active Python environment was when Jupyter was instantiated.

In VS Code, most users don't start it from an active Python environment. Therefore, the above behavior isn't possible. Instead we currently show the list of custom kernels registered by the user (ala kernelspec.json files), in addition to every other Python environment the Python extension can find on the system.

This can be highly confusing to a user who may have been working with Jupyter classic or Lab using the default kernel. When they open it in VS Code and have to choose a kernel they'll be presented with a long list of possibilities. This can be startling and confusing, even IF it makes technical sense.

[rchiodo]

Potential solution talked about:

• Remove all the interpreters for python
• Have a 'Current Python Environment' kernel (which maps to the default kernelspec for jupyter)
  • This would use the active python interpreter
• Also show the other custom kernelspecs that the user has.
• Potentially have a command to generate a kernelspec from another environment

[greazer]

Part of the investigation for this was finding out how many people are using the Python extension alongside Jupyter when viewing or editing ipynb files containing Python cells. The result was overwhelming 99.7% have both installed. This is both because we recommend it but also because it makes logical sense.

So based on that, here's a proposal.

• Change our recommendation to a requirement that the Python extension be installed when an ipynb is opened that contains Python cells and the user attempts to run one or select a kernel.
• The first entry in the kernel selection dropdown is "Python Extension Interpreter"
• The rest of the entries are those discovered by us (kernelspec.json entries)

This mechanism gives a strong hint to the user as to the need to set the Python Extension's default interpreter. I believe that if no interpreter has been set, then the Python extension will prompt the user to do so. Not sure if that would require work on our end. Otherwise the list of kernels will appear to be identical to what they see in any other notebook implementation.

We would register the currently selected interpreter as a registered kernel if and when the user saves the ipynb.

Bonus: If we can detect that the user started VS Code with an activated conda or venv environment, list that as another option to select. Something like "VS Code's Python Environment". Would be great if we could say what the name of that environment is.

Another option is to not make the Python extension a requirement. If we do that, then we still would strongly recommend they install it in the usual locations, but if they choose not to, they'll never see "Python Extension Interpreter" in the kernel selection. They would only see the registered kernels and/or the environment VS Code was started in (If we go the Bonus route)

[rchiodo]

I think we need the telemetry first before we can remove all the custom kernels that show up.

I have an alternate proposal:

• Indicate which interpreters are 'ready' for running notebooks. (Ready means ipykernel is already there)
• Sort these ready interpreters at the top
• Add one that is just 'Current Environment'

[greazer]

What are the custom kernels? what kernels are you referring to? The interpreters that the Python extension finds?

[rchiodo]

Yes. All interpreters are allowed at the moment. Your proposal forces a user to set the 'active' interpreter in Python first before picking the 'Python Extension Interpreter'

[greazer]

I disagree we need know whether users need to see all the python interpreters in our list. If they want to choose a different one for another notebook, they would select a new Python interpreter, then run a cell in the new notebook. We would let Python continue to handle all interpreter operations.

[rchiodo]

Your proposal also makes it impossible to have two notebooks open using different kernels. The telemetry in #7376 is to check if anybody needs this though.

[greazer]

Your proposal also makes it impossible to have two notebooks open using different kernels. The telemetry in #7376 is to check if anybody needs this though.

How is that? I open one ipynb. I start it using the default python interpreter. I open a second ipynb, then choose a different default Python interpreter, then run a cell in it.

[rchiodo]

So changing the active interpreter kills the kernel? Seems off to me.

[greazer]

I'm not following. Why would changing the active interpreter that was used as a kernel in notebook A, need to be killed when starting notebook B after changing the default Python interpreter?

[rchiodo]

You're also going to have to teach people to do that - maybe. Will they understand the active interpreter is the one running in the kernel? What happens when I change the active interpreter? Does it

• Kill all kenrels? Seems bad
• Switch only those notebooks which haven't run yet? How do people know which interpreter is being used?

[rchiodo]

I feel like there'd still be confusion as to what interpreter is being used in a notebook. It's not the active interpreter, that's only what will be used if you actually hit run. However intellisense (at the moment) is using the active one (well if you haven't run yet) or the one that the kernel was started with if you hit run and then switched.

[rchiodo]

There's a disconnect between what's happening in the notebook and what is shown as the 'active' interpreter. I feel like we'd need some way to show this.

Maybe we could update the kernel name as soon as it runs. Not sure if that would work on next open though.

[greazer]

Yeah I hear you. However, the current model isn't very easy to understand either. A couple thoughts. We could register the current interpreter chosen to start the notebook as a new kernel. I think we've done that in the past, but in this case, it would be intended to be seen by the customer (unlike what I think we had before). Second, yeah I think we should indicate the situation in the kernel specification at the top right. Not sure how, but it doesn't seem like a real hard problem to figure out.

Additionally, I thought that another selection in the kernel picker could be "Select default Python Interpreter" which will basically just invoke the Python extension's command.

To be clear, I'm also assuming that if a notebook has focus, the status bar will not show the active interpreter. That should be the case now, but I've seen it show up there at times even when I only have a notebook focused.

The point here is that people who use other notebook products, like Jupyter, essentially select their interpreter up front, by starting the appropriate conda or venv environment shell and running from there. This proposal follows that basic idea. I mean it's not like Jupyter peruses the user's machine to find all the other conda environments that they might want to install jupyter into as part of their kernel picker.

[rchiodo]

However, the current model isn't very easy to understand either

Disagree here. The current model is VERY clear about what interpreter you're starting for your kernel. What's not clear is why it doesn't work or why I need to pick it in the first place.

To be clear, I'm also assuming that if a notebook has focus, the status bar will not show the active interpreter. That should be the case now, but I've seen it show up there at times even when I only have a notebook focused

This sounds even worse? How does the user know which interpreter they're using then? I believe you specified this:

Second, yeah I think we should indicate the situation in the kernel specification at the top right. Not sure how, but it doesn't seem like a real hard problem to figure out.

AFAIK, this is harder than you think. There's problems with changing notebook controllers after they've been given to VS code. We had started down this path before.

Additionally, I thought that another selection in the kernel picker could be "Select default Python Interpreter" which will basically just invoke the Python extension's command.

This sounds even worse than having the picker at the bottom. How do I know what effect this is going to have on my notebook?

I think we need a spec for this idea. I don't think the flow of it works. Or maybe a GIF like Miguel does.

[greazer]

• The current model is not easy to understand. At least for me, and likely thousands of others.
• Right, we'd need to indicate to the user what interpreter their using if it's not a registered kernel, somewhere. In one predictable place.
• If we need to propose a change to the API to enable an experience that customers can readily comprehend and like, then so be it.
• Why is being able to select an interpreter from the kernel picker any different than what we have today? Other than being one level of indirection?
• Sorry, but having a different interpreter shown in the status bar than what is instantiated in a focused, running notebook is supremely confusing. Can't this easily happen if the user opens a notebook that has a valid kernel set for it?

[rchiodo]

Can't this easily happen if the user opens a notebook that has a valid kernel set for it?

All depends upon what the UI we land on looks like. Not clear in my mind what the proposal actually looks like or how it flows through the different conditions.

[greazer]

Can't this easily happen if the user opens a notebook that has a valid kernel set for it?

All depends upon what the UI we land on looks like. Not clear in my mind what the proposal actually looks like or how it flows through the different conditions.

Yes, this can happen today IFF:

1. A bug causes the Python interpreter to appear in the status bar when a notebook has focus. As I mentioned I swear I've seen this, but I cannot repro it at the moment.
2. We decide to keep showing the selected Python interpreter in the status bar even when a notebook is focused and running.

Of course other behaviors are possible, but the above two are closest to what we have today or have had in the past.

[dynamicwebpaige]

I agree with @rchiodo that it would make sense to work with @misolori and the UX team to figure out the best flow for this using VS Code. We should also check in PyCharm, Jupyter Notebooks, and JupyterLab to observe if there are any existing patterns that we could leverage.

As an example, the process of changing a kernel in a Jupyter Notebook lists the default Python version first, and then user-defined conda and virtual environments below:

A bug causes the Python interpreter to appear in the status bar when a notebook has focus. As I mentioned I swear I've seen this, but I cannot repro it at the moment.

Same! I believe I've seen this behavior, too, @greazer :

[greazer]

As an example, the process of changing a kernel in a Jupyter Notebook lists the default Python version first, and then user-defined conda and virtual environments below:

To be clear, the list of kernels that are shown in the Jupyter image above only reflect those kernels that are found to be registered in a specific location on disk. Example for Windows its: %appdata%\jupyter\kernels. It does not reflect all environments the user may have defined in their Python installation(s). To me, this is where the confusion comes into play.

[DonJayamanne]

Suggestions

• Fix sorting for kernels
  • 1. List interpreters first & then kernels as follows
       • First item is always the preferred item, if any
       • Conda <env name> (<todays name without everyting in brackets>)
       • Conda <env name> (<todays name>)
       • Conda: <env name> (<todays name>)
       • Venv: <env name> (<todays name>)
       • Pyenv: <env name> (<todays name>)
       • Python <version>
       • Python <version>
       • Conda <env name> (<kernel name>)
       • Jupyter Kernel: C# 5.5
       • Jupyter Kernel: Java
  • 2. Command to select favorite kernels per workspace
    • Dropdown quickpick with all kernels and checkboxes
    • Provide VSCode with the selected kernels
    • Store excluded items per workspace
  • Later
    • Ability to tag a kernel as favorite (so that favorites are on top)
    • Ability to remove kernels from the list (via kernel management UI)
    • Multi step selection (Conda -> conda items) into core
  • Problems:
    • If naming convention is different from Pythons interpreter, then user will get confused
      • Ans: N/A
• Use new python API for kernel
  • After the above & python API is stable.
  • But after pip
• Side panel to display interpreters with packages (phase 2)
  • List all interpreters as a tree view and packages under them
  • Don - built this as a seprate extension as well as prototype in jupyter extension (works for remote kernels as well)
  • Benefits:
    • User knows what interpreter has their packages (pandas, scikitlearn, tensorflow, etc)
• Exports
  • Export to notebook not use a kernel
  • Export to python script not use a kernel

[IanMatthewHuff]

A new suggestion to add (though the Controller API would need to change).

Would be nice to put somethings just totally off the list like in a "More Kernels" item. Something like interpreters not active workspace interpreter or global kernels that have never been picked.

• Kernel A
• Kernel B
• Interpreter A
• More Kernels

More Kernels would have other items. Anything that was picked from "More Kernels" once would start to show up in the top list then.

[DonJayamanne]

Sample

Grouped (by prefixing each with env type)

Managing favorites (remove items from the list)

[DonJayamanne]

Discussion points:

• Active interpreter will always appear in the list of kernels for Interactive window
  • Assume you have 5 interpreters, you hide all 5, we will always display the active interpreter in interactive window
  • If user changes active interpreetr and creates a new interactive window, then we display that acitve interpreter (& hide the rest, try to honor the users choice to hide)
• Preferred kernels will not work properly
  • Assume you have a notebook with kernel information pointing to conda A
  • If conda A is hidden in your workspace, then that will not appear, hence user will be prompted to select, even though that same notebook will work in another folder.
  • We know users open notebooks without opening folders today (& this could break that scenario)
  • Suggestion: If we find a 100% match, then we should unhide those kernels
    • This is an edge case, but if we want this favorites feature to be widely used, then users are more likely to run into this issue.
• Users could be prompted to select kernels
  • If a user hides a kernel that was previously used, now opening previous notebooks will result in user being prompted to select a kernel as the previously used kernel is no longer registered with VSCode.

Suggestion:

• I think we should roll out the favorites feature via some experiment (opt in or the like), why?
  • So users know its in flux and design can change
  • We know how user will use this (multiroot worksapces, notebooks without worksapces, etc)
  • Else we run into issues with who moved my cheese and difficultly to change it once shipped and announced

[DonJayamanne]

Suggestions:

• Sort by last used
• no need of favorites, sorting
• Store list of exclusions in settings instead of memento (so users can see what has been excluded)

[DonJayamanne]

Problems

• We will not be able to control sort order for first time users (in the future)
  • Currently controllers are listed in the manner they appear in the list
  • Except for preferred kernels
  • Today we get all kernels in one go and list them (hence can sort them),
    • However this is something we were supposed to fix a few months ago and has prevented users from running kernels (as we list kernels very slowly)
  • Hence relying on current technique of being able to solve might not always work
  • Or sorting returning users might not be the best either (if that's the case then we're better off sorting based on usage, not some algorithm we come up with)
• If a kernel is excluded and user opens a notebook (belonging to another folder) that uses a hidden kernel, we do not display that
  • Displaying that is too messy & complicated (just hide if user decided to do so)

[DonJayamanne]

Addressed by microsoft/vscode#135502

[DonJayamanne]

New picker

Filter

[davidanthoff]

Is there a chance that you could move the controller kind property from proposed API to released?

We have the same UI problem with Julia that motivated this issue here in the first place, i.e. a pretty unruly mess of kernels from the Julia extension and the Jupyter extension. Would be great if we could group things in the same way that the Jupyter/Python word can do.