nbresuse reports total Jupyter server memory

[jkleint]

It looks like nbresuse is reporting the memory use for the entire server process, including all notebooks.

class MetricsHandler(IPythonHandler):
    def get(self):
        """
        Calculate and return current resource usage metrics
        """
        config = self.settings['nbresuse_display_config']
        cur_process = psutil.Process()
        all_processes = [cur_process] + cur_process.children(recursive=True)
        rss = sum([p.memory_info().rss for p in all_processes])

Is it possible to report memory use for each notebook individually? I don't know anything about jupyter architecture, but maybe there's a way to get a pointer to the calling notebook or PID?

[mathematicalmichael]

I mean, isn't the total memory usage what you really care about, since bumping up against your system's limits is what the concern is? I think one solution to get what you would like to see is to run dockerspawner through jupyterhub so that each notebook instance is in an isolated container. That way this extension should report the usage as a proportion of the total memory allocated to the container.

[jkleint]

I often run several notebooks at once, on a shared server (not jupyterhub), and the concern is being a good citizen by not overusing my fair share of memory. And if I am, I'd like to know not only which notebook is responsible, but which cell(s) caused a jump in usage. The more general concern is having an idea of the memory usage of my code. That's difficult if other notebooks are causing random usage fluctuations in the background.

[jkleint]

The existing code also doesn't account for memory shared between processes, and so over-reports usage. Accurately measuring memory is actually somewhat tricky, but fortunately there are good packages like psutil that can help.

[krinsman]

@jkleint This issue seems to have been split into two separate issues: multi-process sharing memory -- #16 and per-kernel metric reporting: #31. To avoid having duplicate open issues, I will provisionally close this issue.

If you feel that this issue encompasses something that is not discussed by #16 or #31 then please let me know, and we can create a new issue focusing on that sub-part.

As you noted, currently NBResuse by design only reports memory usage for the entire notebook server. We want to perfect that feature first before attempting to accurately report metrics for more advanced/less uncommon situations/setups. Any PR's fixing this would of course be welcomed however.

[jkleint]

Is per-kernel metrics the same thing as per-notebook? (Or close enough in general, anyway?) If so, I'm fine with tracking that in another issue.

I came upon nbresuse hoping for more granular information than what top can already tell me. When per-notebook metrics are available, I feel total server usage is not a very useful default, for these reasons:

1. Total process memory usage is already easily found in top or ps or any system monitoring utility, which is arguably the natural first "go to" place for memory information.
2. Internal Jupyter- and notebook-specific resource usage cannot be found with general tools, and Jupyter is the first and obvious place to report such info.
3. If your goal is to monitor bumping up against system limits:
   • This is dependent on the usage of other processes / users
   • Usage of Jupyter, other processes, and system limits are effectively reported by familiar tools
   • You would still want to know which notebook is hitting the limits and/or which notebook would give the biggest savings if shut down.
4. It is confusing and perhaps misleading for all notebooks to report the same memory usage: this is a giveaway that nbresuse is either wrong or multiple-counting, and decreases confidence in it. Total usage would be better reported in a general server location location like the File | Open page, rather than repeated in notebook-specific locations.
5. It is difficult to monitor the change in usage for individual cells or pieces of code.

Ideally users could configure the metrics they would like to see, but I think per-notebook metrics make the most sense as the default.

[jtpio]

@jkleint yes per notebook metrics would roughly correspond to per kernel metrics (although there can be multiple notebooks sharing the same kernel).

This is great feedback. Would you like to continue the conversation in #31? (and maybe link to you comment or copy the content over there)