Out of process profiling

[msaroufim]

Investigate running scalene in a seperate process if some environment variable is set https://github.com/plasma-umass/scalene

[msaroufim]

This won't quite work in out of process way. Scalene is a solid profile but works only over the python part of the codebase. so needs to be called like scalene main.py and considering that torchserve is often started with torchserve --start (not a python script) we can't do things out of process.

So the second attempt was to add @profile decorators in scalene but they require an extra --on and --off argument which aren't documented anywhere

And the third attempt was adding scalene.start_profile() manually to code which doesn't play well with torchserve since a new process is spawned to run a handler and was making the metric collector lose track of the PID it needs to work on. I'm not sure why this happens but here are the logs and here's a SO answer that suggests answer has to do with running shell=True when running a subprocess and there's no main function which would be a natural place to start instrumentation.

It's still a cool profiler in particular it shows number of copies, memory peak and time spent in native vs python code but I'm not sure it's the right solution for us. @yqhu as FYI perhaps omni-profiler is a better option here. @yqhu recommends py-spy which needs a process ID so could probably scan it with a grep on ps aux and use it.

Starting to think that htop is already not bad at finding major macro issues

EDIT: So another option is to not profile the torchserve part of the codebase and only profile the handler. So assuming someone creates their own handler.py - they could also something like

if __name__ =="__main__":
  handler = CustomHandler()
  handler.initialize()
  x = torch.randn()
  x = handler.preprocess(x)
  x = handle.inference(x)
  x = handler.postprocess(x)
  print(x)

And in this case you can now just simply run pyspy handler.py or scalene handler.py. Nowhere in our docs do we describe such a workflow and I think we really should. A tutorial on how here's to profile your handler.