Add --gzip option to cli

[methane]

No description provided.

[methane]

@jonashaag How do you think this approach?

[jonashaag]

I'd like to have this available from the python API as well, not only CLI -- you'll have to implement it in the reader anyways (or in the uploaded and shower but then you might as well implement it in the reader)

[methane]

This pull request demonstrated that starting subprocess in Python doesn't have
penalty while profiling.
It's cleaner and more flexible (you can use lz4, lzop, bzip2, snappy, brotli, and any other compression command).

I'd like to have this available from the python API as well, not only CLI.

You can add utility context manager which manage subprocess.

you'll have to implement it in the reader anyways.

done.

$ python3 -m vmprof -o file.prof example.py
$ bzip2 file.prof
$ ls file.*
file.prof.bz2
$ bunzip2 -c file.prof.bz2 | vmprofshow -
100.0%  <module>  100.0%  example.py:2
 91.4% .. main  91.4%  example.py:10
  6.7% .... test_1  7.3%  example.py:2
  6.7% ...... <listcomp>  100.0%  example.py:3
 67.4% .... test_2  73.8%  example.py:6
 67.4% ...... <listcomp>  100.0%  example.py:7

[jonashaag]

I believe you have to use

try:
    yield
finally:
    disable()
    ...

to account for the case that an exception is thrown in the profiled function.

[jonashaag]

done.

That's not really an integration into vmprof. Of course you can always use Unix tools here, but it would be nice to integrate it directly into vmprof.

You can add utility context manager which manage subprocess.

That's no good abstraction. The decorator has nothing to do with gzipping, i.e. both can live without the other. Why not simply use gzip all the time, and make it an implementation detail of the profile format? This way it should live in the enable method or in the C code with a call to subprocess.

[methane]

Oh, I didn't thinking about tightly integrate compression.
There are no /usr/bin/gzip in Windows.

@fijal How do you think?

[jonashaag]

You can always use Python for implementing a gzip "binary", that's the approach I used (but with C)

[fijal]

I would prefer if the file was compressed in a way that is slightly more transparent (e.g. if you want to upload it, you need to ungzip it, then upload it, there is no support for reading). That said, this is a bike color - if you say "that's too much work" I'm happy to merge it as it is

[methane]

@fijal To make it transparent, which do you prefer?

• By filename extension.
  (pros: clear, easy to use another compression format. cons: filename is required for reading data).
• By sniffing starting bytes. (When starting with 00 03 00, it's plain profile data, when starting with 1f 8b 08, it's gzip.)
  (pros: completely transparent, backward compatible, cons: not generic, hard to support another compression format).

[jonashaag]

Maybe we should move this discussion to #88

[fijal]

By sniffing the bytes. Generally speaking I want the reader to transparently recognize the format somehow. Working on windows would be a nice bonus (but not a blocker)

[jonashaag]

@methane byte sniffing is already implemented in #71, so the path forward is merge #88, #71 and replace the C gzip implementation with a Python one

[fijal]

Python is only required where there is no /usr/bin/gzip

[fijal]

If I can ask for one thing, can you please come up with this in one pull request if possible? If not, I'll try to look them bit by bit, but e.g. #71 is closed, so I did not even consider it right now. I'm trying to undig myself from a gigantic pile of other work, so anything that simplifies reviewing would make the merge a lot faster, sorry for being so difficult :-/

[jonashaag]

If I can ask for one thing, can you please come up with this in one pull request if possible?

Absolutely, that's the plan!