I made memo, and just discovered bkt thanks to HN, they are similar

[aktau]

Hi there, I came here from https://news.ycombinator.com/item?id=45690588, where I posted my memo shell script as something I use a lot. I didn't know about bkt (and I think it may postdate memo), but it was interesting to see there were other implementations of the same idea. Just wanted to give you a heads up and a kudos for having a good idea. Sub-process caching is a clear productivity improvement (for me).

[dimo414]

Thanks for sharing @aktau! You might find https://github.com/dimo414/bash-cache interesting to look over as well, since you mention preferring a bash/shell implementation.

I appreciate the callout about compression, it's an interesting suggestion and if you'd be interested in seeing that implemented in bkt please feel free to open a FR. I'd want to figure out some benchmarks before deciding to enable compression by default, but certainly it could be added as a flag.

A quick clarification, bkt doesn't use a database today, though I've wondered how a sqlite backend would perform. Today it works roughly as you describe, writing the data directly to disk. However it is written in a serialized format, so you can't easily use third party tools against the cached data (this is a tradeoff, bash-cache makes the opposite tradeoff and writes stdout+stderr+exit code to separate files). You can compile in debug mode and have the data serialize to JSON which is more human/machine readable, but this is intended as a debugging aid, not core functionality.

[aktau]

Thanks, and hi Michael (I think we've talked before internally, I did not realize we had been acquainted).

You might find https://github.com/dimo414/bash-cache interesting to look over as well, since you mention preferring a bash/shell implementation.

That's indeed more like memo in operation. Cool.

it's an interesting suggestion and if you'd be interested in seeing that implemented in bkt please feel free to open a FR. I'd want to figure out some benchmarks before deciding to enable compression by default, but certainly it could be added as a flag.

As I've mentioned on HN I think, memo works just about perfectly for me, so for the moment I'm not thinking about migrating to anything.

However, just from personal experience: compressing with zstd/lz4 has been a huge win for me ~always (I think both compressors have a transparent mode too if they notice they can't compress the bitstream well). I sometimes memoize gigantic bits of output (gigabytes) and the storage/IO savings are definitely worth it. zstd/lz4 are so fast they are generally not the bottleneck. Especially so because usually we memoize an invocation due to it being slow. As known, zstd/lz4 decompressing is incredibly fast.

The idea of also saving stderr (and exit code) is certainly interesting. I did consider it but have never needed it. One thought I'd have is that the fidelity could be impacted by (in normal operation) stdout/stderr being interleaved in a different way.

As for exit code: I think I tried to make memo simply not memoize when the exit code is non-zero. For one this allows interrupting the command while avoiding the next invocation from "succeeding with nothing". But I know there are some gaps in the implementation here.