Use load_param_mem with ncnn CPU inference

[JeremyRand]

Upstream ncnn now has a Pybind for load_param_mem, so we can utilize that and make things a bit more efficient.

[joeyballentine]

There isn't also one for the bin? That's a shame

[JeremyRand]

@joeyballentine It's... complicated. When I tried to port the load_model_mem Pybind, it gave me a segfault sometime after the method returned. I checked with the ncnn dev, and it turns out that load_model_mem is intentionally zero-copy, which makes using it with Pybind a memory safety violation (Pybind will free the model string before ncnn tries to use it). Hence why upstream ncnn has no Pybind for it. I think the zero-copy behavior doesn't happen with Vulkan mode, which is probably why you haven't had issues with it.

There is an API that will do what we want for load_model, but I haven't tested it yet, whereas this PR is tested and working, so I figured I'd submit this now, and do the load_model changes as a follow-up PR.

[joeyballentine]

That makes sense. If you're in contact with NCNN devs, could you ask them if they could start officially supporting Vulkan in the python bindings? I'd love to no longer have to maintain my fork

[JeremyRand]

@joeyballentine Yes, that is actually already on my TODO list.

[joeyballentine]

Great. There's still other things my fork changes that afaik aren't included in the official bindings, such as throwing python errors rather than just logging errors. This was necessary for the safe OOM catching chaiNNer does. I think that's the last requirement besides Vulkan support that would need to be ported before we could just get rid of my fork entirely

[JeremyRand]

Yeah makes sense. I'm a big fan of killing off forks and upstreaming things whenever possible, so yeah, I intend to spend some effort on this. Not sure how long it'll take me, especially given my lack of familiarity with Vulkan.

[joeyballentine]

If it makes you feel any better, I don't know shit about how Vulkan works. I was just trying things out until it did what I needed it to do 😆

In terms of enabling Vulkan for python, it's just a compiler flag they have off. I hardcoded it to be enabled everywhere in my fork (which I heavily regret now due to merge conflicts every time I try to merge upstream's changes). AFAIK you just add an env var and it compiles with Vulkan enabled

[JeremyRand]

Oh good, if it's just a cmake flag change, then maybe I can deal with that at the same time that I'm getting POWER9 VSX support enabled by default in their Python package.

[JeremyRand]

(VSX by default is not going to happen until an upstream Clang bug gets fixed though -- right now Clang artificially slows down ncnn on POWER9 by around 15% due to a stupid bug in a header file that Clang ships with. So probably will be a few weeks at least before that gets unblocked.)