Move grammar support out of examples? Unify?

[josharian]

I'd like to add Go binding support for grammars (for #1697). It's a bit inconvenient now, because the grammar support is off in an examples directory, and adding stuff from examples to libwhisper.a feels wrong; ditto for the header file.

GBNF seems pretty well established at this point (it is in common in llama.cpp). It'd be nice to make it easier to support.

Could we promote grammar support to core whisper.h/whisper.cpp, similar to llama.cpp, where you simply provide a grammar string?

[ggerganov]

Yes, we should do that. I'm also thinking about moving all the grammar stuff into the ggml core library so that it becomes available everywhere. But the main problem is re-implementing the C++ stuff in C

[josharian]

the main problem is re-implementing the C++ stuff in C

Another option is to hide it all behind a very simple C facade, something like new/init, parse, free. (At least, that would suffice for Go bindings.)

Btw, I've noticed some crashes in the GBNF parser. I am planning to set up some fuzzing for it soon to try to shake them out.

[philipag]

I have a basic implementation at #2838
If someone can add this to the build it should work with minor boiler plate changes.