Benchmark data

[paulpierre]

Any plans on providing benchmarks w/ the top OSS models like Mistral 7b using this as well as benchmarks against fine-tuned models.

[rizerphe]

Thanks! I don't have any plans currently, as it's not something I can think of a way to benchmark this. Fine-tuning, prompt engineering and quantization all affect the results drastically, and this project only constrains the generation, not affecting the overall quality. I'm not aware of any function calling benchmarks either. If you know of any, I'd love to hear about it.