How well do NLU engines perform? CPU% vs correctness (per domain & whole data set)

[secretsauceai]

How do the NLU engines perform?

We are interested in measuring resource utilization vs correctness, scalability, performance per domain, and performance per language.

This might be a good data set for performing this analysis. The utterances seem close to reality, it contains enough variations, most of the labeling is pretty good, and it has roughly 25k utterances. Perhaps translate these into other languages as a rough analysis for other languages?

For each intent matching system, we want to find out:

• CPU% (base: raspi4 aarch64 4gb ram)
• correct intents matched
  • over each domain
  • for whole dataset (scalability test)
  • for other languages (this will be performed after English)

List of NLU engines

(This is a work in progress)

• diet classifier (Rasa)
• Snips
• random forest
• Padatious
• ?

Deliverables

• testing script to run data set on several NLU engines (can the existing scripts used from the dataset be modified, should the snips one be used or should a new one be written?)
• analysis comparing the performance of each intent matcher per domain and over the whole set

This will help people decide given the domains they require for intent matching, how scalable the engine should be, the device they want to run it on, and the language: which intent matcher will perform the best for them.