Quantifying approximation of a model and appropriateness for evaluation

[clegaspi]

This feature probably should be coupled with #214 in whatever way we choose to move forward with tagging/annotating quantities/models/symbols.

We need extreme vetting of our current models to determine their uncertainty and limit their use to the cases where it is appropriate.

There have been several suggestions on ways to add in this kind of information:

• Attach uncertainty values where possible. Possibly having a dynamic uncertainty depending on the properties of the material.
• Have a confidence rating system (e.g. 0-4) to qualitatively assess the validity of a quantity. Not exactly sure how we might determine this on the fly.
  • None: uncertainty of the model is known precisely
  • 0: unable to determine confidence
  • 1: poor confidence in accuracy
  • 2: reasonable confidence in accuracy
  • 3: high confidence in accuracy
  • 4: value is exact

[maxhutch]

Our first-pass approach (prior to having actual distributions everywhere) was to have each model define a measure of its fidelity like your "confidence" system, but continuously. The scale was entirely ordinal: we compared the values from models to decide which to apply next, but didn't actually care what the value of the fidelity was. As our system matured, this continued to be a useful heuristic for dealing with some combinatorial explosions in graph traversal complexity.

[clegaspi]

Let me see if I understand you correctly. You assign the model and any quantities a fidelity score, the closer to zero, the lower the uncertainty. If you applied a model to a set of inputs, the fidelity score of the output(s) would be some sum/combination of the inputs' fidelity scores and the model's fidelity score, increasing as you traversed the graph. Periodically (or at every step?) in the graph traversal, you would trim quantities whose scores were too high so they would stop propagating through the graph.

Something like this?

[maxhutch]

Something like that, yeah.