new metric: Mean Directional Accuracy; new benchmark: Repeat Last Known Value

[ericleonardo]

Hi.. Please, can you add new metric Mean Directional Accuracy?
In financial prediction like Exchange rate (Forex), predicting the direction (up/down) is more important than MSE.
Because if the investor trade in opposite direction he will have loss. If trading in same direction as target value he will profit.

The MDA measures if the predicted value increased/decreased from its previous value following the same target movement.
If MDA is 0.5 (50%), it means random guessing up/down, the model is not predicting direction.
https://datasciencestunt.com/mean-directional-accuracy-of-time-series-forecast/
You can calculate by comparing Predicted delta to Target delta sign, or just multiply both deltas and see if result is > 0 (same signal) or not. Return 1 if both have same signal, or 0 if divergent signal.

Another suggestion:
Add new basic benchmark comparison: Repeat Last Known Value (like if it was the predicted value).
If the current model MSE, MAE, MDA, etc is not better than just Repeating Last Known Value, then model is not useful.
Ex: in Stock or Forex forecasting, many advanced models are just following the last known value, not predicting.
If we repeat last known value, maybe the MSE, MAE are better than the MSE, MAE performed by the model.

Thank you!

[qiu69]

Thank you for your feedback. We are currently busy optimizing our code and do not have time to add your requirement. This requirement has been added to our plan but has a low priority. We will approve it if you can submit the code.