Add prediction metrics tracking documentation

docs/deployments/apis.md

@@ -10,9 +10,9 @@ Serve models at scale.
   model: <string>  # path to an exported model (e.g. s3://my-bucket/exported_model)
   model_format: <string>  # model format, must be "tensorflow" or "onnx" (default: "onnx" if model path ends with .onnx, "tensorflow" if model path ends with .zip or is a directory)
   request_handler: <string>  # path to the request handler implementation file, relative to the cortex root
-  tf_signature_key: <string> # name of the signature def to use for prediction (required if your model has more than one signature def)
+  tf_signature_key: <string>  # name of the signature def to use for prediction (required if your model has more than one signature def)
   tracker:
-    key: <string>  # json key to track if the response payload is a dictionary
+    key: <string>  # key to track, only required if the response payload is a json object
     model_type: <string>  # model type, must be "classification" or "regression"
   compute:
     min_replicas: <int>  # minimum number of replicas (default: 1)
@@ -43,6 +43,10 @@ Request handlers are used to decouple the interface of an API endpoint from its
 
 See [request handlers](request-handlers.md) for a detailed guide.
 
+## Prediction Monitoring
+
+The `tracker` can be configured to collect API prediction metrics and display real time stats in `cortex get <api_name>`. The tracker looks for scalar values in the response payload (after the execution of `post_inference` request handler). If the response payload is a json object, the `key` can be set to extract the desired scalar value. For regression models, the tracker should be configured with `model_type: regression` to collect float values and display regreission stats such as min, max and avg. For classification models, the tracker should be configured with `model_type: classification` to collect integer or string values and display the class distribution.
+
 ## Debugging
 
 You can log more information about each request by adding a `?debug=true` parameter to your requests. This will print:

examples/iris-classifier/cortex.yaml

@@ -5,23 +5,33 @@
   name: tensorflow
   model: s3://cortex-examples/iris/tensorflow
   request_handler: handlers/tensorflow.py
+  tracker:
+    model_type: classification
 
 - kind: api
   name: pytorch
   model: s3://cortex-examples/iris/pytorch.onnx
   request_handler: handlers/pytorch.py
+  tracker:
+    model_type: classification
 
 - kind: api
   name: keras
   model: s3://cortex-examples/iris/keras.onnx
   request_handler: handlers/keras.py
+  tracker:
+    model_type: classification
 
 - kind: api
   name: xgboost
   model: s3://cortex-examples/iris/xgboost.onnx
   request_handler: handlers/xgboost.py
+  tracker:
+    model_type: classification
 
 - kind: api
   name: sklearn
   model: s3://cortex-examples/iris/sklearn.onnx
   request_handler: handlers/sklearn.py
+  tracker:
+    model_type: classification