Make AnalysisTemplate more configurable

[nestorsokil]

Summary

There are a number of fields in AnalysisTemplate/ClusterAnalysisTemplate that are static (consecutiveErrorLimit, interval, count, etc). It would mean a lot of practical benefits to make them dynamically configurable.

Use Cases

Having a single AnalysisTemplate you could define different threasholds for different applications (since they may have different tolerance to failures). The preferred experience IMO would be something like

postPromotionAnalysis:
    templates:
    - templateName: error-rate
      failureLimit: 3
      interval: 5m
      count: 10

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[jessesuen]

Rather than add the values in the template reference, I think we should support these are arguments to the AnalysisTemplate. e.g.:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  args:
  - name: interval
  - name: count
  metrics:
  - name: error-rate
    provider:
      prometheus:
        interval: "{{args.interval}}"
        count: "{{args.count}}"

and just use normal arg substitution to handle this. We'll need to switch the type to use the intstr.IntOrString kubernetes type to support this. Currently I think they are all *int32

[zimmertr]

Rather than add the values in the template reference, I think we should support these are arguments to the AnalysisTemplate. e.g.:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  args:
  - name: interval
  - name: count
  metrics:
  - name: error-rate
    provider:
      prometheus:
        interval: "{{args.interval}}"
        count: "{{args.count}}"

and just use normal arg substitution to handle this. We'll need to switch the type to use the intstr.IntOrString kubernetes type to support this. Currently I think they are all *int32

Does this actually work?

---

With Quotes:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: ...
spec:
  args:
    - name: count
       value: 5
       ...
  metrics:
      ...
      count: "{{ args.count }}"

one or more objects failed to apply, reason: error when patching "/dev/shm/2225555538": AnalysisTemplate.argoproj.io "..." is invalid: spec.args[0].value: Invalid value: "integer": spec.args[0].value in body must be of type string: "integer".

---

Without Quotes:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: ...
spec:
  args:
    - name: count
       value: 5
       ...
  metrics:
      ...
      count: {{ args.count }}

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = kustomize build <path to cached source>/.../.../... --enable-helm --load-restrictor=LoadRestrictionsNone failed exit status 1: Error: map[string]interface {}(nil): yaml: invalid map key: map[string]interface {}{"args.count":""}