MON-1666: CMO deployment: pass enabled-remote-write

[jan--f]

in order to switch telemeter over to Prometheus remote write.

Signed-off-by: Jan Fajerski jfajersk@redhat.com

• I added CHANGELOG entry for this change.
• No user facing changes, so no entry in CHANGELOG was needed.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jan--f

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jan--f]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[arajkumar]

/retest

[jan--f]

/hold

[jan--f]

Generally this seems to work. However some wrinkles need to be ironed out.

• An origin test needs a change.
• The current telemeter endpoint rejects (at least most of) the remote write requests with status 413 : request too big. We might look at either remote write tuning or relax the endpoint restrictions a bit.

cc @simonpasquier @ianbillett

[bill3tt]

Let me repost our DM here for completeness...

The default value of the limit_bytes flag is 512k - in prod we set it to 5.1M.
IIRC we don't actually use the receive endpoint in telemeter at the moment - we use the upload endpoint which is a telemeter specific thing. See the /receive panel in this dashboard

[simonpasquier]

I've noticed from the logs that Prometheus sends metadata by default but I presume that we don't want this for telemeter. I believe that it should be turned off explicitly in the RemoteWrite spec.

[simonpasquier]

IIUC the /api/v1/receive endpoint has a size limit of 15k while the Prometheus could send up to 10,000 samples. That would explain the "request too big" errors.

[jan--f]

I've noticed from the logs that Prometheus sends metadata by default but I presume that we don't want this for telemeter. I believe that it should be turned off explicitly in the RemoteWrite spec.

Yeak makes sense. Tbh I'm not 100% sure yet what the impact is and whether telemeter can actually make use of this metadata. I'll investigate more, but until then lets turn it off.

[jan--f]

Going by the prometheus_remote_storage_bytes_total metric of prometheus, it seems like the requests should be around 70kB large (we have roughly 500kB over 7 requests and ~413kB over 6 requests).

Should we start with a 128kB request limit on the telemeter side?

[matej-g]

Should we start with a 128kB request limit on the telemeter side?

Nice investigation 👍 That limit sounds to be ample and reasonable.

[simonpasquier]

Looking at the number of sent samples, we're at about 2k samples per minute. Knowing that the remote write is configured with a maximum number of samples per send = 10k and a batch deadline of 1m, it means that in the CI runs, we never reach the 10k limit. "Real" environments might generate more samples (e.g. more OLM operators = more telemetry data) and we may hit the 10k samples per send limit, meaning larger requests. I think that we should account for it by increasing the request limit on the telemeter server side (even more than 128k) and/or reducing the number of samples per send.

[jan--f]

Agreed, the idea is to set a "reasonable" default in telemeter and deploy that in the staging environment. Then for production the limit will be explicitly set and likely a lot higher. This would be similar to how the upload endpoint is treated (512kB default and 5.1MB in production.

[jan--f]

/retest

[jan--f]

/retest

[jan--f]

/retest

[openshift-ci]

@jan--f: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-agnostic-operator	27fabbc	link	true	/test e2e-agnostic-operator
ci/prow/e2e-agnostic	27fabbc	link	true	/test e2e-agnostic
ci/prow/e2e-aws-single-node	27fabbc	link	false	/test e2e-aws-single-node

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[jan--f]

/close
Needs further research

[openshift-ci]

@jan--f: Closed this PR.

In response to this:

/close
Needs further research

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.