[Spike] Migrate API Server Source to the Open Telemetry lib

[skonto]

Problem
As discussed here and as a sub-task of #3126 we should start exploring migrating to the new lib.

Persona:
Affects everyone consuming Knative metrics (ops etc)

Exit Criteria
Check if migration is viable/stable/performant and evaluate the new metrics model to drive decision making for next steps and full migration to the new stack.

[lionelvillard]

@kthakar1990 FYI

[skonto]

Apiserver source is recording metrics in three separate places one is memstats as usual eg.

# TYPE apiserversource_go_alloc gauge
apiserversource_go_alloc{name=""} 8.02932e+06

the other is the source specific metrics here eg.

# TYPE apiserversource_event_count counter
apiserversource_event_count{event_source="https://172.25.0.1:443",event_type="dev.knative.apiserver.resource.add",name="unknown",namespace_name="unknown",resource_group="unknown",response_code="202",response_code_class="2xx",response_error="",response_timeout=""} 730

and there are metrics that come from the controller reporter eg.

# TYPE apiserversource_client_latency histogram
apiserversource_client_latency_bucket{name="",le="1e-05"} 0

The plan for this spike is roughly to develop/migrate in parallel the code in knative/pkg that is called above following the new otel model as described here.
In the new model there is support for asynchronous calls (eg. Observer) and there are also plugins that can be used in this project. For example the memstats reporting can be replaced by this plugin, there is a trivial example here.

Regarding the architecture supported, we can try/test two different approaches:
a) use this example as a guide to directly expose metrics for Prometheus to scrape (pull)
b) emit metrics directly to a collector that can be scraped from Prometheus (push).

Goal is to compare with existing code and have enough data to evaluate performance.
For example otel introduces a resource (we already have our own definition for that in current implementation) and it is important to know to what extend we will face issues like here.

I will work on a PR.
/cc @aliok @vaikas @evankanderson @matzew

[kthakar1990]

@skonto we have a requirement for couple of metrics for our source and I had presented about that in a community workgroup meeting today.

They asked me to connect with you to see if I can contribute in this work with you all and get our requirements added along with the same PR that you are working on. This is the issue for your reference.

Please let me know what's the best way to collaborate and have a quick discussion on this. Thank you :)

[skonto]

I did a quick PoC on the new lib for the metrics stuff and tried to emulate the memstats we have in knative/pkg, here is a sample of the latter. Here are the issues I spotted. I didnt want to jump in directly of porting stuff in knative.dev before we know what is working and what is not, what is wip etc.
I will continue with adding tests to check the oteltest lib and other stuff. Then we can migrate to complete this task if we see fit to do it.
/cc @aliok @bharattkukreja @kthakar1990 @evankanderson @vaikas

[bharattkukreja]

@skonto can you share the branch with your changes? (I want to take up other areas for API Server Source migration). I am a little concerned about the issues because otel is supposed to be backwards compatible.

[skonto]

@bharattkukreja the branch is https://github.com/skonto/pkg/tree/otel, this is where we agreed to work on the api server source migration. The otel is backwards compatible at the shim layer wg. for tracing but not 100% check here. In general backwards compatibility for both open-tracing and opencensus is wip. I am sure things will improve in the future but in general we can skip at some point relying on the old opencensus lib.

[lberk]

@skonto looking at the date we first triaged the issue -- moving it back to the backlog for now -- feel free to move it once you get some spare cycles to work on it! Thanks!

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.