[feature] Improve Logging in the KFP components

[DharmitD]

Feature Area

/area backend

What feature would you like to see?

We need to enable logging for KFP components (APIServer, Persistence Agent, Scheduled Workflow) through manifest parameters and inject through environment variables in all pods to enable debugging.
This would involve updating the main.go for each of these components, in order to have a log-level parameter that could either be set to info, error, debug, etc levels, and also updating respective components' dockerfiles.

What is the use case or pain point?

Is there a workaround currently?

---

Love this idea? Give it a 👍.

[chensun]

logging for KFP components (APIServer, Persistence Agent, Scheduled Workflow)

The cluster admin would have access to such logging, I don't think KFP users needs to debug through such information.

[rimolive]

logging for KFP components (APIServer, Persistence Agent, Scheduled Workflow)

The cluster admin would have access to such logging, I don't think KFP users needs to debug through such information.

I think this issue needs clarification. The idea is to give users a way to easily specify the logging level for KFP components if needed. Instructions on how to do that are in the code, so either clarify in the KFP documentation how to set the logging level or add an env var in each KFP component Deployment to do that. If we can do both, the better.

I got a scenario where API Server was failling on DB connection (I knew it was because I had all the console message sequence in my head), but I saw no more than 2 lines in console. No errors returned. And I realized that default logging level is warning.

[DharmitD]

@chensun following up on this thread, any thoughts / suggestions based on @rimolive 's explanation above?

Could we move this task to "ready" from "needs more info"?