[VC-35738] Bubble up the errors from sub-commands and handle errors centrally instead of using log.Fatal at multiple sites

[wallrj]

This is a small part of a larger effort to allow the agent to use structured logging.
This will make it easier for platform administrators to parse the logs and to distinguish errors from info messages.
It also is a small part of an effort to use contextual logging instead of the legacy log module.
This will make it easier to test the log output of our code in unit tests.

The existing code uses log.Fatal at the site of various errors to both print the error and to exit the process with code 1.
Instead the errors should be returned and bubbled back to the root command, which makes it easier to handle the errors in a consistent fashion and makes it easier to test the code that had previously caused the process to exit.

$ git grep -i '\.fatal' | grep -v 't\.Fatal'

Example output:

$ POD_NAMESPACE=venafi ./preflight agent --one-shot --output-path /dev/null --api-token should-not-be-required  --agent-config-file examples/cert-manager-agent.yaml -v 1
I1105 13:28:59.806101   86250 logs.go:158] "Preflight agent version: development ()" source="agent"
I1105 13:28:59.806320   86250 logs.go:158] "Using the Jetstack Secure API Token auth mode since --api-token was specified." source="agent"
I1105 13:28:59.806344   86250 logs.go:158] "Using deprecated Endpoint configuration. User Server instead." source="agent"
I1105 13:28:59.806362   86250 run.go:117] "Healthz endpoints enabled" logger="APIServer" addr=":8081" path="/healthz"
I1105 13:28:59.806389   86250 run.go:121] "Readyz endpoints enabled" logger="APIServer" addr=":8081" path="/readyz"
I1105 13:28:59.806464   86250 run.go:447] "Starting" logger="APIServer.ListenAndServe" addr=":8081"
E1105 13:28:59.807120   86250 logs.go:156] "error messages will not show in the pod's events because the POD_NAME environment variable is empty" source="agent"
I1105 13:28:59.807717   86250 logs.go:158] "starting \"k8s/secrets.v1\" datagatherer" source="agent"
I1105 13:28:59.807854   86250 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1105 13:28:59.807888   86250 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1105 13:29:00.008575   86250 logs.go:158] "successfully gathered 18 items from \"k8s/secrets.v1\" datagatherer" source="agent"
I1105 13:29:00.009067   86250 logs.go:158] "Data saved to local file: /dev/null" source="agent"
I1105 13:29:00.009147   86250 run.go:461] "Shutting down" logger="APIServer.ListenAndServe" addr=":8081"
I1105 13:29:00.009205   86250 run.go:476] "Shutdown complete" logger="APIServer.ListenAndServe" addr=":8081"

$ echo $?
0

$ ./preflight agent --foo
E1105 13:30:18.198379   86557 root.go:51] "Exiting due to error" err="unknown flag: --foo" exit-code=1

$ echo $?
1

$ ./preflight agent
I1105 13:30:52.936075   86581 logs.go:153] "Preflight agent version: development ()" source="agent"
E1105 13:30:52.936271   86581 root.go:51] "Exiting due to error" err=<
        While evaluating configuration: no auth mode specified. You can use one of four auth modes:
         - Use (--venafi-cloud with --credentials-file) or (--client-id with --private-key-path) to use the Venafi Cloud Key Pair Service Account mode.
         - Use --venafi-connection for the Venafi Cloud VenafiConnection mode.
         - Use --credentials-file alone if you want to use the Jetstack Secure OAuth mode.
         - Use --api-token if you want to use the Jetstack Secure API Token mode.
 > exit-code=1

$ echo $?
1

$ ./preflight agent --one-shot --output-path /foo/bar --api-token should-not-be-required  --agent-config-file examples/cert-manager-agent.yaml
I1105 13:31:38.686969   86604 logs.go:153] "Preflight agent version: development ()" source="agent"
I1105 13:31:38.687764   86604 logs.go:153] "Using the Jetstack Secure API Token auth mode since --api-token was specified." source="agent"
I1105 13:31:38.687789   86604 logs.go:153] "Using deprecated Endpoint configuration. User Server instead." source="agent"
E1105 13:31:38.687815   86604 root.go:51] "Exiting due to error" err=<
        While evaluating configuration: 1 error occurred:
                * could not guess which namespace the agent is running in: POD_NAMESPACE env var not set, meaning that you are probably not running in cluster. Please use --install-namespace or POD_NAMESPACE to specify the namespace in which the agent is running.

 > exit-code=1

$ echo $?
1

[wallrj]

As suggested by @maelvls in #593 (comment)

[maelvls]

It seems all good to me. I haven't tested this manually, but I trust the output that you gave.

[wallrj]

This caused a bug.
The group error is discarded, because I was assigning to the err variable assuming that it was the returned error, but it's not.

[wallrj]

Fixed in #606

[maelvls]

For context, we have done the same thing in Firefly: https://gitlab.com/venafi/vaas/applications/tls-protect/dmi/cli/firefly-ca/-/merge_requests/426