OSD-8760: Improve bulk service log error handling

[supreeth7]

Resolves: https://issues.redhat.com/browse/OSD-8760
Relates to: https://issues.redhat.com/browse/OSD-7691

This PR improves error handling when sending a service log to multiple clusters, if a service log failed to post to a single cluster it doesn't halt the command execution and continues to post service log updates to the remaining clusters.

This PR also refactors the service log post sub-command output.

---

When run servicelog post sub-command to post service logs to multiple clusters, the command output will list the clusters the service log was successfully posted along with a list of clusters which encountered errors while sending the service log.

$ osdctl servicelog post ...

INFO[0008] Success: 2, Failed: 1
                       
INFO[0008] Successful clusters:                         
ID                   Status
<clusterID>          Message has been successfully sent to <clusterID>
<clusterID>          Message has been successfully sent to <clusterID>

INFO[0008] Failed clusters:                             
ID                   Status
<clusterID>          Message sent, but wrong severity information was passed (wanted "high", got "Error")

[fahlmant]

/label tide/merge-method-squash

[supreeth7]

It has been noticed that a service log is being sent to the same cluster multiple times, will work on a fix.

[fahlmant]

@supreeth7 what was the fix?

[supreeth7]

@fahlmant It was a wrong pointer reference, a one-liner :)

[fahlmant]

@supreeth7 Awesome. Do you have any tests against stage clusters I can look at?

[supreeth7]

@fahlmant I tested them on two stage clusters which I had created prior, here are the results:

./dist/osdctl_linux_amd64/osdctl servicelog post -t https://raw.githubusercontent.com/openshift/managed-notifications/master/osd/cluster_has_gone_missing.json -p "CLUSTER_UUID=shepherd%"
INFO[0002] The following clusters match the given parameters: 
Name                ID                                 State               Version             Cloud Provider      Region
shepherd-1          1oudlp5pmjparuqfasf16pq8jtbtnptq   ready               4.9.8               aws                 ap-south-1
shepherd-2          1outpcau2bum4mtmkgj1m0q51r4qdunr   ready               4.9.9               aws                 ap-south-1

INFO[0002] The following template will be sent:         
{
  "severity": "Error",
  "service_name": "SREManualAction",
  "cluster_uuid": "${CLUSTER_UUID}",
  "summary": "Action required: cluster not checking in",
  "description": "Your cluster requires you to take action because it is no longer checking in with Red Hat OpenShift Cluster Manager. Possible causes include stopping instances or a networking misconfiguration. If you have stopped the cluster instances, please start them again - stopping instances is not supported. If you intended to terminate this cluster then please delete the cluster in the Red Hat console. Otherwise file a support request.",
  "internal_only": false,
  "event_stream_id": ""
}
Continue? (y/N): y
INFO[0016] Success: 2, Failed: 0
                       
INFO[0016] Successful clusters:                         
ID                                     Status
751e4669-857f-45ce-804f-ece43abbe91c   Message has been successfully sent to 751e4669-857f-45ce-804f-ece43abbe91c
93f72b85-e317-432e-8355-46885c38f68f   Message has been successfully sent to 93f72b85-e317-432e-8355-46885c38f68f

If there is an error in one of the clusters (as an example, here a cluster was in installing state):

❯ ./dist/osdctl_linux_amd64/osdctl servicelog post -t https://raw.githubusercontent.com/openshift/managed-notifications/master/osd/cluster_has_gone_missing.json -p "CLUSTER_UUID=shepherd%"
INFO[0001] The following clusters match the given parameters: 
Name                ID                                 State               Version             Cloud Provider      Region
shepherd-1          1oudlp5pmjparuqfasf16pq8jtbtnptq   ready               4.9.8               aws                 ap-south-1
shepherd-2          1outpcau2bum4mtmkgj1m0q51r4qdunr   installing          4.9.9               aws                 ap-south-1

INFO[0001] The following template will be sent:         
{
  "severity": "Error",
  "service_name": "SREManualAction",
  "cluster_uuid": "${CLUSTER_UUID}",
  "summary": "Action required: cluster not checking in",
  "description": "Your cluster requires you to take action because it is no longer checking in with Red Hat OpenShift Cluster Manager. Possible causes include stopping instances or a networking misconfiguration. If you have stopped the cluster instances, please start them again - stopping instances is not supported. If you intended to terminate this cluster then please delete the cluster in the Red Hat console. Otherwise file a support request.",
  "internal_only": false,
  "event_stream_id": ""
}
Continue? (y/N): y
INFO[0003] Success: 1, Failed: 1
                       
INFO[0003] Successful clusters:                         
ID                                     Status
93f72b85-e317-432e-8355-46885c38f68f   Message has been successfully sent to 93f72b85-e317-432e-8355-46885c38f68f

INFO[0003] Failed clusters:                             
ID                                     Status
751e4669-857f-45ce-804f-ece43abbe91c   Account sbasabat.openshift denied access to perform create on ServiceLog with HTTP call POST /api/service_logs/v1/cluster_logs

I can confirm that the messaged clusters have service logs listed via osdctl list and console.

[supreeth7]

/retest

[wanghaoran1988]

/lgtm

[wanghaoran1988]

@clcollins @fahlmant @georgettica @iamkirkbater Cloud one of you take another look and approve this?

[georgettica]

/approve
looks good overall.
I do still need to test the normal post command to make sure that's ok
/hold

[georgettica]

/approve
works :)

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: georgettica, supreeth7, wanghaoran1988

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [georgettica]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[georgettica]

/unhold

[openshift-ci]

@supreeth7: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.