Improve the logging experience mainly within installation and startup.

[balhar-jakub]

1. In the default setup, if there is no issue, we should log only that the services started, ideally including a point when the whole Zowe is started and works properly. We may include critical information such as version and key configuration details.
2. In the default setup, if there are issues that limit the functionality but don't break things, we need to log them as warnings in a way that's actionable by the users.
3. In the default setup, if there are issues preventing Zowe from working, these will be logged as errors with easy to find error codes and actionable text.
4. Logs created for debugging should go to files instead of the spool so that the diagnostics pax contains them.
5. Debugging logs should contain, by default, the details of the TLS communication. Provide the reasonable true or false across the whole Zowe.

Possibly show some information in the Syslog.

[1000TurquoisePogs]

I think this is going to need help from multiple people across squads, such as timing events that occur between 2 servers, so I'm assigning @DivergentEuropeans but don't expect he's going to solve this alone.

[balhar-jakub]

I expect that we will do a lot on API ML squad part, but it would be great to have @DivergentEuropeans to help us with understanding of the startup of App Server.

[DivergentEuropeans]

Hey @Himani1519 I’m using Zenhub in GitHub, click this link to join my workspace and see other features available in GitHub or download the Zenhub extension and sign up with your GitHub account.
Posted using Zenhub

[DivergentEuropeans]

@balhar-jakub Sure. This task will be owned by myself and Himani (who will help me with testing various enviornments) but I'm always a resource to reach out about App server Q's on Slack if you'd like

[Martin-Zeithaml]

Another better logging experience

As recently found, there is an undocumented feature, which is mixing the time zones and even days.
For example: after the stopping zowe, there is a message, that 2 days before the discovery was started:

2023-05-12 08:18:11 <ZWELNCH:17172296> ZWESVUSR INFO ZWEL0002I component metrics-service stopped
CEE5205S The signal SIGTERM was received.
2023-05-10 11:38:03 <ZWELS:84279636> ZWESVUSR INFO (zwe-internal-start-component) starting component discovery ...
2023-05-12 08:18:10.933 <ZWEADS1:ThreadPoolTaskExecutor-1:67504465> ZWESVUSR WARN  (o.z.a.s.ApimlPoolingHttpClientConnectionManager) ZWEAO105W Gateway HTTP Client per-route connection limit (maxConnectionsPerRoute) of 10 has been reached for the '{s}-&gt;https://hlavniramec.example.com:12345' route.

Another example, these messages are in the correct time order, however there are probably 2 time zones:

 2023-04-27 11:01:21 <ZWELNCH:83886624> ZWESVUSR INFO ZWEL0058I WORKSPACE_DIR is '/var/zowe/workspace'
 node is /IBM/nj/v14r0/IBM/node-v14.20.0-os390-s390x-202208110152/bin/node
 2023-04-27 17:02:53 <ZWELS:50332073> ZWESVUSR INFO (zwe-internal-start-prepare) Zowe version: v2.6.0
...  
 2023-04-27 17:07:18 <ZWELS:50332073> ZWESVUSR INFO (zwe-internal-start-prepare) Zowe runtime environment prepared
 2023-04-27 11:07:19 <ZWELNCH:83886624> ZWESVUSR INFO ZWEL0018I Zowe instance prepared successfully

Ideally when reading the logs, I would appreciate not to examine each timestamp of each message.

[jalel01]

Referencing the APIML Squad Epic that has in progress work around logging:

zowe/api-layer#1181