[WIP] Centralized logging capability. Follow up on the #4108

[soreana]

Description

I have created a PR to enable remote logging in system VM and routerVM in the past (PR: #4108, issue: #4093, ML discussion). This is an upgrade to that pr which enable/disable remote logging in systemvm and routervm using systemvm.remote.logging.enabled global setting.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Screenshots (if appropriate):

Refer to PR: #4108 and issue: #4093

How Has This Been Tested?

[soreana]

@rhtyd @DaanHoogland @weizhouapache

I have added a new global setting to enable/disable the remote logging in system VMs and router VMs. Based on the systemvm.remote.logging.enabled value which is passed to system VMs through cmdline it decide whether send the logs to a remote server or not. One missing part of this puzzle is the changes in nodes and management servers. I wasn't sure where I should add the changes, I need your help with that.

One option is to leave it as is and let the system administrator be in charge of changes in management servers and nodes. Another option is that cloudstack changes the following files in the management server/agent restart. I prefer to automate the procedure as much as possible, but I don't know how to achieve that. I appreciate any help.

Changes in nodes:

1. edit /etc/rsyslog.conf and uncomment/comment the following lines.

module(load="imtcp")
input(type="imtcp" port="514")

module(load="imudp")
input(type="imudp" port="514")

2. Adding below file on each hypervisor to export logs from hypervisor to management server.

# cat /etc/rsyslog.d/00-fwd.conf
if ($fromhost-ip startswith '169.') then {
    *.* @<Management Server IP>:514
    stop
}

Changes in management servers:

1. Just like nodes, below 4 lines have to uncommented in rsyslog configuration file on management server in /etc/rsyslog.conf

module(load="imtcp")
input(type="imtcp" port="514")

module(load="imudp")
input(type="imudp" port="514")

2. Add below file on management server

# cat /etc/rsyslog.d/00-remote.conf
$template remote-incoming-logs,"/var/log/rsyslog/%HOSTNAME%/syslog"
$template remote-incoming-logs-combined,"/var/log/rsyslog/remote"
if ($fromhost-ip startswith '10.' or $fromhost-ip startswith '172.') then ?remote-incoming-logs
if ($fromhost-ip startswith '10.' or $fromhost-ip startswith '172.') then ?remote-incoming-logs-combined
& ~ 

[DaanHoogland]

@soreana You are asking a rather wide question but let me try;
I'd create global settings for

• remote logging for VRs
• remote logging for systemVMs
• remote logging for hosts
  Than based on these settings appropriate changes on the targets can be made
  as for the management server, I'd say that a separate package setting up the remote log server should be installed, this should already exist in most linux distros. Worst case a dependency can be defined and some minor post install script run (i.e. cloudstack-logserver-4.16.0.0.rpm)

Most operators will want to make it a different machine than the management server.

The IP of the remote logging server should be configurable per target as one can have a rather various cloud topography. An alternative would be to install a pass-through logging server that redirects to a central server, but that is only for the advanced version.

I'm pretty sure I didn't answer (all of) your questions but I hope I contributed some.

[soreana]

@DaanHoogland Thanks for the detailed comment. I have added the following global settings to manage remote logging for console proxy VM, storage VM, and router VM.

Name	Type	default value	is dynamic	description
consoleproxy.remote.logging.enabled	Boolean	false	✅	If true, console proxy will forward console proxy logs to <consoleproxy.remote.logging.address>
consoleproxy.remote.logging.address	String	""	✅	Console proxy forwards its logs to this address
secondary.storage.remote.logging.enabled	Boolean	false	✅	If true, secondary storage will forward secondary storage logs to <secondary.storage.remote.logging.address>
secondary.storage.remote.logging.address	String	""	✅	Secondary storage forwards its logs to this address
router.remote.logging.enabled	Boolean	false	✅	If true, Router VM will forward console proxy logs to <router.remote.logging.address>
router.remote.logging.address	String	""	✅	Router VMs forward their logs to this address

I want to change the /etc/rsyslog.conf and /etc/rsyslog.d/00-fwd.conf files on hypervisors as well. Where is the best place to modify these files?

[DaanHoogland]

I want to change the /etc/rsyslog.conf and /etc/rsyslog.d/00-fwd.conf files on hypervisors as well. Where is the best place to modify these files?

is that a one time change or an on-line configuration? the onetime change should be in the install pacckage for the agent. the online configuration should be a libvirt command (if it is only for KVM)

[soreana]

@DaanHoogland As a cloudstack admin, I prefer to have the same configuration for hosts as the ones I defined for system VMs. Something Like:

Name	Type	default value	is dynamic	description
host.remote.logging.forward.enabled	Boolean	false	✅	If true, Host will forward console proxy, storage vm, and router vm logs to <host.remote.logging.forward.address>
host.remote.logging.forward.address	String	""	✅	Host forwards console proxy, storage vm, and router vm logs to this address

I want to change /etc/rsyslog.conf based on the host.remote.logging.forward.enabled value and /etc/rsyslog.d/00-fwd.conf based on the host.remote.logging.forward.address.

I think It is better to go with the libvirt command option. I found lot of samples in plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper directory. Let me know if you are aware of any documentation or tips.

[DaanHoogland]

I think It is better to go with the libvirt command option. I found lot of samples in plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper directory. Let me know if you are aware of any documentation or tips.

I'm having second thoughts about my use of the phrase "libvirt command" as it is nothing to do with libvirt. It would be a command to be executed by the agent though. maybe the execution at the backend doesn't have to be so different as on the SVMs??

[soreana]

@DaanHoogland Honestly, when I tried to change the code to add a libvirt command, I noticed how many changes it requires for such a small task. I searched for other options.

I thought it is better to add some fields in agent.properties and modify the mentioned files in agent init process. My idea is to create a new class in python/lib/cloudutils/serviceConfig.py for the remote logging option. which uses remote.logging.forward.enabled and remote.logging.forward.address in agent.properties.

Let me know what do you think.

[DaanHoogland]

sounds like a plan @soreana

[github-actions]

Hi @${author}, your pull request has merge conflicts. Can you fix the conflicts and sync your branch with the base branch?