The VM status monitoring subsystem is specially designed to send the information about virtual machines from libvirt to VMmanager KVM. The main goals of this subsystem:
- Synchronizes data in the control panel about virtual machines and their actual statuses
- Transmits statuses of virtual machines with minimal time lags
- Registers changes of VM statuses
Monitoring subsystem:
- Participates in OS installation on a virtual machine
- Controls duplicated of virtual machines
Subsystem architecture
Subsystem consists of 2 services: vmwatch-master and vmwatch-node.
A message queue, which is implemented using ZeroMQ library via the PUSH-PULL pattern, is used for interaction between services.
vmwatch-master
This service runs on the node, where the control panel is installed. It receives messages on the IP address and port specified in the configuration file, processes them, and calls the corresponding API functions of a control panel functions to send data about changes in statuses of virtual machines.
vmwatch-node
The service runs on every cluster node. It sends messages to the IP address and port specified in the configuration file. It receives information about changes of VM statuses from libvirtd. If the connection with libvirt is lost, the service will be stopped.
Service configuration
System V (System D for CentOS 7) controls service run and auto-run. vmwatch-master service and its control script are kept in vmmanager-kvm package for VMmanager KVM. vmwatch-node service and its control script are kept in vmmanager-pkg-vmnode package.
Services run automatically when booting the OS or by the control panel during the configuration process.
After VMmanager installation vmwatch-master service is blocked until the first cluster node is added into the cluster.
Service work parameters are added into the vmwatch-master.conf and vmwatch-node.conf configuration files that are kept in /usr/local/mgr5/etc catalog. The configuration files are created by the control panel automatically.
The configuration file is recorded in the NAME_PARAMETER=VALUE format, one parameter on one line.
vmwatch-master.conf configuration file
It may contain the following parameters:
- disabled - If parameter has yes value, it blocks service execution.
- listen - Parameter value in 'protocol://IP:port' format assigns socket where service receives messages.
- secret - Undefined symbol line (A-Z, 0-9), the parameter value is used for incoming messages authentication.
vmwatch-node.conf configuration file
It may contain the following parameters:
- connect - Parameter value in 'protocol://IP:port' format assigns socket where service sends messages
- secret - Parameter value is used for incoming messages authentication. It has to match with a secret parameter value of vmwatch-master configuration file
- node_id - cluster node id which is used in messages for message source identification
Service configuration
The control panel configures the service automatically when needed:
- During the upgrade to the version which supports the monitoring subsystem, and if there are nodes in the cluster;
- When a cluster node is added into the cluster;
IP address assignment, on which vmwatch-master receives messages, is executed in the following way:
- the system checks the VmwatchListenIp parameter value of the configuration file;
- If IP address is not assigned by the parameter, a cluster node where the control panel is installed is searched;
- If there is no local cluster node, IP address scanning is started at all network interfaces of the server where the control panel is installed, it chooses the first IP address from the list which is not a local one.
The port where vmwatch-master receives messages is defined by the VmwatchListenPort parameter of the control panel configuration file. The default value is 1515.
To change the IP address and/or port, add the corresponding parameters into the control panel configuration file, restart the panel, and call API vmwatch.configure function. This function configures all monitoring services according to the indicated parameters.
Service status control
vmwatch-master service status control is run by the panel when periodic API function is called. It is run by the cron every 60 minutes.
If the control function detects that service is stopped, it will register the notification about this problem. When this problem is solved, the system will configure the whole monitoring subsystem once again.
vmwatch-node service status control is run by the panel when periodic API function is called. It is run by the cron every 5 minutes when checking the cluster node status (vmmini).
If the service is stopped, the control function registers the notification about this problem. When the problem is solved, the system will configure the service on this cluster node
VM status check
In the Cluster settings→ Policy you can enable the check of VM statuses.
The system will use statistics data sent by VM monitoring services. The status check function analyses packages that were sent over the network and the number of write requests to a VM disk. Every 10 minutes vmwatch-nodecollects statistics from all domains and sends it to vmwatch-master. If any of the values equals 0, the corresponding warning will be displayed in the list of virtual machines.
Service log
vmwatch-master service log is kept in the /usr/local/mgr5/var/vmwatch-master.log file, vmwatch-node service log is kept in the /usr/local/mgr5/var/vmwatch-node.log file.