Managing servers in the cluster

A cluster is a group of servers in a single location. The key feature of such servers (cluster nodes) is their location and high data transfer speed between them.

Server connection

Before connecting, make sure that the server meets the requirements for the cluster.

Perform the following steps to connect a server to the cluster: 

1. Go to Cluster nodes → Connect a node.
2. Enter a cluster node Name. 
3. Select a Cluster that the server will be connected to.

4. To prevent VMmanager from creating a default bridge on the node, enable the Do not configure network automatically option. Learn more in Network setting on the cluster node.
   

   Enable this option if:

   • you plan to use the VLAN interface for the server IP;
   • the server has a non-standard network configuration — there is no single interface for all traffic to pass through.

   After connecting the node, you will be able to manually configure the necessary network settings.

5. Enter additional information into the Notes field.
6. Enter a server IP address and Port to connect via SSH. The default port is 22. 
7. Enter the User. It is recommended to use root for correct operation of the node.

8. Enter the server User password or add a public SSH-key of the server with VMmanager. To do so, click on I want to use a public ssh-key, copy the key, and add it into the file /root/.ssh/authorized_keys on the server. 

   If you specify a SSH root password, VMmanager connects to the server with this password and add the platform's public SSH key to the list of authorized keys. Next time to connect to the storage, VMmanager will use only the SSH key.
9. Enter the RAM Overselling ratio. Learn more in Overselling. 
10. Select VM limit. When the limit is reached, the creation and migration of virtual machines on the node will be forbidden. If the limits are not set, the virtual machines can be created unless there are sufficient resources on the node.
11. Select vCPU limit — the maximum number of vCPUs that can be allocated to VMs on this node. The default value of the parameter is the number of CPUs on the server multiplied by the overselling ratio.
12. Specify the VM parameters that will be created on the cluster node. For details, see the VM distribution settings section.
13. If you are connecting the server to a cluster with IP-fabric type, specify an IPv6 address for configuring networks.
14. If you connect the server to the cluster with the "Routing" network configuration type, enter the IP for new VM. You can enter single IP addresses, or a mask. Enter every IP address from a new line. 
15. Select the scripts to be executed when the node is connected:
    1. Press Select scripts.
    2. Mark the desired scripts in the list.
    3. Select the order in which the scripts will be executed in the Start Priority column.

    4. Press Select ... scripts.
       

       If the Autorun on clusters option is enabled in the script settings, you cannot disable the script and change the priority. Read more in Creating a script for the node.
16. Click on Connect the node.
17. If you have selected a cluster with two network interfaces, specify the Interface for the main network and/or the Interface for the additional network. Read more in Main and additional network. Press Continue.

Example of server connection

The system will install the required software and configure the server. 

Cluster node dashboard

To view information about a cluster node, open its card: go to Nodes section → select the node. The cluster node card displays:

• node status;
• virtualization type — KVM or LXD;
• CPU — CPU load;
• RAM — RAM used;
• Storage — disk space used;
• overselling ratios;
• information about connected storages;
• Total VMs — total number of VMs on the node;
• number of VMs with a certain status;
• Estimated number — maximum possible number of VMs. The value is calculated dynamically and depends on available node resources;
• cluster name;
• operating system;
• IP-address of the node;
• Uptime — time of continuous node operation;
• Services — services running;
• note.

Possible node statuses:

In a high availability cluster, ha-agent services can assume the following statuses:

• serviceable:
  • Master node — the node is serviceable and has been selected as the master;
  • Active — the node is serviceable and is a participant or in an intermediate running state (master selection or waiting for an update);
• non-serviceable:
  • Network isolation — the node is not accessible via the network, but the node has access to the storage. The status is set if the node is unavailable to ha-agent services on other nodes in the cluster;
  • No connection with storage — the node has no access to the storage, but remains accessible via the network;
  • Unavailable — the node is unavailable via the network and has no access to the storage. The status is set if the platform does not have access to the node;
  • Excluded from HA cluster — the node is unavailable via the network, but has access to the storage. The node's VMs are temporarily excluded from the cluster and are not migrated;
• special:
  • Unknown status — the exact status of the node is unknown. For example, high availability is in the process of being enabled;
  • Error enabling HA — the task of enabling HA on the node failed.

Cluster node dashboard interface

To view detailed information about a node, click the Parameters button in the node card.

Reconnecting a node

If the software on a node does not work correctly, the platform sets the status "Corrupted" for it. You can reconnect the problem node. To do this:

1. On the server with the platform, examine the connection error log file /var/log/node_create/{node_id}.log. Fix the problems that prevent you from adding a node.
2. In the Nodes section, in the row with the problem node, click the message in the Status column → Reconnect node button.

Change connection settings

If the IP address and/or SSH port on the cluster node has changed, VMmanager will lose connection with the node. To reconnect:

1. Go to Nodes → select a node → menu → Change connection settings.
2. Specify the Node IP address.
3. Specify the SSH port.
4. Click Connect to node.

Changing the platform SSH key

To change the platform SSH key:

1. Connect to the server with the platform via SSH.
2. Back up the platform in case of possible failures: 
   

   vm backup

3. Update the platform private SSH key:
   

   docker exec -it vm_box curl 'input:1500/vm/v3/ssh_key_replace' -X POST -H 'internal-auth: on' --data-raw '{}'

4. Restart the platform:
   

   vm restart

   After the restart, the platform will delete the old public SSH keys on the cluster nodes and copy the new ones. 
   
   Replacing SSH keys on cluster nodes will restart the hawatch, node_alert, and monitor services.

The SSH private key update logs are saved to the /var/log/node_ssh_key_replace.log file in the vm_box container.

SSH keys are saved in the /opt/ispsystem/vm/etc/.ssh/ directory in the vm_box container. Files with old keys are saved with .bak extension.

VM distribution settings

When creating a VM, the platform selects the cluster node with the most suitable distribution settings. For example, you can configure the settings so that only VMs with a certain amount of RAM or Storage are created on the node.

To configure distribution settings:

1. Enter Nodes → select the node → VM distribution settings.
2. Enter the RAM оverselling ratio. Read more in Overselling.
3. Specify the VM limit — the maximum number of VMs on the node.
4. Specify the Limit the vCPU number — the maximum number of vCPUs that can be allocated to VMs on this node.
5. Select VM parameters for distribution filters:

   • OS tags — operating system template tags; 

     Some VMs may not have OS tags. For example, if the VM is created without an OS or from an ISO image. For such VMs, filters work as follows:

     • if the filter checks for a match with a tag, the VM cannot be placed on this node;
     • if the filter checks for a mismatch with a tag (the filter value is set in "!some_tag" format), the VM can be placed on this node.
   • Configurations — the names of configurations;
   • RAM resources — the amount of RAM;
   • Storage resources — the volume of the disk;
   • vCPU resources — the number of virtual CPUs;
   • Complex filter — a condition with several parameters.
     When entering conditions, use the filter syntax. If you want to set an additional condition, press Add filter. To delete a condition, press .
     
     If you set multiple conditions, they will be combined with a logical “AND”. I.e, the platform will exclude cluster nodes that do not satisfy at least one of the conditions.
6. Press Save.

Example of setting distribution filters

To configure VM distribution on multiple nodes of a cluster:

1. Enter Nodes → select the nodes.
2. On the bottom menu, press VM distribution settings.
3. In the window that opens, configure the VM distribution filters.
4. Press Save.

Example of setting distribution filters

Forbid users to create VMs on a cluster node

To prevent users from creating virtual machines on a cluster node: 

1. Go to Cluster nodes → menu → Prohibit VM creation.
2. Click on Confirm.

To cancel the settings: 

1. Go to Cluster nodes → menu → Allow VM creation.
2. Click on Allow.