Cluster network configurations

There are three types of cluster network configurations in VMmanager: "Switching", "Routing" and "IP-fabric". They differ in the order in which IP addresses are assigned to virtual machines and in the network topology.

"Switching" configuration type

Operation logic

A pool is allocated for the cluster, combining blocks of IP addresses from physical networks. You can manage IP addresses of this pool in the Networks section. Migration of virtual machines (VMs) between cluster nodes is available for this type of IP address assigning.

Example of network settings

The physical networks 192.0.2.0.0/27 and 192.0.2.32/27 are allocated to the cluster. One of the network addresses is assigned to the vmbr0 network interface. VMmanager assigns addresses from the allocated networks to VMs on the cluster nodes.

"Routing" configuration type

Limitations

The "IP-fabric" configuration type is supported only for KVM cluster nodes with AlmaLinux 8 OS and LXD cluster nodes with Ubuntu 20.04 OS.

Bridge creation is not supported on cluster nodes.

Operation logic

A cluster node in a data center is assigned the primary IP address and a subnet of additional IP addresses. Virtual machines on this node are allocated only the IP addresses from the specified subnet. When connecting a node to the cluster, a range of allowable IP addresses for the virtual machine is specified. The newly created virtual machine on the node is assigned the first available IP address from the range. You can manage IP addresses in the node dashboard: Nodes → node name → IP addresses .

VM routing is performed through the virtual interfaces vnet and the interface of the network bridge vmbr0. This subnet routing type does not allow migrating virtual machines to other cluster nodes.

The Port Security function is enabled on the data center equipment. This function prevents from sending Ethernet frames if the sender MAC-address is not specified as allowed. Since a KVM virtual machine has a separate MAC-address, frames from the virtual machine will be blocked on the network equipment. To avoid this, VMmanager modifies network settings on the node so that the MAC-address in the VM frames will be changed into the MAC-address of the cluster node network interface.

Example of network settings

The server is assigned the primary IP address 123.45.12.48 and the subnet 203.0.113.40/29. As the subnet IP addresses will be used as additional, the virtual machine can be allocated all the addresses from this subnet: 203.0.113.40, 203.0.113.41, ..., 203.0.113.47. The primary IP address of the server on the network interface eno1 is used as the subnet gateway.

Operation scheme

How to configure IPv6 on cluster nodes in the Hetzner and OVH data centers

To configure IPv6 routing, add IPv6 address to the cluster node: Nodes → node name → IP addresses → IPv6 → Add an IP address. For correct operation of IPv6 network, specify the address ::2 and prefix length 64. Example: 2a01:4f8:13b:9a7::2/64. VMmanager will configure the network in the node automatically.

If there is no routing with the provider's network, you can enable IPv6 proxying via NDP:

1. Connect to the cluster node via SSH.

2. Install the ndppd service: 

   dnf config-manager --set-enabled powertools

   dnf install epel-release

   dnf install ndppd

3. Create the configuration file /etc/ndppd.conf: 

   Configuration file example

   proxy enp3s0f0 {
    rule 2a01:4f8:13b:9a7::/64 {
    iface vmbr0
    }
   }

   Comments

   enp3s0f0 — network interface for proxying

   2a01:4f8:13b:9a7::/64 — IPv6 subnet

   vmbr0 — bridge interface on the cluster node

4. Add a rule for the default gateway: 

   ip -6 neigh add proxy <address> dev vmbr0

   Comments to the command

   <address> — IPv6 address of the cluster node. For example, 2a01:4f8:13b:9a7::2

"IP-fabric" configuration type

Limitations

The "IP-fabric" configuration type is supported only for KVM cluster nodes with AlmaLinux 8 OS and LXD cluster nodes with Ubuntu 20.04 OS.

Bridge creation is not supported on IP-fabric cluster nodes.

Operation logic

The "IP-fabric" configuration type allows you to create virtual machines with public IP addresses in a private network. A pool is allocated for the cluster, combining blocks of IP addresses from physical networks. You can manage IP addresses from this pool in the Networks section. Migration of VMs between cluster nodes is available for this type of configuration.

Creating an "IP-fabric" cluster is only possible on a network that supports iBGP routing. Route Reflector equipment is used to transmit information about iBGP routes. Physical or virtual routers and/or servers can be used as such equipment. When creating a cluster, add the addresses of its nodes and BGP autonomous systems numbers to the Route Reflector configuration.

When creating a cluster, you specify:

1. Gateway IP address for the VM. VMmanager will assign this address of the gateway to all virtual interfaces on the cluster node and use it only to route the VM inside the node. As IP address you can specify any address that will not match with those used in the cluster node.
2. BGP community.
3. IP addresses and numbers of autonomous systems of Route Reflector equipment.

When receiving the task to create a VM in the cluster "IP-fabric", VMmanager will:

1. Create <vmname>_net0 virtual interface on the cluster node.

   Comments

   <vmname> — name and id of the VM. For example, for a VM with the name aaa and id 12, the interface will be called aaa12_net0.
2. Create a VM with an IP address from the dedicated pool and the network mask of 255.255.255.255 (/32).
3. Set <vmname>_net0 virtual interface as gateway for the VM. Routing between the VM and the gateway is performed through a point-to-point connection (PtP).
4. Transmit the information about the created route to the Route Reflector equipment through the FRR service.

Example of network configuration

The pool of public IP addresses 198.51.100.0/24 is allocated for the cluster. VMmanager assigns addresses from this pool to VMs on cluster nodes. The names of created VMs are aaa (id 1), bbb (id 2), ccc (id 3), etc. The interfaces aaa1_net0, bbb2_net0, ccc3_net0, etc with IP addresses 10.0.0.1 are used as gateway. The external interfaces of the cluster nodes have private IP addresses in the 172.16.0.0/24 network.

Operation scheme