Connecting a network LVM storage

You can connect a network LVM storage to the cluster. Before that, configure the storage area network (SAN) and connect the storage to the cluster nodes as a block device. Read more in Pre-configuring SAN.

Summary of LVM

Logical Volume Manager (LVM) is a subsystem that allows you to use different areas of the same hard disk and/or areas from different hard disks as one logical volume. The size of file systems of logical volumes is not limited to one disk, as the volume can be located on different disks and partitions.

Main LVM designations:

• PV (Physical Volume) — partitions of disks or entire "unpartitioned" disks;
• VG (Volume Group) — a set of physical PV volumes combined to form a single disk;
• LV (Logical Volume) — partitions created on the physical space of the VG volume group.

LVM storage supports one virtual disk image format — RAW. Read more about LVM in the official documentation.

In VMmanager, LVM storage is used only for storing virtual machine (VM) disks. Each disk is stored in a separate LV with the name in the "<vm_id>_<vm_name>” format.

Configuring the multipath

Multipath is a technology for connecting storage network nodes using multiple routes. If one controller fails, the operating system will use another controller to access the device. This increases system fault tolerance and allows for load balancing. Multipath devices are combined into one using specialized software.

To configure multipath:

1. Install the multipath-tools software:

   Ubuntu

   apt install -y multipath-tools
   

   AlmaLinux

   dnf install device-mapper-multipath

2. In the configuration file /etc/multipath.conf, change the value of user_friendly_names to no:

   user_friendly_names no
   

3. Restart the multipathd service:

   systemctl restart multipathd
   

4. On each storage node, combine all disk connections into a single multichannel connection:

   multipath /dev/sdX
   

   Comments to the command

   /dev/sdX is the device for the first disk. For example, /dev/sda

5. The multipath command will group all paths into a single mapper device. To see the list of grouped devices, run the command:

   multipath -ll
   

   This device will have the same ID on all nodes. Save the identifier value. It will be needed when connecting the storage.

   Example output

   root@node1:~# multipath -ll
   36589cfc0000008af95b3633c714c5c57 dm-0 TrueNAS,iSCSI Disk
   size=550G features='0' hwhandler='1 alua' wp=rw
   `-+- policy='service-time 0' prio=50 status=active
     `- 2:0:0:0 sda 8:0 active ready running

6. On each storage node, add a filter to the devices {...} block in the LVM configuration file /etc/lvm/lvm.conf:

   filter = [ "a|/dev/disk/by-id/wwn-.*|", "r|.*|" ]
   

   Comments

   a|/dev/disk/by-id/wwn-.*| — enables /dev/disk/by-id/wwn-.* devices
   r|.*| — disables all other devices

   If you need to use other disk paths, specify them in the filter. For example, filter = [ "a|/dev/disk/by-id/.*|", "r|.*|" ]

   If the system disk uses LVM, add it to the filter:

   Example config for /dev/sda

   filter = [ "a|/dev/disk/by-id/wwn-.*|", "a|/dev/sda|", "r|.*|" ]
   

7. Verify that the filter and configuration are correct with the command:

   vgs
   

   If everything is configured correctly, the VG list in the command output should contain no errors.

Determining the WWID of a block device

When connecting the storage it is necessary to specify the path to the block device. For stable operation of the storage we recommend to use the path with WWID (WWN). 

To determine the WWID of a block device:

• if multipath is configured in the storage, run the command: 

  udevadm info -q property /dev/mapper/<mapper_id> | grep WWN

  Comments to the command

  <mapper_id> — id of the mapper device
  Example command output

  DM_WWN=0x600c0ff000fc5cb10b040e6701000000

• if multipath is not configured in the storage, run the command: 

  udevadm info -q property <device> | grep -E 'ID_WWN_WITH_EXTENSION='

  Comments to the command

  <device> — block device. For example, /dev/sda
  Example command output

  ID_WWN_WITH_EXTENSION=0x60014053aa5c2bc20ed4cf19d65e6437

The device path will be /dev/disk/by-id/wwn-<device_WWID>. For example, /dev/disk/by-id/wwn-0x600c0ff000fc5cb10b040e6701000000.

How to connect a storage

To connect the storage, enter Clusters → select the cluster → menu → Network storages:

1. Click Add storage → Add a network storage → SAN.
2. Specify its settings:
3. 1. Name.
   2. Storage tags. Read more in the Selecting storage for VM disks. 
   3. Path to block device. We recommend using the WWID path — /dev/disk/by-id/wwn-<device_WWID>.
4. Click Add.

VMmanager will automatically configure the LVM storage on the block device: it will create PV and VG. If a VG has been created on the block device, the storage will not be connected.