KVM Manager: creating and managing virtual guests

Overview

May First uses KVM Manager to manage virtual guests on each of our physical host machines.

KVM Manager uses the qemu-kvm virtualization software.

KVM Manager is a set of simple bash scripts that help us by setting up new virtual server instances and allows us to manage each instance via systemd. The main benefits of KVM Manager are:

  • Use isolation: nothing runs as root on the host. All virtual servers run via systemd as a non-provileged user.

  • Easy console access using familiar tools. All kvm guests provide access to the console via screen. We can grant ssh access to the non-privileged user account on the host and allow any connecting user to connect to the kvm guests via screen -x.

KVM Manager specifically provides the following:

  • Systemd integration: Two template systemd files are provided:

    • kvm@.service: Every kvm instance can be enabled or started via the service kvm@<guest>.service. This services handles setting up and tearing down the necessary network routing for the guest.
    • kvm-screen@.service: Every kvm instance has a corresponding kvm-screen@<guest>.service that controls the screen session providing access to the console.

  • kvm-creator: creates a new virtual guest instances, specifically:

    • Creates a non-provileged user with a home directory.
    • Creates a logical volume to hold the operating system
    • Add udev rules to ensure the proper permissions to access the logical volume are created for the user.

How to use it - general concepts

KVM Manager is fully integrated into ansible. You can indicate which host a server should be created on and all the details of the setup (including the provision of extra data volumes) via the kvm variable.

  • ansible will automatically resize partitions on the host after they have been created, but only if the new size is bigger then the old size. This step is handled by a script called mf-manage-logical-volumes which reads a configuration file /etc/kvm-manager/logical-volumes.yml that is written out by ansible.

  • If a logical volume is deleted and a new logical volume is created, it will have a signature on it and the lvmcreate command will prompt for a confirmation to override it. Bah. That will cause ansible to simply hang. The only solution is to manually create that logical volume and re-run ansible since instructing lvm to auto wipe signatures seems like a bad idea.

Here’s an example of a kvm guest yaml file with comments:

kvm:
  # The host line indicates on which host this guest should be created.
  host: host-001.mayfirst.dev
  # Usually the guest name is the host name of the guest.
  name: flowerhub001
  # How much RAM should be allocated? You can change this later to increase
  # or decreate the RAM. You will need to restart the guest before it takes
  # effect but ansible will handle writing it out to the `env` file.
  ram: 5g
  # These details apply to the disk handed to the guest as the root disk which
  # will appears as /dev/vda or /dev/sda depending on the driver. You can change
  # these after the guest has been created and the disk will be resized, but the
  # change will only take effect after the guest is rebooted.
  root_disk:
    size: 20g
    volume_group: vg0 
    # You have to specify a driver. `virtio-blk-pci` should always be the
    # default and is the best option *unless it's an SSD disk* in which
    # case you should specify `scsi-hd` so the guest can properly using
    # [trim](https://en.wikipedia.org/wiki/Trim_(computing)).
    driver: virtio-blk-pci
    # kvm-manager has to index each disk separately so it can specify HDA,
    # HDB, HDC, etc, however they won't # necessarily show up with this
    # letter on the guest. This letter has to be unique within a given guest,
    # not unique for the host.
    index: A
  # Like the root disk, you can change the size of a disk after it has been created
  # and it will automatically be re-sized, but a reboot is required.
  data_disks:
    - name: data0001
      size: 250g
      volume_group: vg1
      # See driver comment above.
      driver: scsi-hd 
      index: B
    - name: data0002
      size: 250g
      volume_group: vg1
      driver: scsi-hd 
      index: C

NOTE: You have to limit ansible to just the metal server when you first run it, otherwise it will complain that the container guests are not reachable (because they have not yet been created).

What to do specifically

Create a yaml file for the new guest

For common server types (weborigin, mailstore, mysql, webproxy), you can automatically generate a new yaml file with the command (replace “weborigin” with the desired type):

sower --live generate:weborigin --parent <parent>

For all other guests, use the generic template (sower --live generate:generic --parent <parent> --host <fqdn>.

Note: This process will assign an IP address in our default range. If you need an IP address with access to unusual ports, keep the assigned IP and add an additional IP for the service that needs special network access.

Update the hosts.yml file

Next, update the hosts.yml file. Usually you want to find a similar server and look for all the places it is configured and repeat for your new server.

Push changes to the host

The fastest way is to only run the kvm tag:

sower --live playbook --tags kvm host009.mayfirst.org # replace host009 with the actual host

Run the installation

After your first run, the guest will be created and booted into the installer.

You have to ssh into the console ($guest@$host.mayfirst.org) and type: screen -x.

Then, hit enter to start the installer.

You also have to click through the warning about no swap.

Logging in and initializing

Once the guest has booted, type screen -x again and you can login with the username root and no password.

Change the password and add it to keyringer.

Then, run /root/mf-initialize-server, which will ensure the network is setup, install minimal packages to get ansible to work and will publish the sshfp dns records so when you ssh into the server directly, you will know you have the right fingerprint.

Setting partitions

From your laptop, run sower --live generate:fstab --parent <parent> --host <guest>

This command will properly format all data partitions as ext4 and build out the /etc/fstab partition and also enable quotas if you are setting up a weborigin or mailstore guest.

Generate an ssh key

If you are going to back up the server, be sure to generate an ssh key:

ssh-keygen

Then, copy /root/.ssh/id_rsa.pub to the root_ssh_public_key variable in the host yaml file.

Run ansible

Now you should be ready to push ansible to the guest.

Be sure to:

  1. Pass --secrets and enter the ansible password from keyringer. This ensures the new server gets any secrets from our vault necessary for normal operations.

    sower --live playbook --secrets newserver001.mayfirst.org

  2. After running for the new host, be sure to run for the key server, backup server, internal DNS servers, and monitor server (as appropriate depending on the host). For example:

    sower --live playbook --tags keystore,simplemonitor,backup,knot key*.mayfirst.org monitor*.mayfirst.org backup*.mayfirst.org nsauth*.mayfirst.org

Maintenance

Sometimes you want to make a change to an existing KVM guest.

The most common change is to add or modify a logical volume.

To avoid running the full ansible playbook, you can pass --tags lvm to just run the tasks related to provisioning logical volumes.

Trouble shooting

ansible hangs

If, when running ansible on the bare metal server, it just hangs on:

TASK [run kvm creator for each guest on this host]

Then it may mean that lvm is creating a logical volume from a volume group in which there is already a logical volume signature. Out of an abundance of caution, we don’t use the flag to ignore this error.

To get around this problem, you’ll need to manually run the kvm-creator command on the host.

It should be in this format:

kvm-creator create SERVERNAME VOLUMEGROUP DISKSIZE RAM

For example:

kvm-creator create samba001 vg_llosa0 25g 2g

Then, re-run ansible.

Installer says no kernel modules found

If you get an error in the installer saying “No kernal modules were found” then, delete /usr/local/share/ISOs/default.iso and the .iso file named after your guest and re-run ansible. This error means that our base installer is out of date and needs to be refreshed.

Data block device is not available in the guest

If your data block device does not show up in your guest as expected, ensure that the permissions are set correctly on the host:

First, check where your device mapper is symlinked, e.g.:

0 severo:~# ls -l /dev/mapper/vg0-data0002
lrwxrwxrwx 1 root root 7 Mar 26 14:18 /dev/mapper/vg0-data0002 -> ../dm-9
0 severo:~#

Then, check the permissions on the actual device:

0 severo:~# ls -l /dev/dm-9 
brw-rw---- 1 root disk 253, 9 Mar 26 14:18 /dev/dm-9
0 severo:~#

The group ownership should be set to the name of the guest (e.g. container001).

Ensure that the udev rule is correct:

0 severo:~# cat /etc/udev/rules.d/92-kvm-creator-data0002.rules 
ACTION=="change", SUBSYSTEM=="block", ATTR{dm/name}=="vg0-data0002", GROUP="container001"
0 severo:~#

(If it is not correct, you may need to re-run ansible and/or check your configuration.)

Then, re-trigger udev to get it run: udevadm control --reload-rules && udevadm trigger

“No root file system is defined”

This can happen if you specify that the root partition should use the scsi-hd driver instead of virtio-blk-pci. The pre-seed file has /dev/vda hard-coded, so change this to virtio-blk-pci for the install, then you can change it back.

De-commission

To de-commission a kvm instance, follow these steps (assuming the guest is vps999):

  • Avoid spurious alerts: Edit ansible yml file for the guest (e.g. host_vars/vps999.mayfirst.org.yml). Remove the monitor block. Push to our monitor servers: sower --live playbook --tags simplemonitor monitor00[12].mayfirst.org
  • Stop and disable the systemd service: systemctl stop kvm@vps999, systemctl disable kvm@vps999
  • Remove from the guest: Check which volume group the vps999 logical volume is on, then: kvm-creator destroy vps999 vg0 (assuming it’s on vg0). This step should remove the guest settings, user, home directory and main logical volume.
  • Remove data disks: Check the ansible yml file for any data_disks defined. Delete them directly from the host, e.g. lvremove vg0/data9999.
  • Delete from ansible Delete both the yaml file in host_vars and also edit hosts.yml and search for and delete all lines matching the guest.
  • Update control panel Run sower --live red:sync to update the control panel with the changes.
  • Update the membership record in Outreach So they are no longer charged.