OA - Ubuntu
A blog about Ubuntu, mobile GIS and archaeology

KVM, VMBuilder & Puppet - (Really) Automated deployment

Aug 10, 2010 by Yann Hamon

I've spent some time automating the deployment of virtual machines using vmbuilder and puppet. There are two reasons behind this:

  • I give every IT person in the company (about 8-10 of us) a VM for development, to play with new software and try fancy configurations. As a result, those machines often break and need reinstalling.  
  • We do not have enough space on our SAN to put all VMs on it. As a result, there are still VMs on the disks of the KVM servers. If one of those breaks, I need to be able to recreate the machines from backups quickly.

The idea is to use puppet to "describe" the virtual machine, create the machine using vmbuilder, and set itself up on it. This has a lot of advantages in comparison to just making a backup of the whole image: first, it doesn't take any additional disk space, it is configurable (you can change the IP or disk size when doing a deployment) - and it gives you full knowledge of how the VM is setup, instead of having just a "blackbox", that would be very difficult to recreate by anyone but you.

Warning: this is a work in progress. One of the reasons I post this here is to get some feedback :)

Prerequisites: the version of vmbuilder in lucid is badly broken - --tmpfs, --firstbook and --templates do not work properly. I grabbed python-vm-builder_0.12.4-0ubuntu0.1_all.deb  from -proposed.

So, there comes the puppet configuration. Let's define a vm called yhamon-dev on the node kvmhost1.goo.thehumanjourney.net :

node "kvmhost1.goo.thehumanjourney.net" {
virtual_machine {
yhamon-dev :
fqdn => "yhamon-dev.goo.thehumanjourney.net",
ip => "10.0.10.108",
netmask => "255.255.0.0",
dns => "10.0.10.70",
gateway => "10.0.0.153",
memory => "512",
rootsize => "5120",

}
}

 This will call the function virtual_machine which I have defined here (had to split some lines - lines here split by \ are only on line):

define virtual_machine ($fqdn, $ip, $netmask="255.255.0.0", $dns="10.0.0.1", \
$gateway="10.0.0.1", $memory="1024", $rootsize="6144" ) {
exec {"create_vm_${name}":
path => "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
timeout => 3600,
command => "virsh destroy $name ; virsh undefine $name ; /usr/bin/vmbuilder \
kvm ubuntu -d /var/lib/kvm/$name -v -m $memory --cpus=1 --rootsize=$rootsize\
--swapsize=512 --domain=$name --ip=$ip --mask=$netmask --gw=$gateway --dns=$dns\
--hostname=$name --suite=lucid --user=yhamon --name='Yann'\
--rootpass=password --libvirt=qemu:///system \
--install-mirror='http://archive.thehumanjourney.net/ubuntu'\
--install-security-mirror='http://archive.thehumanjourney.net/ubuntu'\
--components=main,restricted --execscript=/var/lib/kvm/postinstall.sh --debug\
--verbose --firstboot=/var/lib/kvm/kickstartpuppet.sh \
--copy=/var/lib/kvm/puppetkeys/$fqdn/files --tmpfs=- --addpkg=puppet \
&& virsh start $name" ,
unless => "/usr/bin/test -d /var/lib/kvm/$name",
}
}

A few words about this function. First, it checks if the folder /var/lib/kvm/$name exists ($name being the name of the vm, in our case yhamon-dev). If that folder doesn't exist, it will stop and undefine any VM running with that name in libvirt. You might not want this - I use it so I can just delete the folder and not bother about removing the vm from libvirt as well.

After this, it will call vmbuilder with the arguments as given previously.  We need to raise the timeout and specify the $PATH. We use the --tmpfs argument and a local APT repository to speed up image creation. The tricky part here is to get puppet installed and configured right, so that it will start at the first boot and configure the server without intervention. Four steps are required here:

  1. The installation of puppet, as done with --addpkg puppet
  2. Preventing puppet from starting on its own, and therefore eventually doing things we don't want it to - this is done by the postinstall.sh script given to --execscript
  3. Copying puppet's SSL keys and configuration files - this is done by the --copy argument
  4. Deploying those configuration files and starting puppet, this is done by the kickstartpuppet.sh script from --firstboot.

The first step is pretty much self-explanatory, so let's start with the second. We override the /etc/default/puppet file to prevent puppet from starting after the first boot, and remove any new SSL keys the install script might have generated:

$ cat postinstall.sh 
#!/bin/sh
chroot $1 rm -rf /etc/puppet/ssl/
chroot $1 /bin/bash -c "echo -e 'START=no\nDAEMON_OPTS=""\n' > /etc/default/puppet"

 

We deploy the keys and puppet configuration that we had properly saved from the previous, existing VM to the KVM host, in /var/lib/kvm/puppetkeys/yhamon-dev.goo.thehumanjourney.net - and copy everything to /root/puppet/ (reason given below). I believe this to be more secure than autosigning puppet certificates.

/var/lib/kvm/puppetkeys/yhamon-dev.goo.thehumanjourney.net$ cat files 
/var/lib/kvm/puppetkeys/yhamon-dev.goo.thehumanjourney.net/ssl/certs/yhamon-dev.goo.thehumanjourney.net.pem /root/puppet/ssl/certs/yhamon-dev.goo.thehumanjourney.net.pem
/var/lib/kvm/puppetkeys/yhamon-dev.goo.thehumanjourney.net/ssl/certs/ca.pem /root/puppet/ssl/certs/ca.pem
/var/lib/kvm/puppetkeys/yhamon-dev.goo.thehumanjourney.net/ssl/certificate_requests/yhamon-dev.goo.thehumanjourney.net.pem /root/puppet/ssl/certificate_requests/yhamon-dev.goo.thehumanjourney.net.pem
/var/lib/kvm/puppetkeys/yhamon-dev.goo.thehumanjourney.net/ssl/public_keys/yhamon-dev.goo.thehumanjourney.net.pem /root/puppet/ssl/public_keys/yhamon-dev.goo.thehumanjourney.net.pem
/var/lib/kvm/puppetkeys/yhamon-dev.goo.thehumanjourney.net/ssl/crl.pem /root/puppet/ssl/crl.pem
/var/lib/kvm/puppetkeys/yhamon-dev.goo.thehumanjourney.net/ssl/private_keys/yhamon-dev.goo.thehumanjourney.net.pem /root/puppet/ssl/private_keys/yhamon-dev.goo.thehumanjourney.net.pem
/var/lib/kvm/puppetkeys/yhamon-dev.goo.thehumanjourney.net/puppet.conf /root/puppet/puppet.conf

 

After the first boot, we copy the files to where they belong:

/var/lib/kvm$ cat kickstartpuppet.sh
#!/bin/bash

cp /root/puppet/puppet.conf /etc/puppet/puppet.conf
puppetd --test --debug
cp -r /root/puppet/ssl/* /var/lib/puppet/ssl/
puppetd

 

There is a ugly hack here that can probably be leveraged - the first time it runs, puppet seems to do all sort of magic, that involves moving the /etc/puppet/ssl/ folder to /var/lib/ssl/, and potentially setting up more configuration. This is why we copy our ssl configuration to /root/ first - puppet would otherwise override our files in /var/lib/ssl the first time it runs should we copy them to there directly. So we run puppet once, let it do its work, copy our files from /root/puppet/ssl to /var/lib/puppet/ssl, and restart it. Improvements to this part welcome...

Finally, in puppet, we need to define the configuration for yhamon-dev.goo.thehumanjourney.net:

node "yhamon-dev.goo.thehumanjourney.net" {
include baseclass_dev_vm
# Add anything here
}

There you go. Now if I want to recreate the virtual machine yhamon-dev, I just remove it (sudo rm -rf /var/lib/kvm/yhamon-dev/), and wait for a few minutes, no intervention required. Hope this will be helpful.

Am of course happy to answer any questions you might have regarding this setup... I'm not sure I've been that clear!



Comments:

Thanks for sharing, Yann, interesting read :)

Posted by Fabian Rodriguez on August 10, 2010 at 08:15 PM GMT+00:00 #

Post a Comment:
Comments are closed for this entry.