Report from the virtualization workshop at Linuxhotel, Essen/Germany

Last weekend, we had a fantastic virtualization hacking workshop at the Linuxhotel in Essen/Germany.

Here's my report on what I and others did there:

I built a FAI USB Stick for Installation of Debian Etch base systems to test Ganeti installation from Scratch

I did an upgrade of an installed Debian Etch system to sid, to be able to test the latest version of libvirt. Here, I realized, that there are some problems with the upgrade from Etch to sid, started again from a newly installed system, and realized that the Etch-Lenny upgrade didn't work either. I know, I have to try it again, from the start on, and create a proper bug report, so this can be fixed, but didn't come that far yet.

In the meantime, I did also a proper Lenny Install with Lenny D-I beta2, which worked well. I wanted to use this VM to test libvirt with qemu inside. Here I realized a bug in Ubuntu's KVM packages, which I reported. While doing so, I realized bug reporting in launchpad is not always obvious and intuitive (if you chose kvm at the very start, you cannot report bugs - you have to chose ubuntu first, and then go to kvm - logical once you know it, as it's about kvm package in Ubuntu, not upstream, but hard to find out once you're in that trap)

And Ubuntu's reportbug has a bug which leads to not being able to report bugs - see https://bugs.launchpad.net/ubuntu/+source/reportbug/+bug/44116

Until that weekend, me, and many others, thought there is no working XenLinux patch for Kernels >=2.6.24, but another participant showed me there is one in Ubuntu. The word goes it is not a solution for dom0 with paravirt_ops, but "only" a forward port of the XenLinux patches from XenSource/Citrix, which they themselves do not release for newer versions than 2.6.18.

I watched, with great joy, how Oliver and Michael did set up a live migration with KVM - which even worked from a 32 bit Intel System to a 64 bit AMD system, easily - admittedly, once on 64 bit, the system wouldn't migrate back again, still impressive.

Then, I realized, that in opposition to earlier tests, suspend to RAM, and especially waking upo after that, did not work too properly on Ubuntu, once I had some VM's running.

Later I discovered, the system works much more stable if I don't use the -rt kernel - it seems that as virtualbox, and kvm virtualization make the system unstable when used in conjunction with thet -rt kernel - I have to report a bug about that.

I showed Ingo how to use xen-tools and FAI combined on his rootserver to set up Xen guests.

I worked quite some time on Ganeti 1.2.5 Installations from sources, on Debian Etch with backports, and was a bit disappointed - it didn't work so easy and out of the box as I had it when I played with it the last time. I had some troubles with ssh keys as well for setting up a new instance, and for adding a new node to the cluster. While I got an instance running, I never got a second node to the cluster. Probably this hint will help: http://code.google.com/p/ganeti/wiki/ClusterKeysReplacement . One problem with Ganeti I realized: it seems the project never got proper traction among non-google people. For user, that means, there is not much community support, basically, if you have trouble outside google office times, your out of luck. Against all odds, I created a more minimal install howto, with comments in German: http://code.google.com/p/ganeti/wiki/GanetiInstallGermanShort

Stuff the others did(from the final status meeting):

Oli did test some Xen stuff on Ubuntu, and it still has some rough edges.

Peter did test the newly free of cost available VMWare ESX and played around with Xen stuff

Matthias wanted to get Libvirt, KVM and Redhat Cluster Manager running nicely together - Which was not too successful. There where still problems with fencing that could not be solved - machines did shut down their counterparts in the node for unclear reasons. Apart from that he helperd Thomas with OpenQRM packaging for Debian.

Guido worked on USB support for libvirt, and could send a patch upstream - even with support for handing over files on the host as USB devices on the guests.

Frank di work on getting D-I, he Debian Installer, running in a paravirtual Xen guest. That worked quite nice, but there where troubles getting the resulting system booted with libvirt and pygrub - it worked when using ext2 instead of ext3 as filesystem...

Florian also tested Xen on Ubuntu - which lead to quite some trouble, especially in conjunction with an Nvidia Graphics adapter. Apart from that, he worked on building Debian packages of VMWare for his infrastructure, for easier installation.

A Hint from Oli for the graphics problem on Ubuntu/Xen/Nvidia: disable restricted drivers before installing and booting Xen - that works much better.

Jacqueline had similar problems with Ubuntu and graphic cards, but finally got it solved, unfortunately the solution wasn't easily reproducible. So Oli's hints seem,s to be working fine. She also helped with the packaging stuff.

Thomas worked mainly on the OpenQRM packages for Debian, being supported by many useful hints from knowledgeable Debian Developer's Guido and Matthias. He learned a lot about proper Deb packaging that way.

Ingo got Xen guests on his rootserver set up with xen-tools and FAI, and got into the complex topic of writing your own vif-script(s) for Xen - so he can have specific vif rules for each vm being started - for example leading to have very clean firewall rules which are only active when the vm needing these rules is really up, and being remove when it is shut down.

Interesting, great proposal from Guido: We had a meeting with a bunch of quite knowledgeable people, and tested the edges of the usefulness of open source virtualization for real production, and high performance, well-managed usage. In virtualization, there are some things that work, and some that don't, and often enough it's a matter of having the right combination of Hardware, base OS virtualization technology, management tools for your usage scenario at hand. We should try to somehow preserve, document, and advance this know how, because then, not everybody has to try out each possible combination of the above to see where and how his own usage wishes can be fulfilled.

What could be the next steps to go from here, especially regarding the last idea:

  • Set up a wiki (or other tools, e.g. for structured storage and display of evaluation and testing criteria and results)
  • Create a mailinglist for further comunication between the participants
  • Organize and encourage concerted, targeted and documented Testing
All that should be open source centric, and vendor neutral.

The next steps would be, to decide for a place to host that stuff (jpberlin came to my mind, otherwise probably a small VPS at hosteurope?), and a name for the initiative(mostly just to have a nice URL to be reached).
How about "vm-knowledge-center.org" - (hmm, too long, most likely)?