Troubles with KVM... Anyone got a version working?

Dear Planet Ubuntu, I've been having issues with KVM for several months now (running Ubuntu Hardy LTS). I was using kvm62, had many issues with it (non fontional SMP, issues with network drivers, ...), then I moved to the kvm84 and libvirt 0.6.1 backports, an still am experiencing many issues - the worst being regular crashes of virtual machines, which isn't exactly fun. I thought I had fixed it (my 32 bits vms were running on 64 bits kvm processors, which should work, though.. but they are now crashing with a 32 bits processor as well).  This is what is produced by munin when the VM crashes.

Dear planet readers and KVM users, what version of Ubuntu are you running with KVM, and with what version of KVM/libvirt? Are you happy with it? I am getting somewhat desperate

Blueprint plug: Within Launchpad use OpenStreetMap instead of Google

The new year is almost upon us, so I thought I'd plug one of my wishes for 2010, the use of OpenStreetMap within Launchpad rather than Google Maps. There's a blueprint here. 

This move would, I believe, be beneficial to both Launchpad and OpenStreetMap; the first gets the use of free mapping whilst the latter gets increased exposure. As far as I can tell, the aim of Launchpad is to provide the tools necessary for collaborating on software projects; OpenStreetMap provides the tools required to collaborate towards a free map of the world. I'm a firm believer that linking the two projects slightly closer together can only be a good thing.

Returning your Palm Pre to health with a Koala

So much has changed; Palm devices now use and support Linux, are cute and very, very neat (oh, and for iPhone users: among other things, the Pre can run more than one application at at time ).

But K/Ubuntu 9.10 did away with /etc/event.d, which means that the debs thoughtfully provided by Palm to facilitate access to your Pre from your Linux box don't work as advertised. So for any Pre owners out there using K/Ubuntu 9.10 or variants, who's just updated their Palm Pre to the shiny new WebOS 1.3, and had it go wrong (like a scary message involving www.palm.com/ROM) here's how to do sort it all out the easy way[1].

First, don't go straight for the excellently cross-platform solution offered by Palm, the WebOS Doctor; you will need it though, so if you have already got it up and running, no matter.

 You need to install the Palm SDK and Novacom tools, which you can get from here:

http://developer.palm.com/index.php?option=com_content&view=article&id=1585 (bottom of page)

Run the installer as directed, but ignore the errors caused by the Novacom install, which are caused by the above referenced deprecation of event.d. To get this vital element working you can just run sudo /opt/Palm/novacom/novacomd (thanks to http://zootlinux.blogspot.com/2009/11/installing-webos-sdk-in-ubuntu-910.html).

Now go for it on the  WebOS Doctor - if you haven't already got it, go here: http://www.palm.com/us/support/downloads/pre/recoverytool/

See now that was not so hard. But it would have been so much easier (particularly for the non-techies) if these things were in the Partner respository. How on earth do things get into there?

[1] Where easy means "not trawling around a whole bunch of websites first"

Koala's LTS

I know that Karmic isn't a LTS release, and that this has very little, if not nothing at all, to do with Ubuntu usage within a modern corporation, but I couldn't help think of the OS I'm using when I read this news.

 Australia's koalas could be wiped out within 30 years unless urgent action is taken to halt a decline in population, according to researchers.

"They say development, climate change and bushfires have all combined to send the numbers of wild koalas plummeting". Chlamydia too is cited as a reason for the decline. After the linux.conf.au attendees supported the Tasmanian Devil earlier in the year, is it time for the Ubuntu community to get behind Koalas?

I'm not too sure what we could do to protect our fussy eater friends from development, climate change, bushfires and chlamydia, however, so if anyone has any good ideas, please post them in the comments.

Making progress part 2: small successes all add up

So, a quick-ish follow up so my previous post doesn't come across as too negative, and to show that progress in this area can be made with just a little effort.

We are having significant success as a relatively small company using awareness-raising tactics with potential suppliers; conversations along the lines of "hey, we liked your product - but we couldn't make it work with Linux and we have a strategy that includes ensuring desktop Linux is an option in our future" are having an effect.

But it is arguably easier in the corporate IT world. Even a small company spends more money than an individual consumer; so at the very least the salesman who loses the sale gets the message. We have had very positive reactions from large companies, such as Dell, and from the most senior level, when raising concerns regarding cross-platform support.

Some areas are basically done with, such as printers. We've recently carried out a refresh of large multi-function devices (black and white and colour copier/printers) in a couple of our offices; it was part of the base requirements that the devices have cross-platform support, and were surprised to find that the selected supplier, Canon, not only had Linux support (despite some Internet postings to the contrary) but had it in the form of a full-featured GPL-licenced open source driver. There are few if any serious printer manufacturers now who do not provide and, as importantly, state Linux support.

Other areas are similarly done with in the sense that they work just fine e.g. digital cameras (we buy quite a few...), but the manufacturers are not admitting it in the documentation or on their websites. These are the fun ones from a purchasing perspective: "does this camera support Linux?" you ask innocently, knowing it is USB mass storage or PPTP; "I'm not sure, does it matter?" responds the salesperson. "Yes, it is a requirement for the purchase, could you find out?" you reply, struggling to keep a straight face as they start to look nervous...

It seems cruel, but there is a purpose. My knowing that the digital cameras, USB hard drive or ISP services, etc., will work just fine with Linux is not the same as it being there in black and white, with instructions (which are mostly very similar to those for the Mac, so it is not a huge cost to provide them) so the potential and not particularly technical Linux user can see that Linux is a real option. This is as true in enterprise decision making as it is for home users; you or I (I'm guessing you as the reader of an Ubuntu blog...) might know what things will work for sure, what should and what needs to be carefully selected (portable music players springs to mind - did I already mention the slimy demon that is Apple? :D ); you or I will know that just about any enterprise level hardware will be just fine, that any corporate function can be reproduced with a Linux platform, that Linux is a real option on the desktop and the server, but many of my peers at an IT decision-making level don't, as they lost contact with tech stuff a while back (and then some!).

Having sales and marketing functions listing Linux support for their hardware, no matter how distro-focused, no matter how qualified, puts it there in black and white "you can use Linux, you can use Linux". Drills the message in. Makes people comfortable with Linux. Makes the application software suppliers uncomfortable so they start considering porting to Java or platform neutral C/QT mixes, etc.. Makes the web-app providers list Linux as a client platform as well as a server platform. Then the FUD will start to really dissipate and proper informed choices made.

 So if you get involved with purchasing, or when buying something, hardware, software or services, for personal use, ask the question, no matter how obvious the answer; and when the answer is no when you know it to be yes, make sure to contact the manufacturer and ask why the information regarding Linux support is not made public, communicated to the sales channel, printed on the box. And if they say no, when it really is no, don't buy it. And make sure to tell them why you didn't buy it...

Making progress part 1: A little advice or a question?

I guess the question needs to be first: do you (the community and individual users) want Ubuntu to be a real alternative for typical home users? You know, not the equivalent of a kit car relative to the every day car of Windows or a Mac, but something the average user can acquire and use without needing technical support to get along on a month by month basis.

If your answer is no, I'm intrigued. Why not? You like it to be hard?Answers on a postcard to "Keep Linux off the desktop", the usual address...

If yes, then I have a little advice. Pages like this:

https://help.ubuntu.com/community/PortableDevices/iPhone

need to start differently. I'll pick on Apple because they have just taken two contented Ubuntu users back to Windows just through the purchase of an iPhone, much to my chagrin, but I guess there are other hardware suppliers out there who because of their ignorance require similar pages  to be up on the help.ubuntu site.

Ok. This is the issue. The first point that has to be made is that any instructions on this type of "how do I use this normal thing with Linux" has to start with a statement along these lines:

 The following instructions are a method for getting your newly purchased/acquired <insert name of gadget> to work with Linux. These instructions may look daunting and to anybody non-technical they probably are daunting; that these instructions have to exist at all is entirely the fault of <insert manufacturer's name>. It would be great if you could contact <insert manufacturers name> and make it clear that you believe they should provide support for Ubuntu and Linux in general as well as for Apple OS and Microsoft Windows.

Maybe you prefer an open solution to every hardware use. Me too, I prefer choice. But where an open solution is lacking, any (Linux) solution is better than no solution at all. The average consumer buys on impulse. They see a thing, they like  a thing, they buy a thing. Said thing should then, like nowadays most any printer, just work when they plug it into their home computer, regardless of OS. Sure, they might need to read a manual and install some software, but they should not have to hack at stuff.

And it is the manufacturers who are at fault, not those who choose or have acquired Linux, and they and the Linux users should understand that. Manufacturers like Samsung have taken to including Linux software and instructions with their printers; surely everyone else should be too?

I'd go further than just that change to the "how to hack my gadget so it works with Ubuntu" pages, and suggest a campaign targetted at Apple et al. where we make our feelings clear. Maybe they don't care. Maybe they shouldn't care. But why not see if we can't make them care? If one Canadian can make United Airlines care, surely millions of Ubuntu users can make Apple cry...

KVM84 in the starting blocks

Remember my post from March? Where, we're nearly there. Dustin announced a few days ago that he was expecting to push kvm84 in -updates next week. I've been beta-testing and chasing bugs on this for some time now, and I am pretty happy with this backport, and all the goodness it brings. So, a short list of features/bugfixes that I've noticed so far:

• Disk speed seems a lot better:

  yhamon@yhamon-dev:~$ time cp SunStudio12ml-solaris-sparc-200709-pkg.tar.bz2  tmp
  real	0m12.528s
  
  VS
  
  yhamon@mirror:~$ time cp SunStudio12ml-solaris-sparc-200709-pkg.tar.bz2 tmp
  real	0m20.159s
  

  yhamon-dev being a VM running on a kvm84 host, and the other one on a standard kvm-62. Both hosts are similar in specs and similarly loaded. I wonder what could trigger such a significant change (cache?), but I ran this cp several times, an every time kvm84 would be significantly faster...

• ACPI2 for windows guests: among other features, it means that windows guests will now be able to reboot "themselves". Until now, when triggering a reboot from a windows guest, it would just shut it down. Now that works fine, too.
• Proper SMP support. In KVM-62, SMP support was quite broken, it would use a lot of CPU on the host - the network would also regularly crash with SMP guests, leaving them without connectivity. Now this seems to work correctly; I've been running SMP windows and Linux guests for a while, and it seems quite stable.

Speaking of Windows guests (believe me, I'd be happier without): I think I found a better way to install those. Instead of using virt-install like I documented a while ago , it is much easier to directly manually create the QCOW2 file:

qemu-img create -f qcow2 windows.qcow2 12G

And to create manually the XML definition file (copy it from a template - don't forget to change UUID and MAC address), and boot it directly with virsh. Here is an example of a libvirt XML file that I use.

This works fine; I had many issues before with virt-install, where the VM just wouldn't restart after the disk had been formatted.

World's most detailed fail

Picked up by the BBC here:

Global Digital Elevation Map covers 99% of the Earth's surface, and will be free to download and use...

"This is the most complete, consistent global digital elevation data yet made available to the world," said Woody Turner, Nasa programme scientist on the Aster mission.

What does it look like? This is pretty exciting stuff! I quickly click to the website, eagerly anticipating the gigabytes of free data that I'll be able to enjoy!

Microsoft OLE DB Provider for ODBC Drivers error '80040e4d'

[Microsoft][ODBC Microsoft Access Driver] Too many client tasks.

/index.asp, line 3

The most complete, consistent global digital elevation data yet made available to the world, failing to be provided by a Microsoft Access ODBC driver. NASA put folks on the moon, then chose Access to deliver an enormous dataset via the Internet. Fail.

Screenshot proof here.

The 100 papercuts and the fleshwound

The 100 papercuts and the fleshwound.

I actually bought support because of this bug, and they fixed it for me by telling me to disable the wacom tablets in the xorg.conf file; I also share responsibility in this not being fixed yet as I didn't spend enough time to do bug tracking once I had a workaround. But someone recently blogged that important bugs that were not being given enough attention could be worth blogging... so... if any xorg developer is reading me...

UNR Jaunty (French) Mini 9

Oh, what a job I was asked to do, install Kubuntu 9.04 on a French Dell Mini 9; I could barely contain my excitement. I'm not a Kubuntu user, I much prefer Gnome over KDE, I speak only the most rudimentary French (although with a little help I recently explained to a contractor how I wanted our new French office cabled) and using an azerty keyboard makes my head and hands hurt.

Chris has recently gone through installing Kubuntu on a Mini 9 in another blog post. Tony's comments on that post, such as the problems connecting to wireless access points and the intrusiveness of KDE Wallet weren't things I wanted to reproduce myself on a tiny little screen. It's that kind of annoyance that made me ditch Kubuntu on my proper sized laptop some time back, there was no way I wanted to see that in miniature. And yes, I'll admit now, I've never installed and used KDE 4.2; I've had a brief play and have been shown its various merits and have had its technical superiority explained to me. Despite this, it still looks like it's going to get in my way, not like this lovely Gnome.

I'd been given my task however, install Kubuntu 9.04 on this little French computer.

A temporary fit of madness must have come over me whilst I perused the available downloads on ubuntu.com; before I could say what was going on I'd downloaded Ubuntu Netbook Remix and had installed usb-imagewriter on my full sized laptop. A quick bug report later and I had the image on a USB stick. I could perhaps file another bug - why does the shortcut to ImageWriter appear under Applications > Accessories whereas USB Startup Disk Creator appear under System > Administration? Couldn't these either appear in the same place or, preferably, be combined into a single application?

Installation is easy; as Chris describes you push 0 on boot, change the boot order, wait for the live image to load, click install, use the laptop until you're told that it's completed, restart and pull the USB stick out. Give that to a friend / colleague so that they can enjoy the UNR experience. Run a quick update and you're ready to go, even Firefox is installed by default!

Following Fabian Rodriguez's suggestion I installed the droid fonts (sudo apt-get install ttf-droid) and changed my settings to suit. I'm becoming a big Android fan, so having this little computer look even a little similar is a good thing. I also quickly installed openoffice.org-base and openoffice.org-sdbc-postgresql and connected to the database that my colleague is currently developing. Easy-peasy! As the French would say.

Ah yes, our friends the French... Coucou madame! So far I'd done all this whilst struggling with the azerty keyboard, however, all the menus, dialogues and applications remained in my native English. A quick trip to System > Administration > Language Support and I had downloaded the French language packs and chosen it as my defaults. I logged in and out again and was still in English, pig dogs, so I rebooted and found myself in a whole world of computing en francais.

I think I'll give this computer to one of our French staff now and ask them what they think of it. It doesn't mean too much to me any more...

Canonical support

Let's introduce a subject that I've rarely seen discussed on planets or forums: Canonical paid-for support. At Oxford Archaeology we have been paying customers for almost a year, and I think it is a good time to look back and see if it was worth it, what worked well, and what could be improved.

Why we chose to buy support

The IT insfrastructure has, over the years, become a critical element in a company like ours. Systems that handle the payslips, the websites, the databases, are very costly when they are not working as expected. You may hire the best sysadmin ever, there will always be the case where a problem/bug occurs that he is plainly unable to solve - and where getting help from the community just takes an unacceptable amount of time.

We purchased support from Canonical to ensure that our road will never be blocked by an unsurmountable bug - and to know that we have a company guaranteeing that the software we deploy will actually work as expected, and that commits itself to fixing it if it doesn't.

How we use Canonical support

I tend to use Canonical support as a very last resort when I really don't know how to solve my problem anymore, or when I have a critical bug that requires a quick fix. This happens fairly rarely, as the support from the community is pretty good - I opened five support cases in the last year.

The support portal for Canonical is http://landscape.canonical.com ; so landscape is not only a monitoring solution, it is also the whole support portal. It integrates salesforce.com for the management of support tickets. Here is what it looks like:

One good point in it is: you do not have to use landscape (the monitoring system) to be supported. We already have a monitoring solution (Munin + Nagios) - and frankly both externalizing our infrastructure and using proprietary services go completely against the very own reason we chose Ubuntu in a first place. Ubuntu gives us the ability to manage our infrastructure ourselves and with free software, and we are happy with it that way.

Quality and responsiveness of the support

The support quality is pretty good. I usually get an answer the next day - the people there are friendly and qualified. Not-so-basic questions on how to use the systems are answered quickly and in a professional manner.

More complex problems (ie: that require a patch / a new version of a package to be pushed) can take more time, but this is normal. I remember encountering a bug affecting apache; I had someone from the support browsing the apache mailing lists to find the correct patch, and making a patched package available to me on a PPA. This was definitely vey much appreciated - the bug was severe, and I wouldn't have been able to fix it myself.

Certified hardware

Now what this actually means remains a mystery to me. What does it mean to run Ubuntu on a "certified server"? Is it a requirement for a server to be supported? Does it mean that Ubuntu will run on it? Does it mean that 100% of the hardware the server contains will work, with opensource drivers? Does it mean that 100% of the hardware will work, but eventually with proprietary drivers? And what about the servers that are configurable, and where you can chose from several RAID cards for example - does it mean all the configurations available will work?

The state of the art here is a lot uglier than what it could be. Take for example the Sun X4100 (from which we bought a few at the time because they were supported):

• The Sun X4100 is certified for use with Ubuntu Dapper 6.06
• but its raid card doesn't work with it - and that has never been fixed... although dapper is still supported.

I think we all remember too well the mess Microsoft got in with its "Vista ready" logo. We should learn from that and not do the same mistakes... If a server is certified, it means it has been tested - it would be really helpful to know what exactly has been tested, and to which extent Canonical is committed to getting Ubuntu to work on it .

Supported software

It is also quite unclear to which extent Canonical will support supported packages. Imagine a software you use has a feature that is badly broken - and that is fixed only 3 or 4 versions of the software later... For example for kvm (yes, I already mentionned that in the past ) - what features of KVM is Canonical committed to support?

• Guest multi-processor support?
• Live migration?
• Paravirtualized network drivers?

All of this is in main, so officially supported. On the other hand I knew when I deployed that some of these features would not work... it is still unclear to me what I can reasonnably expect them to fix. I bet KVM isn't the only package with this issue?

It may sound painful but I think it would be a good idea to explicitly mention in all the packages in main (in the README, or any other text file in the package) which features of the package are *not* supported.

Level of commitment?

Although this never happened to me so far - what if Canonical encountered an issue it is unable to fix? Is Canonical at the very top level of the support chain, or would they eventually hire the services of a company that could fix it, if they reckon they are not able to fix it themselves? I doubt it - but it could be an interesting premium service (just a hint)

Conclusion

I am globally very satisfied with the level of support I am getting - I know I have a bunch of very capable people who would help me whenever I got a question, or who would spend time fixing a bug for me - which is something I may sometimes not be able to do myself. By paying these people to do that I also have the feeling that we contribute (in our own microscropic way) to making Ubuntu a better product, for us, and for all other Ubuntu users.

There are loads of companies out there who are using Ubuntu without paying for support - I hope they will understand that by paying for support, they will allow Canonical to hire more packagers and support staff, which will directly lead to a better product, saving them money in the end.

How much do Canonical job offers tell us about Ubuntu?

I used to look a lot at the job offers on Ubuntu.com, as many times the job offers from a company tell a lot about the direction the company is going. It uses to be very interesting, as you can see how much effort is put into launchpad, Ubuntu mobile, or sometimes even learn about new projects before they get announced (in fact I even found my current job there, as companies proposing jobs related to Ubuntu are allowed to advertise there).

However, I was a bit saddened that a very large part of these jobs were non-technical or not really benefiting the community: business development, sales consultant, system administrators for Canonical's servers, launchpad developer, support, ... it was fueling the idea that some other communities criticized in the past that Canonical was only packaging and selling other people's work, without creating much added value. I am a pretty strong opponent of launchpad and landscape closed-sourceness myself...

So, I was really thrilled to see that Canonical was now hiring a "Desktop Architect - Network experience" person, and a "Desktop Architect – Sound Experience" person. Add this to the few offers from October which sadly are still there (Gnome developper, OpenGL developper), and it seems to me that Canonical finally decided to pass the second gear.

I was a bit afraid to see Canonical go in so many directions (Ubuntu mobile, ARM support, Ubuntu netbook remix, ubuntu server) - so I must say I am very happy to see that Canonical is still committed to providing the best Desktop experience, ever

I wouldn't go as far as saying that it is related to Ayatana, but who knows... Really looking forward to what will come out of all this now! A working network manager, anyone?

Kubuntu Jaunty Mini 9

Ok, not original, but it took me a few trawls around to make all this work, then it worked so easy I thought I would share - these instructions should work for most netbooks (except the bits that say "Mini 9" of course) and other computers (except the bits that are about netbooks...).

Install process

First backup your data - actually irrelevant to my tests as these were brand new out of the box machines for corporate use, but I just like to remind people

Download ISO of Kubuntu Jaunty from the Kubuntu download page. At time of writing, this was actually the beta which was downloaded from the Kubuntu beta page.

Install usb-creator onto your current Kubuntu install (available in Kubuntu 8.10 for sure, not sure about earlier). This excellent application simplifies the creation of bootable USB flash drives to the point I could do it slightly hungover...

Plug decent-ish USB flash drive with 1GB of free space or so into your computer.

Run usb-creator, entering sudo password when prompted; select downloaded ISO and target USB flash drive. Press the Make Startup Disk button.

Wait. Not very long...

Now (re)boot target Mini 9 computer with USB key in it and either modify the BIOS to make USB its first boot device or press "0" during the splash screen to get to boot options menu and select USB from there.

Go through install setup process as suits. Click the button to complete.

Wait. Not very long at all...

Install completes in under 10 minutes. Reboot.

Wait. Not very long at all, at all...

Repeat as necessary (from the (re)boot target step just above).

Aftertweaks
Create your very own Kubuntu NBR by going to system settings and setting all the fonts to small. If likely to use Konsole, configure its fonts to small too.

Add Firefox (sorry guys, but it really should be in there by default. Trust me...).

Open mixer window and ensure speaker volume is turned up.

Done. Enjoy

Things that remain annoying
(i) Getting NetworkManager to use a Bluetooth connected mobile as a modem. This I'm told worked easily with the rather lovely BlueMan and previous versions of NetworkManager, but no longer does, As these netbooks are intended for non-expert users, I really need to get this sorted in a click GUI sort of way. A fix has been committed, I understand, hopefully it will be packaged up shortly.
Update 090423: yes, it does work very easily with Blueman and the older KNetworkManager (after hours of struggling after the rfcomm0 net interface started appearing in the network device list when you used Blueman to connect the dialup servive of the phone...). See bug https://bugs.launchpad.net/ubuntu/+source/plasma-widget-network-manager/+bug/334122 for latest on this, but for now remove the plasma-widget and install KNetworkManager. Now I just need to figure out how to make the older application start automatically...

(ii) Be damned if I can figure out how to make the edge of touchpad work for scrolling. Come to think of it, not sure the edge of the touchpad worked for scrolling with the Dell build either...
Update 090409 - just noticed this is now working, I guess an X update or something...

(iii) The launcher doesn't put the focus into the Search box on the first one I built, did the second, which then stopped doing it again...
Update 090415 - after this morning's update this is working as expected

(iv) be nice if the launcher could be scaled so that the "Leave" section displays all its content without scrolling (picky, I know), although now I've got used to just letting it suspend to RAM by closing the lid and not worrying about it.

Any of these issues may of course be resolved when using the release version rather than the beta or by some clever fellow providing a nifty solution as a comment below. I'll update the post as things progress!

Evolution is better than revolution - unless it aint

We have precious few Linux flame wars here at Oxford Archaeology; on the whole we're just all too nice and well meaning. There's one exception, however, and it's a classic - the good old KDE versus Gnome flame war stalwart.

I'm a Gnome user; I tried KDE for a whole year, but came back to the correct path. I don't think Chris was convinced by my argument (check out the only comment I got), but he's my boss, so I won't push it too far.

Some time has passed since then, although recently Chris penned a popular entry on this blog Evolution is better than revolution. Aha! Admission that the Gnome way was the way forward; nobody likes those ugly KDE releases that change to very different, yet still ugly, KDE releases. Everyone likes to keep it slow and considered. Mellow.

Slashdot picked up on the same theme, discussing Bruce Byfield's article on the "evolutionary advantage"; evolve rather than revolt by starting afresh and ditching support for all the old cruft in your software. It seems that Gnome are now going down the same route, as reported by El Reg.

So, evolution is better than revolution, unless it isn't? Ubuntu has a good core that can be evolved gradually whereas other projects need to be "evolved" in a slightly more revolutionary way? Frankly I'm just pleased that with Gnome 3.0 expected sometime in the future, there's really no reason to consider that awful KDE again

Flames in the comments please!

KVM 84 backported in Hardy

Dustin Kirkland announced a few weeks ago that he was trying to backport KVM-84 to Ubuntu Hardy. This was made following a post on the ubuntuserver blog that described the way KVM-84 would be backported: ~ubuntu-virt PPA -> hardy-backports -> hardy-{proposed|security} -> hardy-updates.
This raises some interesting questions. But let's define the context.

At OA, we've been using KVM in production since the very beginning - we started testing KVM in april 2008, deployed in late may, and started to move critical pieces of infrastructure on it later in the year. Some bugs surfaced after a while: problems with virtio networking, problems with the default NIC as well, performance problems with SMP, networking problems with SMP, need to use QCOW2 for windows VMs (although virt-install doesn't create QCOW images by default)... We managed to workaround most of them, mostly by using e1000 as NIC and living with the SMP slowness. We reported these bugs in launchpad and landscape, and Canonical had quite a tough time helping us with these issues.

In Ubuntu Hardy, the version of KVM used is kvm-62. The current version of KVM is KVM-84 (if not newer already) - and as you can see on the mailing list the development is going on really quickly. KVM is an opensource project created by QUMRANET and now belonging to Redhat. So, when you arrive on the mailing list or the IRC channel saying "I am running a one year old version on a concurrent distro and there is a bug", you'd be lucky to get help from a KVM dev (ie, a redhat engineer).

This is made worse by the development model, which is somewhat weird, as afaik there is no "stable" version (or only stable versions, pick the one you like). New versions get released all the time, fixing bugs in the previous versions. This is making the life of Ubuntu packagers hard, as they need to backport security and critical functionality bugs.

To summarize: It is extremely hard for Canonical to support packages for longer than the upstream would support them. There is no chance for Canonical to support KVM-62 in 3 years time, as some bugs may be fixed so much later that the code would have changed significantly, making the patches not that easy to apply (I am also still wondering how Canonical plans to support PHP4 in Dapper in 2 years time).

Backporting KVM-84 is a truly significant move, that acknowledges the lack of control on the supported packages, but also that confirms the commitment of Canonical to support packages during the whole support period. As a customer we are thrilled to see that. I think it is also the first time Ubuntu will push a new version of such an important package in -upgrades, that introduces so many new changes (but I may be wrong on that). It is quite an interesting precedent if you ask me

Now KVM is at the core of our infrastructure and I just can't afford to upgrade and experience new, significant bugs. I will be helping kirkland in testing kvm-84, reporting as many issues as I can, but I must say I am quite eager (and scared at the same time) to see how Canonical will handle quality assurance on this. I really hope they get it right :]

Announcing our competition winner

Last weekend I asked if anyone could come up with a name for the animal bone recording system we are launching; we were "crowd sourcing", which I think is Web 2.0 speak for "getting others to do it". Still, suggestions poured in and OA Digital staff have been getting a little bit democratic, a little bit creative and a little bit etymological; results are ready...

The winning entry was suggested just two minutes after the competition opened, leaving me feeling a little like ITV. Regardless, Jeremy Ottevanger wins it with zooOS.

From our new, ink's still wet, Launchpad project:

zooOS (pronounced Zeus) is an open source system for recording and analysing animal bones found during archaeological excavations. The project builds on the work done in the York recording system and by English Heritage in extending it. Main technologies are PHP/JS and PostgreSQL.

https://launchpad.net/zooos

zooOS then, a word with mixed etymology meaning animal bone whilst highlighting the importance of Open Source. Sounds like a Greek god too. Well done Jeremy.

Schuyler Erle's Ossuary was a high scoring contender (I think we're going to use that one for something else); I personally also liked ryts' FOSS-il, Sean Gillies' SQLETONS, and Skeletor, as suggested by both Jody Garnett and Benjamin Kay. Benjamin also came up with BONEDb, which made me laugh; RafaMJ's Zoonecropenis was certainly inventive, but sadly not a winner...

If you're interested in bones (who isn't?) and like Open Source Software (again, who doesn't?), then come by our Launchpad site and see how we're doing. Feel free to get involved! Thanks to all who entered, I look forward to some exciting stuff from zooOS.

We need your help

We're starting work on an Open Source animal bones database for use by Archaeologists (and others?) - it's going up on our Open Archaeology Launchpad site, but we've hit a snag - what do we call it?

Chris has emailed about it, Jo has blogged about it, but we're still without a name.

What would you call an Open Source animal bones database that will be finding use all over the world? Immortality is the prize awarded to the winner, there'll be no losers.

Evolution is better than revolution

I originally posted a version of this as a comment on this ZDNet article, but it seemed something that deserved wider exposure than a ZDNet comment...

In so many things. Life. Cultures. Political systems. Motorcycles. And of course operating systems and the applications that run on them. In all these and others, evolution is almost always better than revolution.

Desktop and server alike, actually we (the users, both home and professional) don't want change that is big enough to be unsettling and difficult to deal with. We want improvements. We want the niggles ironed out.[1] We want things to just get a little bit better over time and each upgrade. And like evolutionary pressures everywhere, what is seen as better will tend to change over time.

We want upgrades to be something you just do in your own time without any major concerns, without trepidation. Particularly in an organisational context; evolutionary change between software versions means no huge planning overheads, far less rigorous testing requirements and far less stress for all concerned. Upgrades should be regular and be more like patches rolled into service packs bundled up together, but with no new features held back for nefarious marketing purposes.

In short, we want the development cycle of Ubuntu and OpenOffice, Firefox and (now it seems) KDE. All of us would prefer it, even those still stuck with legacy systems that don't follow such comforting behaviour.

New versions of software and operating systems being revolutionary, a big step-change, is a mindset from the legacy proprietary thinking. We (who have moved to the new way) don't need to be "sold" the upgrade because we don't have to pay for it. Incremental change, more often, is far more comfortable for all involved.

So in fact open source is more people-friendly and better for organisations, even in its development paradigm...

[1] Like why does the shell in Ubuntu still not default to using Page Up pattern matching from history on partial commands? Costs nothing to put it in AFAIK as it is just a config setting. But is incredibly useful.

Edit: markup fixed

KVM - The informations you may be missing

Good morning Ubuntu, we are together today to try to help people who want to deploy KVM. Ubuntu Hardy being the latest LTS version of Ubuntu, it is a good choice for people like me who don't want to spend time upgrading every 6 months. On the other hand, there are some issues with KVM on it - so let's work around them!

Short howtos

You should use the e1000 network NIC (really)

Using the default network NIC, I have noticed tiny but highly annoying ethernet errors. Run a ifconfig -a:

RX packets:50916071 errors:24 dropped:27 overruns:0 frame:0
TX packets:4185949 errors:0 dropped:0 overruns:0 carrier:0 

Oups! That may be the reason why the md5 of some of the isos I copy sometimes don't match. Sooo - I moved to virtio for my network. With not so much more success - virtio seems very unhappy with large file transfers, and the driver would just crash, leaving the VM without connectivity. Not great - luckily the e1000 seems to work fine so far.

So, run "virsh dumpxml virtualmachine" - copy the result and paste it into a new file, the name doesn't really matter. Open that file and look for the section with the interface, where you will add the "model" line:

    <interface type='bridge'>
      <mac address='00:16:3e:00:50:19'/>
      <source bridge='br0'/>
      <model type='e1000'/>
    </interface>

Now, run "virsh define thefile" and restart the VM. Should be much better

The VMs don't start automatically when the host starts. But they could!

A trick found thanks to virt-manager and to the help of people on #ubuntu-virt. Go to /etc/libvirt/qemu/, and there create a folder called "autostart". There, make a softlink to the XML files of the vms you want to start automatically:

cd /etc/libvirt/qemu/
sudo mkdir autostart
sudo chmod 755 autostart
cd autostart
sudo ln -s /etc/libvirt/qemu/virtualmachine.xml virtualmachine.xml

And that should do the job.

Get Windows XP/2000 to run

Slightly tricky. Forums are full of questions about that... Use the Qcow2 file format! Sadly virt-install doesn't have any way to specify the file format (at least in ubuntu Hardy) - so let's start with a normal creation of a VM:

cd /var/lib/kvm/
sudo virt-install --connect qemu:///system -n vm -r 1024 -f vm/windows -s 20 -c /home/yann/win2Kserver.iso --vnc --noautoconsole --os-type windows --os-variant win2k

Then, we quickly stop the VM while it boots *cough* : virsh destroy vm - and convert the image to qcow2:

cd /path/to/your/image 
sudo qemu-img convert windows -o qcow2 windows.qcow2

There are also some tweaks we will need to do to the XML definition of the file. So, as previously, run "virsh dumpxml virtualmachine" and paste the result in a file. There, update the path of the disk:

<source file="/path/to/your/vm/windows.qcow2" />

Update the network NIC to e1000 (see previous section), and add a link to the cdrom (windows reboots after the formatting part, and can't find its cdrom anymore as virt-install attachs the cdrom only for the first boot).

      [...]
      <target dev='hda' bus='ide'/>
    </disk>
    <disk type='file' device='cdrom'>
      <source file='//home/yann/win2Kserver.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
    </disk>

"virsh define file", and "virsh start vm". You should now be able to install windows XP and 2000 (didn't test other versions) without problems!

And some news...

Spice-up your remote desktop

Sorry for the ugly title So, at the moment, we use VNC to connect to our KVM virtual machines. Let's admit it: VNC's performances suck. When Qumranet developped KVM, they also developped a proprietary alternative to VNC called SPICE. The interesting bit is that now that Redhat has bought Qumranet, they want to make it opensource, and apparently it's planned to happen pretty soon. Here is the interesting interview. I've been looking at alternatives to VNC for months, and beyond NX and its single implementation, I've not been very successful - so huge hopes from me on that side.

Some more links for those with real interest

Shamelessly copied from an interesting article on linuxfr.org:

1. Red Hat Sets Its Virtualization Agenda
2. Red Hat Announces Broad ISV Ecosystem is Virtualization-Ready

And Ubuntu?

As Redhat is owning KVM, they are the best place to look for shiny news... But regarding Ubuntu, I heard the current buzzwords were Eucalyptus and Open nebula. Haven't really had time to try any of these yet - would be incredibly grateful for feedback on them

That's all for today - as a strong KVM supported I'd be happy to reply to questions and queries related to it in the comments. Thanks for reading

Zimbra - Presentation, example of an "anonymous mailing list"

Hello, today we are going to talk about Zimbra. This post is the result of my overall satisfaction with it - I hope it won't be considered as an attempt to advertise for them hoping for special favours in return. People who know me know I really like their product, and I believe the people there really deserve some praising.

Introduction

So, what is Zimbra? Zimbra is a complete, multi-platform, easy to install, scalable mail and collaboration software. It is based on opensource software, including mysql, jetty, mailman, spamassassin, postfix, Java, amavis, openldap .. Zimbra bundles all these together, and includes a nice web-2.0-ish web interface to manage everything. Among the features you will find:

• Antivirus/Antispam
• Shared calendars
• Shared task lists
• An integrated jabber server
• A very good and fast search (everything is indexed)
• Shared email folders
• Tags for emails
• Integration with LDAP/Active Directory
• Static HTML and mobile phone interfaces
• POP(s) and IMAP(s) access

Zimbra at OA

The good

At Oxford Archaeology we have gradually moved over the past year from Exchange to Zimbra ZCS 4, and then 5. This involved the transfer of 350 people's mailbox, PSTs, their training, mailing lists, and so on. Thanks to the Exchange import and PST import tools, this went rather smoothly. I just had to write some homemade awk scripts to bulk-import the users and mailing lists, but I think this is now a feature of Zimbra 5.

Before, people in our main offices in Oxford would use Outlook, people in our branch offices Outlook Web access (which really sucked, to be fair, and only worked properly in internet explorer). Now people would access their email in the same way, from the main office, a branch office, or from home; and this is a huge step forward.

On a whole, I must say I am very impressed by their web interface and the features it proposes (you can give it a go here, just press skip registration). More than the product itself, I really like the way the project is managed. Several staff spend time on the forums and provide free support to non paying customers; this is great. The bugzilla is public, and as a customer you can join a support ticket to a bug/feature request, therefore giving it more weight. The support is quick, and even with the lowest level of support you are entitled for 24/7 phone support in case of major breakdown. The quality insurance of the product is very good; bugs do happen, but you can usually live with them, and if serious they are treated with the appropriate speed.

The bad... and the ugly

My biggest concern right now is the fact that Zimbra has been bought by Yahoo. Things have got a lot worse since that happened. Yahoo started over-branding the zimbra interface with their crappy logos, pushed an ugly yahoo search nobody wanted, installed yahoo zimlets by default. The Zimbra Desktop is now Yahoo Zimbra Desktop. The previous conception was that Zimbra would never propose Zimbra hosting, and instead let a pool of partners propose their own solutions instead; these have all been screwed now.

Then it makes me unsure about the free will of the developers, who now uses the YUI libs for ajax development; was this because it was the best option? To this, add the shadow of a purchase by microsoft or AOL - who may not have the best interest in keeping it running as it is. Finally, Zimbra licensing sucks (even the opensource is ad-licensed - you have to keep the original logo). I really hope Yahoo will understand that it is a bad idea to use paid-for Zimbra customers to push their unsuccessful products. That was for the ugly part of Zimbra.Zimbra's power by the example: an anonymous mailing list

Ok let me define what I mean by "anonymous" mailing list. I had the request recently to create a new email address. Several people should be able to read the emails sent to that email address, and to reply to these emails. Mail sent as a reply should appear to be coming from that "mailing list" and never display nor contain the name of the actual sender.

How it used to work

This used to be implemented as a full account, with a password shared among all the person who would have to use it. This is bad because:

• Well, first I need to pay for a new account (you pay for a certain number of mailboxes).
• Someone who would leave the company in bad terms would still be able to log into it, unless you change the password every time someone leaves (regarding the number of mailing lists, this is not really possible).
• The user needs to log into two different mailboxes, and remember more passwords.

The Zimbra way

Let me try to explain how you could do this with Zimbra. Good luck setting this up without weeks of hassle on any other system . This is just ONE way to do this and may not be the best; I found it to be quite flexible.

First, let's create a new user account, which we will call "mailinglists@domain". Let's say the address of the mailing list we want to create is anon@domain. We create an alias "anon@domain" for the mailing list mailinglist@domain.

We do not define a password for the user account mailinglists@domain - and use the admin interface to "su -" into it. In that account, we create a new folder called "anon". We then create a new filter to file any incoming email, addressed to anon@domain, into the folder "anon".

We then right click on that folder, and select "share with". Now you add all the persons that should be able to read these emails.

This is for the first step; now the persons who are able to read these emails need to be able to reply using anon@domain as sender. In the admin interface, go to the profile of a user that should be able to send emails as anon@domain; go to the preferences tab, and "Sending mail" section, at "Allow sending mail only from these " add "anon@domain".

Then, use the admin interface to "su -" to the user account, and go into the preferences tab, accounts sub-tab. Select "Add new Persona", and configure it like this:

Now save. You're done! When sending an email, the user will have a new drop-down list that will allow him to select with which address he wants to send. Following this method will allow you to use only a single zimbra license for all mailing lists, allow any new person to have access to all the archives, and not tie the mailing list to any particular account.

Zimbra and Ubuntu

Ubuntu has been a supported platform for Zimbra since Ubuntu 6.06. Ubuntu 8.04 has become fully supported as well a few months after the release of 8.04 (right, we pushed a bit :P). We believe Ubuntu 8.04 is a platform of choice for Zimbra - and hope Zimbra will continue to support it in the future.

Edit: title typo fixed