Last night knife stopped responding, so we rebooted it this morning and the domUs running on it, and the downtime was 12 hours. On the serial console, we couldn't log in to the dom0 and the dom0 kernel didn't respond to a break signal for the magic sysrq. The hypervisor responded to its escape code (crtl-a 3 times), and we were able to reboot the system from the hypervisor.
We don't know yet what caused the problem, but everyone on knife will get a free month. We are also planning to improve our monitoring system with nagios, and maybe a pager system to notify us more effectively.
see http://book.xen.prgmr.com/mediawiki/index.php/Peth0:_too_many_iterations_%286%29_in_nv_nic_irq_rx. for details.
so I've decided what I'm going to do about the larger domains.
The problem is that people get a 4GiB domain and then are all disappointed, because I/O sucks. The thing is, Virtualization sucks for storage. If I have a mirror, and I run 8 4GiB guests on it, when there is contention for disk, each domain gets a lot less than 1/8th the disk performance they would get if they had a disk to themselves; you see, SATA disk is pretty good for sequential transfers, but it sucks for random access. If you have 8 guys trying to sequentially access the disk, guess what? all that sequential access is now random, in an attempt to service each request in a reasonable amount of time.
(now, I/O also sucks for smaller domains, but people don't complain as much there. I mean, you are paying $8 a month, what do you want? If you get enough ram, the thing ends up being reasonably usable once you warm up the cache.)
So, my plan for the 4GiB domains is to jack up the prices (I plan on staying well below slicehost and linode prices, still) and giving you your own mirror. Current pricing plan is something like the current price ($68/month) plus $0.10 per gigabyte of disk (so around $100 for a 1TB disk) with the caveat that you must buy the entire disk. (you would get a mirror, not an individual disk, so I think $.10 per gigabyte month is a reasonably good deal.)
So, I have a bunch of disk laying about on many of my servers, sitting idle; it is possible I will use some of them as 4TiB or 8TiB guests. It is also possible I'll just start buying servers with 16 drive bays and going at it that way. (maybe with the 500GB 2.5" disks? 500GB is a nicer disk size and disk cost than making people buy a whole TB and a half, which are what the disks I have now are.)
note, as the title implies, these are 'forward looking statements' - I have not tested this out; I have not shaken out the bugs. It's possible I'll never get around to doing this. It's even more likely that I'll run out of free disks in short order and it will be a while before I accept new larger orders.
we've got to replace a disk.
update: it is coming back up now, restoring domains. those of you running debian might need another reboot (I've had problems with debian save/restore)
update: it looks like all domains but one came back up successfully (with a save/restore, no reboot)
we'll be rebuilding the RAID now, so expect disk performance to suck for a while.
Update 2010-03-11 02:28 PST:
wow, disk is sucking a lot more than we expected. I am wondering if the remaining disk also has problems. Either way, we've gotta wait for it to finish rebuilding, and that will take on the order of another 24 hours, according to my best estimate. All users of horn will receive a month worth of credit.
in order to replace a bad disk