Xen hosting: Lessons from the Trenches

Hey, make sure you disable tso and gso in your guest

2013-05-23T22:03:30Z

so yeah, srn found a bug; the NIC offloading stuff has always not worked properly for virtual guests... but with the latest RHEL/CentOS kernel it's gone from 'you drop a few packets every now and then' to "takes down your guest entirely if you send just one packet"

So yeah, uh, we'll change the starting image to add

ethtool -K eth0 tso off gso off

to /etc/rc.local. Please do the same on your guest.

details from srn:

4 separate domu's have been seeing an instance of this bug - probably more will do so as they upgrade:

http://xen.crc.id.au/bugs/view.php?id=3

This behavior on the dom0 side (disconnecting when it sees a packet that is too large) was introduced in 2.6.18-348.4.1.el5. It is not present in .6.18-348.3.1.el5. It is still present in 2.6.18-348.6.1.el5 (latest.)

40 of our servers have 2.6.18-348.4.1.el5.

There is a bug fix:

http://lists.xen.org/archives/html/xen-devel/2013-04/msg01328.html

But I don't know what the status of that is WRT centos. I guess this redhat bug is related:
https://bugzilla.redhat.com/show_bug.cgi?id=957231

But without a redhat account we can't look.

domu's can work around this (apparently with some performance impact) by running

ethtool -K eth0 tso off gso off

Considering we have 40 servers running 4.1 and only 4 people have been affected, is the best thing to do just to send a list out to announce / the blog and throw swatch on the console logs?

I may poke at the centos virt mailing list and ask if they know if there's a timeline for applying the patch to netback I linked to above.

dhcp problems (now fixed)

2013-05-21T21:38:23Z

The following subnets may have had issues acquiring a dhcp lease from approximately 03:30 to 14:30 PST 2013-05-21:

71.19.145
71.17.146
71.19.147
71.19.149
71.19.150
71.19.154
71.19.156
71.19.157

The problem is we are running dhcrelay on those subnets, which forwards dhcp requests to another server, and they were using a cached dns lookup which pointed to an old ip address rather than the current for the dhcp server.

Outage in rack 05-11 - Luke's report

2013-05-18T09:56:44Z

NOTE: please read the report written by srn. It's better:

http://blog.prgmr.com/xenophilia/2013/05/unplanned-downtime-in-santa-cl.html

So, a co-lo customer brings in a desktop so cheap that it has a manual 120-240
switch. My co-lo, of course, is 208.

Everything made in the last 10 years auto-switches 100-240v. But not this
garbage.

Anyhow, he plugged it in, and this destroyed my PDU. We went to the
office and grabbed a spare, mounted it, and you should be back up (the
power cord situation in rack 05-11 is now much, much worse, due to plugging
it back in during a panic.)

This is my fault, of course; I thought this person was pretty competent,
and I had a little room left in 05-11 and nowhere else, so I put him in with
the production stuff. Clearly, this was a mistake.

Interestingly, this was on a sub-pdu (that was plugged in via a c19->c20; my main pdu has several c19 outlets as well as a bunch of c13) - and the sub-pdu works just fine still; it was the main PDU that completely fried. I need to learn more about electricity.

anyhow, you should be back up now. I'm sorry.

unplanned downtime in santa clara

2013-05-18T08:33:54Z

UPDATE 2:49 PST: All servers should be back up.

UPDATE: The problem was a dedicated server customer plugged in a computer with a configurable power supply at the wrong setting (120V instead of 240V) and it blew out the pdu. We picked up another pdu from the office and are installing it right now.

Following servers are down, starting at about 01:10 PST:

sword
chariot
branch
seashell
knife
cauldron
chessboard
horn
fuller
coins
council
bowl
waite
whetstone
taney
mantle
lozenges
rutledge
pearl

Reasons are unknown (to me at least) at this time - will post with further updates as I have them.

/etc/resolv.conf

2013-05-16T12:54:46Z

Please replace any existing nameserver entries with the following:

nameserver 71.19.155.120
nameserver 71.19.145.215

dhcp has been changed to the renumbered ips, dns to change shortly

2013-05-16T11:31:31Z

In short, for those people who have been renumbered

username.xen.prgmr.com is becoming username.old.xen.prgmr.com
username.new.xen.prgmr.com is becoming username.xen.prgmr.com

reverse dns mappings should be preserved.

The old IPs are only around for another day or so, so if you are not pingable on username.new.xen.prgmr.com PLEASE take action.

We intend to send a follow up email for those vps's which are still not responding but you are responsible for making sure your server has the correct network configuration, we do not edit files on your machine.

As a side note, if you do not remember which server your vps is hosted on, current users should be able to log in as @.console.xen.prgmr.com

Support

2013-05-15T02:47:47Z

Just to let you know, support is getting a little backed up with all the move going on and the IP address renumbering. Please be patient. We are attempting to get back to normal as soon as possible.

If you need something urgently, please go to #prgmr on Freenode IRC and ask for prgmrcom (Luke), srn_prgmr (Sarah), danihan (Dan), lifftchi (Chris), nb (Nick), or _will_ (Will) and give us your ticket number, and we will attempt to take care of it as soon as we can. Otherwise, we are attempting to get through the ticket queue as quickly as possible.

Additional unplanned downtime

2013-05-13T07:28:11Z

so yeah, we have a handful of dell C6100s in the bottom of each of the racks. These are me working on a deal where I help unixsurplus lease some of their stuff. Anyhow, this evening around 8:00pm my time, Miles was swapping out one of the units. And here is why it's a bad idea to put these things on the bottom of the rack; the 4 blades all come out the back... and the l6-30 is right there in the way. Well, the upshot is that power got disconnected for the whole rack.

This outage was exacerbated by me just doing the minimum to get servers up last night. some of the servers, for instance, defaulted to non-xen linux in the grub config (we do this, often, when troubleshooting the serial console, for instance, we then manually select the xen kernel, and then fix menu.lst after the box is up. Well, I didn't fix menu.lst when I was configuring these this morning.)

So yeah, what are we going to do to fix this problem? well, first, I think the importance of being careful around power connectors has been impressed upon miles. but more importantly, now we are moving all the c6100 units up to mid rack height.

wow... that was a lot more than an hour.

2013-05-12T15:40:51Z

Wow. That was a lot more than an hour of downtime.   I'm sorry.   Several
things went wrong. You know all those plans I was talking about with
regards to having everything set and ready to go?   Well, first?
the console server wasn't all the way setup.    Next? turns out, I had 5
pre-made opengear dongles, not 10 like I needed; I of course have plenty
of unconfigured db9->rj45 adaptors about, but the instructions I had were
by color (which is stupid, as there aren't standard colurs for the
rj45->db9 dongles, and my new ones are different) and it took me a while
to figure out how to make a proper opengear rj45->db9 dongle.

Then, the rest? mostly just being stupid and tired.   Like we swapped the
network ports (which I had pre-configured) then stood around like idiots
wondering why the thing boots so slowly.

Anyhow, I've got a 5 year lease on my four 5kw racks here in coresite
santa clara, so for you? the ordeal is over. Yes, at some point I'll
want to move you on to better hardware, but that usually comes with better
performance and usually more disk/ram and stuff, and often it can be done
at the user's convenience. (that's what I told the users who moved, anyhow.)

Anyhow, I think I'm not moving anything tonight. I will need to clean up the
co-lo, but most of my mistakes here were of the tired and stupid variety.

so, uh, sorry about breaking networking on wilson

2013-05-11T13:07:34Z

I'm... kindof an idiot. See, I'm up writing this instead of sleeping, when I'm clearly stupid at this point. but yeah. Thing is? the new network consists of a force10 10G switch (4x fxp, 24xcx4) switch for all four racks, plus one woven trx100 (4 port 10g cx4, 48 port 1g rj45) switch per rack. I'm using lacp to aggrigate the 10g links into one 40G link (it actually works pretty well, because you all have different mac address, the traffic is well-balanced.   a common problem with layer two link aggregation is that usually it uses the source and dest mac addresses to figure out which link to send a packet down, so it's easy to get unbalanced traffic... but all you have different mac addresses, so it spreads traffic as well as a much larger network.)

Anyhow, uh, yeah.   lemme check my notes.

okay, on the woven? to put a port on a vlan, you do something like this:

interface 0/3
vlan pvid 30

or so.    Then, on the aggrigate, you do something like:

on the woven:
interface 3/1
vlan acceptframe vlanonly
vlan participation exclude 1
vlan participation include 30,33
vlan tagging 30,33
lacp collector max-delay 0
exit

See, I thought that the 'vlan tagging' thing was for access-mode links, to tag incoming packets that don't already have a vlan header with a particular vlan, so I removed (!) it.    from a working (!) vlan. so clearly, it was a stupid, stupid, undercaffinated mistake.

I figured the problem out shortly after making myself a cup of coffee.

dhcp currently (kindof) broken, will fix after sleep

2013-05-11T13:05:25Z

uh, yeah, set your IPs statically for now. We're getting reports of DHCP being broken in some places but not others. It doesn't make sense to me right now. I'm going to setup a dhrelay server on each subnet and each location, and that should cover it.

An error in judgement

2013-05-05T16:05:08Z

so yeah, I emailed 15 servers of users, told them they'd have an hour of downtime "Saturday Evening"

I figured I'd do 5 servers at a go.   small number.

So yeah, I came up with some spare rails, screwed 'em in, setup the PDU, etc... did all the usual things you do before you move servers.   But these things have a way of taking longer than you expect, and I also had to configure an unfamiliar switch (my force10 10gbe setup, which I still don't have configured)

So after we setup the destination co-lo, we headed to the source co-lo (55 s. market.) - this means getting tickets to remove servers and arranging it so that the security folk know we are coming and verify our tickets before we shut things down (it always takes a few tries to get the ticket right to leave with hardware.) so Anyhow, it was around 4:30AM local time when we started shutting things down. It's 9:00am now.     I really should have called it off around 1:00am or so.

I mean, the rest of our problems have to do with the fact that these are some of the oldest servers (both hardware wise and software wise) in the fleet.   Well, that, and I'm not exactly at my mental best at this point.

Anyhow, we're back up. I'm sorry.

network outage at 4:00AM pst

2013-05-03T11:49:13Z

so, uh, yeah. stupid mistake on my part. approx. 20 mins. downtime (longer for IPv6) you are back now

and council today

2013-05-03T06:37:55Z

UPDATE 2013-05-03 1:42 PST : Drive is replaced, the raid is rebuilding. Estimated completion time is ~ 2013-05-04 10:30 PST.
---

Why isn't it failing out of the raid? and that is an 'enterprise' - weird. anyhow, I failed the drive. (interestingly, this is one of those ancient servers that still has a 'consumer' drive- it was the 'enterprise' not the 'consumer' that is bad.)

anyhow, I'm off to swap a drive.

1.00: (BMDMA stat 0x0)
ata1.00: tag 0 cmd 0x25 Emask 0x9 stat 0x51 err 0x40 (media error)
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: (BMDMA stat 0x0)
ata1.00: tag 0 cmd 0x25 Emask 0x9 stat 0x51 err 0x40 (media error)
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: (BMDMA stat 0x0)
ata1.00: tag 0 cmd 0x25 Emask 0x9 stat 0x51 err 0x40 (media error)
end_request: I/O error, dev sdc, sector 283579720
SCSI device sdc: 2930277168 512-byte hdwr sectors (1500302 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
SCSI device sdc: 2930277168 512-byte hdwr sectors (1500302 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: (BMDMA stat 0x0)
ata1.00: tag 0 cmd 0x25 Emask 0x9 stat 0x51 err 0x40 (media error)
SCSI device sdc: 2930277168 512-byte hdwr sectors (1500302 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
raid1: sdb2: redirecting sector 262614887 to another mirror

rehnquist is back, on fresh hardware.

2013-05-02T11:45:19Z

and yeah. I'm going to take the marvell card out of Wilson and replace it with a LSI or something.

It looks like the dom0 got hit harder by the crash than the DomUs, though I would expect some of you will have to fsck.

http://twitter.com/prgmrcom got many of my updates this time, though as the evening wore, my coherency did, too.