August 2011 Archives

Update:  all customers should be back.   The RAID was broken by the unclean restart, so expect degraded I/O while it finishes rebuilding.  

Update:  I fixed the grub boot records.   The box is back up,   the virtuals will be coming on line one at a time.   

Update: knife isn't rebooting remotely.  (I mean, the pdus cycle power, but knife hangs up somewhere betwen post and boot)  I'm heading down to the co-lo to jerk with it in person.

not a good morning.  I'm rebooting knife right now.

BUG: soft lockup detected on CPU#0!

Call Trace:

 <IRQ> [<ffffffff8025758a>] softlockup_tick+0xce/0xe0

 [<ffffffff8020df48>] timer_interrupt+0x3a0/0x3fa

[Wed Aug 31 00:44:24 2011] [<ffffffff80257874>] handle_IRQ_event+0x4e/0x96

 [<ffffffff80257960>] __do_IRQ+0xa4/0x105

 [<ffffffff88204e6f>] :bridge:br_forward_finish+0x0/0x51

 [<ffffffff8020bd5c>] do_IRQ+0x44/0x4d

 [<ffffffff8034c980>] evtchn_do_upcall+0x19e/0x250

 [<ffffffff80209d8e>] do_hypervisor_callback+0x1e/0x2c

 [<ffffffff8020528a>] hypercall_page+0x28a/0x1000

 [<ffffffff8020528a>] hypercall_page+0x28a/0x1000

 [<ffffffff8036109a>] net_tx_action+0xa82/0x103f

 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc

[Wed Aug 31 00:44:24 2011] [<ffffffff8820593e>] :bridge:br_handle_frame_finish+0x0/0xf8

 [<ffffffff88205ba4>] :bridge:br_handle_frame+0x16e/0x1a2

 [<ffffffff8039edb6>] netif_receive_skb+0x1ca/0x2ea

 [<ffffffff80233b33>] tasklet_action+0xa7/0x133

 [<ffffffff802339f8>] __do_softirq+0x83/0x117

 [<ffffffff8034c980>] evtchn_do_upcall+0x19e/0x250

 [<ffffffff803fc644>] call_softirq+0x1c/0x26

 [<ffffffff8020c055>] do_softirq+0x6a/0xed

 [<ffffffff80209d8e>] do_hypervisor_callback+0x1e/0x2c

 <EOI> [<ffffffff803581ea>] show_rd_sect+0x0/0x68

[Wed Aug 31 00:44:24 2011] [<ffffffff802ebbfc>] __read_lock_failed+0x8/0x14

 [<ffffffff80343f3e>] get_device+0x17/0x20

 [<ffffffff803fc3fd>] .text.lock.spinlock+0x53/0x8a

 [<ffffffff80358211>] show_rd_sect+0x27/0x68

 [<ffffffff802bc351>] sysfs_read_file+0xa5/0x12e

 [<ffffffff8027e3f5>] vfs_read+0xcb/0x171

 [<ffffffff8027e7d4>] sys_read+0x45/0x6e

 [<ffffffff802097b2>] tracesys+0xab/0xb5

http://web.archive.org/web/20100101015856/http://prgmr.com/aup.html

as usual, the live copy is at http://prgmr.com/aup.html

My intent here is to make clear the fact that we have several clauses prohibiting unsolicited bulk mail and spam in addition to the main "no bulk mail of any kind without permission" prohibition.  At Chris' suggestion, I also made more of the clauses stand on their own by appending a "is prohibited" 

Being as I didn't make any substantive changes, I'm not sending out an email, but I wanted to note it here anyhow.  

stables had a problem (xend didn't start??  will figure it out after the carnage is cleaned up)  but it's coming up now.

Ingot is back.  If you are still down, complain loudly.  

here is the official report from he.net:

Please note, I'm adding the new updates at the top;  jewel is up right now.  Read down for the history.  

edit at 20:33:  we've disabled hyperthreading and we've got it on the new kernel and on the e1000e ethernet adaptor.  It should be up and stable;  the raid is still rebuilding, but yeah, I think we are in okay shape for now.    We'll be moving people off this server as we get more capacity up;  email us if you want to move to the front of the line.  

The raid is still rebuilding, so expect less than stellar performance.   

edit at 18:13:   a new crash

I've seen some similar things having to do with the intel hardware virtualization, so I disabled all hardware virtualization, and I disabled hyperthreading.  booting again.  

edit at 17:00:   the kernel was upgraded some time ago, but the system still crashes when we start guests.  We can't get it to crash without starting guests, so we're grasping here;  we're going to use the onboard e1000e rather than the usb, now that we have the good kernel in place.  Nick is en-route to the data center.    

original post:

We're going to update the kernel to latest (the old one we were using was built for our amd mcp55 systems, and it's in a modern intel server right now)  

(XEN) rdx: 00002072659edae4   rsi: ffff830c3fdc7e88   rdi: ffff828c802855a0

(XEN) rbp: ffff8308df024cb0   rsp: ffff830c3fdc7ec8   r8:  0000000000012252

(XEN) r9:  0000000000000004   r10: 0000000000000005   r11: 0000000000000000

(XEN) r12: ffff8308df024d70   r13: 00002072659ec3c7   r14: 0000000000000000

(XEN) r15: ffff828c80221100   cr0: 000000008005003b   cr4: 00000000000026f0

(XEN) cr3: 00000009030ae000   cr2: 00000000000000d4

(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008

(XEN) Xen stack trace from rsp=ffff830c3fdc7ec8:

[Sat Aug  6 14:14:12 2011](XEN)    ffff82ec80173b66 ffff828c8025f900 00000000802

5e900 ffff830c3fdc7f28

(XEN)    ffff828c8025e900 ffff828c8021f5b0 00002072659ec3c7 0000000000000000

(XEN)    ffff828c80138ed7 0000000000002000 ffff8300bf23c000 ffff8300bf0e4000

(XEN)    0000000001301c00 ffffffff8057d160 ffffffff8057c520 ffffffffffffffff

(XEN)    0000000000631918 0000000000000000 0000000000000246 0000000000631918

(XEN)    ffff880001939ee0 ffffffff805cbe48 0000000000000000 ffffffff802083aa

(XEN)    ffffffff80553f28 0000000000000000 0000000000000001 0000010000000000

(XEN)    ffffffff802083aa 000000000000e033 0000000000000246 ffffffff80553f10

(XEN)    000000000000e02b 0000000000000000 0000000000000000 0000000000000000

(XEN)    0000000000000000 0000000000000002 ffff8300bf23c000

[Sat Aug  6 14:14:12 2011](XEN) Xen call trace:

(XEN)    [<ffff828c801207d0>] compat_xen_version+0x3e0/0x420

(XEN)    [<ffff828c80138ed7>] idle_loop+0x47/0xa0

(XEN)    

(XEN) Pagetable walk from 00000000000000d4:

(XEN)  L4[0x000] = 00000009030e8067 000000000001f2cc

(XEN)  L3[0x000] = 00000009030e7067 000000000001f2cd

(XEN)  L2[0x000] = 0000000000000000 ffffffffffffffff 

(XEN) 

(XEN) ****************************************

[Sat Aug  6 14:14:12 2011](XEN) Panic on CPU 2:

(XEN) FATAL PAGE FAULT

(XEN) [error_code=0000]

(XEN) Faulting linear address: 00000000000000d4

(XEN) ****************************************

(XEN) 

(XEN) Reboot in five seconds...

Update:  We decided that it'd be faster to swap the drives into ellsworth, one of our dual quad-core intel servers, which is actually quite a nice server; much newer than Jewel was.   We're having some kernel issues so for now, it's running on Nick's USB ethernet adapter, which I'm not particularly happy about, but it's better than nick and I screwing around with the kernel on this old, touchy setup while we are half-asleep.   

The plan is to move everyone off this box on to newer systems with more and newer drives starting on Sunday, after we're fresh.

Meanwhile, all users on Jewel will get a free month.  Megan is working on that now; credits will be applied, uh, probably within a week.