September 2012 Archives

an inconvienent page

| | Comments (1)
Looks like the disks in Cerberus finally gave out... or something.  thing has hung up.  I rebooted, and it's trying to rebuild.  checking disks now, making sure it's rebuilding on to sane disks, etc. 

Ok, so it looks like we have a serious disk issue.    Cerberus is back up and limping along. 

Chris is working on moving the users we owe upgrades to.  My sunday will be dedicated to moving everyone else off

note, you should have been up for all of last night.  you should be up now.  Chris is starting the upgrades at 20:00, and I'm still working on the new dest server for the rest of you.

for want of a cr2032 (taney down)

| | Comments (0)
so yeah.  Taney crashed, and we were unable to bring it up due to a dead CMOS battery.  During the worst time of the day to travel, and while I was in mountain view.  

Nb and nick couldn't bring it up remotely, so I came down here and swapped out the CMOS battery;  Nick fixed the secondary issues (we somehow forgot to put the bootloader on the disk in p0.
We're moving our router;  it shouldn't be more than 20 minutes downtime.  (it should be well under 5 minutes, but everyone will be down.)  

Note this is the router at svtix, so, uh, that's not most of you.  let's see if I can get a list of netblocks.  Yeah, here we go:

'71.19.150.1/24', '71.19.149.0/24'   plus nearly all of our co-location stuff. 
We are about to make a network change that will take down the connection to he.net briefly. I will update here again when it is over.

Update: It is all set now, and without any downtime I think. -Nick

cauldron is oom-killing itself

| | Comments (0)
very irritating;  this may necessitate a reboot.

Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:30
cpu 0 cold: high 62, batch 15 used:60
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages:        5412kB (0kB HighMem)
Active:152866 inactive:63067 dirty:51 writeback:0 unstable:0 free:1353 slab:17833 mapped:3657 pagetables:1058
DMA free:4012kB min:4kB low:4kB high:4kB active:3840kB inactive:0kB present:1956kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1002 1002 1002
DMA32 free:1400kB min:4044kB low:5052kB high:6064kB active:607624kB inactive:252268kB present:1026160kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4012kB
DMA32: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1400kB
Normal: empty
HighMem: empty
Swap cache: add 26, delete 26, find 0/0, race 0+0
Free swap  = 1048472kB
Total swap = 1048568kB
Free swap:       1048472kB
264192 pages of RAM
22575 reserved pages
145636 pages shared
0 pages swap cached
peth0: too many iterations (6) in nv_nic_irq_rx.
yum-updatesd-he: page allocation failure. order:0, mode:0x20

Call Trace:
 <IRQ> [<ffffffff8025f43f>] __alloc_pages+0x299/0x2b2
 [<ffffffff80279c31>] cache_alloc_refill+0x2b2/0x535
 [<ffffffff80279f64>] __kmalloc+0xb0/0xf0
 [<ffffffff8039ae62>] __alloc_skb+0x5c/0x123
 [<ffffffff88122e27>] :forcedeth:nv_nic_irq_rx+0x3ba/0x5bd
 [<ffffffff80257874>] handle_IRQ_event+0x4e/0x96
 [<ffffffff80257960>] __do_IRQ+0xa4/0x105
 [<ffffffff8020bd5c>] do_IRQ+0x44/0x4d
 [<ffffffff8034c980>] evtchn_do_upcall+0x19e/0x250
 [<ffffffff80209d8e>] do_hypervisor_callback+0x1e/0x2c
 [<ffffffff8020522a>] hypercall_page+0x22a/0x1000
 [<ffffffff8020522a>] hypercall_page+0x22a/0x1000
 [<ffffffff8034b922>] force_evtchn_callback+0xa/0xb
 [<ffffffff80279f93>] __kmalloc+0xdf/0xf0
 [<ffffffff8039a79b>] pskb_expand_head+0x51/0x137
 [<ffffffff8822be00>] :bridge:br_dev_queue_push_xmit+0x13e/0x1ad
 [<ffffffff882300e6>] :bridge:br_nf_post_routing+0x17a/0x195
 [<ffffffff882124d4>] :ip6_tables:ipv6_find_hdr+0x4e/0x1a1
 [<ffffffff803b676d>] nf_iterate+0x41/0x7d
 [<ffffffff8822bcc2>] :bridge:br_dev_queue_push_xmit+0x0/0x1ad
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff8822bcc2>] :bridge:br_dev_queue_push_xmit+0x0/0x1ad
 [<ffffffff8822beae>] :bridge:br_forward_finish+0x3f/0x51
 [<ffffffff8822ff64>] :bridge:br_nf_forward_finish+0xf7/0xff
 [<ffffffff8823080f>] :bridge:br_nf_forward_ip+0x14d/0x15d
 [<ffffffff803b676d>] nf_iterate+0x41/0x7d
 [<ffffffff8822be6f>] :bridge:br_forward_finish+0x0/0x51
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff8822be6f>] :bridge:br_forward_finish+0x0/0x51
 [<ffffffff8822bec0>] :bridge:__br_forward+0x0/0x6d
 [<ffffffff8822bf19>] :bridge:__br_forward+0x59/0x6d
 [<ffffffff8822bc61>] :bridge:br_flood+0x7d/0xc6
 [<ffffffff8822c9d9>] :bridge:br_handle_frame_finish+0x9b/0xf8
 [<ffffffff882305d7>] :bridge:br_nf_pre_routing_finish+0x2ed/0x2fc
 [<ffffffff882302ea>] :bridge:br_nf_pre_routing_finish+0x0/0x2fc
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff882302ea>] :bridge:br_nf_pre_routing_finish+0x0/0x2fc
 [<ffffffff8822c93c>] :bridge:br_pass_frame_up+0x67/0x69
 [<ffffffff882311ee>] :bridge:br_nf_pre_routing+0x611/0x62f
 [<ffffffff803b676d>] nf_iterate+0x41/0x7d
 [<ffffffff8822c93e>] :bridge:br_handle_frame_finish+0x0/0xf8
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff8822c93e>] :bridge:br_handle_frame_finish+0x0/0xf8
 [<ffffffff8822cba4>] :bridge:br_handle_frame+0x16e/0x1a2
 [<ffffffff8039edb6>] netif_receive_skb+0x1ca/0x2ea
 [<ffffffff803a0d6e>] process_backlog+0xd0/0x182
 [<ffffffff803a0fe8>] net_rx_action+0xe3/0x24b
 [<ffffffff802339f8>] __do_softirq+0x83/0x117
 [<ffffffff8034c980>] evtchn_do_upcall+0x19e/0x250
 [<ffffffff803fc644>] call_softirq+0x1c/0x26
 [<ffffffff8020c055>] do_softirq+0x6a/0xed
 [<ffffffff80209d8e>] do_hypervisor_callback+0x1e/0x2c
 <EOI>
Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:30
cpu 0 cold: high 62, batch 15 used:60
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages:        5412kB (0kB HighMem)
Active:152866 inactive:63067 dirty:51 writeback:0 unstable:0 free:1353 slab:17833 mapped:3657 pagetables:1058
DMA free:4012kB min:4kB low:4kB high:4kB active:3840kB inactive:0kB present:1956kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1002 1002 1002
DMA32 free:1400kB min:4044kB low:5052kB high:6064kB active:607624kB inactive:252268kB present:1026160kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4012kB
DMA32: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1400kB
Normal: empty
HighMem: empty
Swap cache: add 26, delete 26, find 0/0, race 0+0
Free swap  = 1048472kB
Total swap = 1048568kB
Free swap:       1048472kB
264192 pages of RAM
22575 reserved pages
145636 pages shared
0 pages swap cached
yum-updatesd-he: page allocation failure. order:0, mode:0x20

Call Trace:
 <IRQ> [<ffffffff8025f43f>] __alloc_pages+0x299/0x2b2
 [<ffffffff80279c31>] cache_alloc_refill+0x2b2/0x535
 [<ffffffff80279f64>] __kmalloc+0xb0/0xf0
 [<ffffffff8039a79b>] pskb_expand_head+0x51/0x137
 [<ffffffff8822be00>] :bridge:br_dev_queue_push_xmit+0x13e/0x1ad
 [<ffffffff882300e6>] :bridge:br_nf_post_routing+0x17a/0x195
 [<ffffffff882124d4>] :ip6_tables:ipv6_find_hdr+0x4e/0x1a1
 [<ffffffff803b676d>] nf_iterate+0x41/0x7d
 [<ffffffff8822bcc2>] :bridge:br_dev_queue_push_xmit+0x0/0x1ad
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff8822bcc2>] :bridge:br_dev_queue_push_xmit+0x0/0x1ad
 [<ffffffff8822beae>] :bridge:br_forward_finish+0x3f/0x51
 [<ffffffff8822ff64>] :bridge:br_nf_forward_finish+0xf7/0xff
 [<ffffffff8823080f>] :bridge:br_nf_forward_ip+0x14d/0x15d
 [<ffffffff803b676d>] nf_iterate+0x41/0x7d
 [<ffffffff8822be6f>] :bridge:br_forward_finish+0x0/0x51
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff8822be6f>] :bridge:br_forward_finish+0x0/0x51
 [<ffffffff8822bec0>] :bridge:__br_forward+0x0/0x6d
 [<ffffffff8822bf19>] :bridge:__br_forward+0x59/0x6d
 [<ffffffff8822bc61>] :bridge:br_flood+0x7d/0xc6
 [<ffffffff8822c9d9>] :bridge:br_handle_frame_finish+0x9b/0xf8
 [<ffffffff882305d7>] :bridge:br_nf_pre_routing_finish+0x2ed/0x2fc
 [<ffffffff882302ea>] :bridge:br_nf_pre_routing_finish+0x0/0x2fc
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff882302ea>] :bridge:br_nf_pre_routing_finish+0x0/0x2fc
 [<ffffffff8822c93c>] :bridge:br_pass_frame_up+0x67/0x69
 [<ffffffff882311ee>] :bridge:br_nf_pre_routing+0x611/0x62f
 [<ffffffff803b676d>] nf_iterate+0x41/0x7d
 [<ffffffff8822c93e>] :bridge:br_handle_frame_finish+0x0/0xf8
 [<ffffffff803b68f2>] nf_hook_slow+0x58/0xbc
 [<ffffffff8822c93e>] :bridge:br_handle_frame_finish+0x0/0xf8
 [<ffffffff8822cba4>] :bridge:br_handle_frame+0x16e/0x1a2
 [<ffffffff8039edb6>] netif_receive_skb+0x1ca/0x2ea
 [<ffffffff803a0d6e>] process_backlog+0xd0/0x182
 [<ffffffff803a0fe8>] net_rx_action+0xe3/0x24b
 [<ffffffff802339f8>] __do_softirq+0x83/0x117
 [<ffffffff8034c980>] evtchn_do_upcall+0x19e/0x250
 [<ffffffff803fc644>] call_softirq+0x1c/0x26
 [<ffffffff8020c055>] do_softirq+0x6a/0xed
 [<ffffffff80209d8e>] do_hypervisor_callback+0x1e/0x2c
 <EOI>
Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:30
cpu 0 cold: high 62, batch 15 used:60
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages:        5412kB (0kB HighMem)
Active:152866 inactive:63067 dirty:51 writeback:0 unstable:0 free:1353 slab:17833 mapped:3657 pagetables:1058
DMA free:4012kB min:4kB low:4kB high:4kB active:3840kB inactive:0kB present:1956kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1002 1002 1002
DMA32 free:1400kB min:4044kB low:5052kB high:6064kB active:607624kB inactive:252268kB present:1026160kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4012kB
DMA32: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1400kB
Normal: empty
HighMem: empty
Swap cache: add 26, delete 26, find 0/0, race 0+0
Free swap  = 1048472kB
Total swap = 1048568kB
Free swap:       1048472kB
264192 pages of RAM
22575 reserved pages
145636 pages shared
0 pages swap cached


yup.  rebooted.