April 2013 Archives

lozenges is down

| | Comments (0)
UPDATE 12:36 PST: The raid is rebuilding right now.  Since everyone is already down and it takes much longer to rebuild the raid when guests are running, we're going to wait another 2-3 hours and let the raid finish rebuilding before bringing up guests.  Everyone on lozenges will be getting a month credit.  We will post when the guests on lozenges have been brought back up.

---

As of 10:15 PST 2 disks in the same mirror were kicked out of the raid.  We're at 55 south market with 2 new hard drives.  Working on it.

---
UPDATE: it appears that one disk fell out of the array at Sat Apr 20 01:51:06 2013, the other at  Sat Apr 20 02:12:00 2013.  We aren't quite sure why there wasn't an alert sent until 10:15 this morning.
--
Commands run:
(Repair filesystem) 73 # mdadm --assemble /dev/md1
mdadm: /dev/md1 assembled from 2 drives - not enough to start the array.
(Repair filesystem) 74 # mdadm --assemble --scan /dev/md1
mdadm: /dev/md1 assembled from 2 drives - not enough to start the array.
(Repair filesystem) 75 # mdadm --assemble --force /dev/md1
mdadm: forcing event count in /dev/sdd2(3) from 15103 upto 15105
mdadm: clearing FAULTY flag for device 0 in /dev/md1 for /dev/sdd2
mdadm: /dev/md1 assembled from 3 drives - not enough to start the array.
(Repair filesystem) 76 # mdadm --assemble --force /dev/md1 --run
mdadm: /dev/md1 has been started with 3 drives (out of 4).
--
Partitioning new /dev/sdc:
(Repair filesystem) 86 # sfdisk --dump /dev/sda > sda.sfdisk
(Repair filesystem) 87 # sfdisk /dev/sdc < sda.sfdisk
(Repair filesystem) 93 # mdadm /dev/md0 -a /dev/sdc1
(Repair filesystem) 94 # cat /proc/mdstat
(Repair filesystem) 95 # mdadm /dev/md1 -a /dev/sdc2
(Repair filesystem) 96 # cat /proc/mdstat
--
UPDATE: another round of emails will be going out within the next few hours, please do not email support until (tomorrow) 2013-04-16 if you have not received an email already.
--
The following IP ranges outside of 71.19.144.0/20 are NOT being renumbered at this time:

  • 216.218.210.64/27
  • 216.218.223.64/26
  • 38.99.2.0/24
  • 64.62.173.0/25
  • 64.62.188.0/24
  • 64.62.205.192/26

71.19.144.0/20 is owned by prgmr.com directly and will not be renumbered at all.

If your IP is outside of these ranges and you have not yet received an email regarding the new IP allocation, please contact support@prgmr.com.

You will need to update your ipv6 address should you be using ipv6.  The new ipv6 address can be found the same way as the ipv4 address by running

host [username].new.xen.prgmr.com dns.xen.prgmr.com