March 2014 Archives

Scheduled downtime on mantle

| | Comments (0)
UPDATE 21:42 PST: guests are booting. Everyone should be back up shortly.
---
UPDATE: grub was not installed on all the drives that were replaced last time and needs to be updated.  New ETA for everyone coming up is 21:50 PST(-0700).
---
There will be scheduled downtime on mantle for Sunday 2014-03-30 20:00 PST(-0700).  We have allocated approximately one hour.  If there is extended down time beyond the scope estimated we will post updates here and/or send out emails to affected users.

Note for Arch Linux users

| | Comments (0)
Leo Franchi, the one user who didn't come back up after the move from taft, had custom compiled his kernel because he was having trouble with one of the ones from Arch.  He has since moved to their xen LTS kernel. If you are using Arch we recommend you try this kernel first unless you have a strong reason to compile your own. Thanks to Leo for pointing this out.


All customers previously on taft have been moved.  Every instance, except for one running what is probably a custom compiled kernel, is up and running.  Affected customers can access the management console by ssh with the credentials <user>@<user>.console.xen.prgmr.com.

taft has serious problems

| | Comments (0)
edit at 20:48-   we are looking good;  two users are restored;  it's just a matter of moving bits at this point.     Don't log in, please, until we email you.  

edit at 19:30 -  progress is being made;  we are moving customer data to mcgrigor,  a newer host.   So far, it looks like the corruption is confined to  the root device (my stuff, not customer stuff)

apparently, I've got a very bad disk and/or disk controller, and some disk corruption.   Taft users are going to be down for a while while srn and I work it out.   




mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
sd 0:0:4:0: Unhandled sense code
sd 0:0:4:0: SCSI error: return code = 0x08100002
Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
sde: Current: sense key: Medium Error
    Add. Sense: Unrecovered read error

Info fld=0x19f36d6
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
sd 0:0:4:0: Unhandled sense code
sd 0:0:4:0: SCSI error: return code = 0x08100002
Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
sde: Current: sense key: Medium Error
    Add. Sense: Unrecovered read error

Info fld=0x19f3ccd
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31110610): originator(PL), code(0x11), sub_code(0x0610)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
sd 0:0:4:0: Unhandled sense code
sd 0:0:4:0: SCSI error: return code = 0x08100002
Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
sde: Current: sense key: Medium Error
    Add. Sense: Unrecovered read error

Info fld=0x19f4459
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
mpt2sas0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000
Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:4:0: Unhandled error code
sd 0:0:4:0: SCSI error: return code = 0x000b0000

cauldron crash

| | Comments (0)
another old socket F opteron server.  I don't know why it crashed.  Effort that would normally have gone to figuring that out is going into moving customers off.   


so I rebooted it, it hung again during the reboot (completely unresponsive;  one disk light stuck solid.)      

I moved the hard drives to a new server, which took longer than usual, 'cause the new one had my new SoL setup on com2, while the old one was physical serial on com1, and I was an idiot and didn't correct that when I corrected the eth0 mac.


Anyhow, it's back now.   Expect an upgrade soon-ish, when I'll move you off the server you are on.  
network issues at rippleweb and/or qts sacramento.   (the hearkulies data location is now owned by qts.)   

There's not much I can do except to finish moving everyone off, which I can't do until the network is up. 

mirror.prgmr.com updated

| | Comments (0)
Now mirror.prgmr.com has Fedora, CentOS, EPEL, and Debian.  It has a cron job which should keep it updated.  At this time I don't have much more space on it's VM to add any other distros, although feel free to send me suggestions and I will consider them should we be able to allocate more space towards our mirror.

Any questions, please email me at nb@prgmr.com

--
Nick Bebout
System Administrator
prgmr.com, Inc.
nb@prgmr.com

mirrors.prgmr.com (old debian mirror) is gone

| | Comments (0)
So, for some reason I couldn't connect to mirrors.prgmr.com and no one has maintained it for a while now, so I shut it down.  Apparently it has been taking care of itself for quite a while and was working just fine, and for some reason I just couldn't see anything there when I was looking to see if it was still up.

On a good note, I am setting up http://mirror.prgmr.com which will have Fedora, CentOS, EPEL (Extra Packages for Enterprise Linux), and will probably be getting Debian on it, and maybe others, disk space permitting.

I suggest for now that you remove mirrors.prgmr.com from your configuration files and use a generic mirror.  I will blog again when I get more added to mirror.prgmr.com

Once again:

mirror (no s) is good, new mirror
mirrors (with s) is the old only debian mirror which is gone.

Sorry

--
Nick Bebout
System Administrator
prgmr.com, Inc.
nb@prgmr.com

About this Archive

This page is an archive of entries from March 2014 listed from newest to oldest.

February 2014 is the previous archive.

April 2014 is the next archive.

Find recent content on the main index or look in the archives to find all content.