ext3, pvgrub and "block error -1 for op 0"

| No Comments
For testing purposes I was creating a large number of instances with very small root file systems. The partitions were formatted using ext3. About one out of twenty would fail to decompress bzImage with the message "block error -1 for op 0".  The files were all read correctly when the partition was mounted in the dom0.

When I increased the disk size from 32MiB to 64MiB and the partition size to 63MiB, the problem went away.  I also made sure the ratio between inodes and blocks was an integer number though this didn't seem to help.

Direct connect to HVM serial ports

| No Comments
Me and Luke recently got this question:

"So with both pv and hvm domains you can pull out the virtual tty via xenstore-read /local/domain/[domain]/console/tty . However, with pv domains you can use the tty directly with something like screen /dev/tty/blah but with a hvm domain that ends up giving you a blank screen. What's even weirder is that when you look at the source of xm console it doesn't seem to differentiate between the two, and yet it works fine on both.

Any ideas?"

We are using xl on xen 4.3, not xm, on our test machine but in theory these should be doing mostly the same thing.

Using the command "strace -f xl console ubuntu-1 &> con" I found this in the output of strace:

[pid 30642] access("/dev/pts/2", R_OK|W_OK) = 0
[pid 30642] open("/dev/pts/2", O_RDWR|O_NOCTTY) = 8

Searching back for /dev/pts/2 I found the following:

[pid 30642] write(5, "/local/domain/356/serial/0/tty\0", 31) = 31

Therefore it looks like the key is


migrate yourself, part 2.

| No Comments
Finally had a successful migration, end-to-end, without administrator intervention.

Several things developed over the course of this testing.

  1. we need config management more than ever.
  2. the migration script doesn't bother with /etc/sudoers.  this is a problem.
  3. full paths, please.
  4. corollary: sudo doesn't source .profile et. al, so your path will stay the same.
  5. we still haven't caught all the error conditions that should cause us to bail out and run the cleanup function.
But still.  Success.  Now, if the rest of you on mares would just get moving, that would be just tops.  (And if you're on mares but didn't get the move email, drop a message on support@prgmr.com.)

(Also, a hearty "thank you" to user aahmedi, who tested the migration script over several frustrating back-and-forth intervals via email.  (Didn't work.  Try again now.  Still doesn't work.  Okay, try again.  Nope.  How about now?  And so forth.)

migrate yourself.

| No Comments
We're trying to move everyone off of mares because we are evil and oppressive hosts.  But I hate scheduling moves, especially since I've got a day job and don't like working nights.

So.  We have decided to throw technology at the problem.  I wrote up a move script based on the previously-existing (but still not rolled out) backup and restore scripts, and enabled it for some lucky users.

(Unfortunately, while this solves the scheduling problem, it introduces a new problem: users either wait for someone else to try it, or they hit amazing, horrible bugs.  Meanwhile I tear my hair out.)

But hey.  It's something.  We really do need to clear people off mares.  So if you got that email, you might want to give the move a shot.  (And your data is totally safe. . . in fact, extra-safe, because the move generates a backup.  Two backups if successful.)
Wow, been a while.  No explanation, no apologies.

Anyway.  Spent about a day working on trying to get the Sentry Power Tower XL working with powerman.  I'm not declaring victory here, because I still haven't got powerman working.  But I'm much farther along, and I got sufficiently annoyed at the lack of documentation to rant at Luke, whose remark was "you can fix this."

So.  Documentation on controlling the Sentry/Servertech Power Tower XL via SNMP, as written by someone who's never used SNMP before.

First, set up the box so that you can connect to it.  I used a Cisco-pinout RJ45 serial cable, 9600 8n1.  Power it up, sign in with the default username and password (admn/admn).  If necessary, reset the firmware by holding in the reset button next to the LCD on the front.

(Some notes: "front" is the side with all the plugs, regardless of how you actually choose to mount it.  Also, the reset button is an unlabeled pinhole.)

The online documentation on resetting was a little inaccurate, at least for the firmware version I originally had.  Nothing will happen when you initially push the button.  Hold down the button until the display changes to 3 horizonal lines, then release.  The display will change to one pair of horizontal lines in the middle of the display.

Now that we can log in, we need to upgrade the firmware.  The mechanism is pretty clever.  The box includes an ftp client that can download the firmware from a remote host and install it.  (It also includes an ftp server, apparently, but that's unrelated.)  We ran into trouble because we're on a masq'd network that doesn't allow active FTP.  If you don't have this problem, you can tell the box to download the firmware from ftp://ftp.servertech.com/pub/firmware/Sentry3/Version_5/v5.3/PTXL-PT2x-PT4x-48xx/ .

We downloaded the firmware locally and set up a quick FTP server.  Whatever.

Either way, set the ftp settings as appropriate.  Our settings are as shown:

Sentry: show ftp                                                               
   FTP Client Configuration                                                    
      Username:   prgmr                                                       
      Password:   ******                                                       
      Directory:  Downloads/                                                   
      Filename:   mrrpm-v53s.bin                                               
   FTP Automatic Update Configuration                                          
      Automatic Updates:  Disabled                                             
      Scheduled Day:      Everyday                                             
      Scheduled Hour:     12 AM                                                
   Command successful              

Then issue a 'restart ftpload'.  The box will download and flash its firmware.

Cool. Now you have a fully armed and operational Sentry Power Tower XL.  (Incidentally, much of this probably also applies to the Switched CDU.)

SNMP is going to be a little tougher, especially if you've never used it.  I had to install the appropriate software (on Ubuntu, apt-get install snmp worked fine.).  Then I had to download MIBs, which translate the numeric keys returned by snmp into human-readable values.  To do this:

 * download mib file: wget ftp://ftp.servertech.com/pub/SNMP/sentry3/Sentry3.mib
 * copy mib file to snmp's search path: cp Sentry3.mib /usr/share/snmp/mibs
 * specify that you want to use this file: I did this by adding a "-m +Sentry3-MIB" to my snmp commands.  There are better ways.  Note that you use the name listed in the file, not the filename itself.

Now you should be able to get useful information.  Try:

$ snmpwalk -v2c -c public -m +Sentry3-MIB

This should give you a long list of keys, including stuff like power usage.  Congratulations.

You should also be able to control the outlets.  This was where I got extremely annoyed and started ranting, because I had to resort to guessing magic numbers to figure this out.  There's probably some documentation somewhere, but I couldn't find it.  To spare you the trouble:

The OID you want is: Sentry3-MIB::outletControlAction.1.1.<outlet number>

The magic numbers are: 1 (on), 2 (off), 3 (reboot).

In our case, the approriate invocation to reboot outlet 4 was:

$ snmpset -v2c -c private -m +Sentry3-MIB Sentry3-MIB::outletControlAction.1.1.4 i 3

That's the translated OID, "i" for integer, and the value we want to set, "3".

Now to actually figure out how to put powerman in front of this.

New update on prgmr.com servers

| No Comments
blog.prgmr.com and wiki.prgmr.com may now be accessed over IPv6.  We also now have SSHFP records in dns for all of the dom0's.  If you notice any missing, please let support know.

birds down hard

| No Comments
replacing a drive;  older system without hot swap. 

packet loss at svtix.

| No Comments
we've had packet loss at svtix all day.  my provider tells me:

" Looks like there is packet loss reaching the IP through HE.net. We are investig\
ating the issue."

edit:  looks like it's a problem is our fault, or at least is in our equipment.   we're working on it now.

fair and balanced.

Reminded Luke that someone who didn't know us might suspect from his blog that we run an extremely unstable service!  That would be a travesty.  I suggested that he make a daily post: "nothing to report" or "doin' fine" or "10 days without a fatal accident" or something like that.

archaic unix lore.

| 1 Comment
So I've been working on stuff at home, and I want to backup stuff on a publicly accessible server.  Ordinarily, I use a tar pipe for the purpose:

$ ssh user@host "cd parentdir ; tar -cvf - sourcedir" | tar xvf -

Problem is, I'm trying to back up home directories, most of which are inaccessible to my user.  I can't log in as root, because I have root login via SSH disabled like a good sysadmin.  I might be able to use su, but I've never gotten that to work.  Sudo might as well not exist.  This is Solaris.

Normally, I ssh to the remote host and push the tar rather than pulling it, but the home machine is in NAT hell.  Setting up port forwarding sounds like work, and totally inelegant besides.   Luke suggested bouncing the tar through another machine, but yuck.  I mean, really.

Solution: make a fifo and use that.

On the source host:

# mkfifo /transfer
# chmod 666 /transfer
# cd /export ; tar cvf /transfer export

The tar process will begin, then pause while it waits for something to read from the fifo.  On the destination host:

$ ssh user@host cat /transfer | tar xvf -

Problem solved.  Don't forget, putting absolute paths in a tar can ruin your day.

Recent Comments

  • luke: also note, it will whine forever.... lsc@beholder:~$ snmpwalk -v2c -c read more
  • luke: note, if you have a 'b' slave unit, it's a read more
  • chris t: It wouldn't help with the basic problem of poor publicity. read more
  • matt40k: Why don't you use one of the site uptime services read more
  • Matt Howard: It looks like you were just doing this for a read more
  • pileofrogs: Hi, I'm playing around with doing exactly this. Do you read more
  • luke: The external-device-migrate brokenness, I believe, is rhel specific. Redhat supports read more
  • luke: The biggest problem with the xen 'Chinese Wall' is that read more
  • luke: Tip from the Computer Janitor: When doing development within a read more
  • chris t: Movable Type also created a comment for me as well read more

Recent Assets

  • 01-incoming_traffic.gif
  • edit.jpg
  • 02-migrate.gif
  • day-shaped.png

Find recent content on the main index or look in the archives to find all content.