"Ah, but I was so much older then, I'm younger than that now."
My Boog Pages
Wednesday, April 5
twentyfourseven
Sometimes I feel that's how much I work, or at least how much I've worked lately. It all started last Thursday night. I had been asleep about half an hour when the on-call phone rang. It was the help desk: "Uh, the power's off in the data center. Do you think you could come in and check it out?"
Crap. Just the Monday before I had finished installing ten servers, only to be told that I couldn't turn them on. I guess they weren't kidding when they said we were out of power.
I suppose the outage that the rest of the crew had to deal with while I was on vacation should have been a tip-off, too. That one burst a couple of batteries in our UPS*. This time the batteries held - for ten seconds.
So I dragged in to work, arriving at midnight on the dot. Pretty much all the groups had someone on-site - Operations, Network, Unix, etc. - so at least I wasn't lonely while I waited for them to turn the power back on. Once they did I had to run around turning off servers until they could get the external storage arrays back online. Once that was taken care of I discovered that one of our VMWare servers - hosting a dozen mission-critical virtual servers - had a dead disk controller. We have four IBM server of that model and this was the THIRD ONE to go dead in a four month period.
Fortunately, one of the servers that I couldn't turn on had the same controller, so I swapped it out and then strutted around like a hero. For five minutes, then it was back to work. I had to manually check the 200 or so other servers to make sure they'd booted properly. About a dozen hadn't, so I had to grub around and figure out what went wrong.
I finally had everything back online around 7:00am, just as the first of the rest of the team was coming in. I turned it over to him and went home to crash. But of course I had a few other calls that kept me up, then the kids came home from school, and after that, well, NOBODY in Fort Worth was sleeping. I don't know where they learned to be so LOUD. Although my wife says she knows...
So, I survived to the weekend. Unfortunately work wasn't done yet, and while I wish I could quit you, cowboy, I had to deal with it. Seems there was one server that kept dropping its connection. To one other server. And not all the time. And no other servers had this problem. So I spent two or three hours on the phone and in front of my computer at the house without ever geting anything accomplished. I hoped maybe the problem would go away on its own. But like the supporting cast of Seinfeld, it just sort of hung around, stinking up the joint.
Sunday morning started off quiet, until I received a call about a test and dev server that was down. I wanted to shout TEST AND DEV IS NOT PRIORITY ONE, YOU JACKASS!!! but instead I walked the tech through pressing the F1 key at the boot screen.
After that everything seemed copacetic, and I even took Graham Jr. (actually a girl) on a trip to Recycled Books in Denton. She wanted a coloring book, and I had to explain that used coloring books are really not much good. Then I got The Call and we had to go.
The server was still acting up. If we didn't get it fixed 60 people would be idle Monday morning. So I had to come up to work and build a new one. Which went off without a hitch, except that it took four hours, plus I had to wait until the application group installed and tested their programs. Upshot: I got home at two in the morning.
And was back at work at nine. Why did I rush in? Two reasons. 1) To make sure everything was working properly, and 2) to hand off the on-call phone to the next victim.
Mission accomplished.
The Fabulous Flying Powells. I collect famous people with my last name, such as Dick Powell, William Powell, Colin Powell, Talmadge Powell, Megan Powell, and now Richard Powell, author of Say It With Bullets, just reprinted by Hard Case Crime. Lots of other people have blogged about this one, but it is a tremendously entertaining book, full of wit and humor. My favorite: the protagonist needs to pick a fight, but everyone he meets is just so damn nice. If you like Richard S. Prather you'll love this one.
As an added bonus, today James Lileks mentioned a boogie woogie piano player from the 50s named Bud Powell. So I'm off to iTunes...
