gregh  2008-02-14 12:47  system_administration  work   

When I graduated with my undergraduate degree, I had lousy grades, some significant history -- though limited formal training -- with technology, and a lousy job as a secretary at Mellon Bank in Pittsburgh.

Back then, you could actually read newsgroups regularly, as spam had not quite hit, and one of the groups I watched with interest was misc.jobs.offered.entry. For whatever reason, it his me that I should be able to find this in Google Groups. Sure enough:

Date: Jul 2 1995, 11:00 pm
Subject: BOSTON-Unique Oppy - Liberal Arts grads who love computers - Train OTJ
To: misc.jobs.offered.entry

OPENENVIRONMENTCORPORATION

OpenEnvironmentCorporation is seeking individuals for the following
positions. These positions are full-time, permanent positions in our
Boston, MA offices. OEC offers software development toolkits that
facilitate rapid development of three-tiered client-server applications.
Our customers are Fortune 500 companies, and our company is fast-paced and
rapidly growing. OEC is an equal opportunity employer. We are located in
Allston, a ten-minute walk from Harvard Square.

THE POSITION

OpenEnvironmenthas a unique opening for a bright, motivated individual
who is "computer literate" and wishes to train on the job for a technical
position. We are seeking a person with strong organizational skills and
a love of learning. S/he must be energetic, willing to learn, with a "can-do"
attitude.

The initial responsibilities for this individual will involve
setting up and maintaining a software and hardware purchasing and inventory
system. Through this, he or she will have an opportunity to learn about
Unix and PC hardware and software, work with the Systems Engineers, and
use his or her strong organizational skills. Over time, we intend to train
this person in system administration tasks and have him/her serve as the
dispatcher for hotline requests.

The salary range is in the low to mid 20's depending on experience and
technical knowledge.

Requirements:

Bachelor's degree
Excellent organizational and communication skills
User-level familiarity with PCs and common PC applications
Fearless attitude about learning complex technology
Willing to "work his/her way up"

HOW TO APPLY

Please send resumes and an optional cover letter to:

. . .

"low to mid 20's" was $23K. Fortunately, shortly after I arrived, the two full-time sysadmins supporting our complex development environment (we made DCE-based enterprise middleware) left, and I was dropped into the fire. $23K became $32K in a few months, OEC was purchased by Borland, and I eventually moved on. It was a good job. I learned a heck of a lot, and I got plunged deep in the world of developing complex web applications, before they had really taken off.

gregh  2007-05-10 07:47  Computing  system_administration  work   

SILICON VALLEY USERS GUIDE: How do I get my sysadmin to do anything?:

No one sets out to be a professional systems administrator -- do you ever see kids wearing toy pagers playing sysadmin? For those fluent with computers but uninterested in writing huge software programs, it's a pretty good job. Except for the part where they have to deal with you.
...

• Say hello. Even when you don't need something. You say a big Hi to the pretty lady at the front desk every time you pass. Why not treat the tech guy the same? Sure, he doesn't say anything back. Don't be fooled. Sysadmins are like cats -- he won't acknowledge you, but he's mentally keeping score.
• Don't question what he does all day. Systems administrators are like firemen and cops. If you don't have a couple of bored ones hanging around, you'll be sorry when there's an emergency.

Much of this is just a caricature of sysadmins. However, the most important points here, which management overlooks time and again in nearly every organization, are that there is often no visible work output from what we do and often we're working even when it may look like we're not. It's important to remember that it's when it seems we have nothing to do that we're doing our jobs best. Trying to fill that "underutilized" time simply prevents the ongoing smooth running of operations.

gregh  2006-08-31 12:37  hosting_services  system_administration  Website   

Maybe you noticed that my server has been unavailable lately?

Since Saturday, my server has been randomly going down. I hadn't been able to figure out why. My logs bore no indications, I keep the system relatively up-to-date, and I don't run much on the server other than the webserver, ssh, and a tightly controlled proxy. I figured there might be some random hardware problem, or perhaps there was an unpatched denial of service targeted at my kernel, which I don't update as often.

Neither was particularly attractive to address. The reason I try to avoid touching the kernel is that the server is a rented machine, hosted by EV1Servers (yes, the company that took so much heat for buying an IP license from the SCO Group a couple of years ago.) I've not had an opportunity to test them with a tough problem, and hosing up booting by killing the system wasn't something I wanted to try. So, I've only patched kernels when it's been critical. Because the server lives in Houston, TX, and is actually owned by the host, that's another reason that troubleshooting potential hardware issues is difficult.

Of course, on-campus interviewing submissions had to be out Tuesday morning, so I didn't spend much time on it over the weekend. There's work and school during the week. I'm going out of town this weekend. In short, I wasn't sure how or when I was going to resolve this, and it's been driving me nuts.

And so, today it went down again, just as it had first on Saturday, then again on Monday morning, and Tuesday afternoon. EV1 provides a remote reboot capability, but it's never brought the server back up. I've always had to request a manual reboot. It never made much sense to me; after all, it's supposed to be power cycling the server. I just figured maybe it didn't really do what they claimed it did.

Things started to get odd when the second of my requests received a response like this:

8/28/2006 5:15:18 PM
DataCenter
Dear Gregory Haverkamp,
We are closing this ticket now for ther seem to be another issue being addressed on this server at this time. Please feel free to look into the other open tickets on this server for further details of the status of your server.

I reopened the ticket, noted that I had no outstanding trouble tickets, and that I really need the server rebooted. They complied. On Tuesday's request, no "other tickets" were mentioned. That subject did come up again today after I tried to reopen their closed ticket:

8/31/2006 2:25:10 PM
DataCenter
Dear Gregory Haverkamp,
We are closing this ticket now for ther seem to be another issue being
addressed on this server at this time. Please feel free to look into
the other open tickets or customer service on this server for further details of the status
of your server.

Thank You
joyce o
Webhosting Server Support Technician
Ev1Servers

8/31/2006 2:42:46 PM
DataCenter
Dear Gregory Haverkamp,
Your server is under abuse investigations right now.It is for this reason that we are unable to reboot your server.We are sorry for any inconvience costed.Please refer to escalation # 1307120.Thanks you for your patience.

Thank You
joyce o
Webhosting Server Support Technician
Ev1Servers

And so I contacted customer service:

[16:26:06] Janelle W : unfortunately, I believe there has been a
mistake made on your account. The ticket number that they have
refrenced does pull up an abuse issue, but it is not on your account.
It is on someone else's account.
[16:26:40] Greg : Hehe. It's probably the same mistake they made
two days ago, too, when they initially told me they couldn't do it.
[16:27:01] Greg : Could this also be involved in my server appearing
to occasional drop off the network?
[16:27:09] Janelle W : let me check one more thing, and if I still
cant figure it out, I will forward you to technical support, since
they are the ones that e-mailed you
[16:30:13] Janelle W : I believe we might have figured out what the
problem is. can you hold on just one moment while I confirm it?
[16:30:26] Greg : Certainly.
[16:34:08] Janelle W : thank you so much for your patience, it will
be just one more moment
[16:40:15] Janelle W : Thank you so much for waiting. Our abuse
department is investigating this issue and someone will get back with
you either by trouble ticket, or e-mail

And so, around 4 hours afters it originally went down again:

8/31/2006 3:53:58 PM
Abuse
Dear Customer,

Your server is online and responding to ping and ssh. After reviewing the situation, it appears that your server was unplugged incorrectly due to human error. We sincerely apologize for any inconvenience this issue has caused you. No AUP violation has occured on your server, and we will be notating your account to indicate that. Once again, we apologize for the error and will be reviewing the matter so that it does not reoccur.

And so, they had mistakenly tied my server (but not my account) to an abuse investigation. This apparently resulted in repeatedly unplugging my server without any notification. From what I can tell of the reboot history of this machine since Saturday, others probably could have noticed something was amiss and said something about it, but no one did.

Mistakes do happen, and I'm glad they cleared it up quickly once they discovered there was a problem. Right now, I'm just hopeful that this ends the recent string server deaths I was seeing, that this actually was my problem.