punkwalrus (punkwalrus) wrote,
punkwalrus
punkwalrus

I am a little sleepy, but okay

This weekend, I was paged every night, and each time, it was some big issue, yet was created by some complicated problems within problems within problems. Suffice to say, I get paid extra to be on call 17 weeks out of the year, so I am not complaining to be employed, and the "experience" I get is that my company still has a ways to go to fix some issues.

I doubt most of you know what a Radius server is, so I will spare you that explanation, but don't worry: *I* barely know what it is. But it went down, slowly, in some agonizing death.

Let's just say that we run a service that we also outsource to other companies so they can run the same service. Kind of like owning 2 cars, and renting one out to friends. Well, someone who uses this service decided to make a change Saturday afternoon, but instead of putting what he was supposed to put in the online form, he put in the technical equivalent of signing the exact words, "Put your signature here" where your signature goes on a form. As a result, this caused a process to fail, but instead of failing and asking him to put his REAL signature instead of the example provided, it just took it. This caused a database to replicate the errors to a configuration file that gets sent to several other servers, which caused them to fail. This would be like the car you rented to a friend who put sugar in the gas tank, and as the sugar spread through the engine, it caused everything to seize. The argument could be made your friend is an idiot for putting sugar in the gas tank, and it could also be made that you were an idiot for not putting a sticker over the gas cap that said, "Do not put sugar in the gas tank." In either case, you don't have a working car anymore.

I was working about 18 hours this weekend on work. Most of those 18 hours were looking through logs, and going blind in the attempt to find that one small piece of information that would be the clue to why things were failing. During this time, I also managed to set up all but two of my SETI array with Pocket Linux, a kind of mini-Slackware distro. The guest room is now host to the SETI thing once again. Currently I have 10 machines in the array, with 2 more coming in soon.

My apologies to anyarm and fuzzywhuze for having to work while they were at my house. I have to say I am jealous at where fuzzywhuze works. I mean, not "envy and hate" jealous, just "Aw... that's so cool, man...." jealous. At least I didn't get paged with another highlight of the weekend, going out with the whole Heare family for sushi:

The Heares eating sushi

In September 2005, we all went to Sakura in Fairfax.

From left to right: my son's hand reaching for sushi, Keiran, Cheezy_reads not picking her nose (really), Stodgyspike (in car seat) happy to be on Motrin Pediatric (he's teething), Stodgycat offering another set of chopsticks, Scarlet using the chopsticks like a medieval person used their boot knife (you period people will get that joke), and Chance with her scarf, trying to look like a skinny white Aunt Jemima.
The Heares eating sushi

Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 0 comments