I doubt most of you know what a Radius server is, so I will spare you that explanation, but don't worry: *I* barely know what it is. But it went down, slowly, in some agonizing death.
Let's just say that we run a service that we also outsource to other companies so they can run the same service. Kind of like owning 2 cars, and renting one out to friends. Well, someone who uses this service decided to make a change Saturday afternoon, but instead of putting what he was supposed to put in the online form, he put in the technical equivalent of signing the exact words, "Put your signature here" where your signature goes on a form. As a result, this caused a process to fail, but instead of failing and asking him to put his REAL signature instead of the example provided, it just took it. This caused a database to replicate the errors to a configuration file that gets sent to several other servers, which caused them to fail. This would be like the car you rented to a friend who put sugar in the gas tank, and as the sugar spread through the engine, it caused everything to seize. The argument could be made your friend is an idiot for putting sugar in the gas tank, and it could also be made that you were an idiot for not putting a sticker over the gas cap that said, "Do not put sugar in the gas tank." In either case, you don't have a working car anymore.
I was working about 18 hours this weekend on work. Most of those 18 hours were looking through logs, and going blind in the attempt to find that one small piece of information that would be the clue to why things were failing. During this time, I also managed to set up all but two of my SETI array with Pocket Linux, a kind of mini-Slackware distro. The guest room is now host to the SETI thing once again. Currently I have 10 machines in the array, with 2 more coming in soon.
My apologies to anyarm and fuzzywhuze for having to work while they were at my house. I have to say I am jealous at where fuzzywhuze works. I mean, not "envy and hate" jealous, just "Aw... that's so cool, man...." jealous. At least I didn't get paged with another highlight of the weekend, going out with the whole Heare family for sushi: