As for system reboots, we always gave advance warning and would place a courtesy call to anyone still logged in before cut off transmissions. As expected, the job killer reduced the number of users in the system — and our hair.
Catching the culprit ... or not
We ran into a snag. Once this utility was in place, overall it seemed to work well. However, on a few occasions it mysteriously died. Everything checked out fine as I narrowed down the possibilities. Eventually I came to a conclusion: In all likelihood, someone with privileged access had shut down the process killer.
To find the wrongdoer, I used a trick I'd heard about at a recent computer symposium. I had learned that one could dump the command buffer for any current user on the system to see what commands were issued. Often you could see every command the user issued on their terminal since logging into the system.
It was limited only by the total size of the collection of commands because the size of the command history area in system memory for each user was fixed. If the user logged out, the command history would be gone. For once, it worked to our advantage that some users did not want to log off. The next time my process died, I looked at the command buffers for several users and found the command that stopped my process. I had caught the person red-handed and presented a printout of these commands to my boss.
We confronted the user and asked him not to do this, but he said he had never entered those commands and someone else must have used his account to do the deed. It wasn't worth pressing the issue any further at that point, so my boss gave the user the benefit of the doubt and gently asked him to keep his terminal and password secure. Whether or not that user was guilty, we didn't see the problem again.
Besides being a fun project, this endeavor taught me a lesson: Just because I could show an action originated from a particular user account does not always prove it was done by that particular user. More evidence may be required to tie an event to an actual person. Proceed cautiously — it could be their job on the line.
Sign up for CIO Asia eNewsletters.