Categories

Get Started Now

IT Horror Stories: Favorite of Week Two

featured-2

Week Two continued the trend of awesome, awesome submissions for our IT Horror Story contest. All of them qualified for Favorites of the Week, Part Two, our prize for the best stories of each week.

There were so many good submissions this week that we awarded three stories. Even then, there were plenty of stories that barely missed the cut, such as a Texas IT worker discovering a telecommunications site filled with snakes.

The submissions below won our second weekly Prestigious Poet’s Pen and Paper Pack.

All submissions are still eligible for Best Overall Story and Second Best Overall Story. Submit your story by July 28 to qualify for a QuickFire XT mechanical keyboard, HyperX Cloud headset, or an Amazon Paperwhite.

Favorite 1: “Files and Tribble-ations” or “Disk Trouble with Tribbles”

A long, long time ago, in a galaxy really close by…a workflow management system existed in a delicate state of homeostasis on the planet Iota Geminorum IV. The workflow management system, affectionately called W.T.F., by all who used it, functioned as well as you might expect, until one dark day (cue ominous music here).

A well meaning humanoid (doing his job at warp speed, as many humanoids must do) sent a file into the black depths of the workflow system. Due to a bug (or as our vendor often referred to such bugs -“a feature”), any filename with a space in it cloned itself repeatedly at an exponential rate. The tribbles, I mean files, quickly filled many gigabytes of server space, shutting down said server and all systems residing therein. Disk space truly is the final frontier. The reproductive process was stopped as soon as was humanoidly possible and the tribble-izing files stopped procreating. Unfortunately, though, this process happened repeated for several months, disrupting the balanced ecosystem that is any good workflow infrastructure.

All was not lost, however. After many attempts at eradication, the bug/feature was eliminated. Files stopped tribble-izing, no matter their naming convention, leaving all sentient beings (or at least partially sentient beings) free to add files to the workflow management system. Peace and justice had been restored to the Galaxy.

Favorite 2: Major Malfunction

There he stood, a Major General of the Air Force, next to him was a very tired looking Chief Master Sergeant. The time was shortly past midnight, and I had already been on shift for nine hours, and exhaustion was slowly creeping in, three hours to go until I could clock out. Ever since leaving the military, rank no longer intimidated me, I was nearly untouchable if they decided to lose their cool. The General was impatient and baffled by my stubbornness, what he wanted was specialized access to a sensitive folder on the government shared drive. Something about him made me uneasy, my background in cyber security was unconsciously invoking a skeptical reaction to his firm demands.

The whole situation was irregular, I could understand a late night worker, I could understand why the gentleman would need access, but what was strange was why he was at the front desk in the first place. Usually general officers had aides to do this sort of grunt work, and they would provide some sort of documentation. This man showed no physical ID, but did give an accurate identification number, no paperwork and was starting to get belligerent. My heart raced, but I stood fast, as he named dropped my head boss’s name, demanded my name, and information, and as his voice raised to inconsolable heights.

“No.” I repeated, “Sir, with all due respect, I need some form of identification, your Chiefs is not enough. I also need documentation from your security officer with reasoning why you need the access. I cannot help you unless you provide me with this information.” My tone was clear and calm, I was not playing his game tonight. “Let me get my boss.”

My shift lead, also a contractor, came out of his office, clearly struggling to stay awake through his double shift. The General explained, in a very direct, and harsh tone what he needed from him that instance. He nodded, and then scolded me on why I should de-escalate situations such as this in front of the customer. He said he would personally grant the General access. The General smiled abruptly.

“Are you sure, sir, this will not be a problem?”

“All it takes is a few clicks of the button, what is your number so I can find you.” The General chuckled quietly, and gave him a number that was not the same as the one given to me.

A few moments went by, and I watched my supervisors face go white. The General was not a Major General, he was a civilian inspector, and had given the identification number to an account whose first name was “You Failed,” last name “The Inspection.”

The man apologized for his rudeness to me, and needless to say, my supervisor, was no longer my supervisor shortly after that. Ever since then, I have always trusted my instincts, and I can thank my wonderful teachers and mentors over the years for that ingrained behavior.

Favorite 3: Pop

My first IT job was almost by accident. The company I had worked for had a head IT guy, but he was busy with family issues and immigration, so he frequently had to travel outside of the country for extended periods of time. When I started at this company, I was hired to pretty much do data entry, and help with basic stuff like printers not working, or someone needing to know how to scan…just general help desk items.

Anyhow, over the course of three years, I slowly took more and more under my belt. Working on servers, ordering supplies, some light networking. When possible, I tried to figure out how people were doing things, and see if there was room to improve. They weren’t always big things. Sometimes it was just something that might save me some effort, or make someone else’s life a little easier.

One thing I noticed was that we didn’t have much in the way of monitoring. When things weren’t working right, it wasn’t usually discovered until it caused a problem, which could be weeks later, often needing a lot of work to fix any issues that may have cropped up in that time. I slowly worked to figure out better ways to do things. I couldn’t really setup a monitoring system myself, as I lacked the resources and knowledge at the time, so I just set to getting as much information to myself as possible. I checked whatever logs I could find. Server error logs, firewall error logs, I just wanted to get into the habit of checking and get ahead of things when possible.

One thing I found that I was curious about were our backup power supplies. One had our handful of physical servers (two that had a handful of VM’s on them) plugged into them, while the other had the network equipment. I didn’t get to set it up, so I didn’t get to really spread the servers between the two. Noticing this, I was a bit worried that they all might go down with a power outage. So I’m looking around the UPS and I notice it has an RJ-45 spot on the back of it. I’m thinking to myself “Perfect, I can plug it right into the network!” and so I dig around in a box to find a spare Cat6 cable (Note to anyone reading this…always at least take 5 minutes to search around online for product manuals…) and I quickly find one and walk back over to the UPS and switch.

I grab the cable, plug it into a spare port on the switch, and then reach back to plug it into the UPS. As soon as I hear the click of the plastic head of the cable locking into the UPS, I hear a following POP from inside the UPS. It sounds like a circuit being killed…and much to my dismay, the Cisco switches quickly become the loudest things in the room…each server that was plugged into the UPS is now sitting idle. The switches on the other UPS continue whirring away, but the lights and sounds from these servers have stopped. I panic…hundreds of users were in the middle of their 2pm work, where most everyone has returned from lunch and is slowly getting back into the swing of things.

I walk outside of the server room, and it feels like everyone is looking at me…most probably were, since they all knew I was the only one who could explain what could possible have caused the SQL server, Exchange server, and pretty much everything else to go down. From where I stood, I could hear my phone ringing at my desk, and the receptionist desk starting to ring, as people who had previously been remotely connected now stared at their disconnected sessions. To make matters worse, the owner walked out of her office and down the building to find me in the server room, frantically pacing in front of the servers that had finally started back up, but were moving painfully slowly through their boot processes.

She asked what happened and I just had to say “I’m so sorry…I plugged the wrong thing into the wrong place, and it reset everything. I’m working to get everything back up…”. Visibly frustrated, but satisfied with my answer, she signed and walked away. After 20 minutes that felt like 5 hours, all of the servers came back, and everyone picked up where they left off.

But I never forgot to always check the documentation first.

The contest has one more week! Submit your stories by July 28th, 2016 here!

 

Not a CBT Nuggets subscriber? Start your free week now.

CBT Nuggets has everything you need to learn new IT skills and advance your career — unlimited video training and Practice Exams, Virtual Labs, validated learning with in-video Quizzes, Accountability Coaching, and access to our exclusive community of IT professionals.

Learn more about the CBT Nuggets Learning Experience.

 

Comments are closed.